Scaling a node out of a kubernetes cluster in kops

In this example we will be scaling out the test-nodes pod from one of our KOPS cluster.

Draining the node of pods before deletion

Login to the KOPS node for one of the clusters and run this command to find the node:

kubectl get nodes --show-labels | grep test-nodes

That should show you a single node’s name, will be of something like ip-10-99-229-241.ec2.internal once you know the name you can drain the node

kubectl drain ip-10-99-229-241.ec2.internal

At first you will receive two errors:

  • cannot delete DaemonSet-managed Pods

    To understand a DaemonSet: Imagine a deployment which had a rule which is “you must run at most one pod on a worker node” and “you must run one pod on each worker node where you can run” IE where label selectors say you can run effectively: That’s what a DaemonSet is.
    What this error message is saying is that: I can’t evict a DaemonSet as by its nature it can only run here
    We can ignore this behaviour by using: –ignore-daemonsets

  • cannot delete Pods with local storage

    In pods you can mount storage – usually central – into a pods volumeMounts, you can also mount local storage – either through specific local dirs – or through emptyDirs (which mount a temp volume)
    What this error message is saying that the contents of those won’t be ported across – that is fine in our case (and should be in all kubernetes systems) as if you need persistent storage there is central storage for that.
    We can ignore this behaviour by using:  –delete-emptydir-data

Run the command again with the two flags to ignore the errors:

kubectl drain ip-10-99-229-241.ec2.internal --ignore-daemonsets --delete-emptydir-data

Once the node has drained you should get a list of pods which have been evicted, to make sure those pods come back to life elsewhere in the cluster, we can view the status of all pods:

kubectl get pods -A | less

If you see any pods restarting just run this command again after a while to ensure all pods are in either a completed or running state, once everything is fine you can move onto the next step:

Deleting the node

Export the IAM creds for KOPS as normal (pipeline admin ones) and then edit the config file of the node you are scaling out (in our case test-nodes)

kops edit ig test-nodes

there will be a set of vars

 maxSize: 1
 minSize: 1

(In our case they should always be set to the same value as we haven’t enabled auto-scaling)

set both to then run

kops update cluster

If all has gone to plan it should show a single change where it scales from 1 => 0, if so you can now run:

kops update cluster --yes

Congratulations you have just scaled a node out of a cluster

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.