Scaling nodes in Kubernetes on a schedule.
When autoscaling can’t solve all of your problems.
Kubernetes is one of the hottest technologies on the market for running infrastructure that can scale. The ability to meet demand as it comes, and scale back as it goes down is one of the hallmarks of going to the Cloud. However, is there ever a time when letting a system scale for you might not always work? And if so, what should we do instead?
Hi, I’m Maher and I work as a Cloud Customer Engineer at Google. I’m writing this post to show a niche solution to a niche problem, but if this is a problem that you have, then I hope this guide provides some insights onto a single way to solve this issue.
Part 0: Preliminary Kubernetes information
Kubernetes is a container orchestration platform. Containers provide a way to isolate an applications environment, and allows one to specify system configuration and libraries without worrying about anything else running on the same physical machine.
Docker is a popular container platform; more info about platforms can be found here:
So why Kubernetes? What happens when your container dies? or you need a way to have multiple containers communicate with one another? There are many issues with managing containers on your own, and having a platform to orchestrate your containers is a massive appeal for Kubernetes.
Part 1: Introduction
Kubernetes is a great platform for orchestrating and managing containers, and one of the major benefits of managed Kubernetes platforms like Google Kubernetes Engine is the ability to autoscale your nodes.
As a refresher, Kubernetes runs containers inside something known as pods, and Kubernetes schedules pods into what are known as nodes. This is all to ensure an easier way to orchestrate your containers.
autoscaling based on demand is a wonderful feature and for the majority of use-cases, this is the preferred best practice to do. However, if you know you’ll be faced with high demand at certain times, then preparing the nodes beforehand might be a path that you’d want to take. And in this blog post, I’m going to cover a simple way to do this using GCP and Google Cloud Scheduler.
Part 2: Initial setup
Step 1: Create a GCP Project environment if you don’t already have one
Step 2.1: Enable the Kubernetes Engine API
Step 2.2: Enable the Cloud Scheduler API
Step 3: Create a Node pool in Kubernetes and don’t enable auto-scaling
Step 4: Wait for the Cluster to be created.
Step 5: Create a service account with the correct IAM permissions.
In our case, we’ll create a service account called gke-nodes-scaler and give it the Kubernetes Engine Cluster Admin role. In practice, you’d want to follow the idea of least-privilege when it comes to giving IAM permissions, but for this demo, we’re using the Kubernetes Engine Cluster Admin role to ensure things work properly.
Step 6: Create a Cloud Scheduler and use setSize to configure the size of your Node Pool.
Cloud Scheduler takes your frequency of the job in a format known as unix-cron. More reading on it here
0 8 * * * is the unix-cron format to schedule job every day at 8am.
The URL should be the setSize API which is an HTTP Post request:
This is the api we’re using. Filling it out with my personal project info, it looks like this:
Step 6.1 Continue filling out the Cloud Scheduler and add HTTP Headers
Create a few HTTP headers to pass along information to the Cloud Scheduler. More reading on HTTP headers here
Content-Type: application/octet-stream
User-Agent: Google-Cloud-Scheduler
Step 6.2 Attach a message to the API to set the node size of our Kubernetes cluster
In the body of the message, attach the JSON to configure the nodeCount. In our case, we’ll scale it from one node, to two.
{
“nodeCount”: 4
}
This will scale up our cluster to 4 nodes, and as an inverse, if we put 0, it’ll allow us to later scale it down.
Step 7: Test scaling up and scaling down the Kubernetes cluster
We can go ahead and do a test run of it, and see if it truly scales up our nodes.
And as you can see, we can scale up our nodes. If we do this again for another job on Cloud Scheduler, we can scale down on a set schedule.
So let’s try scaling it down now as well.
Conclusion
And that’s everything! It’s a very simple solution to a rather niche problem, but if it’s an issue that you are having, then I hope that this guide helps you.
