Save Costs by Scheduling your Google App Engine Flex Instance to Switch Off and On
As a Data Scientist building ETL pipelines, you will often have to break your pipelines up into various stages that could potentially each be run in a serverless environment, such as Cloud Functions or Cloud Run on GCP.
Sometimes you can run into a situation where it is impossible to deploy a specific stage of your ETL pipeline on the very cheap and easy-to-use Cloud Functions serverless infrastructure (your code may take longer than the allotted maximum 9 minutes to run, it might need more memory than the available 4 GB, or it might require system packages to be installed).
In some of these cases, the slightly more flexible Cloud Run environment will come to your rescue. At least in this environment, you can build a custom Docker image and have access to a bit more time (15 minutes, or up to 60 minutes in beta) before your response times out, as well as access to the same 4 GB of memory as Cloud Functions, but sometimes even that will not suffice.
Enter App Engine.
Google App Engine’s Flexible environment allows you to deploy services that cannot be deployed as Cloud Functions or via Cloud Run; either because they are too memory intensive, take too long to execute, or require custom system packages to be installed.
Within this wonderful environment, you are able to allocate as much RAM and as many CPU cores as you’d like, but this comes at a cost. App Engine’s Flex environment scales up to accommodate peak times of higher traffic, but can unfortunately not scale down to less than one instance. While this would be fine if you needed to serve traffic 24/7, there are times where you only really need your instances to be available for an hour or two every day, while you run your ETL pipelines to get the delta from the previous day, for example.
Wouldn’t it be amazing if you could just switch off your App Engine services on a set schedule and avoid all those Flex Instance Core Hours and Flex Instance RAM costs that haunt your billing page?
Here’s how you can do it…
First, create a Service Account key-file, according to this guide and save it in 2 separate folders in your repository, name the one folder “switchon” and the other “switchoff”.
Next, you will need two bash scripts, each of which will use functionality from the Google Cloud SDK, but no need to worry about that, we’ll see how Docker has us covered in just a bit.
The first file, switchoff.sh shown below, should be placed in the switchoff folder:
The second file, switchon.sh shown below, should be placed in the switchon folder:
Next, we need a way to call each of these scripts, right? So, we’ll use a very simplistic Flask app to do just that. Call this file main.py and place it in both the switchoff and switchon directories (note the comment in the gist to replace the call to ‘switchon.sh’ with ‘switchoff.sh’ in the switchoff directory):
Lastly, we need a Docker environment with both python and the Google Cloud SDK installed, in order to run our code. Again, drop this file into both the switchoff an switchon directories:
Awesome, now all you need to do is deploy each of these to Cloud Run, by calling the following on your terminal in each directory, replacing ‘switchoff’ with ‘switchon’ as needed:
The final step is to use Cloud Scheduler to schedule switching your instances on and off on a schedule that works for you, you can read up more about how to do this here.
Thanks for reading!
