Slow Time
It's well-known that there are time thresholds that affect how we perceive work and affect our flow. If a task takes too long to do, we put it aside or drift to something else. For example, in a recent internal project, we were using AWS Lambda to handle a serverless web REST API. The problem was that running the integration tests would take more than 15 minutes. This directly impacted how often we were willing to the run the tests: 15 minutes is just too long to be run after every code small change. Running tests meant pausing everything else. It also affected the cost of writing new tests: every new test would make the integration testing even slower.
I found this situation unacceptable. And, on some level, incomprehensible: why were the tests taking so long? So I profiled the code, starting from the client side. It turned out that almost the entire time was spent opening sockets and then waiting for responses. So, I then profiled the server side of the code. Turned out, the server code was very fast.
Where was the time spent?
It became evident that the problem was with the AWS framework. What I discovered is a good illustration of a bad engineering choice. One of the selling point of the AWS serverless framework is that the server code is always hot-loaded, nothing needs to be done to use the latest changes you've made to your code. The problem is, this is implemented by spinning up and shutting down the docker instance for each and every web request! For the convenience of not having to do this manually to try new code, AWS makes every request extremely slow. To save a few second every now and then, it adds 2 to 3 seconds to every request. It's an acceptable compromise if you modify your code very often and only test a single end-point from time to time. But this is not how I work: I like to write a full feature and test it thoroughly afterward. AWS was getting in the way.
Fast Time
The solution I came up with was to avoid using the AWS tools when developing the web back-end and instead use a AWS serverless emulator. Since the back-end was written in Python, I decided to write this simple emulator using the Flask web module. The AWS Lambda emulator bridges Python's Flask with the Lambda API. It takes the flask request and response and converts them into the corresponding Lambda event, context and result.
This allows writing a small flask app that emulates the AWS Lambda by calling the same implementation code. In our integration tests, we've found that this yields about 80x (yes eighty times) the speed of running AWS Lambda locally in docker. As explained before, the reason for the amazing speed-up is that AWS Lambda starts and stops the docker instance for every request! In our application, using the AWS emulator cuts the testing time from 15 minutes down to 10 seconds. Now, 10 seconds is short enough that we could run the entire test suite at any time without affecting our workflow. It also lifted the cost of writing additional tests.
Each new test were now adding less than 30ms each. This led us to write enough tests to achive 99% code coverage, which is the kind of metrics that really helps drive the quality of the end product.
The code for the Python Flask AWS emulator is available on GitHub. It is part of a toy demonstration application that also shows some other nice utlities for Python web development. The GitHub project is available here. The emulation code is contained in the file src/flask-app/aws_emulator.py. It has three functions:
get_event(): creates an AWS Lambda event from the current flask request.get_context(): creates a dummy AWS Lambda context.convert_result(): converts an AWS Lambda result dictionary into a flask response.
The example Flask application that uses the emulation to call on AWS-Lambda-like functions is contained in the file example_app.py. It simply calls get_event(), get_context() and convert_result() in each single-line web end-point.
It does requires to add each Lambda end-point to the Flask app in parallel to adding them to the AWS Lambda description YAML, but it is very straight-forward and always follows the same pattern. A future possible improvement would be to generate this Flask app from the AWS Lambda description instead! But given the tremendous speed-up, this small amount of work was worth it. If you are using AWS Lambda with a Python back-end, you should give it a look.
