Deploy Jupyter Notebook to AWS Lambda

deploy jupyter notebook to AWS Lambda

More often I find myself opening Jupyter Notebook when facing a mathematical or algorithmic problem. Maybe it is also true for you. At first, you think like: “I’ll make only research/visualization part in a notebook and then move to plain python.” But after some time you end up finishing algorithm in Jupyter Notebook. A few years earlier when there were no ”cloud lambdas” you will end up moving code somewhere. However, nowadays it is possible to deploy your function written in Jupiter Notebook in less than a minute.

Before we start

We will use Terraform to define infrastructure as code, so we end up making less DevOps routine. And of course, you need to have an AWS account. All the code for this example you can find in the repository.

Steps

1. Make lambda function ready to deploy
2. Create infrastructure with AWS Lambda and API Gateway
3. Test our function
4. Create a bash script for deployment

Make lambda function ready to deploy

At first, we need to add handler function to our notebook which will receive a request and return the result of calculations.

As you can see this function parse JSON, run algorithm_on_steroids and then return the result in JSON format. Next, we need to create a file with the name libs.txt. Where we will add names of all third-party libs we use(numpy, matplotlib, pandas, …). Then we add bash script which will create zip file ready to be deployed to lambda. What script does:

1. create a directory
2. convert the notebook to python file and move it to the directory
3. run pip install for each lib specified in libs.txt
4. zip folder

No more procrastination. You have time for goals! Increaser will guide you.

Realize your ambitions!

Create infrastructure with AWS Lambda and API Gateway

In order to use AWS lambda, we need to create infrastructure. It includes:

1. S3 for saving the function
2. Lambda for running the function
3. API Gateway to communicate with the function

For simplicity, we will omit best practices and put everything in one terraform file. But before you can run it you need to specify credentials in the file. Special attention to this piece of the file:

Create Lambda and test it!

Open terminal in the directory with lambda and all files mentioned earlier.

$ . ./cook_notebook.sh
$ terraform init
$ terraform apply

It seems like magic. Three commands and you have deployed function. After running terraform apply at the end of the output you will see a green line with the URL. It is the URL by using which we can run our function. Let’s test it by making a POST request.

$ curl --request POST --data '{"a": 3, "b": 4}' <URL_FROM_OUTPUT>/function

OK, but what if you make changes to the function and want to see a new version deployed? Let’s write a script for deployment.

Run it by typing:

$ . ./deploy.sh tf-lambda tf-lambdas function.zip

Conclusion

In this article, we made the automation tool for deploying Jupyter Notebook function. It is not ideal, for example, it would be good to add a script that will automatically find third-party libs in the notebook and add them to libs.txt, but this is out of the scope of this article. If you want to delete from AWS everything we made so far you can run terraform destroy. And poof — everything deleted from AWS. Hope this article was useful for you:)

Reach the next level of focus and productivity with increaser.org.

Increaser