AWS Python Serverless

An easy, step-by-step guide to deploy Python code to AWS Lambda using Serverless Framework.

1. Serverless Setup

1. Make sure Node has been installed before installing Serverless. Otherwise, go to Node official page and follow the instructions on how to install Node.

2. Verify that both Node and npm have been installed on your computer.

   node --version
   

   Example Output

   v16.15.0
   

   npm --version
   

   Example Output

   8.5.5
   

3. Install Serverless globally

   npm install -g serverless
   

4. Verify that Serverless has been successfully installed

   serverless --version
   

   Example Output

   Framework Core: 3.0.1 (standalone)
   Plugin: 6.0.0
   SDK: 4.3.0
   

5. Initialize Serverless Python project

   serverless
   

6. Select any Python template that you are interested in. If you are new to AWS and Serverless, "AWS - Python - Starter" may be a good starting point.

7. Next, name your project (ideally no whitespace between words). Then, Serverless will automatically create a directory with given name and include necessary boilerplate and template files.

8. Select no when you are prompted to login or deploy the project.

9. Change to project directory and start developing your Serverless project.

   cd <PROJECT_DIR>
   

2. Lambda Function Setup

3. How to Add Third-Party Packages

Two of the most common ways to add third-party packages to Serverless Python project.

1. pip + virtualenv + serverless-python-requirements
2. pipenv + serverless-python-requirements (recommended)

I personally prefer the second option for the following reasons.

1. pipenv allows developer to split development and production dependencies (similar to package.json in Node project)
2. Similar to package.json, pipenv will store the project configuration in Pipfile. The file allows granular configuration and control, which is apparently not available in requirements.txt file.
3. pipenv locks the version of the dependencies and store the details in Pipfile.lock to guarantee consistent, manageable dependencies.
4. serverless-python-requirements supports pipenv and will only pick up the production dependencies upon packaging and deployment. Hence, it significantly reduces the size of Lambda functions and layers.
5. The dependencies are installed in $HOME/.cache/pip/wheel/ directory, outside project directory. It means your project directory size remains small and is not bloated with dependencies.

However, feel free to choose any option that works best for you.

3.1. Option 1: pip + virtualenv + serverless-python-requirements

3.1.1. Windows

1. Create a new virtual environment inside project folder, e.g. venv

   Command

   python -m venv <VIRTUAL_ENVIRONMENT_NAME>
   

   Example

   Create a new virtual environment, venv inside project folder

   python -m venv .\venv
   

2. Activate the virtual environment

   Command

   .\<VIRTUAL_ENVIRONMENT_NAME>\Scripts\activate
   

   Activate virtual environment named venv

   Example

   .\venv\Scripts\activate
   

3. Check whether pip is installed

   Command

   pip -V
   

4. (Optional) Upgrade pip to the latest version

   python -m pip install --upgrade pip
   

   Note

   If pip is accidentally removed, or not installed, it can be re-installed using the following command

   python -m ensurepip --upgrade
   

5. Check the list of installed Python packages

   pip list
   

   Note

   The list should only contain minimum installation, most likely pip and setuptools. If it contains any third-party packages, it indicates that the virtual environment has not been activated.

6. Create requirements.txt file and list the needed third-party packages

   Example

   requests
   pyyaml
   numpy
   

   Note

   The file may also contain requirement specifiers if you want to narrow down or select specific version to be installed in the virtual environment.

7. Install these packages by running the following command

   pip install -r requirements.txt
   

   Note

   -r flag tells pip to install recursively (in case of listing multi packages in the requirements)

8. Check whether these packages are successfully installed

   pip list
   

   Example output

   If requests package is successfully installed, it will be listed on the output

   Package            Version
   ------------------ ---------
   ...                 ...
   requests            2.26.0
   ...                 ...
   

9. Initialize Node application

   npm init
   

   Note

   package.json should exist in the project directory if the initialization is successful.

10. Install serverless-python-requirements plugin learn more

    npm install --save serverless-python-requirements
    

11. Add these lines to our serverless.yml after installation is complete

    plugins:
      - serverless-python-requirements
    
    custom:
      pythonRequirements:
        dockerizePip: true

3.2. Option 2: pipenv + serverless-python-requirements (recommended)

1. Install pipenv

   pip install pipenv
   

2. Check for pipenv version to ensure the package is successfully installed.

   python -m pipenv --version
   

   Example Output

   pipenv, version 2022.5.2
   

3. (Optional) Install pipenv inside virtual environment to enable pipenv command

   python -m pipenv install pipenv --dev
   

   Note

   As this is the first third-party package installation, pipenv will automatically generate Pipfile and Pipfile.lock. The --dev option tells pipenv to install the local pipenv as development dependency. It will be excluded during deployment.

4. Activate virtual environment

   python -m pipenv shell
   

5. Check the packages installed inside the virtual environment

   pip list
   

   Alternatively, pipenv also comes with a feature to check dependency graph.

   pipenv graph
   

   If the virtual environment is configured properly, it should only contain pip, setuptools, pipenv and their dependencies.

6. Install all necessary dependencies.

   Production dependency

   pipenv install <PACKAGE_NAME>
   

   Development dependency

   pipenv install <PACKAGE_NAME> --dev
   

7. Initialize Node application

   npm init
   

   Note

   package.json should exist in the project directory if the initialization is successful.

8. Install serverless-python-requirements plugin learn more

   npm install --save serverless-python-requirements
   

9. Add the following config to serverless.yml after installation is complete

   plugins:
     - serverless-python-requirements
   
   custom:
     pythonRequirements:
       dockerizePip: true

4. (Optional) Lambda Layer Configuration

If the Lambda functions of Serverless application use the same dependencies, you can take full advantage of Lambda layer to further reduce the function size by packing it as centralized dependencies.

Before

┌─────────────────────┐         ┌─────────────────────┐
│ Function 1          │         │ Function n          │
├─────────────────────┤         ├─────────────────────┤
│ Main function  5kB  │         │ Main function 10kB  │
├─────────────────────┤   ...   ├─────────────────────┤
│ Dependencies   2MB  │         │ Dependencies   2MB  │
└─────────────────────┘         └─────────────────────┘
 Total size 2.005MB               Total size 2.010MB

After

┌─────────────────────┐         ┌─────────────────────┐
│ Function 1          │         │ Function n          │
├─────────────────────┤         ├─────────────────────┤
│ Main function  5kB  │   ...   │ Main function 10kB  │
└─────────────────────┘         └─────────────────────┘
 Total size 5kB                   Total size 10kB
         │                                 │
         └─────────────────┬───────────────┘
                           │
                ┌─────────────────────┐
                | Lambda Layer        |
                ├─────────────────────┤
                │ Dependencies   2MB  │
                └─────────────────────┘

Layer configuration in serverless.yml

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true
    layer:
      name: ${self:provider.stage}-pythonLambdaLayer
      description: Python requirements lambda layer
      #...

# explicitly filter and select necessary files and directories
package:
  individually: true
  excludeDevDependencies: true
  patterns:
    - "!./**" # exclude all files and directories
    - "./src/**" # select only 'src' directory and its files and subdirectories

# link lambda function to dependency layer
functions:
  myhandler:
    handler: src/path/to/my.handler
    #..
    layers:
      - Ref: PythonRequirementsLambdaLayer

5. Deployment to AWS

Command

serverless deploy [--aws-profile <AWS_PROFILE_NAME>] [--stage <DEPLOYMENT_STAGE>]

Note

Both aws-profile and stage are optional, however, you are expected to have at least 1 profile (default profile) in your machine. Otherwise, Serverless will never know where to deploy nor gain access to your AWS account programmatically.

Below is how Serverless looks for the AWS profile, sorted by priority (high to low):

1. Command line arguments (--aws-profile <AWS_PROFILE_NAME>)

2. Deployment script (typically serverless.yml)

3. Default AWS profile

   • Linux & MacOS: $HOME/.aws/{credentials, config}
   • Windows: C:\Users\<USER_NAME>\.aws\{credentials, config}

6. (Optional) Final Check

To make sure the third-party package has been installed and deployed, check the .serverless directory and it should have similar content as shown below.

your-function-name
├── .serverless
│   ├── requirements
│   │    ├── ...
│   │    └── ...
│   ├── cloudformation-template-create-stack.json
│   ├── cloudformation-template-update-stack.json
│   ├── pythonRequirements.zip
│   ├── function_1.zip
│   ├── ...
│   ├── function_n.zip
│   ├── requirements.txt
│   └── serverless-state.json
├── node_modules
│   ├── ...
│   └── ...
├── venv                 # virtual environment config + local dependencies
│   ├── ...              # (not exist if configuration uses pipenv)
│   └── ...
├── ...
├── src
│   ├── function_1
│   │    ├── ...
│   │    └── handler.py
│   ├── ...
│   └── function_n
│        ├── ...
│        └── handler.py
├── handler.py
├── ...
├── package.json
├── Pipfile              # (not exist if configuration uses pip + venv)
├── Pipfile.lock         # (not exist if configuration uses pip + venv)
├── requirements.txt     # (not exist if configuration uses pipenv)
└── serverless.yml