Python version discrepancy when deploying sandbox vs production

[jsanders5]

Environment information

System:
  OS: Linux 5.15 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
  CPU: (16) x64 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
  Memory: 22.09 GB / 31.20 GB
  Shell: /usr/bin/zsh
Binaries:
  Node: 23.9.0 - ~/.nvm/versions/node/v23.9.0/bin/node
  Yarn: undefined - undefined
  npm: 10.9.2 - ~/.nvm/versions/node/v23.9.0/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/auth-construct: 1.7.0
  @aws-amplify/backend: 1.15.0
  @aws-amplify/backend-ai: Not Found
  @aws-amplify/backend-auth: 1.6.0
  @aws-amplify/backend-cli: 1.4.12
  @aws-amplify/backend-data: 1.5.0
  @aws-amplify/backend-deployer: 1.1.19
  @aws-amplify/backend-function: 1.13.0
  @aws-amplify/backend-output-schemas: 1.5.0
  @aws-amplify/backend-output-storage: 1.2.0
  @aws-amplify/backend-secret: 1.3.0
  @aws-amplify/backend-storage: 1.3.0
  @aws-amplify/cli-core: 1.4.0
  @aws-amplify/client-config: 1.6.0
  @aws-amplify/data-construct: 1.16.0
  @aws-amplify/data-schema: 1.20.4
  @aws-amplify/deployed-backend-client: 1.6.0
  @aws-amplify/form-generator: 1.0.4
  @aws-amplify/model-generator: 1.1.0
  @aws-amplify/platform-core: 1.7.0
  @aws-amplify/plugin-types: 1.9.0
  @aws-amplify/sandbox: 1.2.11
  @aws-amplify/schema-generator: 1.2.7
  aws-amplify: 6.13.3
  aws-cdk: 2.1003.0
  aws-cdk-lib: 2.190.0
  typescript: 5.8.2
No AWS environment variables
No CDK environment variables

Describe the bug

I have a python Lambda in my Amplify Gen 2 app that depends on the pillows python package. When I set the runtime to PYTHON_3_10, my lambda function returns the following error:

[gen-serials] 12:05:59 PM INIT_START Runtime Version: python:3.10.v64   Runtime Version ARN: arn:aws:lambda:us-west-1::runtime:d56bdc8857b2109a058de2ab50bfb80ac2aa600be6f7855a436859944e5fe54b
[gen-serials] 12:06:00 PM [ERROR] Runtime.ImportModuleError: Unable to import module 'index': cannot import name '_imaging' from 'PIL' (/var/task/PIL/__init__.py)
Traceback (most recent call last):
[gen-serials] 12:06:00 PM INIT_REPORT Init Duration: 1108.12 ms Phase: init     Status: error   Error Type: Runtime.ImportModuleError
[gen-serials] 12:06:01 PM [ERROR] Runtime.ImportModuleError: Unable to import module 'index': cannot import name '_imaging' from 'PIL' (/var/task/PIL/__init__.py)
Traceback (most recent call last):
[gen-serials] 12:06:01 PM INIT_REPORT Init Duration: 1045.59 ms Phase: invoke   Status: error   Error Type: Runtime.ImportModuleError
[gen-serials] 12:06:01 PM START RequestId: 2d4e53c7-ab1f-4729-9b21-9d45d79539fb Version: $LATEST
[gen-serials] 12:06:01 PM Unknown application error occurred
Runtime.ImportModuleError
[gen-serials] 12:06:01 PM END RequestId: 2d4e53c7-ab1f-4729-9b21-9d45d79539fb
[gen-serials] 12:06:01 PM REPORT RequestId: 2d4e53c7-ab1f-4729-9b21-9d45d79539fb        Duration: 1046.27 ms  Billed Duration: 1047 ms        Memory Size: 3008 MB    Max Memory Used: 40 MB

If I manually change the runtime to v3.9 in the AWS Lambda console, the error goes away. I believe the issue is due to my local python version. When I deploy the lambda via sandbox, it packages the lambda code using Python 3.9, my local machine python version. Even though I've set the runtime to v3.10 in the resource.ts, it is still going to package it using my local python version.

I have a similar issue in production. When the python runtime is set to 3.9 in the resource.ts, then i deploy to production via github CI/CD, I will get a similar error because the lambda zip was packaged on the remote build server using Python 3.10, but the runtime is set to 3.9.

The current workaround is to set the runtime to 3.10 such that production works, then locally change the runtime to 3.9 and deploy my sandbox.

My thought is that the build process could use docker to package the lambda code using the appropriate python runtime specified in resource.ts which would remove the dependency on the build machine altogether.

Reproduction steps

You will need to be able to deploy from the sandbox from a machine that uses python 3.9. I use WSL on Debian 11.11.

1. Create a python lambda function, setting the runtime to 3.10
2. Import and use the pillows package: from PIL import Image
3. Deploy to sandbox environemnt
4. Invoke the python lambda

[sobolk]

@jsanders5 Could you share a snippet that shows how do you define python function in your app ?

It is possible to use docker with Amplify Gen2 apps, see https://docs.amplify.aws/react/build-a-backend/functions/custom-functions/#docker .

[jsanders5]

@sobolk,

My function is defined below.

I found this: https://gallery.ecr.aws/lambda/python.

So to bundle with python 3.10, would I just change DockerImage.fromRegistry("dummy") to DockerImage.fromRegistry("public.ecr.aws/lambda/python:3.10.2025.04.28.12")?

export const functionHandler= defineFunction(
    (scope) =>
        new Function(scope, "function", {
            handler: "index.handler",
            runtime: Runtime.PYTHON_3_10,
            memorySize: 3008,
            timeout: Duration.minutes(14),
            code: Code.fromAsset(functionDir, {
                bundling: {
                    image: DockerImage.fromRegistry("dummy"), // replace with desired image from AWS ECR Public Gallery
                    local: {
                        tryBundle(outputDir: string) {
                            execSync(
                                `python3 -m pip install -r ${path.join(functionDir, "requirements.txt")} -t ${path.join(outputDir)} --platform manylinux2014_x86_64 --only-binary=:all:`
                            );
                            execSync(`cp -r ${functionDir}/* ${path.join(outputDir)}`);
                            return true;
                        },
                    },
                },
            }),
        }),
    {
        resourceGroupName: "data" // Optional: Groups this function with auth resource
    }
);

[jsanders5]

@sobolk,

My function is defined below.

I found this: https://gallery.ecr.aws/lambda/python.

So to bundle with python 3.10, would I just change DockerImage.fromRegistry("dummy") to DockerImage.fromRegistry("public.ecr.aws/lambda/python:3.10.2025.04.28.12")?

export const functionHandler= defineFunction(
    (scope) =>
        new Function(scope, "function", {
            handler: "index.handler",
            runtime: Runtime.PYTHON_3_10,
            memorySize: 3008,
            timeout: Duration.minutes(14),
            code: Code.fromAsset(functionDir, {
                bundling: {
                    image: DockerImage.fromRegistry("dummy"), // replace with desired image from AWS ECR Public Gallery
                    local: {
                        tryBundle(outputDir: string) {
                            execSync(
                                `python3 -m pip install -r ${path.join(functionDir, "requirements.txt")} -t ${path.join(outputDir)} --platform manylinux2014_x86_64 --only-binary=:all:`
                            );
                            execSync(`cp -r ${functionDir}/* ${path.join(outputDir)}`);
                            return true;
                        },
                    },
                },
            }),
        }),
    {
        resourceGroupName: "data" // Optional: Groups this function with auth resource
    }
);

This did not work, below is the updated code:

export const functionHandler = defineFunction(
    (scope) =>
        new Function(scope, "function", {
            handler: "index.handler",
            runtime: Runtime.PYTHON_3_10,
            memorySize: 3008,
            timeout: Duration.minutes(14),
            code: Code.fromAsset(functionDir, {
                bundling: {
                    image: DockerImage.fromRegistry("public.ecr.aws/lambda/python:3.10.2025.04.28.12"), // replace with desired image from AWS ECR Public Gallery
                    local: {
                        tryBundle(outputDir: string) {
                            execSync(
                                `python3 -m pip install -r ${path.join(functionDir, "requirements.txt")} -t ${path.join(outputDir)} --platform manylinux2014_x86_64 --only-binary=:all:`
                            );
                            execSync(`cp -r ${functionDir}/* ${path.join(outputDir)}`);
                            return true;
                        },
                    },
                },
            }),
        }),
    {
        resourceGroupName: "data" // Optional: Groups this function with auth resource
    }
);

[sobolk]

The local option is suppressing docker build. unless tryBundle returns false. See https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.BundlingOptions.html#local .

I was able to follow this post aws/aws-cdk#24595 to get something working.

I.e. I moved commands to command option (described here https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.BundlingOptions.html#command).

The code that worked for me looks like this.

export const sayHelloFunctionHandler = defineFunction(
    (scope) =>
        new Function(scope, "say-hello", {
            handler: "index.handler",
            runtime: Runtime.PYTHON_3_10, // or any other python version
            timeout: Duration.seconds(20), //  default is 3 seconds
            code: Code.fromAsset(functionDir, {
                bundling: {
                    image: Runtime.PYTHON_3_10.bundlingImage, // replace with desired image from AWS ECR Public Gallery
                    command: ['sh', '-c', 'python -m pip install -r requirements.txt -t /asset-output && cp ./* /asset-output']
                },
            }),
        }),
    {
        resourceGroupName: "auth" // Optional: Groups this function with auth resource
    }
);

[jsanders5]

Hmm, I'm not able to get this to work, and not sure I want to spend more time on it.

Not sure exactly what this error message means:

Failed to instantiate custom function provider
Caused By: Failed to bundle asset amplify-arkmfg-jsanders-sandbox-89f9dbbfdc/data/gen-serials/Code/Stage, bundle output is located at /home/jsanders/workbench/git/ark/ark-mfg/.amplify/artifacts/cdk.out/asset.84badee46d66c5cca0c28c6b1d3da0b07b6ac55932da8cd985cefc80c3ca34b4-building: Error: docker exited with status 125
--> Command: docker run --rm -u "1000:1000" -v "/home/jsanders/workbench/git/ark/ark-mfg/amplify/functions/gen-serials:/asset-input:delegated" -v "/home/jsanders/workbench/git/ark/ark-mfg/.amplify/artifacts/cdk.out/asset.84badee46d66c5cca0c28c6b1d3da0b07b6ac55932da8cd985cefc80c3ca34b4-building:/asset-output:delegated" -w "/asset-input" "public.ecr.aws/sam/build-python3.10" sh -c "python -m pip install -r requirements.txt -t /asset-output && cp ./* /asset-output"

code:

code: Code.fromAsset(functionDir, {
                bundling: {
                    image: Runtime.PYTHON_3_10.bundlingImage, // replace with desired image from AWS ECR Public Gallery
                    command: ['sh', '-c', ``python -m pip install -r ${path.join(functionDir, "requirements.txt")} -t /asset-output && cp ./* /asset-output\``]
                },
            }),

[sobolk]

This error message may indicate that docker daemon is not running on your machine. If this is the case the console output should have Cannot connect to the Docker daemon somewhere above the error.

You can run docker info in the terminal to verify that docker daemon is running or not. It prints error message if docker is not running.

Other than that. The code should look more like.

Code.fromAsset(functionDir, {
                bundling: {
                    image: Runtime.PYTHON_3_10.bundlingImage, // replace with desired image from AWS ECR Public Gallery
                    command: ['sh', '-c', ``python -m pip install -r requirements.txt -t /asset-output && cp ./* /asset-output``]
                },
            }),

When bundling with docker. the function directory is copied into the container and container command is executed in that directory. I.e. ${path.join(functionDir, "requirements.txt")} is not necessary.

Please note that command used for bundling is an example and may need fine tuning depending on how you structure python sources. In general docker based bundling works like this:

1. functionDir is copied into the container.
2. Container executes command in directory where functionDir was copied.
3. The command should execute steps necessary to build and bundle function code.
4. The results of bundling are picked up from /asset-output location in the container.