"sam build" omits cpython DLLs?

[eighttails]

Description

I want to deploy pytorch library with my Lambda function.
Official pytorch library in pip repository is too large to bundle with Lambda.
So I try to use this one https://github.com/ryfeus/lambda-packs/tree/master/pytorch which is not available via pip.
I extracted pytorch libs to Lambda function dir and did sam build but deployed package seems not contain cpython native libraries.

Steps to reproduce

The following script demonstrates the issue.

#!/bin/bash

sam init -r python3.6
cd sam-app/hello_world
echo "psycopg2" > requirements.txt

#extract pytorch modules to the same folder as app.py
wget -q https://github.com/ryfeus/lambda-packs/raw/master/pytorch/pack.zip
unzip -q pack.zip
rm pack.zip
#list of extracted cpython modules
find . |grep x86_64 > ../../source.txt

cd ..
sam build -u --debug
#list of deployed cpython modules
find .aws-sam|grep x86_64 > ../dest.txt

./torch/_dl.cpython-36m-x86_64-linux-gnu.so and ./torch/_C.cpython-36m-x86_64-linux-gnu.so
in source.txt are necessary to import pytorch but they are not contained in dest.txt(built package).
So import torch in Lambda function makes error.

2019-02-25 22:12:57 Invoking app.lambda_handler (python3.6)

Fetching lambci/lambda:python3.6 Docker container image......
2019-02-25 22:12:59 Mounting /home/tadahito/develop/sam-issue/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro inside runtime container
START RequestId: 325e0d78-3f8f-4579-bd70-0683ba13b6e1 Version: $LATEST                                                           
No module named 'torch._C': ModuleNotFoundError
Traceback (most recent call last):
  File "/var/task/app.py", line 35, in lambda_handler
    import torch
  File "/var/task/torch/__init__.py", line 84, in <module>
    from torch._C import *
ModuleNotFoundError: No module named 'torch._C'

END RequestId: 325e0d78-3f8f-4579-bd70-0683ba13b6e1
REPORT RequestId: 325e0d78-3f8f-4579-bd70-0683ba13b6e1 Duration: 11 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 19 MB

{"errorMessage": "No module named 'torch._C'", "errorType": "ModuleNotFoundError", "stackTrace": [["/var/task/app.py", 35, "lambda_handler", "import torch"], ["/var/task/torch/__init__.py", 84, "<module>", "from torch._C import *"]]}    

While .aws-sam/build/HelloWorldFunction/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so installed via requirements.txt is correctly contained in the package.

Observed result

[+] Initializing project structure...

Project generated: ./sam-app

Steps you can take next within the project folder
===================================================
[*] Invoke Function: sam local invoke HelloWorldFunction --event event.json
[*] Start API Gateway locally: sam local start-api

Read sam-app/README.md for further instructions

[*] Project initialization is now complete
2019-02-25 21:36:30 Using SAM Template at /home/tadahito/develop/sam-issue/sam-app/template.yaml
2019-02-25 21:36:30 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-02-25 21:36:30 Changing event name from before-call.apigateway to before-call.api-gateway
2019-02-25 21:36:30 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-02-25 21:36:30 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-02-25 21:36:30 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-02-25 21:36:30 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-02-25 21:36:30 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-02-25 21:36:30 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-02-25 21:36:30 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-02-25 21:36:30 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-02-25 21:36:30 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-02-25 21:36:30 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-02-25 21:36:30 Changing event name from before-call.apigateway to before-call.api-gateway
2019-02-25 21:36:30 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-02-25 21:36:30 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-02-25 21:36:30 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-02-25 21:36:30 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-02-25 21:36:30 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-02-25 21:36:30 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-02-25 21:36:30 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-02-25 21:36:30 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-02-25 21:36:30 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-02-25 21:36:30 'build' command is called
2019-02-25 21:36:30 Starting Build inside a container
2019-02-25 21:36:30 Looking for credentials via: env
2019-02-25 21:36:30 Looking for credentials via: assume-role
2019-02-25 21:36:30 Looking for credentials via: shared-credentials-file
2019-02-25 21:36:30 Found credentials in shared credentials file: ~/.aws/credentials
2019-02-25 21:36:30 Loading JSON file: /home/tadahito/develop/sam_sample/venv/lib/python3.7/site-packages/botocore/data/endpoints.json
2019-02-25 21:36:30 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fcb395a11e0>
2019-02-25 21:36:30 Loading JSON file: /home/tadahito/develop/sam_sample/venv/lib/python3.7/site-packages/botocore/data/serverlessrepo/2017-09-08/service-2.json
2019-02-25 21:36:30 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x7fcb395e39d8>
2019-02-25 21:36:30 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-25 21:36:30 Setting serverlessrepo timeout as (60, 60)
2019-02-25 21:36:30 Loading JSON file: /home/tadahito/develop/sam_sample/venv/lib/python3.7/site-packages/botocore/data/_retry.json
2019-02-25 21:36:30 Registering retry handlers for service: serverlessrepo
2019-02-25 21:36:30 No Parameters detected in the template
2019-02-25 21:36:30 2 resources found in the template
2019-02-25 21:36:30 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-02-25 21:36:30 Trying paths: ['/home/tadahito/.docker/config.json', '/home/tadahito/.dockercfg']
2019-02-25 21:36:30 No config file found
2019-02-25 21:36:30 Trying paths: ['/home/tadahito/.docker/config.json', '/home/tadahito/.dockercfg']
2019-02-25 21:36:30 No config file found
2019-02-25 21:36:30 Building resource 'HelloWorldFunction'
2019-02-25 21:36:30 Trying paths: ['/home/tadahito/.docker/config.json', '/home/tadahito/.dockercfg']
2019-02-25 21:36:30 No config file found
2019-02-25 21:36:30 Trying paths: ['/home/tadahito/.docker/config.json', '/home/tadahito/.dockercfg']
2019-02-25 21:36:30 No config file found
2019-02-25 21:36:30 http://localhost:None "GET /v1.35/images/lambci/lambda:build-python3.6/json HTTP/1.1" 200 None
2019-02-25 21:36:30 Looking for auth config
2019-02-25 21:36:30 No auth config in memory - loading from filesystem
2019-02-25 21:36:30 Trying paths: ['/home/tadahito/.docker/config.json', '/home/tadahito/.dockercfg']
2019-02-25 21:36:30 No config file found
2019-02-25 21:36:30 Looking for auth entry for 'docker.io'
2019-02-25 21:36:30 No entry found
2019-02-25 21:36:30 No auth config found
2019-02-25 21:36:34 http://localhost:None "POST /v1.35/images/create?tag=build-python3.6&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:build-python3.6 Docker container image......
2019-02-25 21:36:34 Mounting /home/tadahito/develop/sam-issue/sam-app/hello_world as /tmp/samcli/source:ro inside runtime container
2019-02-25 21:36:35 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2019-02-25 21:36:35 http://localhost:None "GET /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/json HTTP/1.1" 200 None
2019-02-25 21:36:35 http://localhost:None "GET /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/json HTTP/1.1" 200 None
2019-02-25 21:36:35 http://localhost:None "POST /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/start HTTP/1.1" 204 0
2019-02-25 21:36:35 http://localhost:None "GET /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/json HTTP/1.1" 200 None
2019-02-25 21:36:35 http://localhost:None "POST /containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/attach?stdout=1&stderr=1&logs=1&stream=1&stdin=0 HTTP/1.1" 101 0
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch
Full dependency closure: {psycopg2==2.7.7(wheel)}
initial compatible: {psycopg2==2.7.7(wheel)}
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: {psycopg2==2.7.7(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {psycopg2==2.7.7(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {psycopg2==2.7.7(wheel)}
Final compatible: {psycopg2==2.7.7(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
2019-02-25 21:36:37 Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
2019-02-25 21:36:37 Build inside container was successful. Copying artifacts from container to host
2019-02-25 21:36:37 http://localhost:None "GET /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/json HTTP/1.1" 200 None
2019-02-25 21:36:37 Copying from container: /tmp/samcli/artifacts/. -> /home/tadahito/develop/sam-issue/sam-app/.aws-sam/build/HelloWorldFunction
2019-02-25 21:36:37 http://localhost:None "GET /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/archive?path=%2Ftmp%2Fsamcli%2Fartifacts%2F. HTTP/1.1" 200 None
2019-02-25 21:36:39 http://localhost:None "GET /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403/json HTTP/1.1" 200 None
2019-02-25 21:36:39 http://localhost:None "DELETE /v1.35/containers/9f8389b27fe2ad1074e0cb024a6fc95ded3403398d04ce5642ab338c9a623403?v=False&link=False&force=True HTTP/1.1" 204 0
2019-02-25 21:36:39 Build inside container succeeded

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Expected result

All cpython DLLs are contained in deploy package.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS:debian testing x64
2. sam --version:SAM CLI, version 0.11.0

Add --debug flag to command you are running

[sriram-mv]

you could always create a layer with pytorch. There's an example here: https://github.com/mattmcclean/sam-pytorch-example and see if that works.

[tomotway]

Hi, just to state that I hit the same issue today which led me here. I was able to work around by duplicating one of the packages I was bundling up into the requirements.txt but it caused me confusion for a while. I notice that the "aws cloudformation package" command seems to keep these files, not sure if this is relevant at all....

[pickfire]

Same issue for linux libraries *.so files which was omitted with sam build, I wonder if there is a way to do this without using layers.

[CausticYarn]

I am having the same issue with a custom extension I created. No *.so files are copied into the build folder. I know this used to work as I build an application that packaged lxml. I can't use layers for this at the moment due to our build pipeline.

[mndeveci]

Lambda builders explicitly exclude .so files which was configured here; https://github.com/aws/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/workflow.py#L33

You can use custom makefile builds to fix this issue, which was explained here: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/building-custom-runtimes.html