[Bug] - lambda/nodejs:20 AL2023 based container: aws command doesn't work after installing awscli-2

[gtach2o]

Describe the bug
aws command doesn't work after installing

To Reproduce
Steps to reproduce the behavior:

FROM public.ecr.aws/lambda/nodejs:20

RUN dnf install -y awscli-2

RUN aws --version

build the image

docker build --platform=linux/amd64 --pull -t lambda-node -f Dockerfile --progress=plain .

fails with error

#6 [3/3] RUN aws --version
#6 0.470 Traceback (most recent call last):
#6 0.470   File "/usr/bin/aws", line 19, in <module>
#6 0.471     import awscli.clidriver
#6 0.471   File "/usr/lib/python3.9/site-packages/awscli/clidriver.py", line 21, in <module>
#6 0.471     import botocore.session
#6 0.471   File "/usr/lib/python3.9/site-packages/awscli/botocore/session.py", line 27, in <module>
#6 0.472     import botocore.client
#6 0.472   File "/usr/lib/python3.9/site-packages/awscli/botocore/client.py", line 16, in <module>
#6 0.472     from botocore import UNSIGNED, waiter, xform_name
#6 0.472   File "/usr/lib/python3.9/site-packages/awscli/botocore/waiter.py", line 17, in <module>
#6 0.472     from botocore.docs.docstring import WaiterDocstring
#6 0.472   File "/usr/lib/python3.9/site-packages/awscli/botocore/docs/__init__.py", line 15, in <module>
#6 0.472     from botocore.docs.service import ServiceDocumenter
#6 0.472   File "/usr/lib/python3.9/site-packages/awscli/botocore/docs/service.py", line 13, in <module>
#6 0.472     from botocore.docs.bcdoc.restdoc import DocumentStructure
#6 0.473   File "/usr/lib/python3.9/site-packages/awscli/botocore/docs/bcdoc/restdoc.py", line 15, in <module>
#6 0.473     from botocore.compat import OrderedDict
#6 0.473   File "/usr/lib/python3.9/site-packages/awscli/botocore/compat.py", line 31, in <module>
#6 0.473     from urllib3 import exceptions
#6 0.473   File "/usr/lib/python3.9/site-packages/urllib3/__init__.py", line 7, in <module>
#6 0.474     from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
#6 0.474   File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 11, in <module>
#6 0.474     from .exceptions import (
#6 0.474   File "/usr/lib/python3.9/site-packages/urllib3/exceptions.py", line 2, in <module>
#6 0.474     from .packages.six.moves.http_client import IncompleteRead as httplib_IncompleteRead
#6 0.474   File "/usr/lib/python3.9/site-packages/urllib3/packages/__init__.py", line 3, in <module>
#6 0.474     from . import ssl_match_hostname
#6 0.474   File "/usr/lib/python3.9/site-packages/urllib3/packages/ssl_match_hostname.py", line 1, in <module>
#6 0.474     from ssl import match_hostname, CertificateError
#6 0.474   File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
#6 0.475     import _ssl             # if we can't import it, let the error propagate
#6 0.475 ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0
#6 ERROR: process "/bin/sh -c aws --version" did not complete successfully: exit code: 1
------
 > [3/3] RUN aws --version:
0.474     from .exceptions import (
0.474   File "/usr/lib/python3.9/site-packages/urllib3/exceptions.py", line 2, in <module>
0.474     from .packages.six.moves.http_client import IncompleteRead as httplib_IncompleteRead
0.474   File "/usr/lib/python3.9/site-packages/urllib3/packages/__init__.py", line 3, in <module>
0.474     from . import ssl_match_hostname
0.474   File "/usr/lib/python3.9/site-packages/urllib3/packages/ssl_match_hostname.py", line 1, in <module>
0.474     from ssl import match_hostname, CertificateError
0.474   File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
0.475     import _ssl             # if we can't import it, let the error propagate
0.475 ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0

Expected behavior
aws --version prints it's version

[elsaco]

It's not the awscli-2 that's causing the issue. In a python shell try importing ssl:

bash-5.2# python
Python 3.9.16 (main, Sep  8 2023, 00:00:00)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0

[gtach2o]

dnf install -y awscli-2 installs a lot if stuff like perl etc. Why ssl is not a dependency for awscli-2?

[gtach2o]

also it is not possible to use pip because there will be an error that ssl is not enabled or something like that.

[nmeyerhans]

As far as I can tell, this seems to be unique to the lambda containers. Note the following:

public.ecr.aws/amazonlinux/amazonlinux:2023-minimal

bash-5.2# ldd /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fffe25df000)
        libssl.so.3 => /lib64/libssl.so.3 (0x00007fb0033f4000)
        libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fb002fbb000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb002db3000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb002d99000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb0034d1000)
bash-5.2# rpm -qf /lib64/libcrypto.so.3
openssl-libs-3.0.8-1.amzn2023.0.10.x86_64

public.ecr.aws/lambda/nodejs:20

bash-5.2# ldd /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffcc5f58000)
        libssl.so.3 => /var/lang/lib/libssl.so.3 (0x00007fe4689d2000)
        libcrypto.so.3 => /var/lang/lib/libcrypto.so.3 (0x00007fe46835d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe468155000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe468150000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe46814b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe468ca2000)
bash-5.2# rpm -qf /var/lang/lib/libcrypto.so.3
file /var/lang/lib/libcrypto.so.3 is not owned by any package

So the lambda containers are installing some other libcrypto?

[elsaco]

openssl-libs is present in the lambda container also. If you copy /usr/lib64/libsso.so.3 and /usr/lib64/libcrypto.so.3 and override the files in /var/lang/lib/ then all symbols are resolved and aws --version works:

bash-5.2# aws --version
aws-cli/2.14.5 Python/3.9.16 Linux/5.15.146.1-microsoft-standard-WSL2 source/x86_64.amzn.2023 prompt/off

However, the files in /var/lang/lib/ are runtime libraries for the lambdas and might nuke the container!

[gtach2o]

openssl-libs is present in the lambda container also. If you copy /usr/lib64/libsso.so.3 and /usr/lib64/libcrypto.so.3 and override the files in /var/lang/lib/ then all symbols are resolved and aws --version works:

bash-5.2# aws --version
aws-cli/2.14.5 Python/3.9.16 Linux/5.15.146.1-microsoft-standard-WSL2 source/x86_64.amzn.2023 prompt/off

However, the files in /var/lang/lib/ are runtime libraries for the lambdas and might nuke the container!

I don’t think this can be a solution.

[elsaco]

The lambda container has LD_LIBRARY_PATH environment variable set to /var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib so libraries in /var/lang/lib/ will take precedence over the ones in/usr/lib64/

Run printenv or echo $LD_LIBRARY_PATH to show the contents.

[gtach2o]

It's a docker image. The echo output is the same.
My question was that SSL is not properly configured. It is not even possible to run pip command.
IS this only docker image reflated issue. Should I better ask there https://github.com/aws/aws-lambda-base-images ?

[stewartsmith]

It's a docker image. The echo output is the same. My question was that SSL is not properly configured. It is not even possible to run pip command. IS this only docker image reflated issue. Should I better ask there https://github.com/aws/aws-lambda-base-images ?

The image in question is made by the Lambda team on top of the base AL2023 minimal container image - so cutting an issue over there is likely a good idea. Let's keep this one open here as well until we resolve the issue.

[gtach2o]

Maybe they messed it up by setting LD_LIBRARY_PATH environment variable value.
I got it working with ENV LD_LIBRARY_PATH="" workaround.

[elsaco]

The nodejs20 is built on top of aws-lambda-base-images and the Dockerfile sets some environment variables:

FROM scratch
ADD x86_64/22f00718d013305cd61f4906577540cc6804d6603602567f1bff83781ebe6893.tar.xz /
ADD x86_64/88f2931da348e4df07deb80db8d3d1f42f27bac499f4d786543b5d3520b360c8.tar.xz /
ADD x86_64/8ac98ca8d59bf7ca6c4889d72b65a954cf14af58b8e5672c42469dc0c36cd8a4.tar.xz /
ADD x86_64/9a75445380a50339a32e7a9710a586be0a0203193182c5c4a65789c540e99318.tar.xz /
ADD x86_64/e8312a6d39ae266ae459772d2cea9fac729fe7ed9f58f28bc0cccf6d7b227c4c.tar.xz /
ADD x86_64/feb815887f22ce6ea2814b716320e5a1b255f00f3ed05a1f21f0d8640b117208.tar.xz /

ENV LANG=en_US.UTF-8
ENV TZ=:/etc/localtime
ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin
ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib
ENV LAMBDA_TASK_ROOT=/var/task
ENV LAMBDA_RUNTIME_DIR=/var/runtime

WORKDIR /var/task

ENTRYPOINT ["/lambda-entrypoint.sh"]

See details at https://raw.githubusercontent.com/aws/aws-lambda-base-images/nodejs20.x/Dockerfile.nodejs20.x

LD_LIBRARY_PATH is of importance to the node's runtime but not to aws shell. So when invoking the shell just run LD_LIBRARY_PATH='' aws --version w/out having to mess the environment set by the publisher.

[dtbaker]

Oof, this took me a while to find. Hopefully this gets picked up by Google to save someone else time:

If you are on Amazon Linux 2023 and pip install is failing with an error message:

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

Then the issue is python unable to import the ssl module, as explained above. To verify the issue, start python and try to import the ssl module:

$ python
>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/ssl.py", line 99, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: /usr/lib64/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so: undefined symbol: COMP_get_type, version OPENSSL_3.0.0

The work around for this issue is to prefix pip install with LD_LIBRARY_PATH='' as explained above. Such as:

LD_LIBRARY_PATH='' pip install your_package_here