`local start-api --container-env-vars` option does not work

[milenkovicm]

Description:

It is not clear (to me) what --container-env-vars does when used with sam local start-api.

My understanding is that it should set environment variable to underlying docker container which runs lambda (of PackageType::Zip) in sam local environment without having that environment variable defined in template.yaml

when I run:

sam local start-api --container-env-vars sam_local_envs.json --warm-containers EAGER

sam does not set environment variables to container it starts.

Also, it is not clear what kind the format that file should have, is it same format like --env-vars ?

Steps to reproduce:

sam local start-api --container-env-vars sam_local_envs.json --warm-containers EAGER

where sam_local_envs.json is defined like (hoping that at least one key will work) :

{
    "P1": "V1",
    "Parameters": {
        "P2": "V2",
    }
}

function part of template.yaml is something like:

  LambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: bootstrap
      Runtime: provided.al2
      CodeUri: .
      Events:
        HelloApi:
          Type: Api
          Properties:
            Path: /{path_param}
            Method: get
            RestApiId:
              Ref: ApiGatewayApi
    Metadata:
      BuildMethod: makefile

Observed result:

docker inspect of container which sam started returns:

          "Env": [
              "AWS_SAM_LOCAL=true",
              "AWS_LAMBDA_FUNCTION_MEMORY_SIZE=128",
              "AWS_LAMBDA_FUNCTION_TIMEOUT=10",
              "AWS_LAMBDA_FUNCTION_HANDLER=bootstrap",
              "AWS_LAMBDA_FUNCTION_NAME=[REDACTED]",
              "AWS_LAMBDA_FUNCTION_VERSION=$LATEST",
              "AWS_LAMBDA_LOG_GROUP_NAME=aws/lambda/[REDACTED]",
              "AWS_LAMBDA_LOG_STREAM_NAME=$LATEST",
              "AWS_REGION=us-east-1",
              "AWS_DEFAULT_REGION=us-east-1",
              "AWS_ACCESS_KEY_ID=[REDACTED]",
              "AWS_SECRET_ACCESS_KEY=[REDACTED]",
              "AWS_ACCOUNT_ID=[REDACTED]",
              "RUST_BACKTRACE=1",
              "RUST_LOG=trace",
              "LANG=en_US.UTF-8",
              "TZ=:/etc/localtime",
              "PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin",
              "LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib",
              "LAMBDA_TASK_ROOT=/var/task",
              "LAMBDA_RUNTIME_DIR=/var/runtime"
          ],

Expected result:

environment variables which exists in --container-env-vars file should be set in container

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS
2. sam --version: SAM CLI, version 1.43.0
3. AWS region:

Add --debug flag to command you are running

[milenkovicm]

sam local start-api --help

says

  --container-env-vars PATH       JSON file containing environment variables
                                  to be set within the container environment

Which I read that container-env-vars is the only option needed to set container environment variables, debug-function or debug-port not needed to be set.

Looking at the code, I believe that problem is caused by having container_env_vars as part of DebugContext

Debug context should be changed at line

aws-sam-cli/samcli/commands/local/lib/debug_context.py

Line 28 in 0322c61

return bool(self.debug_ports)

from:

return bool(self.debug_ports)

to:

return bool(self.debug_ports) or bool(self.container_env_vars)

as debug context must be valid when it only has container_env_vars set (without port)

From there, next issue is at:

aws-sam-cli/samcli/local/docker/lambda_container.py

Line 261 in 0322c61

if not debug_ports:

where:

        debug_ports = debug_options.debug_ports
        container_env_vars = debug_options.container_env_vars
        if not debug_ports:
            return entry, {}

should be changed to:

        debug_ports = debug_options.debug_ports
        container_env_vars = debug_options.container_env_vars
        if not debug_ports:
            return entry, container_env_vars

we have no port, but container_env_vars should be preserved.

lastly,

aws-sam-cli/samcli/local/lambdafn/runtime.py

Line 388 in 0322c61

if debug_context and debug_context.debug_function != function_config.name:

        if debug_context and debug_context.debug_function != function_config.name:
            LOG.debug(
                "Disable the debugging for Lambda Function %s, as the passed debug function is %s",
                function_config.name,
                debug_context.debug_function,
            )
            debug_context = None

should be changed to (or similar)

            if debug_context and debug_context.debug_function != function_config.name:
            # or we check if `container_env_vars` is before we eliminate `debug_context`
            #if debug_context and debug_context.debug_function != function_config.name and not bool(debug_context.container_env_vars):    
            LOG.debug(
                "Disable the debugging for Lambda Function %s, as the passed debug function is %s",
                function_config.name,
                debug_context.debug_function,
            )
            #debug_context = None

as we can't kill the debug_context when there is container_env_vars

After updates, environment variables will be set from --container-env-vars file.

Regarding file format used for --container-env-vars its a dictionary:

{
    "P1": "V1",
    "P2": "V2",
}

values can be complex as well, as they will be stringified.

Hope this helps,
Regards

[hawflau]

Thanks for reporting. We will look into it.

[milenkovicm]

Just updated issue description adding Runtime: provided.al2 as its important for reproducibility

[qingchm]

I really appreciate the deep dive you did into our code! I do believe that there are some problems with our documentation on the option that is confusing you! As described in https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-lambda.html the --container-env-vars option is meant for debugging purposes. And for start-api command only when you specify a debug port we will treat this as running in debug mode. So this is actually an expected behaviour for

sam local start-api --container-env-vars sam_local_envs.json --warm-containers EAGER

If you believe that --container-env-vars needs to have wider application other than debugging purposes, we are open for some discussion to understand the use case and maybe we can start expanding this option from there! Thanks again for paying close attention to how our code works and deep diving!
Looking forward to continuing this discussion with you

[qingchm]

@milenkovicm

[milenkovicm]

Hi @qingchm
thank you for your feedback.

I have no other use case for --container-env-vars than debugging.

I'm not sure if agree that port should be mandatory to enable debuging, tools like java flight recorder will provide a file output.

Anyhow if thats the case then

aws-sam-cli/samcli/local/docker/lambda_debug_settings.py

Line 22 in b8f1064

class LambdaDebugSettings:

should be updated to allow runtime: provided.al2 as, if I remember correctly, I was getting exception from

aws-sam-cli/samcli/local/docker/lambda_debug_settings.py

Line 155 in b8f1064

raise DebuggingNotSupported("Debugging is not currently supported for {}".format(runtime)) from ex

saying that runtime (provided.al2) is not supported.

I guess line

aws-sam-cli/samcli/local/docker/lambda_debug_settings.py

Line 152 in b8f1064

if not runtime:

should changed from

aws-sam-cli/samcli/local/docker/lambda_debug_settings.py

Lines 151 to 155 in b8f1064

	except KeyError as ex:
	if not runtime:
	LOG.debug("Passing entrypoint as specified in template")
	return DebugSettings(entry + debug_args_list, _container_env_vars)
	raise DebuggingNotSupported("Debugging is not currently supported for {}".format(runtime)) from ex

to

        except KeyError as ex:
            #if not runtime:
            if runtime:
                LOG.debug("Passing entrypoint as specified in template")
                return DebugSettings(entry + debug_args_list, _container_env_vars)
            #raise DebuggingNotSupported("Debugging is not currently supported for {}".format(runtime)) from ex
            raise DebuggingNotSupported("Debugging is not currently supported for unspecified runtime") from ex

or something along those lines :)

This would help to work around current limitations, but ergonomics of script will not be great.

Also, help message for --container-env-vars should be updated to say it should be combined with port and function option in order to work.

Log statement at line

aws-sam-cli/samcli/local/lambdafn/runtime.py

Lines 389 to 393 in 0322c61

	LOG.debug(
	"Disable the debugging for Lambda Function %s, as the passed debug function is %s",
	function_config.name,
	debug_context.debug_function,
	)

should also update from debug to warning (or execution halted) as it silently drops option for no good reason (at least not documented in help message), and it will be hard to notice.

wdyt @qingchm ?

[qingchm]

Thanks for your detailed feedback! Let me take a look at this more closely!

[milenkovicm]

Hey @qingchm is there anything else I can help with?