Bug: `sam local invoke` fails when `Globals.Function.Layers` contains `AWS::NoValue`

[rariyama]

Thanks for the great tool. I'll report the issue about sam local invoke.

Description:

sam local invoke fails when Globals.Function.Layers contains !Ref AWS::NoValue.
I tried sam deploy and build and both are successfully finished.

Steps to reproduce:

1. Create a template as an example below.
2. Execute example command.

Command example

sam local invoke "FunctionExample" \
    -e event.json \
    --template-file template.yaml \
    --profile awsprofile \
    --parameter-overrides Env=dev

Template example

The template example is as follows.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  Env:
    Type: String

Conditions:
  IsDev: !Equals [ !Ref Env, 'dev' ]

## Global Section
Globals:
  Function:
    Timeout: 30
    MemorySize: 256
    Handler: main.lambda_handler
    Runtime: python3.8
    Layers:
      - Fn::If:
          - IsDev
          - !Ref AWS::NoValue
          - arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25

Resources:
  FunctionExample:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: "function-example"
      CodeUri: example/
      Role: !Sub "arn:aws:iam::${AWS::AccountId}:role/function-example-role"

It's also possible to reproduce this issue by simply adding AWS::NoValue to Layers like below.

    Layers:
      - !Ref AWS::NoValue
      - arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25

Observed result:

The logs are partly masked because parts of them are confidential information.

2023-11-05 20:50:33,632 | Config file location: /Users/samconfig.toml                                                                                                                                                
2023-11-05 20:50:33,633 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in config file at '/Users/samconfig.toml'...                                         
2023-11-05 20:50:33,634 | Configuration values successfully loaded.                                                                                                                                                                                                  
2023-11-05 20:50:33,634 | Configuration values are: {'profile': 'xxx-developer', 'parameter_overrides': ['Env=dev', 'DynamoUrl=http://host.docker.internal:58000']}            
2023-11-05 20:50:33,652 | Using config file: samconfig.toml, config environment: default                                                                                                                                                                             
2023-11-05 20:50:33,652 | Expand command line arguments to:                                                                                                                                                                                                          
2023-11-05 20:50:33,653 | --template_file=/Users/template.yaml --event=event.json --parameter_overrides={'Env': 'dev'}                                   
--function_logical_id=FunctionExample --no_event --layer_cache_basedir=/Users/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1                                                                                        
2023-11-05 20:50:33,653 | local invoke command is called                                                                                                                                                                                                             
2023-11-05 20:50:33,656 | Collected default values for parameters: {'Project': 'xxx', 'Role': 'xxx'}                                                                                                                                                          
2023-11-05 20:50:33,669 | There is no customer defined id or cdk path defined for resource FunctionExample, so we will use the resource logical id as the resource id                                                                                                
2023-11-05 20:50:33,670 | Unable to resolve property Layers: [OrderedDict([('Fn::If', ['IsDev', {'Ref': 'AWS::NoValue'}, 'arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25'])]), OrderedDict([('Fn::If', ['IsDev', {'Ref':                
'CommonCustomLibrary'}, 'arn:aws:lambda:ap-northeast-1:123456789012:layer:ExampleLibrary:1'])])]. Leaving as is.                                                                                                                                               
2023-11-05 20:50:33,671 | 0 stacks found in the template                                                                                                                                                                                                             
2023-11-05 20:50:33,671 | Collected default values for parameters: {'Project': 'xxx', 'Role': 'xxx'}                                                                                                                                                          
2023-11-05 20:50:33,682 | There is no customer defined id or cdk path defined for resource FunctionExample, so we will use the resource logical id as the resource id                                                                                                
2023-11-05 20:50:33,682 | Unable to resolve property Layers: [OrderedDict([('Fn::If', ['IsDev', {'Ref': 'AWS::NoValue'}, 'arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25'])]), OrderedDict([('Fn::If', ['IsDev', {'Ref':                
'CommonCustomLibrary'}, 'arn:aws:lambda:ap-northeast-1:123456789012:layer:ExampleLibrary:1'])])]. Leaving as is.                                                                                                                                               
2023-11-05 20:50:33,683 | 1 resources found in the stack                                                                                                                                                                                                             
2023-11-05 20:50:33,683 | layer "OrderedDict([('Fn::If', ['IsDev', {'Ref': 'AWS::NoValue'}, 'arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25'])])" is not recognizable, it might be using intrinsic functions that we don't support yet. 
Skipping.                                                                                                                                                                                                                                                            
2023-11-05 20:50:33,684 | layer "OrderedDict([('Fn::If', ['IsDev', {'Ref': 'CommonCustomLibrary'}, 'arn:aws:lambda:ap-northeast-1:123456789012:layer:ExampleLibrary:1'])])" is not recognizable, it might be using intrinsic functions that we don't support   
yet. Skipping.                                                                                                                                                                                                                                                       
2023-11-05 20:50:33,684 | Found Serverless function with name='FunctionExample' and CodeUri='example/'                                                                                                                                             
2023-11-05 20:50:33,685 | --base-dir is not presented, adjusting uri example/ relative to /Users/template.yaml                                                                                    
2023-11-05 20:50:33,696 | Found one Lambda function with name 'FunctionExample'                                                                                                                                                                                      
2023-11-05 20:50:33,696 | Invoking main.lambda_handler (python3.8)                                                                                                                                                                                                  
2023-11-05 20:50:33,697 | Loading AWS credentials from session with profile 'xxx'                                                                                                                                                                             
2023-11-05 20:50:33,703 | Resolving code path. Cwd=/Users/xxx/xxx-xxx, CodeUri=/Users/example                                                                  
2023-11-05 20:50:33,703 | Resolved absolute path to code is /Users/example                                                                                                                         
2023-11-05 20:50:33,703 | Code /Users/example is not a zip/jar file                                                                                                                                
2023-11-05 20:50:36,406 | Local image is up-to-date                                                                                                                                                                                                                  
2023-11-05 20:50:36,427 | Using local image: public.ecr.aws/lambda/python:3.8-rapid-x86_64.                                                                                                                                                                          
                                                                                                                                                                                                                                                                     
2023-11-05 20:50:36,428 | Mounting /Users/example as /var/task:ro,delegated, inside runtime container                                                                                              
2023-11-05 20:50:36,774 | Starting a timer for 30 seconds for function 'FunctionExample'                                                                                                                                                                             
START RequestId: 3b4dc886-5fd1-4234-8200-78215c64c544 Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'jwt'
Traceback (most recent call last):
END RequestId: 3b4dc886-5fd1-4234-8200-78215c64c544
REPORT RequestId: 3b4dc886-5fd1-4234-8200-78215c64c544  Init Duration: 0.86 ms  Duration: 491.40 ms     Billed Duration: 492 ms Memory Size: 256 MB     Max Memory Used: 256 MB
{"errorMessage": "Unable to import module 'index': No module named 'jwt'", "errorType": "Runtime.ImportModuleError", "stackTrace": []}
2023-11-05 20:50:38,450 | Cleaning all decompressed code dirs                                                                                                                                                                                                        
2023-11-05 20:50:38,451 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                                                                                 
2023-11-05 20:50:38,483 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                                                                                 
2023-11-05 20:50:38,484 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '4c0f8e42-c28e-4973-8974-d5236ffdf30f', 'installationId': '7d1fc9b0-399a-43c6-89d4-58cc4320aa75', 'sessionId': 'e49789f4-362f-44dc-98d7-9437e4b18441', 'executionEnvironment':
'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.100.0', 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': 'ap-northeast-1', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None,   
'projectName': 'xxx', 'initialCommit': None}, 'duration': 4799, 'exitReason': 'success', 'exitCode': 0}}]}                                                                                              
2023-11-05 20:50:38,484 | Unable to find Click Context for getting session_id.                                                                                                                                                                                       
2023-11-05 20:50:38,487 | Sending Telemetry: {'metrics': [{'events': {'requestId': '3ce5f928-11a2-452a-a75f-2e6a28f719a5', 'installationId': '7d1fc9b0-399a-43c6-89d4-58cc4320aa75', 'sessionId': 'e49789f4-362f-44dc-98d7-9437e4b18441', 'executionEnvironment':    
'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.100.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '508556a40c7c404f926441d27a60a2d0',  
'time_stamp': '2023-11-05 11:50:33.631', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '44cc4177912045c8862e3e2e37a85ff3', 'time_stamp': '2023-11-05 11:50:33.652', 'exception_name': None}]}}}]}           
2023-11-05 20:50:38,999 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                                                                                                           
2023-11-05 20:50:39,003 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                                                                                                           
2023-11-05 20:50:39,004 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                                                                                                 
2023-11-05 20:50:39,004 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': '781d71d7-5ce0-4914-bf3e-60ae8677eed4', 'installationId': '7d1fc9b0-399a-43c6-89d4-58cc4320aa75', 'sessionId': 'e49789f4-362f-44dc-98d7-9437e4b18441',                     
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.100.0', 'runtimes': ['python3.8']}}]}                                                                                                                                         
2023-11-05 20:50:39,563 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                                                      

I guess the following log is related to this issue.

2023-11-05 20:50:33,683 | layer "OrderedDict([('Fn::If', ['IsDev', {'Ref': 'AWS::NoValue'}, 'arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25'])])" is not recognizable, it might be using intrinsic functions that we don't support yet. 
Skipping.                                                                                                                                                                                                                                                            

Expected result:

Command is successfully finished and returns 200.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS 13.2.1
2. sam --version: 1.100.0
3. AWS region: ap-northeast-1

# Paste the output of `sam --info` here
{
  "version": "1.100.0",
  "system": {
    "python": "3.8.13",
    "os": "macOS-13.2.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.22",
    "aws_cdk": "Not available",
    "terraform": "1.6.2"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[hawflau]

Hi @rariyama thanks for raising the issue.

Please note that sam local invoke and other sam local commands do not support resolving Fn::If. I'll mark this as a feature request, and bring up with team.

For the error below:

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'jwt'
Traceback (most recent call last):

Where is the jwt module from? If you are trying debug this error, I'd recommend you not using any Fn::If and conditions to isolate and identify the root cause. Please let me know if you have any further question.

[rariyama]

Hi @hawflau thank you for your comments. I'll reply to your comments below.

Please note that sam local invoke and other sam local commands do not support resolving Fn::If. I'll mark this as a feature request, and bring up with team.

I'm sorry in advance if I misunderstand but I confirmed sam local invoke worked with the following example template.

Conditions:
  IsDev: !Equals 
           - !Ref Env
           - 'dev'

## Global Section
Globals:
  Function:
    Timeout: 30
    MemorySize: 256
    Handler: main.lambda_handler
    Runtime: python3.8
    Layers:
      - Fn::If:
          - IsDev
          - !Ref CommonCustomLibrary
          - 'arn:aws:lambda:ap-northeast-1:123456789012:layer:CommonCustomLibrary:1' # Custom lambda layer being deployed on AWS.

Resources:
  ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: "example-function"
      CodeUri: example-function/
      Role: !Sub 'arn:aws:iam::${AWS::AccountId}:role/example-function-role'

  CommonCustomLibrary:
    Condition: IsDev
    Type: AWS::Serverless::LayerVersion
    Properties:
      RetentionPolicy: Delete
      LayerName : common-custom-library
      Description: 'v1.0.0'
      ContentUri: 'common-custom-source-dir/source'
      CompatibleRuntimes:
        - python3.8
    Metadata:
      BuildMethod: python3.8

The command I executed is as follows.

sam local invoke "ExampleFunction" \
    -e events.json \
    --template-file template.yaml \
    --profile aws-profile \
    --parameter-overrides Env=dev

I think this issue is related AWS::NoValue can't be used in Globals.FunctionLayers rather than !If.

For the error below:

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'jwt'
Traceback (most recent call last):
Where is the jwt module from? If you are trying debug this error, I'd recommend you not using any Fn::If and conditions to isolate and identify the root cause. Please let me know if you have any further question.

I'm sorry but this error caused by my code.
The source code of lambda function I executed contained import jwt. jwt was installed on custom lambda layer but layer mount failed then module import also failed. This is not related to this problem, so please ignore it.