Multiple lambdas not communicating locally in sam local start-api

[blmayer]

Description

First, invocations done by the go-lambda-sdk using thelambda.invoke() function go to deployed AWS lambda instead of the local ones created by sam local start-api.

Second, only lambdas that receives events are being actually mounted on the local container for testing, I think this is not the desired behaviour. I expected my local test environment call other lambdas that are also local.

Steps to reproduce

Create a template.yml with 2 lambdas: one receives the event from the api, the other is just defined on the file. In the code of the first lambda use the sdk to call the second lambda:

lambda1 code:

package main

import (
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-lambda-go/lambda"
        lam "github.com/aws/aws-sdk-go/service/lambda"
        "github.com/aws/aws-lambda-go/events"
)
var svc = lambda.New(session.New())

func Handler(req events.APIGatewayProxyRequest) (e events.APIGatewayProxyResponse, err error) {
        input := &lambda.InvokeInput{
                FunctionName: "lambda2",
                Payload:  `{"some": "json"}`,
        }

        response, err := svc.Invoke(input)
        if err != nil {
               log.Println(err.Error())
               return e, err
        }
        return e, err
}

func main() {
        lam.Start(Handler)
}

This is a Golang example, I haven't tried with other languages. The second lambda just needs to receive and return some JSON:

lambda2 code:

  package main
  
  import (
          "log"

          "github.com/aws/aws-lambda-go/lambda"
  )
  
  // Handler function is needed by AWS lambda functions, it can receive the
  // data passed to it from the API Gateway
  func Handler(req interface{}) (interface{}, error) {
          log.Println("lambda2 triggered")
          return `{"another":"JSON"}`, nil
  }
  
  func main() {
          // Start the lambda handler
          lambda.Start(Handler)
}

Then build and start the local API, the request from the first lambda never hits the second lambda.

Observed result

Addressing the mounting issue, this is what I get:

% sam local start-api --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local start-api command is called
No Parameters detected in the template
2 resources found in the template
Found Serverless function with name='lambda1' and CodeUri='lambda1'
Found Serverless function with name='lambda2' and CodeUri='lambda2'
No Parameters detected in the template
2 resources found in the template
Found '1' API Events in Serverless function with name 'lambda1'
Found '0' API Events in Serverless function with name 'lambda2'
Detected Inline Swagger definition
Lambda function integration not found in Swagger document at path='/cloud/{proxy+}' method='x-amazon-apigateway-any-method'
Found '0' APIs in resource 'ServerlessRestApi'
Removed duplicates from '0' Explicit APIs and '1' Implicit APIs to produce '1' APIs
1 APIs found in the template
Mounting lambda1 at http://127.0.0.1:3000/cloud/{proxy+} [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
Localhost server is starting up. Multi-threading = True
2020-06-19 18:15:31  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Constructed String representation of Event to invoke Lambda. Event: {"httpMethod": "POST", "body": "{}", "resource": "/cloud/{proxy+}", "requestContext": {"resourceId": "123456", "apiId": "1234567890", "resourcePath": "/cloud/{proxy+}", "httpMethod": "POST", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "accountId": "123456789012", "stage": "Prod", "identity": {"apiKey": null, "userArn": null, "cognitoAuthenticationType": null, "caller": null, "userAgent": "Custom User Agent String", "user": null, "cognitoIdentityPoolId": null, "cognitoAuthenticationProvider": null, "sourceIp": "127.0.0.1", "accountId": null}, "extendedRequestId": null, "path": "/cloud/{proxy+}"}, "queryStringParameters": null, "multiValueQueryStringParameters": null, "headers": {"Content-Type": "application/json", "Authorization": "Bearer mytoken", "User-Agent": "PostmanRuntime/7.25.0", "Accept": "*/*", "Cache-Control": "no-cache", "Postman-Token": "f1aca52c-80dc-4058-86eb-10f5a844901e", "Host": "localhost:3000", "Accept-Encoding": "gzip, deflate, br", "Connection": "keep-alive", "Content-Length": "38", "X-Forwarded-Proto": "http", "X-Forwarded-Port": "3000"}, "multiValueHeaders": {"Content-Type": ["application/json"], "Authorization": ["Bearer mytoken"], "User-Agent": ["PostmanRuntime/7.25.0"], "Accept": ["*/*"], "Cache-Control": ["no-cache"], "Postman-Token": ["f1aca52c-80dc-4058-86eb-10f5a844901e"], "Host": ["localhost:3000"], "Accept-Encoding": ["gzip, deflate, br"], "Connection": ["keep-alive"], "Content-Length": ["38"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Port": ["3000"]}, "pathParameters": {"proxy": "lambda1"}, "stageVariables": null, "path": "/cloud/lambda1", "isBase64Encoded": false}
Found one Lambda function with name 'lambda1'
Invoking lambda1 (go1.x)
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=mypath, CodeUri=lambda1
Resolved absolute path to code is mypath/lambda1
Code mypath/lambda1 is not a zip/jar file
Skipping building an image since no layers were defined

Fetching lambci/lambda:go1.x Docker container image......
Mounting mypath/lambda1 as /var/task:ro,delegated inside runtime container
Starting a timer for 5 seconds for function 'lambda1'
START RequestId: 66bbf3ec-63e2-1bd5-8326-14d2ed3fbca2 Version: $LATEST
END RequestId: 66bbf3ec-63e2-1bd5-8326-14d2ed3fbca2
REPORT RequestId: 66bbf3ec-63e2-1bd5-8326-14d2ed3fbca2	Init Duration: 1101.25 ms	Duration: 700.66 ms	Billed Duration: 800 ms	Memory Size: 128 MB	Max Memory Used: 35 MB	
2020-06-19 18:18:14 127.0.0.1 - - [19/Jun/2020 18:18:14] "POST /cloud/lambda1 HTTP/1.1" 200 -

And this is when I put an API event on the two lambdas:

% sam local start-api --debug       
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local start-api command is called
No Parameters detected in the template
14 resources found in the template
Found Serverless function with name='lambda1' and CodeUri='lambda1'
Found Serverless function with name='lambda2' and CodeUri='lambda2'
No Parameters detected in the template
2 resources found in the template
Found '1' API Events in Serverless function with name 'lambda1'
Found '1' API Events in Serverless function with name 'lambda2'
Detected Inline Swagger definition
Lambda function integration not found in Swagger document at path='/cloud/{proxy+}' method='x-amazon-apigateway-any-method'
Lambda function integration not found in Swagger document at path='/somedummypath' method='x-amazon-apigateway-any-method'
Found '0' APIs in resource 'ServerlessRestApi'
Removed duplicates from '0' Explicit APIs and '2' Implicit APIs to produce '2' APIs
2 APIs found in the template
Mounting lambda1 at http://127.0.0.1:3000/cloud/{proxy+} [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
Mounting lambda2 at http://127.0.0.1:3000/somedummypath [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
2020-06-19 17:33:47  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Constructed String representation of Event to invoke Lambda. Event: {"httpMethod": "POST", "body": "{}", "resource": "/cloud/{proxy+}", "requestContext": {"resourceId": "123456", "apiId": "1234567890", "resourcePath": "/cloud/{proxy+}", "httpMethod": "POST", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "accountId": "123456789012", "stage": "Prod", "identity": {"apiKey": null, "userArn": null, "cognitoAuthenticationType": null, "caller": null, "userAgent": "Custom User Agent String", "user": null, "cognitoIdentityPoolId": null, "cognitoAuthenticationProvider": null, "sourceIp": "127.0.0.1", "accountId": null}, "extendedRequestId": null, "path": "/cloud/{proxy+}"}, "queryStringParameters": null, "multiValueQueryStringParameters": null, "headers": {"Content-Type": "application/json", "Authorization": "Bearer mytoken", "User-Agent": "PostmanRuntime/7.25.0", "Accept": "*/*", "Cache-Control": "no-cache", "Postman-Token": "6f7027b2-aae8-4bca-9bf7-992a112153f4", "Host": "localhost:3000", "Accept-Encoding": "gzip, deflate, br", "Connection": "keep-alive", "Content-Length": "38", "X-Forwarded-Proto": "http", "X-Forwarded-Port": "3000"}, "multiValueHeaders": {"Content-Type": ["application/json"], "Authorization": ["Bearer mytoken"], "User-Agent": ["PostmanRuntime/7.25.0"], "Accept": ["*/*"], "Cache-Control": ["no-cache"], "Postman-Token": ["6f7027b2-aae8-4bca-9bf7-992a112153f4"], "Host": ["localhost:3000"], "Accept-Encoding": ["gzip, deflate, br"], "Connection": ["keep-alive"], "Content-Length": ["38"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Port": ["3000"]}, "pathParameters": {"proxy": "lambda1"}, "stageVariables": null, "path": "/cloud/lambda1", "isBase64Encoded": false}
Found one Lambda function with name 'lambda1'
Invoking lambda1 (go1.x)
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=mypath, CodeUri= lambda1
Resolved absolute path to code is mypath/lambda1
Code mypath/lambda1 is not a zip/jar file
Skipping building an image since no layers were defined

Fetching lambci/lambda:go1.x Docker container image......
Mounting mypath/lambda1 as /var/task:ro,delegated inside runtime container
Starting a timer for 5 seconds for function 'lambda1'
START RequestId: 12fbc7e7-5aef-191d-64b6-dfe851da1728 Version: $LATEST
END RequestId: 12fbc7e7-5aef-191d-64b6-dfe851da1728
REPORT RequestId: 12fbc7e7-5aef-191d-64b6-dfe851da1728	Init Duration: 2026.12 ms	Duration: 865.23 ms	Billed Duration: 900 ms	Memory Size: 128 MB	Max Memory Used: 35 MB	
2020-06-19 17:34:17 127.0.0.1 - - [19/Jun/2020 17:34:17] "POST /cloud/lambda1 HTTP/1.1" 200 -

EDIT: I chaned the token text, the real name of my lambdas and the path of lambdas' code to not show the real one.

Expected result

I expect that the lambdas on my local environment invoke other lambdas in my environment instead of the deployed ones.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac catalina
2. SAM CLI, version 0.53.0:

Thanks a lot for this yet great tool!

[sriram-mv]

Were you able to setup sam local start-lamba ? This would allow you to call into lambda locally.

[blmayer]

Yes I set that up, but same behaviour, the first lambda is called locally, but it calls the second lambda of the AWS cloud.

[timoschilling]

@blmayer can you post your template.yaml please

[blmayer]

Here's my template.yaml.

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  cloud

  My cloud lambda services

Resources:
  lambda1:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: "lambda1"
      Handler: lambda1
      Runtime: go1.x
      CodeUri: lambda1/
      Events:
        GatewayApiProxyGet:
          Type: Api
          Properties:
            Path: /{proxy+}
            Method: get

 lambda2:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: "lambda2"
      Handler: lambda2
      Runtime: go1.x
      CodeUri: lambda2/

[blmayer]

Anything?

[godrose]

@blmayer Hi. Did you find any workaround for this?

[jakeberg]

I'm wondering this same thing. It seems there should be an option to test all the invoked lambdas locally. Anything new on this?

[mpiltz]

I'm also working with similar problem.
We have a setup were we have ApiGW with lambda-functions and then we have a proxy lambda in our private VPC to connect to private resources. Call through ApiGW goes to "public" lambda function which invokes the proxy lambda.

My problem is how to test this locally.

I've started sam local start-lambda and i'm able to invoke my local proxy lambda from command line.
But when i start sam local start-api and try to invoke the same proxy lambda from another lambda run by start-api i get connection error and it seems that the call is going to AWS Lambda and not to my local Lambda. I've tried to follow this document https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-lambda.html.

I get this error message:

{"message":"Inaccessible host: 127.0.0.1'. This service may not be available in the eu-west-1' region.","code":"UnknownEndpoint","region":"eu-west-1","hostname":"127.0.0.1","retryable":true,"originalError":{"message":"connect ECONNREFUSED 127.0.0.1:3001","errno":-111,"code":"NetworkingError","syscall":"connect","address":"127.0.0.1","port":3001,"region":"eu-west-1","hostname":"127.0.0.1","retryable":true,"time":"2021-06-08T13:09:26.027Z"},"time":"2021-06-08T13:09:26.027Z"}

In my ApiGW lambda i tried to initialize the Lambda service like this:
const Lambda = new AWS.Lambda({ endpoint: "http://127.0.0.1:3001", sslEnabled: false, maxRetries: 0 })

[santelelle]

I am having the same issue, and news?

EDIT: I actually could solve it on linux by running:

$sam local start-lambda --host 172.17.0.1

and in my python file:

import boto3
client = boto3.client('lambda', endpoint_url='http://172.17.0.1:3001', use_ssl=False, verify=False)
client.invoke(FunctionName="TestFunction", InvocationType='RequestResponse')

The 172.17.0.1 ip comes from
https://stackoverflow.com/questions/48546124/what-is-linux-equivalent-of-host-docker-internal/61001152

Unfortunately I still have not a deep understanding on how this solved my issue.