Configure docker image for aws stepfunctions local [guidance]

[austinben]

As mentioned in a issue I submitted to aws/aws-cli#6615, I am struggling to configure the docker build that occurs when running the aws stepfunction start-execution command, which I believe invokes a docker build in the other terminal running sam local start-lambda as per https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-lambda.html .

If anyone could provide insight into how to set up local testing of stepfunctions with multiple lambda functions with various modules and dependencies? The current structure that is used for local functional testing with sam build sam local invoke is as follows:

├── index.py
├──modules1
│   ├── some_module.py
│   └── other_module.py
├──modules2
│   ├── some_module2.py
│   └── other_module2.py
├── requirements.txt
├── template.yaml
├── Dockerfile.yaml

template.yaml :

Parameters:
  EnvironmentName:
    Type: String
    Description: Environment name to prefix on resource names in this stack
  Memory:
    Type: Number
    Description: Memory (MB) allocated the lambda. This directly affects the CPU allocated
    Default: 512
  Timeout:
    Type: Number
    Description: Timeout (seconds)
    Default: 300

Resources:
  MyLambdaFunction:
    Type: AWS::Serverless::Function
    Metadata:
      DockerTag: python3.8-v1
      DockerContext: ../../../
      Dockerfile: ./lambdas/path/to/Dockerfile
    Properties:
      Description: My Lambda Function
      PackageType: Image
      Environment:
        Variables:
          AWS_ENVIRONMENT_NAME:
            Ref: EnvironmentName
      MemorySize:
        Ref: Memory
      Timeout:
        Ref: Timeout

Dockerfile :

FROM amazon/aws-lambda-python:3.8

COPY lambdas/path/to/index.py lambdas/path/to/requirements.txt ./
ADD ./modules/somemodules ./somemodules
ADD ./modules/somemodules2 ./somemodules2
RUN python3.8 -m pip install -r requirements.txt -t .

CMD ["index.lambda_handler"]

This works correctly just using sam build and sam local invoke -e for functional testing, (and for debugging in VSCode after creating a launch.json.

I have multiple of these Lambda functions, each composing a portion of a step function defined in a .json template, along with everything else in a cloudformation .json template.

Is it possible to use the aws stepfunctions local tool with multiple lambdas defined in this way? Currently, it seems to just build the image from some default settings, and I cant seem to configure it as I have with the individual tests.

If this is too off-topic for this issue I am happy to open a new issue, I was directed here as the build process seems to be from the SAM CLI but the stepfunctions portion of the workflow if from the AWS CLI, please advise! Thanks :)

[c2tarun]

Hi @austinben,

Thanks for reaching out, can you please share the definition you used to create a state-machine? My understanding is that the definition file can have multiple tasks with separate lambda function for each task.

Thanks

[austinben]

Hi @c2tarun, thanks for responding.

The definition is roughly as follows (some stuff redacted), but logic is the same.:

{
    "Comment": "My state machine",
    "StartAt": "Step1",
    "States": {
      "Heartbeat/Alert Only?": {
        "Choices": [
          {
            "BooleanEquals": true,
            "Next": "Heartbeat/Alert Processing Done",
            "Variable": "$.step1_step_output.heartbeat_or_alert_only"
          },
          {
            "BooleanEquals": false,
            "Next": "Step2",
            "Variable": "$.step1_step_output.heartbeat_or_alert_only"
          }
        ],
        "Type": "Choice"
      },
      "Heartbeat/Alert Processing Done": {
        "Type": "Succeed"
      },
      "Step2": {
        "InputPath": "$",
        "Next": "ParallelPipelineSection",
        "Resource": "arn:function:-Step2-Filter",
        "ResultPath": "$.step2_filter_step_output",
        "Retry": [
          {
            "BackoffRate": 2,
            "ErrorEquals": [
              "ESClientGeneralException",
              "PostgresClientGeneralException",
              "Lambda.ServiceException",
              "State.Timeout",
              "Lambda.AWSLambdaException",
              "Lambda.SdkClientException",
              "Lambda.TooManyRequestsException",
              "Lambda.Unknown"
            ],
            "IntervalSeconds": 5,
            "MaxAttempts": 3
          }
        ],
        "Type": "Task"
      },
      "ParallelPipelineSection": {
        "Branches": [
          {
            "StartAt": "Step3-1Analysis",
            "States": {
              "Step3-1Analysis": {
                "InputPath": "$",
                "Next": "Step3-2-1AnalysisEnabled?",
                "Resource": "arn:function:-Step3-1-Analysis",
                "ResultPath": "$.step3-1_analysis_step_output",
                "Retry": [
                  {
                    "BackoffRate": 2,
                    "ErrorEquals": [
                      "ESClientGeneralException",
                      "PostgresClientGeneralException",
                      "Lambda.ServiceException",
                      "State.Timeout",
                      "Lambda.AWSLambdaException",
                      "Lambda.SdkClientException",
                      "Lambda.TooManyRequestsException",
                      "Lambda.Unknown"
                    ],
                    "IntervalSeconds": 5,
                    "MaxAttempts": 3
                  }
                ],
                "Type": "Task"
              },
              "Step3-2-1Analysis": {
                "InputPath": "$",
                "Next": "3-1-3BranchDone",
                "Resource": "arn:-Step3-2-1-Analysis",
                "ResultPath": "$.step3-2-1_analysis_step_output",
                "Retry": [
                  {
                    "BackoffRate": 2,
                    "ErrorEquals": [
                      "ESClientGeneralException",
                      "PostgresClientGeneralException",
                      "Lambda.ServiceException",
                      "State.Timeout",
                      "Lambda.AWSLambdaException",
                      "Lambda.SdkClientException",
                      "Lambda.TooManyRequestsException",
                      "Lambda.Unknown"
                    ],
                    "IntervalSeconds": 5,
                    "MaxAttempts": 3
                  }
                ],
                "Type": "Task"
              },
              "Step3-2-1AnalysisEnabled?": {
                "Choices": [
                  {
                    "BooleanEquals": true,
                    "Next": "Step3-2-1Analysis",
                    "Variable": "$.step3-1_analysis_step_output.step3-2-1_analysis_enabled"
                  },
                  {
                    "BooleanEquals": false,
                    "Next": "3-1-3BranchDone",
                    "Variable": "$.step3-1_analysis_step_output.step3-2-1_analysis_enabled"
                  }
                ],
                "Type": "Choice"
              },
              "3-1-3BranchDone": {
                "Type": "Succeed"
              }
            }
          },
          {
            "StartAt": "Step3-1-1Analysis",
            "States": {
              "Step3-1-1Analysis": {
                "InputPath": "$",
                "Next": "Step3-1-1BranchDone",
                "Resource": "arn::-Step3-1-1-Analysis",
                "ResultPath": "$.step3-1-1_analysis_step_output",
                "Retry": [
                  {
                    "BackoffRate": 2,
                    "ErrorEquals": [
                      "ESClientGeneralException",
                      "PostgresClientGeneralException",
                      "Lambda.ServiceException",
                      "State.Timeout",
                      "Lambda.AWSLambdaException",
                      "Lambda.SdkClientException",
                      "Lambda.TooManyRequestsException",
                      "Lambda.Unknown"
                    ],
                    "IntervalSeconds": 5,
                    "MaxAttempts": 3
                  }
                ],
                "Type": "Task"
              },
              "Step3-1-1BranchDone": {
                "Type": "Succeed"
              }
            }
          }
        ],
        "Comment": "Parallel execution section",
        "End": true,
        "Type": "Parallel"
      },
      "Step1": {
        "InputPath": "$",
        "Next": "Heartbeat/Alert Only?",
        "Resource": "arn::function:Step1LambdaFunction",
        "ResultPath": "$.step1_step_output",
        "Retry": [
          {
            "BackoffRate": 2,
            "ErrorEquals": [
              "ESClientGeneralException",
              "PostgresClientGeneralException",
              "Lambda.ServiceException",
              "State.Timeout",
              "Lambda.AWSLambdaException",
              "Lambda.SdkClientException",
              "Lambda.TooManyRequestsException",
              "Lambda.Unknown"
            ],
            "IntervalSeconds": 5,
            "MaxAttempts": 3
          }
        ],
        "Type": "Task"
      }
    }
  }

I do not believe the definition is the result of my issues. I think it stems from the cloudformation/sam template in template.json, where each lambda from the step function is similar to the following definition:

"MyLambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "CodeUri": "./mylambda1",
        "FunctionName": {"Fn::Sub": "${EnvironmentName}-Lambda"},
        "Handler": "index.lambda_handler",
        "Role": {"Fn::GetAtt": ["ProcessorLambdaRole", "Arn"]},
        "Runtime": "python3.8",
        "MemorySize": {"Ref": "Memory"},
        "Timeout": {"Ref": "Timeout"},
      }

Using this, the stepfunctions tool pulls the public image public.ecr.aws/sam/emulation-python3.8:rapid-1.31.0, builds, attempts to invoke, and subsequently fails due to No module named index or similar. It seems to be unable to see local packages and such.

I have also experienced it timing out on the Mounting path/to/my/stepfunctions as /var/task:ro,delegated inside runtime container step.

Due to errors in the past, I have previously converted my cloudformation template to something along the lines of the following in order to do local testing with sam local invoke and similar:

"MyLambdaFunction": {
      "Type": "AWS::Serverless::Function",
      "Metadata":{
        "DockerTag": "python3.8-v1",
        "DockerContext": "./path",
        "Dockerfile": "./path/to/Dockerfile"
      },
      "Properties": {
        "PackageType": "Image",
        "ImageUri": "amazon/aws-lambda-python:3.8",
        "Description": "my lambda",
        "Environment": {
          "Variables": {
            "AWS_ENVIRONMENT_NAME" : {"Ref": "EnvironmentName"}
          }
        },
        "MemorySize": {"Ref": "Memory"},
        "Timeout": {"Ref": "Timeout"}
      }
    },

But when I use the stepfunctions tool with the above format, it seems to ignore the dockerfile altogether and attempts to pull the default public image.

I was hoping to get guidance into why the process may have issues with timeouts, or the issues with missing modules, or hopefuly see if the process can be forced to use a specified dockerfile and dockercontext for each step.

[austinben]

I have been following https://medium.com/@ozamusic/how-to-test-aws-step-functions-locally-9507cc164044, minus the DynamoDB portion. I can not seem to find a combination that allows the correct communication between sam local and stepfunctions local. I have also read through https://www.codeproject.com/Articles/5276527/A-Note-on-AWS-Step-Function-CDK-SAM-Local-Miscella .

Another issue I am running into is that when I attempt to configure the setup as outlined in the above two links, I run into an issue where the create-state-machine step fails to work, due to Could not connect to the endpoint URL: "http://localhost:8083/". The commands I am using are as follows:

1. Configured credentials.txt as follows:

AWS_ACCOUNT_ID=[my id]
AWS_DEFAULT_REGION=us-east-1
AWS_ACCESS_KEY_ID=[my key id]
AWS_SECRET_ACCESS_KEY=[my secret key id]
LAMBDA_ENDPOINT=http://127.0.0.1:3001

2. sam local start-lambda -t template.json -p 3001
3. docker run -p 8083:8083 --network host --env-file credentials.txt amazon/aws-stepfunctions-local
4. aws stepfunctions --endpoint http://localhost:8083 create-state-machine --definition "cat definition.json" --name localpipeline --role-arn "arn:aws:iam::012345678901:role/DummyRole"

Where step 4 fails.

Replacing with LAMBDA_ENDPOINT=http://host.docker.internal:3001 ensures that the create-state-machine step does not instantly fail, but results in the same socket error as mentioned above during the Mounting path/to/my/stepfunctions as /var/task:ro,delegated inside runtime container step. But with this setup, I am able to run create-state-machine and the subsequent start-execution commands. Although I am not sure if this is the correct path to follow.

Any help regarding this communication issue would be amazing. Thanks.