StartExecutionManager fails to invoke worker if jobs exceed Lambda payload size limit

[jtherrmann]

Jira: https://asfdaac.atlassian.net/browse/TOOL-3344

Note: The above link is accessible only to members of ASF.

---

StartExecutionManager fails to invoke the worker function if the batch of jobs exceeds the Lambda payload size limit:

[ERROR]	2024-12-06T20:44:44.763Z	99a0084a-4074-4b29-87d6-19e0ad896161	Unhandled exception
Traceback (most recent call last):
  File "/var/task/lambda_logging/__init__.py", line 18, in wrapper
    lambda_handler(event, context)
  File "/var/task/start_execution_manager.py", line 35, in lambda_handler
    response = invoke_worker(worker_function_arn, jobs)
  File "/var/task/start_execution_manager.py", line 16, in invoke_worker
    return LAMBDA_CLIENT.invoke(
  File "/var/task/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/task/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (RequestEntityTooLargeException) when calling the Invoke operation: 334391 byte payload is too large for the Event invocation type (limit 262144 bytes)

This is consistent with https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html which shows "256 KB (asynchronous)" for "Invocation payload".

@asjohnston-asf points out that we may no longer have any requirement for the throughput increase granted by the manager/worker model. We'll have to discuss further.

Relates to #1272

[jtherrmann]

Confirmed that this is still an issue for the INSAR_ISCE_MULTI_BURST job type after reducing the max number of burst pairs per job to 15. You can run the following script to reproduce the issue:

from hyp3_sdk import HyP3

hyp3 = HyP3(api_url="https://hyp3-multi-burst-sandbox.asf.alaska.edu")

MOJAVE_REF = [
    "S1_370315_IW3_20240227T134440_VV_0B42-BURST",
    "S1_370315_IW2_20240227T134439_VV_0B42-BURST",
    "S1_370314_IW3_20240227T134437_VV_0B42-BURST",
    "S1_370314_IW2_20240227T134436_VV_0B42-BURST",
    "S1_370313_IW3_20240227T134434_VV_0B42-BURST",
    "S1_370313_IW2_20240227T134433_VV_0B42-BURST",
    "S1_370312_IW3_20240227T134432_VV_0B42-BURST",
    "S1_370312_IW2_20240227T134431_VV_0B42-BURST",
    "S1_370311_IW3_20240227T134429_VV_4BDF-BURST",
    "S1_370311_IW2_20240227T134428_VV_4BDF-BURST",
    "S1_370310_IW3_20240227T134426_VV_4BDF-BURST",
    "S1_370310_IW2_20240227T134425_VV_4BDF-BURST",
    "S1_370309_IW3_20240227T134423_VV_4BDF-BURST",
    "S1_370309_IW2_20240227T134422_VV_4BDF-BURST",
    "S1_370308_IW3_20240227T134421_VV_4BDF-BURST",
]
MOJAVE_SEC = [
    "S1_370315_IW3_20240322T134440_VV_A2EB-BURST",
    "S1_370315_IW2_20240322T134439_VV_A2EB-BURST",
    "S1_370314_IW3_20240322T134438_VV_A2EB-BURST",
    "S1_370314_IW2_20240322T134437_VV_A2EB-BURST",
    "S1_370313_IW3_20240322T134435_VV_A2EB-BURST",
    "S1_370313_IW2_20240322T134434_VV_A2EB-BURST",
    "S1_370312_IW3_20240322T134432_VV_A2EB-BURST",
    "S1_370312_IW2_20240322T134431_VV_A2EB-BURST",
    "S1_370311_IW3_20240322T134429_VV_645B-BURST",
    "S1_370311_IW2_20240322T134428_VV_645B-BURST",
    "S1_370310_IW3_20240322T134427_VV_645B-BURST",
    "S1_370310_IW2_20240322T134426_VV_645B-BURST",
    "S1_370309_IW3_20240322T134424_VV_645B-BURST",
    "S1_370309_IW2_20240322T134423_VV_645B-BURST",
    "S1_370308_IW3_20240322T134421_VV_645B-BURST",
]


def main():
    for _ in range(10):
        prepped_jobs = [
            {
                "job_parameters": {
                    "apply_water_mask": False,
                    "looks": "20x4",
                    "reference": MOJAVE_REF,
                    "secondary": MOJAVE_SEC,
                },
                "job_type": "INSAR_ISCE_MULTI_BURST",
                # max name length of 100
                "name": "----------------------------------------------------------------------------------------------------",
            }
            for _ in range(30)
        ]
        jobs = hyp3.submit_prepared_jobs(prepped_jobs)
        print(f"Submitted {len(jobs)} jobs")


if __name__ == "__main__":
    main()

Then in the StartExecutionManager logs for hyp3-multi-burst-sandbox you'll see:

Invoking worker for 250 jobs

...

botocore.exceptions.ClientError: An error occurred (RequestEntityTooLargeException) when calling the Invoke operation: 466261 byte payload is too large for the Event invocation type (limit 262144 bytes)

Based on this, it looks like the number of bytes per job is 466261 / 250 ≈ 1866, which would give us a limit of floor(262144 / 1866) = 140 jobs.

If we modify the above script to submit 140 jobs, then StartExecutionManager successfully starts all 140 jobs:

Invoking worker for 140 jobs
Got response status code 202

If we then modify the above script to submit 150 jobs, it fails:

Invoking worker for 150 jobs

...

botocore.exceptions.ClientError: An error occurred (RequestEntityTooLargeException) when calling the Invoke operation: 279760 byte payload is too large for the Event invocation type (limit 262144 bytes)

Note that the job size will slightly decrease when renaming this job type to INSAR_ISCE_BURST (6 fewer characters), but the max number of jobs that can be started at a time will still be limited to floor(262144 / 1860) = 140.

[jtherrmann]

Looks like we previously addressed this issue (see #2241) by reducing the batch size to 250 in #2242.

[jtherrmann]

Our solution will likely be to revert to the old StartExecution lambda architecture.

[asjohnston-asf]

For reference, we initially implemented the worker/manager pattern in hyp3 v2.21.4. Going back to a single lambda function would look a lot like reverting the related changes in v2.21.3...v2.21.4

[asjohnston-asf]

Deploying from combine-start-execution (currently ec2a07f) works as expected. The new streamlined start-execution lambda can submit 500 INSAR_ISCE_BURST jobs in ~30 seconds.

Also had no trouble submitting a back of 300 INSAR_ISCE_MULTI_BURST jobs per @jtherrmann 's original test case, suggesting this issue is resolved by our PR.