Pass CWL content via env var to avoid ARG_MAX limit in input staging job

[Copilot]

When CWL workflows have many inputs, the base64-encoded content embedded directly in the /bin/sh -c argument can exceed the kernel's ARG_MAX limit, causing the input staging job to fail.

Changes

• calrissian.py: Replace inline embedding of cwl_serialized in the shell command with a CWL_CONTENT Kubernetes env var; reference it via $CWL_CONTENT in the script
• test_calrissian.py: Update test_create_input_staging_job_manifest assertions to expect the new env field and updated args

# Before — base64 blob embedded in the -c argument, subject to ARG_MAX
args=["-c", f"set -euxo pipefail; echo '{cwl_serialized}' | base64 -d > {cwl_path}"],

# After — content passed via env var, bypassing ARG_MAX
args=["-c", f"set -euxo pipefail; printf '%s' \"$CWL_CONTENT\" | base64 -d > {cwl_path}"],
env=[kubernetes.client.V1EnvVar(name="CWL_CONTENT", value=cwl_serialized)],

[EmileSonneveld]

@copilot Can you find a fix for when the input is larger than 2Mb? environment variables won't support that

[Copilot]

@copilot Can you find a fix for when the input is larger than 2Mb? environment variables won't support that

Good point. I've updated the approach: instead of passing the content via an environment variable, the driver now uploads the CWL content directly to S3 (cwl-staging/{name}.cwl) using put_object, generates a 1-hour presigned URL, and passes only that URL to the container via $CWL_PRESIGNED_URL. The container then fetches it with wget. This works for files of any size since the data goes through S3 rather than through Kubernetes/Linux process argument limits.

[EmileSonneveld]

@copilot Can you add a unit test where a 2Mb file is created that gets passed like this?

[Copilot]

@copilot Can you add a unit test where a 2Mb file is created that gets passed like this?

Done — added test_create_input_staging_job_manifest_large_cwl in commit $(git rev-parse --short HEAD). The test builds a ~2 MB CWL string, passes it through create_input_staging_job_manifest, then asserts that:

1. The full content was stored in S3 under cwl-staging/.
2. The container manifest's args uses wget -qO- "$CWL_PRESIGNED_URL" (not inline content).
3. The env carries a CWL_PRESIGNED_URL pointing to the S3 object.