Problem
When a Modal function or sandbox fails after several retries, child processes, volume reads, or GPU setup steps, the useful context is spread across logs and the user's source code. Reproducing the failure usually means rebuilding the environment by hand. That makes intermittent failures in inference and batch workloads especially expensive to investigate.
The SDK already knows part of the execution boundary. It knows the image and function identity, resource request, sandbox command, mounted objects, and the calls made by the client. That information is currently lost as a single replayable object.
Proposal
Add an opt-in execution failure passport that can turn one failed invocation into a redacted, portable replay bundle.
A passport should contain:
- the invocation and child-execution graph
- function, image, SDK, Python, and dependency digests
- GPU and resource configuration
- retry and timeout decisions
- volume and uploaded-artifact identities
- the first failing boundary and the relevant exception chain
- explicit side-effect boundaries
- redacted inputs and references needed for a replay
The important part is not another log export. The bundle should say what can be replayed, what must be mocked, and what cannot be reproduced. A local inspector could then show the execution graph and generate a replay command for the reproducible part.
Suggested first slice
Keep the first version backend-neutral and opt-in:
- Define a versioned passport schema in the SDK.
- Collect the client-side execution and resource metadata without capturing secrets or raw payloads.
- Add deterministic redaction and stable artifact references.
- Provide a local inspector that validates a passport and explains its replay boundaries.
- Add fixtures for a retry, timeout, child-process failure, and partial upload.
The first slice does not need to rerun a remote job automatically. It should establish a stable artifact that the backend and CLI can build on later.
Why this matters
This would make a production failure useful after the original process is gone. It could also give Modal a common debugging boundary across inference endpoints, Sandboxes, and large fan-out jobs instead of adding separate diagnostics for each feature.
I have worked on incident traces, runbook retrieval, remediation gates, and audit trails across 2,500 incident scenarios. I have also built an LLM serving runtime where cold-start and latency behavior had to be measured rather than guessed. I would be interested in implementing the SDK-side schema, redaction, inspector, and replay fixtures if this direction fits Modal's roadmap.
Would the Modal team be open to discussing the schema and the smallest useful first slice before implementation?
Problem
When a Modal function or sandbox fails after several retries, child processes, volume reads, or GPU setup steps, the useful context is spread across logs and the user's source code. Reproducing the failure usually means rebuilding the environment by hand. That makes intermittent failures in inference and batch workloads especially expensive to investigate.
The SDK already knows part of the execution boundary. It knows the image and function identity, resource request, sandbox command, mounted objects, and the calls made by the client. That information is currently lost as a single replayable object.
Proposal
Add an opt-in execution failure passport that can turn one failed invocation into a redacted, portable replay bundle.
A passport should contain:
The important part is not another log export. The bundle should say what can be replayed, what must be mocked, and what cannot be reproduced. A local inspector could then show the execution graph and generate a replay command for the reproducible part.
Suggested first slice
Keep the first version backend-neutral and opt-in:
The first slice does not need to rerun a remote job automatically. It should establish a stable artifact that the backend and CLI can build on later.
Why this matters
This would make a production failure useful after the original process is gone. It could also give Modal a common debugging boundary across inference endpoints, Sandboxes, and large fan-out jobs instead of adding separate diagnostics for each feature.
I have worked on incident traces, runbook retrieval, remediation gates, and audit trails across 2,500 incident scenarios. I have also built an LLM serving runtime where cold-start and latency behavior had to be measured rather than guessed. I would be interested in implementing the SDK-side schema, redaction, inspector, and replay fixtures if this direction fits Modal's roadmap.
Would the Modal team be open to discussing the schema and the smallest useful first slice before implementation?