docs(sandbox): OpenSandbox best-practices and debugging guides - #3062
Draft
hemildesai wants to merge 16 commits into
Draft
docs(sandbox): OpenSandbox best-practices and debugging guides#3062hemildesai wants to merge 16 commits into
hemildesai wants to merge 16 commits into
Conversation
Field-tested guidance for running Gym environments on OpenSandbox at scale, aimed at people bringing up their first large run: - resources are LIMITS; scheduling requests live under provider_options.resource_requests — and setting only resources makes the server apply that map as both, so a 4 CPU / 32Gi run occupies ~30x the cluster capacity of a 0.5 / 512Mi one and pushes everyone's creates into Pending - host-core fan-out (nproc / os.cpu_count report the host), what derive_cpu_env covers and what it cannot (old jest, plain make, py<3.13 multiprocessing), and why group-OOM kills the whole sandbox - which timeout you are actually waiting on; the N-minute failure cadence that is a create timeout echoing back from a full cluster - create retries vs command_retries: a 502 on POST /command does not mean the command never started - a table of failure signatures that are not bugs (lazy-load phantom ENOENT, startup 502 pings, terminate 404s, stale-IP 404s, PTY zombie attachments) - tenant hygiene: attribution labels, cleaning up a run by id, no apt-get per sandbox, teardown off the critical path - sizing quick reference Linked from the sandbox index and the provider page. Signed-off-by: Hemil Desai <hemild@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-hemild-opensandbox-best-practices.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
Say each thing once: fold the intro note into the lead, drop the sizing table that repeated the resources table, merge the two requests-vs-limits justifications, and cut single-run numbers that do not change a decision. Signed-off-by: Hemil Desai <hemild@nvidia.com>
use_server_proxy, domain, transport_backend, max_connections, keepalive_expiry_s, protocol — each with the failure you get when it is wrong. Signed-off-by: Hemil Desai <hemild@nvidia.com>
Egress is open by default (no sidecar without a network_policy); show the per-sandbox provider_options.network_policy shape, dns vs dns+nft enforcement, and the allow-what-the-harness-needs trap. Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Replace the one-line background_exec bullet with how submit/poll/fetch works and what each poll knob controls, including the request-rate math. Signed-off-by: Hemil Desai <hemild@nvidia.com>
Eight one-line practices, each linking to the section that explains it. Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Promote background execution to its own section (absorbing the timeout material) and move the failure table after tenant hygiene. Signed-off-by: Hemil Desai <hemild@nvidia.com>
…ction Signed-off-by: Hemil Desai <hemild@nvidia.com>
Lead with 'check what your sandbox reports' (host cores vs LXCFS-overlaid limits) instead of asserting one behavior; reword the short-list item to match. Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
How to read the sandbox's own cgroup (limits, live usage, peak, throttling, OOM events), how a whole-sandbox OOM surfaces to the client vs a child-process kill, how to count OOMs across a run from rollout records, and the kubectl checks for those with cluster access. Linked from the best-practices page and the sandbox index. Signed-off-by: Hemil Desai <hemild@nvidia.com>
hemildesai
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two new docs pages under
infrastructure/sandbox/, distilled from several weeks of running thousands of concurrent OpenSandbox sandboxes for SWE-bench / Terminal-Bench / DeepSWE evals. Every recommendation traces to a real incident or a measured A/B.opensandbox-best-practicesOpens with an eight-item short list (each linking to its section), then in that order:
use_server_proxy: true(direct mode dials pod IPs; unreachable from outside the cluster),domainviaOPENSANDBOX_DOMAIN,transport_backend: aiohttp,max_connections: null,keepalive_expiry_s,protocol.resourcesbecomes the pod limits; requests live underprovider_options.resource_requests; setting onlyresourcesapplies it as both, so a4 CPU / 32Girun occupies ~30× the capacity of a0.5 / 512Mione and pushes everyone's creates intoPending.derive_cpu_envcovers and cannot, why group-OOM kills the whole sandbox, bash timeout as the recovery mechanism.create.timeout_svsready_timeout_s,ttl_s, and the "failures every N minutes" create-timeout echo.create.retriesandoperations.retries(idempotent / provably-never-started ops) vscommand_retries: 0: a 502 onPOST /commanddoes not mean the command never started.network_policy); the per-sandbox policy shape;dnsvsdns+nftenforcement; allow what the harness needs and test with one rollout.NEMO_GYM_*inside k8s), reaping a run by id withcleanup_sandboxes.py, noapt-getper sandbox, teardown off the critical path.ENOENT, startup502pings, terminate404s, stale-IP404 unknown execution, PTY zombie attachments, group-OOM137.opensandbox-debuggingcpu.max,memory.max, and whatnprocsees), live usage and peak (memory.current,memory.peak,cpu.statthrottling ratio), ceiling events (memory.events), and a one-shot end-of-rollout snapshot.return_code 137, sandbox alive) vs whole-sandbox kill (SandboxBackendUnreachableErrorcarrying the server'sOOMKilledreason;sandbox.status()→ERROR/STOPPED; the PTY variant), not confusing it withTimeoutError, and counting OOMs across a run fromng_agent_observations.kubectlchecks for those with cluster access, and a signals-to-action table.Both pages are linked from the sandbox index (
<Card>) and from each other; the provider page gets a<Tip>pointing at the best-practices guide. Pages in this folder are auto-discovered, so no nav change.Test plan
pre-commit run --fileson the changed pages (no applicable hooks fail)fern check+ preview link (verify in-page anchors resolve)🤖 Generated with Claude Code