Investigation Summary
NemoClaw sandbox using nvidia/nemotron-3-ultra-550b-a55b reports managed inference as unhealthy with HTTP 503 from https://inference.local/v1/models, although route reachability is reported as reachable.
Deep Agents Code execution failed with status=agent_failure, response=null, response_bytes=0, correlation ID 01a04478-c0f4-7493-bf38-1452116f0adb.
The same ~111,564-character engineering-review prompt succeeds when sent directly to NVIDIA https://integrate.api.nvidia.com/v1/chat/completions using the same Ultra model, returning a complete ~22 KB response.
A separately onboarded nvidia/nemotron-3-super-120b-a12b Deep Agents Code sandbox reports managed inference as healthy.
This isolates the apparent fault to the NemoClaw/OpenShell/Deep Agents Code managed-inference path for Ultra rather than the NVIDIA credential, model endpoint or prompt. Exact underlying cause remains unknown.
Description
We are evaluating NemoClaw for a controlled software-engineering and adversarial code-review workflow.
A Deep Agents Code sandbox configured for:
nvidia/nemotron-3-ultra-550b-a55b
is created successfully, but its status reports:
Model: nvidia/nemotron-3-ultra-550b-a55b
Provider: nvidia-prod
Inference: unhealthy (https://inference.local/v1/models)
Inference gateway did not serve an inference request:
sandbox inference invocation probe returned HTTP 503.
Inference (route reachability): reachable
An attempted agent invocation also failed with:
managed non-interactive error:
error_class=unknown
category=unknown
retryable=false
correlation_id=01a04478-c0f4-7493-bf38-1452116f0adb
Unexpected error
status=agent_failure
exit_code=1
response=null
response_bytes=0
To isolate the fault, we bypassed NemoClaw and Deep Agents Code and sent the same engineering-review workload directly to NVIDIA's OpenAI-compatible endpoint:
https://integrate.api.nvidia.com/v1/chat/completions
using:
model: nvidia/nemotron-3-ultra-550b-a55b
The prompt was approximately 111,564 characters. The direct request succeeded and returned a complete substantive response of approximately 22 KB.
This demonstrates that:
the NVIDIA credential is valid;
the Ultra model endpoint is operational;
the prompt is accepted by the model;
the model can return a full response normally;
the failure appears specific to the NemoClaw managed-inference / inference.local / Deep Agents Code path.
For comparison, we onboarded a separate Deep Agents Code sandbox using:
nvidia/nemotron-3-super-120b-a12b
That sandbox reports:
Inference: healthy
Inference (route reachability): reachable
Environment:
OpenShell: 0.0.101
LangChain Deep Agents Code: v0.1.34
Harness: langchain-deepagents-code
Runtime: Docker
Host GPU: none
Sandbox GPU: disabled
Cloud NVIDIA inference
Expected behaviour:
A Deep Agents Code sandbox configured for Nemotron 3 Ultra should be able to invoke the model through NemoClaw managed inference if the model is supported.
Actual behaviour:
The managed route is reachable, but the invocation probe returns HTTP 503 and Deep Agents Code can terminate with an agent failure and zero response bytes.
We would particularly appreciate confirmation of:
whether nvidia/nemotron-3-ultra-550b-a55b is currently supported through NemoClaw managed inference with Deep Agents Code;
whether the correlation ID above can be traced;
whether there is a known compatibility issue involving OpenShell 0.0.101, Deep Agents Code v0.1.34 and Nemotron 3 Ultra;
whether additional gateway/provider configuration is required for Ultra;
and what additional sanitized logs would be useful.
Separately, we also encountered a 32,768-byte CLI command-argument limit when passing a large review prompt directly as an argument. We worked around that using stdin/file transfer, so we do not believe that is the primary fault reported here.
No API keys or credentials are included in this report.
Reproduction Steps
-
Onboard a NemoClaw LangChain Deep Agents Code sandbox using NVIDIA cloud inference and select:
nvidia/nemotron-3-ultra-550b-a55b
-
Confirm the sandbox reaches Ready state.
-
Run:
nemoclaw evistate-code status
-
Observe:
Inference: unhealthy (https://inference.local/v1/models)
Inference gateway did not serve an inference request:
sandbox inference invocation probe returned HTTP 503.
Inference (route reachability): reachable
-
Attempt a Deep Agents Code task against the sandbox.
-
Observe an agent failure. One affected execution returned:
error_class=unknown
category=unknown
retryable=false
correlation_id=01a04478-c0f4-7493-bf38-1452116f0adb
status=agent_failure
exit_code=1
response=null
response_bytes=0
-
Bypass NemoClaw/Deep Agents Code and submit the same workload directly to:
https://integrate.api.nvidia.com/v1/chat/completions
with model:
nvidia/nemotron-3-ultra-550b-a55b
-
Observe that the direct NVIDIA API request succeeds and returns a complete model response (~22 KB for our ~111,564-character test prompt).
-
As a comparison, onboard a fresh Deep Agents Code sandbox using:
nvidia/nemotron-3-super-120b-a12b
-
Run its status command and observe:
Inference: healthy
Inference (route reachability): reachable
No API key is required to reproduce from this report; use a normal valid NVIDIA credential.
Environment
Host:
- Linux Google Cloud VM
- x86_64
- Docker container runtime
- 4 vCPU
- ~15.6 GiB RAM
- No swap
- No NVIDIA GPU detected
- Sandbox GPU: disabled
- NVIDIA cloud inference
NemoClaw / sandbox:
- OpenShell: 0.0.101
- LangChain Deep Agents Code: v0.1.34
- Harness: langchain-deepagents-code
- Runtime: terminal
- Policies: npm, pypi, huggingface, brew, github
Affected sandbox:
- Name: evistate-code
- Model: nvidia/nemotron-3-ultra-550b-a55b
- Provider reported by status: nvidia-prod
- Inference route: https://inference.local/v1/models
- Route reachability: reachable
- Invocation probe: HTTP 503
Comparison sandbox:
- Name: evistate-super
- Model: nvidia/nemotron-3-super-120b-a12b
- Provider reported by status: nvidia-prod
- Inference: healthy
- Route reachability: reachable
NemoClaw onboarding uses NVIDIA Build/cloud inference.
API credentials intentionally omitted.
Debug Output
nemoclaw debug --quick was run against the affected evistate-code sandbox.
Relevant output:
System:
- Linux x86_64 Google Cloud VM
- RAM: 15,987 MB
- Swap: 0
- No local NVIDIA GPU
OpenShell:
- Gateway: nemoclaw
- Server: https://127.0.0.1:8080
- Status: Connected
- Authentication: Authenticated (mTLS transport)
- Version: 0.0.101
Affected sandbox:
- Name: evistate-code
- Phase: Ready
- Sandbox ID: 391dbd6a-bfb7-4d03-bb3c-559603f108a9
- Revision: 4
Managed inference policy explicitly permits:
- inference.local:443
- POST /v1/chat/completions
- POST /v1/completions
- POST /v1/embeddings
- GET /v1/models
- GET /v1/models/**
Relevant router diagnostics:
[sandbox] NET:OPEN ALLOWED inference.local:443
[sandbox] [openshell_router] routing proxy inference request endpoint=https://integrate.api.nvidia.com/v1 method=GET path=/v1/models
and during agent execution:
[sandbox] NET:OPEN ALLOWED inference.local:443
[sandbox] [openshell_router] routing proxy inference request (streaming) endpoint=https://integrate.api.nvidia.com/v1 method=POST path=/v1/chat/completions
Multiple subsequent streaming POST /v1/chat/completions requests are visible in the debug output.
The sandbox and gateway therefore appear operational and network policy permits managed inference; the failure occurs after the managed inference request is routed toward the NVIDIA endpoint.
No API credentials are included in this extract. Full nemoclaw debug --quick output is available if maintainers require it.
Logs
Affected Deep Agents Code execution:
managed non-interactive error:
error_class=unknown
category=unknown
retryable=false
correlation_id=01a04478-c0f4-7493-bf38-1452116f0adb
Unexpected error
correlation_id=01a04478-c0f4-7493-bf38-1452116f0adb
status=agent_failure
exit_code=1
response=null
response_bytes=0
Affected sandbox status separately reported:
Model: nvidia/nemotron-3-ultra-550b-a55b
Provider: nvidia-prod
Inference: unhealthy (https://inference.local/v1/models)
Inference gateway did not serve an inference request:
sandbox inference invocation probe returned HTTP 503.
Inference (route reachability): reachable
Isolation test:
The same Nemotron 3 Ultra model and substantive workload succeeded when called directly through NVIDIA's integrate.api.nvidia.com/v1/chat/completions endpoint.
Related issues reviewed before filing include #5744, #9895 and #6913; none appears to reproduce this specific Deep Agents Code + NVIDIA Endpoints + Nemotron 3 Ultra managed-inference failure.
Checklist
Investigation Summary
NemoClaw sandbox using nvidia/nemotron-3-ultra-550b-a55b reports managed inference as unhealthy with HTTP 503 from https://inference.local/v1/models, although route reachability is reported as reachable.
Deep Agents Code execution failed with status=agent_failure, response=null, response_bytes=0, correlation ID 01a04478-c0f4-7493-bf38-1452116f0adb.
The same ~111,564-character engineering-review prompt succeeds when sent directly to NVIDIA https://integrate.api.nvidia.com/v1/chat/completions using the same Ultra model, returning a complete ~22 KB response.
A separately onboarded nvidia/nemotron-3-super-120b-a12b Deep Agents Code sandbox reports managed inference as healthy.
This isolates the apparent fault to the NemoClaw/OpenShell/Deep Agents Code managed-inference path for Ultra rather than the NVIDIA credential, model endpoint or prompt. Exact underlying cause remains unknown.
Description
We are evaluating NemoClaw for a controlled software-engineering and adversarial code-review workflow.
A Deep Agents Code sandbox configured for:
nvidia/nemotron-3-ultra-550b-a55b
is created successfully, but its status reports:
Model: nvidia/nemotron-3-ultra-550b-a55b
Provider: nvidia-prod
Inference: unhealthy (https://inference.local/v1/models)
Inference gateway did not serve an inference request:
sandbox inference invocation probe returned HTTP 503.
Inference (route reachability): reachable
An attempted agent invocation also failed with:
managed non-interactive error:
error_class=unknown
category=unknown
retryable=false
correlation_id=01a04478-c0f4-7493-bf38-1452116f0adb
Unexpected error
status=agent_failure
exit_code=1
response=null
response_bytes=0
To isolate the fault, we bypassed NemoClaw and Deep Agents Code and sent the same engineering-review workload directly to NVIDIA's OpenAI-compatible endpoint:
https://integrate.api.nvidia.com/v1/chat/completions
using:
model: nvidia/nemotron-3-ultra-550b-a55b
The prompt was approximately 111,564 characters. The direct request succeeded and returned a complete substantive response of approximately 22 KB.
This demonstrates that:
the NVIDIA credential is valid;
the Ultra model endpoint is operational;
the prompt is accepted by the model;
the model can return a full response normally;
the failure appears specific to the NemoClaw managed-inference / inference.local / Deep Agents Code path.
For comparison, we onboarded a separate Deep Agents Code sandbox using:
nvidia/nemotron-3-super-120b-a12b
That sandbox reports:
Inference: healthy
Inference (route reachability): reachable
Environment:
OpenShell: 0.0.101
LangChain Deep Agents Code: v0.1.34
Harness: langchain-deepagents-code
Runtime: Docker
Host GPU: none
Sandbox GPU: disabled
Cloud NVIDIA inference
Expected behaviour:
A Deep Agents Code sandbox configured for Nemotron 3 Ultra should be able to invoke the model through NemoClaw managed inference if the model is supported.
Actual behaviour:
The managed route is reachable, but the invocation probe returns HTTP 503 and Deep Agents Code can terminate with an agent failure and zero response bytes.
We would particularly appreciate confirmation of:
whether nvidia/nemotron-3-ultra-550b-a55b is currently supported through NemoClaw managed inference with Deep Agents Code;
whether the correlation ID above can be traced;
whether there is a known compatibility issue involving OpenShell 0.0.101, Deep Agents Code v0.1.34 and Nemotron 3 Ultra;
whether additional gateway/provider configuration is required for Ultra;
and what additional sanitized logs would be useful.
Separately, we also encountered a 32,768-byte CLI command-argument limit when passing a large review prompt directly as an argument. We worked around that using stdin/file transfer, so we do not believe that is the primary fault reported here.
No API keys or credentials are included in this report.
Reproduction Steps
Onboard a NemoClaw LangChain Deep Agents Code sandbox using NVIDIA cloud inference and select:
nvidia/nemotron-3-ultra-550b-a55b
Confirm the sandbox reaches Ready state.
Run:
nemoclaw evistate-code status
Observe:
Inference: unhealthy (https://inference.local/v1/models)
Inference gateway did not serve an inference request:
sandbox inference invocation probe returned HTTP 503.
Inference (route reachability): reachable
Attempt a Deep Agents Code task against the sandbox.
Observe an agent failure. One affected execution returned:
error_class=unknown
category=unknown
retryable=false
correlation_id=01a04478-c0f4-7493-bf38-1452116f0adb
status=agent_failure
exit_code=1
response=null
response_bytes=0
Bypass NemoClaw/Deep Agents Code and submit the same workload directly to:
https://integrate.api.nvidia.com/v1/chat/completions
with model:
nvidia/nemotron-3-ultra-550b-a55b
Observe that the direct NVIDIA API request succeeds and returns a complete model response (~22 KB for our ~111,564-character test prompt).
As a comparison, onboard a fresh Deep Agents Code sandbox using:
nvidia/nemotron-3-super-120b-a12b
Run its status command and observe:
Inference: healthy
Inference (route reachability): reachable
No API key is required to reproduce from this report; use a normal valid NVIDIA credential.
Environment
Host:
NemoClaw / sandbox:
Affected sandbox:
Comparison sandbox:
NemoClaw onboarding uses NVIDIA Build/cloud inference.
API credentials intentionally omitted.
Debug Output
nemoclaw debug --quickwas run against the affectedevistate-codesandbox.Relevant output:
System:
OpenShell:
Affected sandbox:
Managed inference policy explicitly permits:
Relevant router diagnostics:
[sandbox] NET:OPEN ALLOWED inference.local:443
[sandbox] [openshell_router] routing proxy inference request endpoint=https://integrate.api.nvidia.com/v1 method=GET path=/v1/models
and during agent execution:
[sandbox] NET:OPEN ALLOWED inference.local:443
[sandbox] [openshell_router] routing proxy inference request (streaming) endpoint=https://integrate.api.nvidia.com/v1 method=POST path=/v1/chat/completions
Multiple subsequent streaming POST /v1/chat/completions requests are visible in the debug output.
The sandbox and gateway therefore appear operational and network policy permits managed inference; the failure occurs after the managed inference request is routed toward the NVIDIA endpoint.
No API credentials are included in this extract. Full
nemoclaw debug --quickoutput is available if maintainers require it.Logs
Checklist