Skip to content

SKILL.md

Latest commit

 

History

History
127 lines (100 loc) · 5.56 KB

SKILL.md

File metadata and controls

127 lines (100 loc) · 5.56 KB
Raw
name check-inference
description Probe a model inference WebSocket server (e.g. `serve_policy`) and validate the response — using the `geniesim benchmark check-inference` CLI verb, which wraps the benchmark package's `check_inference.py`. Trigger: When the user asks to "check inference", "校验模型推理", "test inference server", "verify policy server", "ping the model", or provides an IP/port and wants to confirm a serve_policy / WebSocket inference server is working before running benchmarks.
license MPL-2.0
metadata
author version
genie-sim
2.0
prerequisites
geniesim_cli:fresh-machine-setup
inputs
name desc required
payload
Path to a corobot `.pkl` payload (defaults to the bundled corobot_payload.pkl)
false
name desc required
infer_host
IP:PORT of the inference WebSocket server
true
name desc required default
iters
Number of consecutive requests to send (catch flakiness)
false
1
outputs
desc
Pass/fail verdict with per-dim min/max/mean/std/NaN/Inf checks and round-trip latency; non-zero exit on failure

When to Use

  • Sanity-check whether a running inference server actually accepts requests and returns valid actions, before launching a full task.
  • User provides ip:port and a payload, and asks to verify connectivity / output validity.
  • Quick smoke test in CI / pre-deploy.
  • Diagnosing "the benchmark hangs / outputs garbage" — this probe surfaces protocol mismatches and NaN/Inf in actions before you sink time into a full simulator launch.

Do not use for:

  • Running the benchmark itself → use the run-benchmark skill.
  • Submitting jobs to the Challenge platform → challenge-submit-job.

What This Skill Does

Sends a saved corobot .pkl payload to ws://<HOST>:<PORT> and validates the reply:

  1. Loads the payload.
  2. Connects to the WebSocket server using msgpack-numpy.
  3. Sends one request, receives one action chunk.
  4. Validates: schema (key presence), per-dim min/max/mean/std, NaN/Inf flags, out-of-range checks (kind-aware: JOINT_ABS in radians, EEF_ABS in meters+quat, gripper in [0,1]), large jumps from the input state.

The payload is a corobot JSON-RPC envelope — {"method": "infer", "params": {...}} — and the server replies with {"result": {"left_arm": …, "right_arm": …, …}} (or {"error": …}).

Required Input

The user must provide:

  • HOST — server IP (e.g. 127.0.0.1).
  • PORT — server port (e.g. 8999).

PAYLOAD is optional — it defaults to the bundled corobot_payload.pkl. Pass a path only to override it (see Generating a payload below). If host/port are missing, ask the user before running.

How to Run

# Bundled payload — just point it at the server
geniesim benchmark check-inference --infer-host=<HOST>:<PORT>

# Override with your own payload (positional)
geniesim benchmark check-inference debug_preview/debug_0001.pkl \
    --host <HOST> --port <PORT>

If geniesim isn't on $PATH (the launcher wasn't installed), substitute python3 -m geniesim_cli benchmark check-inference … — same args, same behaviour.

Optional flags (forwarded to check_inference.py)

Flag Effect
--iters N Send N consecutive requests (default 1). Use 5–10 to catch flakiness.
--max-dims N Max idx rows printed per array (default 64).

Generating a payload

A canonical corobot_payload.pkl ships next to the script and is used by default, so you usually don't need to supply one. To probe with a fresh / task-specific observation, run a benchmark task with the corobot policy's debug dump enabled — it writes debug_preview/debug_NNNN.pkl (a {"payload": …, "obs": …} wrapper the probe unwraps automatically) — then pass that path.

Interpreting Output

The script prints structured sections:

Section What it tells you
📦 Payload Payload loaded and recognised as corobot.
🔌 Connecting Connected to ws server (or refused).
📡 …response Server returned a reply, schema check.
📊 <key> Per-dim min/max/mean/std + flags, per output (left_arm/right_arm/…).
⏱ latency Round-trip latency.
Outcome Meaning
✅ PASS — … Server is up and returning sane actions.
❌ Connection refused / timed out Server isn't listening on that host:port. Verify it's running and the firewall is open.
❌ … NaN / Inf … Server responded but model output is broken. Check the policy checkpoint and normalization stats — not a network problem.
❌ response missing 'result' dict Server schema mismatch — it isn't speaking the corobot JSON-RPC protocol the probe expects.
❌ server error: … The server returned a JSON-RPC error; read the message.
⚠️ OOB[…] flags Action is finite but outside the kind's expected range. Could be a units bug (radians vs degrees) or an unnormalized output.

Dependencies

The script needs (in the Python env that runs python3):

  • msgpack
  • numpy
  • websockets

It does not need Isaac Sim — pure-Python deps only. The CLI deliberately uses python3 instead of omni_python here so the probe is snappy.

Resources

  • Script source: source/geniesim_benchmark/src/geniesim_benchmark/scripts/check_inference.py (resolved via the geniesim_benchmark package)
  • CLI dispatcher: source/geniesim_cli/src/geniesim_cli/commands/benchmark.py (_do_check_inference)
  • Payload dump hook: source/geniesim_benchmark/src/geniesim_benchmark/benchmark/policy/corobotpolicy.py