Skip to content

SKILL.md

Latest commit

 

History

History
145 lines (117 loc) · 5.16 KB

SKILL.md

File metadata and controls

145 lines (117 loc) · 5.16 KB
Raw
name run-benchmark
description Launch a geniesim_benchmark task locally (typically inside the GUI Docker container) against a user-provided inference server, using the `geniesim benchmark run` CLI verb. Trigger: When the user asks to "run geniesim", "本地跑仿真", "启动仿真任务", "run a benchmark", "launch <some>_<config>.yaml", or wants to execute a benchmark task config (anything under `geniesim_benchmark/config/*.yaml`) against a remote inference host (ip:port).
license MPL-2.0
metadata
author version
genie-sim
1.0
prerequisites
geniesim_cli:fresh-machine-setup
inputs
name desc required
config
Task config (basename, full path, or unique substring)
true
name desc required
infer_host
IP:PORT of the inference server
true
name desc required default
headless
GUI off (required on remote / batch hosts)
false
false
name desc required
num_episode
Override episode count
false
name desc required
seed
RNG / instance-sampling seed
false
name desc required default
record
Persist episode logs to output_dir
false
false
outputs
desc
Per-episode pass/fail summary; logs under `output_dir` if `record=true`

When to Use

  • User wants to run a geniesim_benchmark task on their workstation (not the Challenge platform).
  • User has an inference server already running somewhere reachable and provides its ip:port.
  • User references any task config under source/geniesim_benchmark/src/geniesim_benchmark/config/.

Do not use for:

  • Submitting jobs to the Challenge platform → challenge-submit-job.
  • Verifying an inference server is healthy → use the check-inference skill.
  • Adding a new benchmark task → add-benchmark-task.

Critical Patterns

  1. Always collect three required inputs first:
    • Task config (basename, full path, or substring — the CLI resolves all three).
    • Inference IP.
    • Inference port.
  2. The runtime needs omni_python / Isaac Sim on the host. Inside the Genie Sim Docker image (geniesim docker up → geniesim docker into), that's already the case. Outside the container the user needs Isaac Sim installed system-wide.
  3. Working directory: anywhere under the repo root works — the CLI walks up to find scripts/ and uses find_spec to locate the benchmark package.
  4. Confirm before launching. The task spawns a full simulator and typically holds a GPU; ask before kicking it off if there's any ambiguity.

Workflow

Step 1 — Collect inputs

If the user hasn't named a config, list candidates:

geniesim benchmark categories      # show category counts
geniesim benchmark robots          # show robot counts
geniesim benchmark list --robot=<R> --category=<C>

Then ask via AskUserQuestion:

  • Task config: free-text (use the basename — e.g. g2op_if_pick_block_color).
  • Inference host as ip:port.

Step 2 — Probe inference (optional but recommended)

Before sinking minutes into Isaac Sim startup, sanity-check the server (uses the bundled corobot payload):

geniesim benchmark check-inference --infer-host=<IP>:<PORT>

See the check-inference skill to override the payload.

Step 3 — Run the task

Inside the GUI container (geniesim docker into):

geniesim benchmark run <CONFIG> --infer-host=<IP>:<PORT>

Example:

geniesim benchmark run g2op_if_pick_block_color --infer-host=<IP>:<PORT>

Step 4 — Pass-through overrides (when asked)

geniesim benchmark run forwards any unknown --key=value to the benchmark's ParameterServer. Common ones:

Flag Meaning
--app.headless=true No GUI (required on remote / batch hosts)
--benchmark.num_episode=N Override episode count
--benchmark.seed=N RNG / instance-sampling seed
--benchmark.record=true Persist episode logs to output_dir
--benchmark.policy_class=… Use a different policy class

Full schema: source/geniesim_benchmark/src/geniesim_benchmark/config/params.py.

Commands (copy-paste summary for the user)

# Host — start the container (GUI by default; add --headless on remote/batch hosts)
cd /path/to/main
geniesim docker up

# Host — drop into a shell inside the running container
geniesim docker into
# inside container:
geniesim status                                       # verify the stack is healthy
geniesim benchmark check-inference --infer-host=<IP>:<PORT>
geniesim benchmark run <CONFIG> --infer-host=<IP>:<PORT>

Notes

  • If geniesim isn't on $PATH (the launcher wasn't installed), substitute python3 -m geniesim_cli benchmark … — same args, same behaviour.
  • <CONFIG> accepts the bare basename (g2op_if_pick_block_color), a full path, or a unique substring.
  • For batch evaluations, prefer geniesim benchmark batch --category=… --robot=… over a shell loop — it forwards extras consistently and prints a per-config pass/fail summary.
  • The new CLI replaces the older ad-hoc omni_python app/app.py --config … invocation. The new form normalizes interpreter selection, host shorthand, and config resolution.