[Feature] Unify job run orchestration for single, multi-task, and full dataset runs

[berstpander]

Feature Category

• SDK & API
• CLI
• Envhub
• Documentation & Examples

Problem Statement

rock job run currently focuses on running a single sandbox job. Dataset-scale workflows need users or upper-level wrappers to manually enumerate tasks, submit jobs one by one, track local progress, and recover interrupted runs.

Key gaps:

1. No unified CLI entrypoint for single-task, multi-task, and full dataset execution.
2. Full dataset execution should remain a CLI-level orchestration capability, while the SDK should only expose reusable atomic primitives.
3. Interrupted local processes cannot reliably resume a run and reconnect to still-running sandbox jobs.
4. There is no run-level metadata that maps run_id -> task_id -> job_name, making status/show queries difficult.
5. Upper-level wrappers need JSONL progress events for automation.

Proposed Solution

1. Extend rock job run as the unified entrypoint:

   • --task <id> for one explicit task
   • --tasks <id1,id2> for selected tasks
   • --all for full dataset split execution
   • --resume <run_id> for interrupted runs
   • --concurrency, --limit, and --jsonl for execution control and automation

2. Keep SDK/CLI responsibilities separated:

   • SDK provides atomic job planning, job execution, metadata repository, and reconnect/wait primitives.
   • CLI composes these primitives into single, multi-task, and full dataset orchestration.

3. Add two-level metadata:

   • Run metadata under the experiment directory, recording run_id, mode, task list, task/job mapping, status, and score summary.
   • Job metadata under each job directory, recording sandbox id, session, pid, task id, status, score, error, and timestamps.

4. Support resume semantics:

   • Skip completed tasks.
   • Reconnect to still-running sandbox jobs when sandbox_id/session/pid are available.
   • Mark unrecoverable previous attempts and rerun when sandbox/process metadata is missing or unusable.

5. Add run-level query commands:

   • rock job runs
   • rock job status --run-id <id>
   • rock job show --run-id <id> --task-id <task>