[iris] Child job submission silently accepted when parent is absent from DB

[rjpower]

Describe the bug

When the Iris controller restarts and restores from an S3 checkpoint, child jobs whose parent rows were not captured in the checkpoint are accepted silently with a broken hierarchy: parent_job_id is set to NULL but depth is computed from the job name path. The root job never appears in the dashboard (which filters depth = 1), even though the child tasks are running correctly.

To Reproduce

1. Submit a multi-level job hierarchy (root → depth-2 → depth-3 zephyr tasks).
2. Controller OOMs before the hourly checkpoint captures the root and depth-2 rows.
3. Running processes (e.g. zephyr) reconnect and resubmit depth-3 child jobs.
4. submit_job in transitions.py:980–984 detects the parent is absent and sets parent_job_id = None rather than rejecting the submission.
5. launch_job in service.py:846–853 only rejects submissions whose parent is terminated, not missing (_job_state() returns None for absent parents, which passes the guard silently).
6. Jobs are inserted with correct depth but null parent_job_id; dashboard query WHERE depth = 1 finds no root → "0 jobs" for that run.

Expected behavior

Submitting a child job whose parent does not exist in the DB should be rejected with FAILED_PRECONDITION, the same as submitting to a terminated parent.

Additional context

• service.py:846: if job_id.parent: / if parent_state is not None and parent_state in TERMINAL_JOB_STATES — missing branch for parent_state is None
• transitions.py:983: if parent_exists is None: parent_job_id = None — silently drops the link instead of raising
• Fix: in launch_job, treat parent_state is None (parent absent) as a hard error, matching the terminated-parent case.

[claude]

🤖 Plan:

• Read the issue and locate launch_job parent guard and submit_job parent handling
• Duplicate-work preflight: no open PRs/issues found
• Post Research + Proposed Fix
• Branch agent/20260409-fix-4559, add regression test
• Update launch_job parent guard + drop silent null fallback in transitions.submit_job
• Fix tests broken by the new invariant (test_requeued_task_maintains_priority_position, test_child_job_inherits_parent_env)
• Run lint + tests/cluster/controller/ tests/e2e/test_env_propagation.py (720 passed)
• Open PR [iris] Reject child submissions when parent is absent from the DB #4560 and monitor CI

Research

After a controller restart, _job_state(db, parent) returns None for a parent absent from the checkpoint. service.py:846-853 only rejected when parent_state is not None and parent_state in TERMINAL_JOB_STATES, so missing parents fell through. transitions.py:980-984 then silently nulled parent_job_id while still computing depth from the name path, leaving orphaned rows the WHERE depth = 1 dashboard query never surfaces.

Relevant code

• service.py#L846-L853 — parent-terminated guard, missing branch for absent parent
• service.py#L321-L325 — _job_state returns None for absent jobs
• transitions.py#L980-L984 — silent null-parent fallback on submit
• test_service.py#L650-L671 — existing terminated-parent rejection test

Proposed Fix

Treat a non-root submission whose parent row is absent as FAILED_PRECONDITION at the launch_job boundary, matching the terminated-parent case. Drop the if parent_exists is None: parent_job_id = None fallback in transitions.submit_job (raise ValueError instead for defense-in-depth). Regression test launches /test-user/absent-parent/new-child without launching the parent and asserts FAILED_PRECONDITION. PR #4560 monitoring CI.