Skip to content

exp105: contacts-and-coordinates-v1 data run — fix iris client freshness gate, finalize launch#119

Open
timodonnell wants to merge 8 commits into
mainfrom
claude/exp105-ccoord-v1-datagen
Open

exp105: contacts-and-coordinates-v1 data run — fix iris client freshness gate, finalize launch#119
timodonnell wants to merge 8 commits into
mainfrom
claude/exp105-ccoord-v1-datagen

Conversation

@timodonnell

@timodonnell timodonnell commented Jul 8, 2026

Copy link
Copy Markdown
Member

What

Prep + infra fix for the exp105 at-scale generation of the
contacts-and-coordinates-v1 corpus (same ~4.2M proteins as contacts-v1,
reusing exp53's selection manifest verbatim). Issue #105.

The format/generator/pipeline already merged (PR #106; 1 Å / max_depth=3 per
PR #118). This PR makes the pipeline actually launchable against the current
live iris cluster and records the smoke-gate result.

The blocker this fixes

The pinned marin-zephyr branch alxmrs/stamp-iris-build-date is gone
(marin PR #5990 closed unmerged, branch deleted). More importantly, the live
controller now enforces a 14-day client-freshness gate on root job
submissions (marin PR #5108), and all marin-*-latest release wheels are
frozen at 2026-05-29 — so any client built from them is rejected at submit
("marin-iris client is too old; minimum 2026-06-24").

Key insight from iris/cluster/controller/service.py: the gate fires only on
job_id.is_root. Worker/child tasks are exempt, and the worker env is
uv sync'd from this pyproject independently. So:

  • Worker env → keep the frozen marin-*-latest wheels (via find-links).
  • Submitting client → must be a fresh editable marin-iris (an editable
    install reports its git-commit date as the client revision). Documented in
    the README; on this workstation /home/bizon/git/marin/.venv/bin/iris
    (currently 2026-07-07) works.

Smoke gate (✅, 2026-07-08, us-central1)

100 docs from train/shard_00000.parquet → JOB_STATE_SUCCEEDED:

  • 100/100 docs, 0 drops (AFDB single-chain + manifest pre-filtered length).
  • Schema = metadata_row() + provenance passthrough; structure = contacts-and-coordinates-v1; max_depth=3.
  • num_tokens 32,764–32,768 (mean 32,766) — budget-filling as projected.
  • sha1 == sha1(document) for all rows; byte-identity to generate_document
    pinned by tests/test_generate_rows.py.
  • Generation: ~109 s / 100 docs on one core (incl. one-time rotamer download).
  • One cosmetic issue: the 2026-05-29 worker finelog client 404s on the
    dashboard status-text push → non-fatal "Failed to report task status text"
    warnings. Pipeline runs + writes correct output regardless.

Changes

  • pyproject.toml: drop dead git-branch pin; find-links to marin-*-latest;
    add marin-iris; keep marinfold git dep; commit uv.lock.
  • README.md: fresh-editable-client requirement + $IRIS launch commands;
    tokenizer published to the HF bucket (not a model repo); finelog caveat.

Not in this PR (awaiting go-ahead — large billable run)

The full train/val/test run + HF publish. Projection: ~1,200 core-hours
(~2.5 h wall on 512 workers), ~229 GB parquet / ~138 B tokens.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk

timodonnell and others added 8 commits July 8, 2026 17:41
@timodonnell @claude
exp105: fix marin client freshness gate + finalize launch procedure
138a2fc 
The live iris controller now rejects root LaunchJob submissions from
marin-iris clients older than 14 days (marin PR #5108). The pinned
marin-zephyr branch (alxmrs/stamp-iris-build-date) was closed unmerged and
deleted, and the marin-*-latest release wheels are frozen at 2026-05-29 —
both too old to submit with.

- pyproject: drop the dead git-branch pin; install marin packages for the
  worker/launcher env from the marin-*-latest release assets via find-links
  (workers/child tasks are exempt from the freshness gate). Add marin-iris.
  Keep marinfold as a git dep (contacts_and_coordinates_v1 is on main via
  PR #106). Commit uv.lock (smoke-validated resolution).
- README: document that the *submitting* client must be a fresh editable
  marin-iris (git-date >= today-14); switch launch commands to $IRIS; move
  the tokenizer to the HF bucket (open-athena model-repo creation 403s and
  the corpus-tokenizer-in-bucket convention supersedes it); note the
  non-fatal finelog dashboard-status warnings.

Iris smoke (100 docs, us-central1) verified: JOB_STATE_SUCCEEDED, 100/100
docs (0 drops), schema + sha1 consistent, num_tokens ~32,766.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: on-demand workers + repeatable --region; document CPU-capacit…
3feee71 
…y blocker

The marin cluster has NO preemptible CPU scale group, so --preemptible worker
requests register zero autoscaler demand and strand the job on the ~dozen
incidental on-demand CPU VMs. Fixes:
- cli.py: --preemptible now defaults to False (on-demand); --region is
  repeatable (action=append) so the on-demand pool can span US regions without
  exp53's trans-Atlantic spill.
- README: on-demand launch commands (multi-US-region), plus a capacity caveat
  (autoscaler peak_demand ~6 on us-central1-a; a 1024-way fan-out only
  materialises when the shared, TPU-heavy cluster is quiet).

Full run remains BLOCKED on CPU capacity: on-demand + 6 US regions still placed
only 2 workers (0/2067 shards) against a cluster running 1324 other tasks.
Escalating to a human for a cluster-ops decision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: use current marin from PyPI (fixes worker-env skew that stall…
77123f8 
…ed the run)

Root cause of the at-scale stall: the frozen marin-*-latest GitHub release
wheels (0.99.dev20260529) are incompatible with the upgraded live controller.
Worker tasks got scheduled but never registered with the zephyr coordinator
(finelog register_table 404s), so zephyr saw ~0 live workers and 0 shards were
produced — regardless of cluster capacity (verified against a quiet cluster).

Fix: marin publishes a fresh co-versioned dev release of the whole stack to
PyPI on ~every push. Depend on marin-iris/zephyr >=0.2.40.dev202607080801
(2026-07-08, matching the controller); fray/rigging/finelog/finelog-server
resolve transitively from PyPI (incl. the native finelog-server wheel). Drop the
stale -latest find-links.

Bonus: the PyPI marin-iris wheel stamps a fresh BUILD_DATE, so `uv run iris`
from this env passes the client-freshness gate directly — no separate editable
marin checkout needed for submission. requires-python bumped to >=3.12 (marin).

Realizes the "run from the marin workspace" intent (current marin stack) via
PyPI wheels instead of a workspace restructure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: README — document the PyPI-marin fix (self-contained iris cli…
7502ab8 
…ent; worker-registration)

Replace the editable-client instructions with the PyPI approach; the launcher
and workers both get current marin, so uv run iris submits directly and workers
register with the control plane. Launch commands use uv run iris + on-demand
multi-US-region.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: add NOTES.md — cluster/env debugging log (freshness gate, wor…
1d080dc 
…ker-registration, PyPI-marin fix, CPU capacity)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: README Results — val + test done + published (0 drops); train…
bbc6da0 
… capacity-blocked

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: add local_generate_train.py — off-cluster train generation/re…
35a0cea 
…sume

The AFDB gcs_uri bucket is publicly readable, so read_object_bytes fetches fine
locally (no requester-pays billing). This generates train shards on the
workstation's cores (spawn pool — fork segfaults the numba/pyconfind/gemmi C
exts), writing each with its ORIGINAL index to the same GCS train/ dir,
skip-existing so it composes with the concurrent iris grind and is restartable.
Runs descending (--reverse) so the local + cluster sweeps meet in the middle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
@timodonnell @claude
exp105: README Results — COMPLETE (train 4,129,682 + val + test, 0 dr…
01fbcb1 
…ops, ~138B tokens)

train ran fast after all: the CPU autoscaler scaled to ~1000 workers for the
2067-shard request (~2h), plus local_generate_train.py on the workstation. Full
corpus published. Corrected the earlier 'capacity-blocked' framing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddz1BSAfmjxCAgdxT681nk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@timodonnell