What this is: the current verification scorecard for every skill, scored against the
ladder in docs/verify.md. verify.md is the protocol (the rules);
this file is the status (where each skill actually stands today).
Legend: ✅ verified · N/A rung doesn't apply · TODO not yet done.
Ladder rungs (see verify.md for detail): 1 official-helper match · 2 tutorial number · 3 conservation closure · 4 physical sanity · 5 internal cross-check · 6 regression tests (teeth-verified) · 7 standing adversarial review.
"Done" bar: a science skill is done only when every applicable rung is satisfied
(or its N/A justified), a teeth-verified regression test exists, and it has passed a
dedicated Rung-7 adversarial pass — with evidence recorded in its references/.
Two independent axes — don't conflate them:
- AI-verified / "DONE" (the rungs + the ✅/
⚠️ /🔴 above) = the skill has cleared this project's automated + adversarial verification ladder. This is what the rung tables track. - Scientist Verified (⏳ Pending / ✅ Verified) = a domain oceanographer has personally
signed off on the calculation as scientifically correct and fit-for-purpose. This is a
separate, higher bar that AI verification cannot substitute for (AI confidence is not
expert sign-off — the founding principle of this project). Every science skill is
currently
Scientist Verified: ⏳ Pending, even the AI-"DONE" ones — no human-expert review has happened yet. Flip a skill to✅ Verifiedonly when an oceanographer has actually reviewed it, and record who/when.
Last updated: 2026-08-06.
| Rung | Status | Evidence / gap |
|---|---|---|
| 1 official helper | N/A | No OHC/volume-weighted-mean helper in ecco_v4_py 1.8.1 (checked dir(), scalar_calc). Cell-volume cross-checked instead (Rung 5). |
| 2 tutorial number | ✅ | Reproduces the scalar-quantities tutorial's published total ocean surface area = 3.58E+08 km² exactly ((rA·maskC).isel(k=0).sum()), now an automated test in test_validation.py. (Validates the grid geometry underpinning the OHC volume weighting; the tutorial publishes no OHC scalar itself.) |
| 3 conservation | N/A | Snapshot heat content, not a budget. |
| 4 physical sanity | ✅ | Volume-mean THETA 3.594 °C (≈3.5 known); THETA range [-1.97, 31.94] °C; runtime L3 guard. |
| 5 cross-check | ✅ | Ocean volume 1.335e18 m³, within 0.4% of literature; hFacC confirmed to zero land. |
| 6 regression | ✅ | scripts/test_validation.py (10) + ecco-common/tests/ (13); teeth verified. |
| 7 adversarial | ✅ | Dedicated "disprove OHC" pass (2026-07-25, independent agent): zero confirmed errors; verified volume formula, no hFac/maskC double-count, constants, potential-temp handling, NaN handling, benchmarks-not-luck, change computation, and that no official helper is being skipped. Three caveats raised (not errors) — 2 documented, 1 fixed. |
Status: DONE. All applicable rungs cleared. The adversarial pass raised three
caveats (not errors): (i) fixed geometry omits z*/SSH volume term — now documented in
SKILL.md; (ii) snapshot aliasing — documented; (iii) loose L3 volume-mean band — fixed
(tightened [0,10] → [2,6] °C). Record: compute-ocean-heat-content/references/acceptance.md.
Scientist Verified: ⏳ Pending — AI-verification complete; no domain-oceanographer sign-off yet.
| Rung | Status | Evidence / gap |
|---|---|---|
| 1 official helper | ✅ | Reproduces ecco_po_tutorials.geos_vel_compute to <1e-9 m/s over 2,237,682 points (vendored @ 3f0fcca). Reproducibility — a shared bug would pass; see Rung 5. |
| 2 tutorial number | N/A | Tutorial publishes figures/arrays, not a scalar. |
| 3 conservation | N/A | Diagnostic velocity, not a budget. |
| 4 physical sanity | ✅ | Surface geostrophic speed median ~0.029 m/s off-equator; WBC-box max ~0.32 m/s. |
| 5 cross-check | ✅ | Independent: matches ACTUAL model UVEL/VVEL at ~350 m (corr 0.998; median norm-diff 0.032; 45,745 pts). Different variable + code path → rules out a bug shared with the reference. Strongest correctness evidence. |
| 6 regression (teeth) | ✅ | test_geostrophic.py (Rung-1 match + independent-velocity check + 5 guards); teeth verified. |
| 7 adversarial | ✅ | Independent disprove-pass (2026-07-25): zero confirmed errors. Its one fair critique (overstated Rung-1 claim) is fixed — added the Rung-5 independent test + corrected language. |
Status: DONE. The adversarial pass found no correctness errors; acting on its critique
strengthened the evidence (independent velocity check added). Documented limitations
(not errors): model-axis output, coastal NaN-bleed, tile-seam extend — all inherited
from the official reference. Record: compute-geostrophic-balance/references/acceptance.md.
Scientist Verified: ⏳ Pending — AI-verification complete; no domain-oceanographer sign-off yet.
| Rung | Status | Evidence / gap |
|---|---|---|
| 1 official helper | N/A | No thermal_wind_compute in ecco_po_tutorials.py (only geos_vel_compute; confirmed by listing all module functions). Tutorial spreads it across cells. |
| 2 tutorial number | ✅ (regime) | Reproduces the tutorial's 26°N reconstruction-vs-actual normalized-difference diagnostic (median 0.231, 100–1000 m). Tutorial publishes a curve, not a scalar. |
| 3 conservation | N/A | Diagnostic shear, not a budget. |
| 4 physical sanity | ✅ | Off-eq |
| 5 cross-check | ✅ | (1) identity: shear ≈ ∂/∂z of geostrophic velocity, corr 0.999 (1.40M pts). (2) independent: predicted vs ACTUAL velocity shear, corr 0.64/0.85 (1.41M pts) — different variable/path. |
| 6 regression (teeth) | ✅ | test_thermal_wind.py: 3 data cross-checks + 6 offline guards. Teeth verified: sign flip fails checks 1–3; dropping g fails check 3. Reconstruction threshold tightened 0.6→0.35 after eval-4 (caught a 1.5× magnitude bug that slipped 0.6). |
| 7 adversarial | ✅ | Independent Sonnet disprove-pass (2026-08-06, docs/eval4.md): could not disprove; zero confirmed errors; all 5 acceptance numbers reproduced. |
Status: ✅ DONE — all applicable rungs cleared, incl. a clean Rung-7 adversarial pass
(one loose-threshold caveat found and fixed). Record:
compute-thermal-wind/references/acceptance.md; review log: docs/eval4.md.
Scientist Verified: ⏳ Pending — AI-verification complete; no domain-oceanographer sign-off yet.
| Rung | Status | Evidence / gap |
|---|---|---|
| 1 official helper | N/A | No curl helper in vendored module or ecco_v4_py. Partial: CS/SN rotation core is bit-identical to official vector_calc.UEVNfromUXVY (max|Δ|=0). |
| 2 tutorial number | ✅ (operator) | Two-rotation pipeline from the official native-grid gradient/curl tutorial; publishes maps not scalars. |
| 3 conservation | N/A | Diagnostic, not a budget. |
| 4 physical sanity | ✅ | Curl ~1e-7 Pa/m; correct sign (N. Pacific subtropical gyre negative → downwelling); w_E O(1e-6 m/s). Equator masked. |
| 5 cross-check | ✅ | (a) rotation == UEVNfromUXVY (bit-identical). (b) independent physical: Ekman w_E vs actual WVEL @~30 m: corr 0.738, sign-agree 0.89 (48,383 pts). |
| 6 regression (teeth) | ✅ | test_curl.py: rotation + Ekman-vs-WVEL + a teeth test (2nd rotation is load-bearing, ~30% shift) + 6 offline guards. Sign flip → WVEL corr −0.56 (fails). |
| 7 adversarial | ✅ | Independent Sonnet disprove-pass (2026-08-06, docs/eval5.md): could not disprove; zero confirmed errors; both historical rotation bugs confirmed blocked; all numbers reproduced. |
Status: ✅ DONE — all applicable rungs cleared, incl. a clean Rung-7 adversarial pass
(one loose-teeth-threshold caveat found and fixed, 0.05→0.20). Build also fixed two
design-doc errors (oceTAUX/Y are on faces not tracer points; the two rotations use the same
formula). Record: compute-curl/references/acceptance.md; review log: docs/eval5.md.
Scientist Verified: ⏳ Pending — AI-verification complete; no domain-oceanographer sign-off yet.
compute-steric-height (Recipe 3-steric) — ✅ DONE (all applicable rungs cleared; Rung-7 pass 2026-08-06)
| Rung | Status | Evidence / gap |
|---|---|---|
| 1 official helper | N/A (integral) | EOS anchor: vendored JMD95 densjmd95(35.5,3,3000) = 1041.83267 == published (automated). |
| 2 tutorial number | ✅ (operator) | Pipeline transcribed from Steric_height.ipynb; publishes maps not scalars. |
| 3 conservation | N/A | Diagnostic, not a budget. |
| 4 physical sanity | ✅ | Global-mean-removed anomaly range ≈ [-3.2, 2.2] m; high in warm subtropics, low in Southern Ocean (matches SSH). Land + too-shallow (<2000 dbar) masked. |
| 5 cross-check | ✅ | (a) sum-of-parts: thermo+halo ≈ full, median residual 0.005 m, corr 0.9998. (b) INDEPENDENT: steric ≈ SSH, corr 0.921 (different collection; residual = non-steric mass part). |
| 6 regression (teeth) | ✅ | test_steric.py (6 tests): EOS check-value + sum-of-parts + steric-vs-SSH + thermo/halo label guard (vs SST) + specvol-sign teeth + offline guards. |
| 7 adversarial | ✅ | Independent Sonnet disprove-pass (2026-08-06, docs/eval6.md): could not disprove; zero confirmed errors; all numbers reproduced. |
Status: ✅ DONE — all applicable rungs cleared, incl. a clean Rung-7 adversarial pass
(one caveat found and fixed: added a thermo/halo label-swap guard, which sum-of-parts
couldn't catch). Vendored the MITgcm JMD95 EOS (no EOS was available); a de-meaning/masking
bug was fixed during the build. Record: compute-steric-height/references/acceptance.md;
review log: docs/eval6.md.
Scientist Verified: ⏳ Pending — AI-verification complete; no domain-oceanographer sign-off yet.
(Not science calculations — the physics rungs 2–5 don't apply. Verified via official
helpers where relevant + the ecco-common regression suite.)
| Skill | Status | Evidence | Gaps |
|---|---|---|---|
ecco-setup (set-up mode) |
✅ | Wheels-only install, --reset re-resolves xgcm<0.10, auto-runs verify, guided "no Python" stop — tested on macOS/arm64/3.12.13. |
Linux/Windows testing TODO. |
ecco-setup (verify mode) |
✅ | Exercises official ecco.get_llc_grid on real geometry; passes from any CWD, and setup→verify handoff works from the new in-skill path. Consolidated from the former ecco-setup-verify skill 2026-08-06 (capability unchanged). |
— |
load-grid |
✅ | Builds grid via official ecco.get_llc_grid (Rung 1); regression-covered; runs from any CWD. |
— |
load-field |
✅ | CMR pagination, size-guard-by-filename, month/day midpoint selection, backfill, offline reuse, selector validation — 13-test suite, teeth-verified. | Download checksum verification not yet implemented (checksum captured only). |
plot-ecco-field |
✅ | Wraps official ecco_v4_py plotters (plot_tile/plot_tiles/plot_proj_to_latlon_grid); verified by producing a physically-correct global SST map + model-orientation tile. Headless (Agg → PNG). |
Visual output not auto-regression-tested (would need image hashing); relies on the official plotter's own correctness. |
| Suite | Count | Teeth-verified? |
|---|---|---|
ecco-common/tests/test_ecco_common.py |
13 | ✅ (reintroducing the size-guard sidecar bug fails it) |
ecco-common/tests/test_grid_ops.py |
5 (Level-1 primitives) | ✅ (grad_to_center matches the inline diff/interp sequence; coriolis signs) |
ecco-common/tests/test_preflight.py |
5 (environment guard) | ✅ (bogus-lib probe fails; healthy passes; exits non-zero on unhealthy) |
compute-ocean-heat-content/scripts/test_validation.py |
10 (+1 Rung-2 tutorial check) | ✅ (bad-input cases fail; land-NaN passes) |
compute-geostrophic-balance/scripts/test_geostrophic.py |
Rung-1 match + 5 guards | ✅ (breaking equatorial mask fails it) |
compute-thermal-wind/scripts/test_thermal_wind.py |
3 cross-checks + 6 guards | ✅ (sign flip fails checks 1–3; dropping g fails check 3) |
compute-curl/scripts/test_curl.py |
rotation + Ekman-vs-WVEL + teeth + 6 guards | ✅ (2nd-rotation drop = 30% shift; sign flip → WVEL corr −0.56) |
compute-steric-height/scripts/test_steric.py |
EOS + sum-of-parts + steric-vs-SSH + teeth + 5 guards | ✅ (specvol sign flip → steric-vs-SSH corr −0.92) |
Run all: .claude/skills/run_all_tests.py |
all of the above | — |
Not yet automated (open, per verify.md / evals): CI wiring; exact dependency lockfile; cached-file checksum verification; index locking for concurrent runs.
Every new/changed science skill gets an independent adversarial review before it's
called done — an external AI instance instructed to disprove it against the tutorials
and installed helpers. Prior rounds logged in docs/eval1.md, docs/eval2.md,
docs/eval3.md, and — all 2026-08-06 — docs/eval4.md (thermal wind), docs/eval5.md
(curl), docs/eval6.md (steric). Keep this loop running — it has been the highest-yield
check. All five science skills have now passed Rung-7 (none had a confirmed error; each
of evals 4–6 found exactly one test-hardening improvement, all fixed). Next skills to build
get the same pass before "done".
Update this dashboard in the same step a skill's verification status changes (new
rung satisfied, new gap found, adversarial pass completed) — same living-doc rule as
design.md. When a skill reaches "done", change its SKILL.md
header.