Skip to content

Coherence is wired but unused by this cycle's harnesses — and 'change the ceiling' is a gate that doesn't exist yet: a four-rung ladder that measures the signal before building on it #40

Description

@OriNachum

The finding

coherence-cli is a live base dependency (coherence-cli>=0.6, approved
under deviation d2), embodiment/continuity.py wraps coherence.assess
behind a never-raise guard, and loop.py already defines the three boundaries
#2 asked for —
BOUNDARY_ACTION, BOUNDARY_COMPLETION, BOUNDARY_MEMORY — plus a
DEGRADED_CONTINUITY code so a failing seam records a transition.

The import guard resolves on this rig today. It works.

But none of the orchestrator-worker cycle's harnesses wire it. Not
arch_arms.py, arch_league.py, arch_vision.py, challenge_coding.py, or
orchestrator_tools.py. Six older harnesses do (greenhouse, selftest,
arena_series, echo_probe_workspace, league_seat, devague_legs), so this
is a coverage regression between cycles rather than a capability we never
had — the new harnesses were composed from t5's seam, and nobody's brief
mentioned continuity, including the ones I wrote.

Consequence: the architecture series measures all four arms with coherence
assessment switched off. Uniformly off, so it is not a confound between arms —
but it means the series says nothing about whether reasoning held together, only
whether answers were right.

The structural fact that shapes any experiment here

coherence.assess reports; it does not gate.

assess(path: str | Path, *, embed_fn=…, reference_date=…) -> dict

It takes a path, runs its domains, and returns scores plus an unavailable
map. There is no threshold parameter. And continuity.py is explicit that it
"reads no score and computes none" — it relays the report and records a
degradation when domains are unavailable, nothing more.

So "change the coherence ceiling" is not a knob that exists to be tuned. A
gate would have to be built
, and building one before knowing whether the
underlying signal discriminates would be assuming the result. That ordering is
the whole design below.

Two integration facts worth knowing before anyone estimates cost:

  • assess reads a file, so a reasoning trace must be written to disk at each
    boundary. Three boundaries × N turns × M cells is real I/O and real latency,
    against a cortex already measured at ~57 s median per reasoning turn.
  • meaning and investiture need the embedding endpoint; without it they
    land in unavailable and assess still returns normally. AssessOutcome.ok
    alone must never be read as "nothing degraded" — continuity.assess already
    guards this, and any experiment must report per-domain availability, not a
    single pass/fail.

Why this is worth doing at all

The series' verdicts rest entirely on correctness graders we wrote ourselves,
and this repo's most-repeated lesson is that our graders are where the defects
live — four failed in a single cycle
(corrections.md).
next-cycle-candidates.md M2 asks for a grader we did not author.

Coherence is exactly that, assessing the thing the architecture series claims to
be about. And there is a specific, plausible hypothesis it could test that
nothing currently can: an orchestrated arm may produce a correct answer whose
reasoning is less coherent
— more delegated fragments, more seams between
minds, more stitching. Correctness would tie while coherence separated. That is
precisely the kind of effect four prior ceiling-bound experiments could not see.

Proposed ladder — escalate, and stop early if the instrument is blind

Each rung has a stop rule. Rungs 0 and 1 cost no GPU at all — they score
artifacts that are already committed.

Rung 0 — does the instrument discriminate anything?

Score reasoning traces whose quality we already know, from committed data:

  • the muse-challenge golden's five fixture classes — a genuine challenge, a
    paraphrase, a paraphrase with challenge words bolted on, fluent contrarianism
    aimed at nothing, and agreement wearing challenge vocabulary (the trap an
    earlier grader of ours scored as counsel);
  • association-work's transcripts, where 5 of 6 failures end on a definite,
    wrong final answer
    — confidently wrong, not confused;
  • correct versus incorrect traces from the challenge problems.

Stop rule: if coherence scores do not separate these, stop here and record
it.
A signal that cannot tell a genuine challenge from agreement-in-challenge-
vocabulary will not tell a good architecture from a bad one, and every rung
above is unfalsifiable. This is #18's
lesson applied up front: vocabulary with no discriminating power certifies
nothing.

Rung 1 — does it discriminate on our series' artifacts?

Post-hoc, free: score the orchestrator-worker series' committed per-call records
and transcripts. Two questions:

  1. does coherence score correlate with correctness on the same cells?
  2. does it differ between flat and orchestrated arms, especially where
    correctness ties?

Question 2 is the interesting one, and a CEILING on correctness does not block
it — that is the point. Stop rule: no separation and no correlation → record
the negative and stop.

Rung 2 — build a gate, then sweep its ceiling

Only reached if rungs 0–1 show signal. Now the operator's proposal becomes
measurable: build a threshold at BOUNDARY_ACTION / BOUNDARY_COMPLETION and
sweep it, measuring a dose–response rather than a binary:

threshold expected
very permissive gate never fires → byte-identical to today (the vacuity control)
mid sweep fires sometimes → the interesting band
very strict fires constantly → refuses everything, cost without benefit

Report at each setting: firing rate, wall-clock cost, token cost,
and whether outcomes changed. A binary on/off design would very likely
ceiling exactly as t18 and t19 did; a sweep produces a curve, and a curve can be
wrong in ways a tie cannot.

Two controls this rung must carry:

  • the permissive end is the vacuity control — if the gate never fires there,
    and outcomes are identical, that proves the mechanism is wired and inert
    rather than absent;
  • a random-gate arm firing at the same rate on coherence-blind criteria.
    Without it, "the gate helped" is indistinguishable from "interrupting the loop
    sometimes helps", which is a genuinely different finding.

Rung 3 — adversarial: does the gate catch planted incoherence?

Inject traces built to be confidently wrong and internally inconsistent, and
measure whether the gate fires. A gate that passes everything is a gate nobody
knows works — the exact failure mode PROVOKERS allowed in
#18, where dead
vocabulary passed an exhaustiveness guard indefinitely.

Risks that must ride along

  • The echo chamber. If a coherence verdict feeds back into the loop, it
    arrives wearing measured-instrument authority — and
    memory-echo-chamber.md
    measured one planted record flipping the loop 6/6, with source labelling
    working correctly. "Coherence says your reasoning is incoherent" is a
    claim, not a fact, and must reach the mind as counsel that proposes, never
    as a verdict that decides.
  • Permission is not coherence (#2).
    Coherence asks whether an action makes sense; the capability layer decides
    whether it is permitted. A gate that starts refusing actions has quietly
    become an authority layer, and that is a different design needing a different
    review.
  • Latency. File I/O plus an embedding round-trip at up to three boundaries
    per turn, on a rig where a cortex turn already costs ~57 s. Rung 2 must report
    the cost even where it reports a benefit.
  • Never as a hidden variable. Whatever ships, the setting is recorded per
    cell — configurations.md exists because temperature was a hidden variable
    through an entire prior series.

Not scheduled, and deliberately not started mid-series

The architecture series is dialling right now against the very harnesses
this would touch. Changing an instrument between observations invalidates
everything measured before the change, so nothing here is wired until that
series completes and its results are written.

Filed so the gap is recorded rather than rediscovered, and so the ordering
argument — measure whether the signal discriminates before building a gate on
top of it
— survives the conversation it came from.

  • embodiment (Claude)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions