Measure bracket survival on real PDF→markdown before committing the hard scope filter

[medelman17]

Measure bracket survival on real PDF→markdown before committing the hard scope filter

Version: eyecite-ts@0.28.1

Summary

The scope-before-recency design treats parenthetical scope as a hard candidate filter, which assumes the bracket structure of (quoting …) parentheticals survives extraction. If ( / ) are frequently dropped or garbled by PDF→markdown, abstains would be driven by parse failure rather than true ambiguity, and a hard filter could hide correct antecedents.

This is the single highest-priority unknown before committing the hard filter.

Measure

1. Bracket survival — on a real corpus sample, how often does an inner (quoting …) cite retain intact, balanced delimiters after extraction? Break down by source (native-text vs OCR'd PDF).
2. Error split — of observed Id. / supra misattributions, what fraction are scope errors (wrong candidate set) vs intra-scope salience errors (right set, wrong rank)? This gates whether a learned ranker is ever needed beyond a thin deterministic scorer.

Outcome

Decides (a) hard vs. degrade-to-soft-on-balance-failure for the scope filter, and (b) whether to invest in stage-3 ranking at all.

Related

• Design backbone: docs/research/2026-06-02-shortform-resolution-02-scope-and-binding.md, …-03-bracket-parsing-error-recovery.md.

[medelman17]

This was generated by AI during triage.

Agent Brief (ready-for-human)

Category: enhancement
Summary: Measure, on a real corpus, how often (quoting …) bracket structure survives PDF→markdown extraction — the empirical gate that decides whether parenthetical scope can be a hard candidate filter or must degrade to soft on balance failure.

Why this is ready-for-human (not delegable as-is):
It requires a real-world OCR'd / PDF→markdown corpus sample (data acquisition) and interpretation of the result into a design decision (hard vs. degrade-to-soft). Both are access/judgment calls. An agent can build and run the measurement harness, but cannot source a representative messy-PDF corpus or own the resulting policy choice.

What to measure:

1. Bracket survival — on a real corpus sample, how often does an inner (quoting …) cite retain intact, balanced delimiters after extraction? Break down by source: native-text vs OCR'd PDF.
2. Error split — of observed Id./supra misattributions, what fraction are scope errors (wrong candidate set) vs intra-scope salience errors (right set, wrong rank)?

Decision it gates:

• (a) Scope filter = hard (delete parenthetical-internal cites from the candidate set) vs degrade-to-soft-on-balance-failure (if survival is poor, warn instead of hard-delete) — i.e. how Replace the linear paren-depth counter with a bounded-depth bracket stack (+ island reconstruction) #809's substrate signal is consumed.
• (b) Whether a learned ranker (Build antecedent selection as a candidate-list scorer (deterministic now, ML-ready seam) #811) is ever worth it beyond a thin deterministic scorer (only if intra-scope salience errors are material).

Acceptance criteria:

• A reproducible measurement harness/script under scripts/, runnable against a provided corpus path.
• Reported bracket-survival rate, split native-text vs OCR/PDF.
• Reported scope-vs-salience error split for Id./supra.
• A written recommendation: hard vs degrade-to-soft, and a go/no-go on Build antecedent selection as a candidate-list scorer (deterministic now, ML-ready seam) #811.

Out of scope: implementing the filter (substrate is #809; the canonical bug fix shipped in #798) and the scorer (#811).

Design backbone: docs/research/2026-06-02-shortform-resolution-02-scope-and-binding.md, …-03-bracket-parsing-error-recovery.md (uncommitted — see #812). Roadmap: #812. This is the roadmap's highest-priority unknown — run it before committing #809's hard-filter policy.

[medelman17]

This was generated by AI during triage.

Measurement done (within available data) — decision needed

Harness + findings: #817 (scripts/measure-bracket-survival.ts, …-09-bracket-survival-measurement.md).

The raw OCR-survival rate remains unmeasurable in-repo (fixtures are native text: 9 (quoting…) parentheticals, 0 OCR) — a real OCR'd-PDF corpus is still the genuine data gap. But the decision no longer depends on that number, because #809's balanceOk detects 100% of dropped-bracket damage:

Outer (quoting Inner). Id.	Id. → outer	flagged balanceOk=false
balanced	100%	n/a
opening ( dropped	80% (recovered, #798)	100% (#809)

No silent-mis-scope tail: every damaged case is recovered or flagged.

Recommendation (your call to accept): degrade-to-soft on balance failure — keep the parenthetical-child exclusion hard where balanceOk=true, but where balanceOk=false lower confidence / warn (optionally abstain via #800's idConfidenceFloor) instead of hard-dropping. This is correct regardless of the unmeasured raw survival rate.

Follow-ups if accepted: (1) wire balanceOk into the abstain/confidence path (small, builds on #800/#809); (2) keep #811's learned ranker deferred (scope dominates ranking); (3) a real OCR corpus is still worth getting to quantify the soft-path hit-rate, but is no longer a blocker.