Skip to content

Measure the writers this category harms, and let the boundary move - #76

Merged
peopleworks merged 2 commits into
mainfrom
pelic-corpus
Sep 1, 2026
Merged

Measure the writers this category harms, and let the boundary move#76
peopleworks merged 2 commits into
mainfrom
pelic-corpus

Conversation

@peopleworks

Copy link
Copy Markdown
Owner

What changes

  • Corpus: 206 essays by adult learners of English join the calibration corpus (PELIC, Pittsburgh, 2006-2012, CC BY-NC-ND 4.0 - measured, never redistributed). fetch --source pelic reproduces the exact selection: first version, writing classes, 662+ words, one text per student by lowest id. Verified: the tool and an independent Python selection produce the same 206 ids and identical SHA-256s.
  • The published number: boundary 25 -> 30/100. At 25 the learners were flagged 9 of 206 (4.4%, interval 2.3-8.1%) against 0 of 90 published texts. At 30 it is 2 of 296 overall (0.2-2.4%). English now supports its own threshold (271 texts, bound 1.4%); Spanish still borrows. Floor 662 -> 649 words.
  • Report prose that stopped being true: the paragraph claiming no group sits above the rest is now computed at the recommended boundary and names the group, its count and its interval. The two caveats that said "measured on published articles rather than student work" now say what the corpus holds, in both languages with recomputed sourceHash.
  • Defect found by the new group: chat.eager-opener accepted Of course, with a comma (an ordinary concession) alongside Certainly!. Eleven learner essays fired it; zero published ones had, by register rather than merit. Both packs now require !; four tests added and shown to fail against the old pattern.
  • Docs carrying the numbers: README, SKILL (items 3 and 5), student sheets EN/ES, corpus README, code comments in VerdictBands, PublishedCalibration, EvidenceReport, RulePack, fixtures and tests. Blog articles and PARAPHRASE.md keep their figures: dated studies.
  • Version 0.6.0 in the six places. First change since 0.5.0 that touches Core, so this one carries a number. Merging obliges v0.6.0 and then desktop-v0.6.0, since the download page already points at it.

What deliberately does not change

The human-rate gates (humanRatePer1000) stay fitted on the 90 published texts. Refitting on the pooled corpus was tried: with learners as the majority, twelve English rules lose their gate entirely and the connector gates inflate two- to seven-fold - learner rates spent on everyone, the population trap #59 documents. A gate per group is issue #75, with the numbers.

Verification

  • 332 Core + 72 Documents + 7 Onnx + 31 Desktop tests green; boot tests unaffected.
  • run regenerates Docs/CALIBRATION.md and the embedded calibration at engine 0.6.0; SkillDocumentTests pins SKILL.md to it.
  • Mutation: reverting the regex fails exactly the four new tests.

Closes nothing; opens #75.

🤖 Generated with Claude Code

https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF

The calibration corpus gains 206 classroom essays by adult learners of
English (PELIC, University of Pittsburgh, 2006-2012): first submitted
version, writing classes, 662+ words, one text per student, selected by
a fixed rule so `fetch --source pelic` yields the same ids and hashes on
any machine. The affiliation proxy stands down; the population is here.

At the boundary the corpus supported before they joined, 25/100, the
tool flagged 9 of their 206 essays and none of the 90 published texts.
The boundary is now 30/100 (2 of 296, interval 0.2-2.4%), English has
its own threshold for the first time, and the report paragraph that
said no group sits above the rest now computes who is flagged at the
recommended boundary and says so.

The learners also caught a defect: chat.eager-opener, admitted on zero
hits in published prose, fired in eleven essays because its pattern
took "Of course," with a comma. It now requires the exclamation mark,
in both packs, with tests that the old pattern fails.

Not done, deliberately: refitting the human-rate gates on the pooled
corpus. With learners as the majority, twelve rules lose their gate and
the connector gates inflate up to seven-fold. That is issue #75.

Version 0.6.0: the first change since 0.5.0 that touches Core.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF
… the step below shown

Fable: the report quoted a language's best bound beside its recommended
threshold. The two coincided while nothing was flagged anywhere, so the
defect was invisible until English earned its own threshold with two
texts flagged at it: "at 30/100, under 1.4%" when the bound at 30 is
2.7%. PublishedLanguage now carries the bound at the threshold and the
report quotes that; a test asserts it and fails against the old code.
PELIC's creators were named nowhere: the generated page now carries a
Sources table with the citation, the manifest carries the DOI, and the
teaching documents that still described a corpus of ninety are current.
The page also shows what happened one step below the boundary, because
that step is why the boundary moved.

Copilot: `run` no longer succeeds with manifest texts missing; a
download is validated before it earns its name and a cached non-CSV is
discarded; a group with no row at the boundary is no longer reported as
flagged nothing; duplicate answer ids are refused. The CLI's inability
to write a Spanish report is #77.

The learners' 4.4% at 25 and the pooled 30 are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF
@peopleworks

peopleworks commented Sep 1, 2026

Copy link
Copy Markdown
Owner Author

Committee round 1 (Fable: method · Copilot: code · DeepSeek: prose, pending)

Both delivered request changes; every claim was verified before acting.

Fixed in cb42a5e:

  • Fable, high — the evidence report quoted the language's best bound (1.4%, at ≥35) beside the recommended threshold (30, where the bound is 2.7%). Latent since the report was written; activated the day a language first earned its own threshold with texts flagged at it. PublishedLanguage.RateHighAtThreshold added, report quotes it, mutation-tested.
  • Fable, high — CC BY attribution: PELIC's creators were named nowhere. Generated page now has a Sources table with the citation; manifest carries doi:10.5281/zenodo.3991977.
  • Fable, mediumDocs/Teaching/committee.{en,es}.md, Docs/Teaching/README.md, SKILL item 6 still described a corpus of ninety. Updated.
  • Fable, follow-up — the page now shows the step below the boundary (9 of 206 at 25), computed, not narrated.
  • Copilot, mediumrun returned 0 with manifest texts missing; now fails unless --allow-missing.
  • Copilot, medium — a 200 with HTML would be cached as the CSV for ever; validated before the move, cached junk discarded.
  • Copilot, mediumSpread() reported a group with no row at the boundary as flagged nothing; distinguished.
  • Copilot, low — duplicate answer_id refused rather than resolved by row order.

Not changed, with reasons:

@peopleworks
peopleworks merged commit ffd3e5f into main Sep 1, 2026
4 checks passed
@peopleworks

Copy link
Copy Markdown
Owner Author

Committee round 2 (DeepSeek: prose, honesty, Spanish)

Verdict request changes; applied in 6bf054d except one, with the reason.

Applied: every surface in this PR now says "before 2022" rather than "before generative models existed" — GPT-3 is from 2020, so the old phrase overclaimed on the most-read pages; the rest of the repository is #78. The Spanish student sheet no longer presents a bound with no supported threshold in the same shape as a rate at one ("medida sobre solo veinticinco textos y todavía sin un umbral respaldado"). "Tightens as it grows" promised a direction; README and the generated page now say the numbers follow the data either way. The step-below sentence is computed: it says "the only flags anywhere in the corpus at that boundary" only when that is true, and "why the boundary does not sit at 25" rather than "why it sits where it does". The report caveats name the encyclopedia revisions they had omitted, and the Spanish ones say "de esta versión", "a un umbral", "acotar con precisión", with agreement fixed. Plus "almost none", "studies report", the learner interval in SKILL item 3, and the two Spanish idioms.

Not applied as proposed: dropping the 5% from SKILL item 3. The 5% is not an unsupported number; it is the target the boundary is derived from (the lowest score whose interval stays under it), and the calibration page leads with the same words. What was wrong is that the skill quoted it with no provenance, so an assistant would repeat it bare. It now reads: "The boundary is the lowest score whose 95% interval stays under the project's 5% target … at 30/100 the published build flagged 2 of 296 … interval reaching 2.4%. Quote the interval; never the point estimate or the target alone."

Three reviewers, three lenses, every claim verified. Ready to merge once CI is green; release order in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant