Scale public hosted corpus with explicit license/OA gates

[rhowardstone]

Supervisor directive from Codex/user: the Render deployment should become the best possible public-facing atlas, not stay at the initial 10-paper baseline.

Goal:

• Expand the hosted public corpus toward the largest useful set we can safely expose, ideally hundreds of protocol-relevant organoid papers over time.
• Keep the public site/license posture clean: no full-text redistribution, short evidence snippets only, and explicit provenance for every claim.

Required gates before adding papers to the public export:

• Verify PMC OA/license status through the PMC OA service or an equivalent primary source, not only Europe PMC candidate metadata.
• Record license, PMCID, DOI, ingest_status, public_export_status, and reason when excluded.
• Reject or quarantine reviews, chips/microtissues-only papers, protocol-by-reference papers, disease-model papers without extractable protocol details, and adjacent non-organoid items unless explicitly marked as candidates.
• Preserve Tier-3 inherited-protocol caution: no authoritative inherited protocol ingestion without human review.
• Export only structured rows and short evidence snippets; do not commit or serve full methods text, paper bodies, or local/private generated artifacts.
• Add/update manifest tests so public counts, license filters, excluded counts, and snippet-only constraints are checked in CI.
• Update /llms.txt and the landing page counts from the manifest, preferably generated rather than hand-maintained once the public set grows.

Suggested staged plan:

1. Add public export status fields to the corpus/candidate manifest and tests.
2. Build an OA/license verification step with captured fixtures for external service responses.
3. Promote the safest CC-compatible protocol papers from data/corpus/incoming/organoid_corpus_candidates_180.csv into a verified public manifest.
4. Rebuild exports/public/* from that manifest and prove the hosted DB counts match.
5. Add browser/rendered QA for the larger public site, including /, /llms.txt, /atlas/protocols, /atlas/reagents, /heatmap, and /-/ask no-model behavior.

This should remain subordinate to issue #8 S-tier priorities when they conflict, but public scale-up is now an explicit product objective.

[rhowardstone]

Codex supervisor audit (2026-06-21):

Current public corpus state: 578 / 860 papers exported — already at the CC-eligible ceiling from the current extraction corpus.

corpus.tsv license breakdown:

• CC-BY: 578 → all 578 in public manifest ✓
• CC-BY-NC: 247 → correctly excluded
• author-manuscript: 17 → correctly excluded
• unknown: 16 → correctly excluded
• CC-BY-NC-ND: 2 → correctly excluded

To expand further, new CC-BY papers must be ingested into the extraction corpus (atlas.db), which requires @rhowardstone coordination per Issue #3 (Tier-1 ingestion policy). Candidates are in data/corpus/incoming/organoid_corpus_candidates_180.csv.

PR #41 merged: Added two new CI tests to test_public_deploy_contract.py that guard this invariant as the corpus grows:

• test_public_manifest_includes_all_corpus_cc_papers() — catches CC papers added to corpus.tsv but not exported
• test_public_manifest_excludes_non_cc_corpus_papers() — catches license policy violations

Steps 1 (export status fields) and 4 (manifest tests) from the Issue #14 staged plan are now covered. Steps 2-3 (OA verification + candidate promotion) remain pending @rhowardstone ingestion authorization.

[rhowardstone]

Status update:

• ✅ License gates implemented: is_public_license() in export_public.py, CC-BY/CC0 only
• ✅ Snippet cap: evidence_quote truncated at 500 chars in public export (PR fix(#33): cap evidence_quote at 500 chars in public reagents export #38)
• ✅ Corpus coverage CI tests: test_public_manifest_includes_all_corpus_cc_papers and test_public_manifest_excludes_non_cc_corpus_papers (PR test(#14): corpus license-coverage cross-validation tests #41)
• ✅ Semantic retrieval (R3) merged: dense-vector index can now find papers keyword search misses — substrate for corpus expansion past current saturation
• 🔄 PR feat(corpus): expand extraction corpus +43 CC-BY papers (903 total) #44 open: +43 CC-BY papers (903 total corpus) — held because CI correctly flags the export hasn't been regenerated yet

To unblock PR #44: run python pipeline/tier1_extract.py locally (uses your UConn ollama/Gemma3:12b), commit updated exports/public/, push to the PR branch. I can then merge.

Next after that: R3b hybrid scorer wired into discover_candidates to actively grow the corpus past keyword saturation — the actual payoff of the semantic index.