webuddhist-library-data-pipeline

Turns raw Buddhist root texts into backend-ready datasets — annotated markdown with citable block IDs and a nested table of contents, validated, and uploaded as a text + edition + table-of-contents triple.

Ported (lean) from OpenPecha/data-pipeline: only the skills the annotate → upload path actually invokes, plus an input/-folder workflow with a done-tracking ledger.

Two pipelines, split at exactly one file (annotated.md):

• Pipeline 1 — annotation. LLM-driven, run interactively in Claude Code / Cowork via /annotate. Raw text → cleaned → verse-segmented → block-ID'd → headed/TOC-anchored → frontmatter'd → annotated.md.
• Pipeline 2 — lint → parse → upload. Deterministic Python, no model calls, run via /upload or tools/run_upload.py directly. annotated.md → validated text_input payload → text.json/edition.json/toc.json → three POSTs to the backend.

Full walkthrough: docs/00-overview.md. Every skill/tool is catalogued in skills/SKILLS-CATALOG.md.

---

How you run it

1. Drop raw text files into input/ — one file per text (.txt or .md; the filename becomes the default text-id).

2. In Claude Code / Cowork, run:

   /annotate
   

   With no arguments it scans input/, checks ledger.json for what's already done, reports the queue, and processes the next pending file — stopping at the two human-review checkpoints (after verse IDs, and before final promotion).

3. Review when asked. Each text ends as texts/<text-id>/annotated.md, and its ledger entry flips to "annotated".

4. Upload (when the backend is real):

   /upload texts/<text-id>/
   

   Dry-run + --offline by default; a real --execute run needs your explicit confirmation in the conversation, and flips the ledger entry to "uploaded".

/annotate path/to/file.txt [text-id] [language] still works for a one-off file outside input/.

The ledger

ledger.json (repo root) is the pipeline's memory: one entry per input file, in_progress → annotated → uploaded, with dates. A no-arg /annotate never re-processes a file whose entry is already annotated/uploaded — delete or edit an entry only to deliberately redo a text.

---

Setup

pip install -r tools/requirements.txt   # PyYAML, rdflib, pyewts
cp config/settings.example.env .env     # then edit if/when the backend is real

Python 3.8+. No other runtime dependency — the linter/parser/uploader are otherwise stdlib-only.

All backend URLs are read from environment variables via tools/common/config.py — see config/settings.example.env (DATA_PIPELINE_API_BASE, BDRC_LOOKUP, BDRC_SEARCH_URL) and docs/08-upload.md for what each one does. DATA_PIPELINE_API_BASE defaults to an unreachable dummy host, so nothing uploads anywhere real until you deliberately point it at a backend — --offline is effectively required until then.

---

Repo layout

webuddhist-library-data-pipeline/
├── README.md                   this file
├── CLAUDE.md                   agent entry point
├── ledger.json                 done-tracking ledger (input file → status)
├── input/                      drop raw text files here (not committed)
├── config/settings.example.env DATA_PIPELINE_API_BASE + BDRC toggles
├── docs/                       stage-by-stage walkthrough + reference specs
├── skills/                     Pipeline 1 skill library + 2 orchestrators
├── texts/<text-id>/            per-text working dirs (not committed)
├── tools/                      Pipeline 2: linter, parser, uploader
└── tests/                      golden fixture + raw sample + stub server

Scope / non-goals

Root texts only — translations, commentaries (alignment payloads), and any CMS path are out of scope for now, matching the source repo's scope at the time of the port.