Skip to content

workflow.md

Latest commit

 

History

History
267 lines (196 loc) · 10.5 KB

workflow.md

File metadata and controls

267 lines (196 loc) · 10.5 KB
Raw

Current Workflow

This workflow builds source-backed draft article briefs for a Natick Town Meeting folder.

1. Build Or Refresh The Manifest

Fetch the official meeting page and create manifest.json:

python3 tools/ingest_natick_meeting.py \
  --meeting-id "SATM 2026" \
  --title "2026 Spring Annual Town Meeting" \
  --url "https://www.natickma.gov/2347/2026-Spring-Annual-Town-Meeting" \
  --meeting-dir "data/meetings/SATM 2026"

To regenerate from already archived HTML:

python3 tools/ingest_natick_meeting.py \
  --meeting-id "SATM 2026" \
  --title "2026 Spring Annual Town Meeting" \
  --url "https://www.natickma.gov/2347/2026-Spring-Annual-Town-Meeting" \
  --meeting-dir "data/meetings/SATM 2026" \
  --html-file "data/meetings/SATM 2026/sources/official-meeting-page.html"

TMT4 Footnote Sources

Refresh the official TMT4 footnote web-resource registry:

python3 tools/ingest_tmt4_footnotes.py

Outputs:

  • data/sources/tmt4-footnotes.json
  • data/sources/tmt4-footnotes.md

If an official source is not publicly available, add it to TODO.md instead of using an unofficial replacement.

Natick Historical Research Sources

Build the Natick-specific historical research set from the synced moderator / Finance Committee working files:

python3 tools/build_natick_history_research.py

Outputs:

  • data/towns/natick/history/README.md
  • data/towns/natick/history/<yyyy>-<xatm>.md
  • data/towns/natick/history/source-manifest.json

The source Google Drive folders are treated as read-only and source files are not copied into this repository. By default, the script builds a lightweight research index from folder names, file names, file types, and source categorization. Optional text extraction can be enabled for targeted deeper review with --text-limit, but reviewer confirmation against source files is still required before using these notes for floor preparation.

This history is Natick-specific. When the toolkit is adopted by another town, prompt that moderator to provide a comparable read-only body of past warrants, motions, presentations, minutes, debate materials, and action reports before generating prior-practice, tradition, or decorum examples.

For a deeper local extraction pass, run the monitorable background job:

/Users/jeffalderson/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 \
  tools/deep_extract_natick_history.py

This processes meeting folders in reverse chronological order and writes local derived output under data/towns/natick/history/deep-extraction/. That directory is gitignored because it can contain raw extracted text from synced working files. Monitor progress with:

python3 -m json.tool data/towns/natick/history/deep-extraction/status.json

Generate the public-safe synthesis layer from the completed local extraction:

/Users/jeffalderson/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 \
  tools/build_public_natick_synthesis.py

Outputs:

  • data/towns/natick/public-synthesis/README.md
  • data/towns/natick/public-synthesis/history-synthesis.json
  • data/towns/natick/public-synthesis/meetings/<yyyy>-<xatm>.md
  • data/towns/natick/public-synthesis/moderator-question-examples.md

Ask prototype moderator-preparation questions against the public-safe synthesis:

python3 tools/answer_natick_history_question.py \
  "What prior examples should I review for a zoning article with possible amendments?"

2. Archive High-Value Official Sources

Download and checksum the official warrant and Finance Committee recommendation book:

python3 tools/archive_manifest_sources.py \
  --manifest "data/meetings/SATM 2026/manifest.json" \
  --categories warrant finance_committee \
  --official-only

3. Extract PDF Text

Use the bundled Codex Python runtime because it includes pypdf:

/Users/jeffalderson/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 \
  tools/extract_archived_pdf_text.py \
  --manifest "data/meetings/SATM 2026/manifest.json" \
  --categories warrant finance_committee

4. Parse Warrant Articles

python3 tools/parse_warrant_articles.py \
  --meeting-id "SATM 2026" \
  --input "data/meetings/SATM 2026/working/extracted-text/2026-satm-warrant-approved-3-2-2026-docx-1-1.txt" \
  --output "data/meetings/SATM 2026/working/warrant-articles.json"

5. Build Article Source Index

python3 tools/build_article_source_index.py \
  --manifest "data/meetings/SATM 2026/manifest.json" \
  --output "data/meetings/SATM 2026/working/article-source-index.json"

6. Parse Finance Committee Recommendations

python3 tools/parse_fincom_recommendations.py \
  --meeting-id "SATM 2026" \
  --input "data/meetings/SATM 2026/working/extracted-text/fincom-recommendation-book-satm26.txt" \
  --output "data/meetings/SATM 2026/working/fincom-recommendations.json"

The parser extracts:

  • Recommendation.
  • Vote quantum.
  • Unanimity status.
  • Date voted.
  • Motion vote threshold.
  • Discussion text.

If the FinCom book header does not match the meeting ID, generated briefs will show a source warning and will not merge the mismatched recommendation records.

7. Parse Motion And Amendment Documents

Archive official motion and amendment PDFs:

python3 tools/archive_manifest_sources.py \
  --manifest "data/meetings/SATM 2026/manifest.json" \
  --categories motion_or_amendment \
  --official-only

Extract PDF text:

/Users/jeffalderson/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 \
  tools/extract_archived_pdf_text.py \
  --manifest "data/meetings/SATM 2026/manifest.json" \
  --categories motion_or_amendment

Parse motion records:

python3 tools/parse_motion_documents.py \
  --manifest "data/meetings/SATM 2026/manifest.json" \
  --output "data/meetings/SATM 2026/working/motions.json"

The parser classifies motion records as motion, procedural_motion, substitute_motion, amendment, memo, handout, or document. Archived consent agenda motions are kept as meeting source material, but the toolkit does not assemble or recommend consent agendas.

8. Parse Final Actions From Official Minutes

Archive official minutes/action PDFs:

python3 tools/archive_manifest_sources.py \
  --manifest "data/meetings/FATM 2025/manifest.json" \
  --categories minutes_or_actions \
  --official-only

Extract PDF text:

/Users/jeffalderson/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 \
  tools/extract_archived_pdf_text.py \
  --manifest "data/meetings/FATM 2025/manifest.json" \
  --categories minutes_or_actions

Parse official final actions:

python3 tools/parse_action_documents.py \
  --manifest "data/meetings/FATM 2025/manifest.json" \
  --output "data/meetings/FATM 2025/working/actions.json"

The parser currently extracts article outcomes from official session-minutes documents. Raw voting-system reports are archived and extracted, but are not treated as final article outcomes unless a later parser can reliably map the vote columns to article actions.

9. Generate Draft Article Briefs

python3 tools/generate_article_briefs.py \
  --warrant-articles "data/meetings/SATM 2026/working/warrant-articles.json" \
  --article-source-index "data/meetings/SATM 2026/working/article-source-index.json" \
  --fincom-recommendations "data/meetings/SATM 2026/working/fincom-recommendations.json" \
  --motions "data/meetings/SATM 2026/working/motions.json" \
  --actions "data/meetings/SATM 2026/working/actions.json" \
  --output-dir "data/meetings/SATM 2026/articles"

10. Generate Moderator HTML Reports

Generate a report for every meeting folder from already archived and parsed local artifacts:

python3 tools/generate_meeting_reports.py --all

Or generate one meeting report:

python3 tools/generate_meeting_reports.py "data/meetings/SATM 2026"

Each report is written to data/meetings/<MEETING>/report.html and includes:

  • AI-generated local-news style meeting preview based on parsed local artifacts.
  • Meeting overview and source-coverage counts.
  • Moderator preparation table by article.
  • Article-level links to briefs, Finance Committee recommendation status, motions, final action status, and source traceability.
  • Manifest source cards with local archived artifacts, extracted text, and original source URLs when available.
  • Open source, parser, and reviewer items.

Current Review Notes

  • SATM 2026 parsed 18 warrant articles.
  • SATM 2026 parsed 18 Finance Committee article recommendation sections.
  • SATM 2026 parsed 9 official motion/amendment documents: 4 substantive parsed records and 5 blank motion-form templates requiring review.
  • SATM 2026 archived and extracted 2 official voting-system reports, but no official session-minutes prose was available in the manifest for final-action parsing.
  • FATM 2025 parsed 33 warrant articles.
  • FATM 2025 parsed 33 Finance Committee article recommendation sections from the Google Drive alternate source. The original Natick page link downloaded a Spring Annual Town Meeting 2026 recommendation book and remains recorded in the manifest as source_mismatch.
  • FATM 2025 parsed 21 official motion/amendment documents: 14 substantive parsed records, 5 records needing OCR/manual text review, and 2 blank motion-form templates.
  • FATM 2025 parsed 60 official final-action outcome records from session minutes, covering all 33 articles.
  • SATM 2025 parsed 25 warrant articles.
  • SATM 2025 parsed 25 Finance Committee article recommendation sections.
  • SATM 2025 parsed 10 official motion/amendment documents: 6 substantive parsed records and 4 blank motion-form templates requiring review.
  • SATM 2025 parsed 59 official final-action outcome records, covering 8 articles.
  • SATM 2025 also parsed 25 accepted unofficial vote-result records from the Finance Committee Google workspace spreadsheet linked on the official meeting page. The source remains marked official: false and accepted_unofficial: true.
  • Draft article briefs are conservative. They now include parsed FinCom recommendation fields where the source passes the meeting-header check, parsed motion/amendment details where text is available, and parsed final actions where official session minutes are available.
  • FATM draft article briefs now include parsed final-action rows from official session minutes.
  • Draft for/against bullets are heuristic extraction aids and require reviewer confirmation.
  • Moderator HTML reports have been generated for SATM 2026, SATM 2025, and FATM 2025 from local artifacts only.