handshake-level2-level3.md

Handshake: level 2 + 2.5 output -> level 3 input

What level 3 (file analysis, CONTEXT.md's "budget line item" stage) should read to turn one processed source file into classified, unit-normalized budget line items. Written against real output -- every file referenced here was generated by actually running the level-2 and level-2.5 code in this repo against all 10 files in budget_examples/, not hand-transcribed. See docs/examples/level2-processing/ and docs/examples/level2.5-scope-filter/ for the full set.

Status of both producer stages: level 2 (src/muni_budget_analysis/processing) is built and tested (prd.md, all 7 tasks complete). Level 2.5 (docs/adr/0003-llm-scope-classification-level-2-5.md) is POC-only -- scripts/run_scope_classify.py is a batch runner over real data, not a production module with its own tests. Treat scoped.json's shape as provisional until level 2.5 is formalized; normalized.json's shape is stable.

Files, per processed document

Under data/processed/{muni_id}/{filename_stem}/ at runtime (see docs/examples/level2-processing/{muni_id}/ for committed samples):

File	Producer	Required?
manifest.json	level 2	always
normalized.json	level 2	always (level 3's primary input)
document.md	level 2 (docling-routed files only)	absent for excel_native
docling_native.json	level 2 (docling-routed files only)	absent for excel_native; docling's raw export, level 3 should not need this if it consumes normalized.json
scoped.json	level 2.5	only if the level-2.5 batch has run over this document; may not exist, and even when it does, only covers a sample of the document's tables, not all of them (see below)

manifest.json

Provenance + outcome, not content. Fields: muni_id, source_filename, source_sha256, detected_type, pipeline_used (docling_pdf | docling_pdf_ocr | excel_native), docling_version, processed_at, status (success | partial | failed), page_count (null for Excel), warnings, error, outputs (dict of output-kind -> filename).

Read status before reading anything else. failed means no usable normalized.json exists at all -- don't try to open it. partial means normalized.json exists but has non-empty warnings; level 3 can still read it but should surface the warnings, not silently trust the data.

normalized.json -- the real contract

{
  "muni_id": int,
  "source_filename": string,
  "pipeline_used": "docling_pdf" | "docling_pdf_ocr" | "excel_native",
  "sections": [ {section} ],
  "tables": [ {table} ]
}

section:

{
  "section_id": string,        // e.g. "sec-3"
  "title": string,             // heading/sheet-name text, RTL Hebrew
  "page_range": [int, int] | null,   // null for Excel
  "table_ids": [string]        // table_ids belonging to this section
}

table:

{
  "table_id": string,          // e.g. "table-12", globally unique within the document
  "section_id": string | null, // join key back to `sections`
  "page_range": [int, int] | null,
  "sheet_name": string | null, // Excel only; null for PDF-sourced tables
  "num_rows": int,
  "num_cols": int,
  "rows": [ [cell] ]           // list of rows, each a list of cells in column order
}

cell:

{
  "text": string,      // as extracted -- RTL Hebrew word order is sometimes
                        // reversed by docling (e.g. "המים מפעל" instead of
                        // "מפעל המים"); level 3 must be robust to this, not
                        // assume clean text
  "row_span": int,
  "col_span": int,
  "is_header": bool    // NOT fully reliable on OCR'd tables -- row 0 of a
                        // table is effectively always a header regardless
                        // of this flag; see pdf_pipeline.py's own caveat
}

Notes for anything consuming this:

• Flat, uninterpreted. No code hierarchy, no fiscal-year/amount-type parsing, no category classification -- that's level 3's job (CONTEXT.md's "structural extraction" vs. "budget line item" split). A hierarchical code like 6111 is just cell text.
• Multi-page table continuations are already merged into one logical table by level 2 (pdf_pipeline.py's merge_multipage_tables) -- level 3 does not need to re-detect repeated header rows across pages.
• No unit/currency normalization. "19,640,900", "-", "105.29%" are all just cell text as extracted; level 3 (or a level-3-adjacent helper -- see pipeline/analysis/docling_rows.py's _detect_unit for prior art) must parse and unit-normalize.
• sheet_name/page_range are mutually exclusive in practice: PDF tables have page_range set and sheet_name: null; Excel tables have the reverse.
• elad_2022 (muni_id=903, pipeline_used="excel_native") is the only committed Excel example. Excel tables come from gap-based region segmentation (docs/adr/0002-*), not docling -- same output shape, but no docling_native.json/document.md exist alongside it. Don't assume those files are always present.

scoped.json -- level 2.5's output (provisional, partial coverage)

{
  "muni_id": int,
  "source_filename": string,
  "target_year": int,             // parsed from source_filename, e.g. "..._2026.pdf" -> 2026
  "tables_total": int,            // normalized.json's total table count
  "tables_classified": int,       // how many of those this file actually covers
  "table_scopes": [ {table_scope} ]
}

tables_classified is usually much smaller than tables_total. Classifying every table is one live LLM call each; tel_aviv_2026 alone has 353 tables. scripts/run_scope_classify.py caps classification at --max-tables-per-doc (default/used here: 3) per document as a cost/quota guard -- see docs/examples/level2.5-scope-filter/'s per-document scoped.json for exactly which table_ids got covered. A table_id in normalized.json with no matching entry in scoped.json's table_scopes was never classified -- absence is not a judgment, it's "not run yet." Level 3 must handle tables with no scope info at all (the common case today), not assume every table has one.

table_scope -- one of two shapes, keyed by year_axis:

Column-axis (the common case) or row-axis (e.g. multi-year forecast tables):

{
  "table_id": string,             // join key back to normalized.json's tables
  "section_title": string,
  "year_axis": "column" | "row",
  "columns": [ {"index": int, "header": string, "detected_year": int | null, "keep": bool} ],
    // present when year_axis == "column" -- one entry per table column, in order
  "rows": [ {"index": int, "row_label": string, "detected_year": int | null, "keep": bool} ],
    // present when year_axis == "row"
  "confidence": float             // 0-1
}

No-year-signal case (whole table unusable for fiscal-year-specific extraction):

{
  "table_id": string,
  "section_title": string,
  "year_axis": "none",
  "keep": false,
  "reason": string,
  "confidence": float
}

Real examples of both: docs/examples/level2.5-scope-filter/901/scoped.json (column-axis) and .../903/scoped.json (none-axis).

What keep: true means beyond the obvious target-year column/row: per ADR-0003's POC findings, the model also keeps columns/rows it judges "structural" -- not just the bare label/code column, but things like execution-rate percentages or year-over-year delta columns that aren't literally the target year. This is a live judgment call the model makes without explicit instruction either way (see docs/examples/level2.5-scope-filter/poc/FINDINGS.md, "Judgment calls worth level-3-team sign-off"). Level 3 should not treat keep: true as "this is target-year data" -- cross-check detected_year too: detected_year == target_year is target-year data; detected_year: null + keep: true is context the model chose to retain, which may span other years.

Known schema deviation (from the same POC findings, not yet fixed): for a row-axis table, the model has been observed to return both columns (every header, all keep: true, detected_year: null) and rows, despite the prompt saying columns is only present for year_axis == "column". None of the 10 documents processed for this handshake doc happened to hit year_axis == "row" live (see docs/examples/level2.5-scope-filter/poc/jerusalem_2026_debt_forecast_row_axis.response.json for a hand-fed example that did). Level 3 should key off year_axis and ignore whichever list doesn't match it, rather than assuming the non-matching key is absent.

How level 3 should join the two

1. Read manifest.json, bail if status == "failed".
2. Read normalized.json -- this is always present and always the structural source of truth.
3. If scoped.json exists, index its table_scopes by table_id. For each table in normalized.json:
   • No matching entry -> not yet classified, level 3 decides its own fallback (process the whole table, or skip until level 2.5 covers it -- not specified by this handshake, a level-3 design decision).
   • year_axis == "none" -> table has no usable fiscal-year signal, keep is always false for all of it.
   • year_axis == "column" / "row" -> use the per-column/per-row keep to select which slice of the table to interpret; every kept column/row still needs level 3's own code/fiscal-year/amount-type resolution (level 2.5 does not do that, per ADR-0003 -- it only says which slice is in-scope for the target year, not what any of the values mean).
4. If scoped.json does not exist at all, level 3 has no scope information and must decide its own default (the production-ready module src/muni_budget_analysis/analysis/ classifies all rows with values).

Gap: Resolved (July 2026)

The migration of the level-3 analyzer pipeline is fully complete and verified. The production Stage 3 processor (src/muni_budget_analysis/analysis/run.py and build_output.py) now natively reads and consumes normalized.json and joins it with scoped.json's keeps and targets (using heuristic fallbacks when no scope classification metadata is found), and outputs the standardized, pre-normalized line_items.json output contract. Raw native.json is no longer used by the pipeline.

Additionally, Stage 3 has been optimized to execute upfront table and row pre-filtering before invoking the LLM, and to run all LLM calls concurrently as async batches (via client.aio and asyncio semaphores), vastly accelerating execution and minimizing API token consumption.

muni_id caveat

Every muni_id in the examples referenced here (901-910) is a synthetic placeholder, not a real semel-yishuv code -- no level-1 scraper exists yet to assign real ones (tests/test_run_e2e.py already established this convention for 901-903; extended here for 904-910). Level 3 should not assume muni_id < 1000 means anything semantically; treat it as an opaque join key until level 1 exists.