Add bulletproof verification JSON system

- Extend fetch_cansim_enhanced.R with --simple mode for single-series
  indicators, supporting --filter and --name flags
- Delete save_verification_json.R (functionality merged into enhanced fetcher)
- Rewrite check_verification_coverage.R to read from article frontmatter
  instead of hardcoded mappings
- Add scripts/validate-verification.js for build-time validation
- Update npm build to run validation before building (fails if missing)
- Add npm validate command for checking without building
- Update SKILL.md and data-workflow.md with new workflow

Verification JSON requirement:
- Every article must declare verification_json in frontmatter
- Referenced JSON file must exist
- Build blocked until all 122 articles have valid verification

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: dshkol

.claude/skills/the-daily-generator/SKILL.md

@@ -76,121 +76,100 @@ Before finalizing ANY article, verify all checks pass.
 
 ```
 1. FETCH DATA
+   For complex tables (CPI, LFS, GDP, Retail):
    Rscript r-tools/fetch_cansim_enhanced.R <table-number> output
    → output/data_<table>_enhanced.json
 
-2. VERIFY JSON EXISTS (MANDATORY)
-   Check that verification JSON file exists in output/
-   If missing, create it using save_verification_json.R:
+   For simple single-series indicators:
+   Rscript r-tools/fetch_cansim_enhanced.R <table-number> output --simple \
+     --filter "GEO=Canada" \
+     --filter "Column Name=Filter Value" \
+     --name "indicator_name"
+   → output/indicator_name.json
 
-   Rscript -e "source('r-tools/save_verification_json.R'); ..."
-
-   See "Verification JSON Requirement" section below.
-
-3. READ AND CONFIRM DATA (MANDATORY)
+2. READ AND CONFIRM DATA (MANDATORY)
    Before writing ANY article content:
    - Read the JSON file completely
-   - State the headline value: "latest.yoy_pct_change is X.X%"
-   - State the reference period: "metadata.reference_period is YYYY-MM"
+   - State the headline value: "yoy_pct is X.X%"
+   - State the reference period: "ref_date is YYYY-MM"
    - If JSON doesn't exist or is stale, STOP - do not proceed
 
-4. CREATE ENGLISH ARTICLE
+3. CREATE ENGLISH ARTICLE WITH FRONTMATTER (MANDATORY)
    docs/en/<slug>/index.md
 
-5. CREATE FRENCH ARTICLE
-   docs/fr/<slug-fr>/index.md
-
-6. LINK ARTICLE TO JSON (MANDATORY)
-   Record which JSON file this article uses in the article's source-info div:
+   MUST include verification_json in frontmatter:
+   ---
+   title: Manufacturing sales down 1.0% in October 2025
+   verification_json: output/manufacturing_sales.json
+   toc: false
+   ---
 
-   **Verification JSON:** `output/<indicator>.json`
+4. CREATE FRENCH ARTICLE
+   docs/fr/<slug-fr>/index.md (same frontmatter requirement)
 
-   This creates the audit trail from article → data source.
-
-7. UPDATE LANGUAGE MAP
+5. UPDATE LANGUAGE MAP
    Add slug pair to src/lang-map.js
 
-8. UPDATE INDEX PAGES
+6. UPDATE INDEX PAGES
    Add entry to docs/en/index.md and docs/fr/index.md
 
-9. PREVIEW AND VERIFY
+7. PREVIEW AND VERIFY
    npm run dev → http://localhost:3000
 ```
 
 ## Verification JSON Requirement
 
-**Every article MUST have a corresponding JSON verification file.** This enables:
+**Every article MUST declare its verification JSON in frontmatter.** This enables:
+- Build-time validation (fails if JSON missing)
 - Audit trail for data provenance
-- Post-publication verification
 - Detection of fabricated data
 
-### For Tables with Enhanced Fetcher Support
+### Fetching Data
+
+**One tool for all tables** - use `fetch_cansim_enhanced.R`:
 
-Tables like CPI (18-10-0004), LFS (14-10-0287), Retail (20-10-0008) use the enhanced fetcher:
 ```bash
+# Complex tables with subseries/provincial breakdowns
 Rscript r-tools/fetch_cansim_enhanced.R 18-10-0004 output
-```
-This automatically creates `output/data_18_10_0004_enhanced.json`.
-
-### For Other Tables (Simpler Indicators)
 
-Use the verification JSON utility:
-```r
-source("r-tools/save_verification_json.R")
-
-# Fetch and save in one step
-fetch_and_save_verification(
-  series_name = "Manufacturing Sales",
-  table_number = "16-10-0047",
-  GEO == "Canada",
-  `Seasonal adjustment` == "Seasonally adjusted",
-  `Principal statistics` == "Sales of goods manufactured (shipments)",
-  `North American Industry Classification System (NAICS)` == "Manufacturing",
-  unit = "millions"
-)
+# Simple single-series indicators
+Rscript r-tools/fetch_cansim_enhanced.R 16-10-0047 output --simple \
+  --filter "GEO=Canada" \
+  --filter "Seasonal adjustment=Seasonally adjusted" \
+  --filter "Principal statistics=Sales of goods manufactured (shipments)" \
+  --filter "North American Industry Classification System (NAICS)=Manufacturing" \
+  --name "manufacturing_sales"
 ```
 
-Or manually:
-```r
-source("r-tools/save_verification_json.R")
+### Frontmatter Declaration
 
-# Your custom fetch
-data <- get_cansim("16-10-0047") %>%
-  filter(...) %>%
-  select(REF_DATE, VALUE) %>%
-  arrange(REF_DATE)
-
-# Save verification JSON
-save_verification_json(
-  series_name = "Manufacturing Sales",
-  table_number = "16-10-0047",
-  data = data,
-  unit = "millions",
-  article_slug = "manufacturing-sales-october-2025"
-)
-```
+Every article must include `verification_json` in its YAML frontmatter:
 
-### JSON File Naming Convention
+```yaml
+---
+title: Manufacturing sales down 1.0% in October 2025
+verification_json: output/manufacturing_sales.json
+toc: false
+---
+```
 
-| Series Name | JSON Filename |
-|-------------|---------------|
-| Manufacturing Sales | `manufacturing_sales.json` |
-| Consumer Price Index | `data_18_10_0004_enhanced.json` |
-| EI Claims | `ei_claims.json` |
-| Industrial Product Prices | `ippi.json` |
+This is **enforced at build time** - the site will not build if:
+- `verification_json` field is missing
+- The referenced JSON file doesn't exist
 
-### Verification Before Publishing
+### Verification Audit
 
-Before marking an article complete:
-1. Confirm JSON file exists in `output/`
-2. Confirm article values match JSON values
-3. Confirm article period ≤ JSON reference period
+Run the coverage checker to see which articles have valid verification:
+```bash
+Rscript r-tools/check_verification_coverage.R
+```
 
 ## Article Structure
 
 ```markdown
 ---
 title: Consumer prices up 2.2% year over year in November 2025
+verification_json: output/data_18_10_0004_enhanced.json
 toc: false
 ---
 
@@ -327,15 +306,14 @@ import * as Plot from "npm:@observablehq/plot";
 ## Quality Checklist
 
 Before publishing:
-- [ ] **Verification JSON exists** in `output/` for this indicator
+- [ ] **`verification_json` in frontmatter** pointing to valid JSON file
 - [ ] All values from fetched JSON (no made-up data)
 - [ ] Headline leads with key statistic
 - [ ] Charts render with #AF3C43 color
 - [ ] Language switcher works (slug in lang-map.js)
 - [ ] Voice is neutral (no "surged", "plummeted")
 - [ ] French uses comma decimals (2,2 %)
 - [ ] R code reproducibility section included with correct table number
-- [ ] **Verification JSON path** noted in source-info div
 
 ## Review Mode
 

.claude/skills/the-daily-generator/references/data-workflow.md

@@ -349,73 +349,64 @@ Never substitute synthetic data.
 - If `output/<indicator>.json` doesn't exist → audit trail gap
 
 **Prevention**:
-1. **EVERY article MUST have a verification JSON file**
-2. Use `save_verification_json.R` for indicators not covered by enhanced fetcher
-3. Record JSON path in article's source-info div
-4. Pre-commit hook should verify JSON exists for each article
+1. **EVERY article MUST declare `verification_json` in frontmatter**
+2. Use `fetch_cansim_enhanced.R --simple` for single-series indicators
+3. Build fails if verification JSON is missing or invalid
 
 ## Verification JSON System
 
-Every article must have a corresponding JSON file to enable post-publication verification.
+Every article must declare its verification JSON in frontmatter. Build-time validation enforces this.
 
-### Required JSON Files
+### Fetching Data
 
-| Indicator | JSON File | Source |
-|-----------|-----------|--------|
-| CPI | `data_18_10_0004_enhanced.json` | Enhanced fetcher |
-| LFS | `lfs_real.json` | Enhanced fetcher |
-| GDP | `gdp_real.json` | Enhanced fetcher |
-| Retail | `retail_real.json` | Enhanced fetcher |
-| Trade | `trade_real.json` | Enhanced fetcher |
-| Manufacturing | `manufacturing_sales.json` | save_verification_json.R |
-| Food Services | `food_services.json` | save_verification_json.R |
-| IPPI | `ippi.json` | save_verification_json.R |
-| RMPI | `rmpi.json` | save_verification_json.R |
-| Electricity | `electricity.json` | save_verification_json.R |
-| EI Claims | `ei_claims.json` | save_verification_json.R |
+**One tool for all tables:**
 
-### Using save_verification_json.R
+```bash
+# Complex tables (CPI, LFS, GDP, Retail)
+Rscript r-tools/fetch_cansim_enhanced.R 18-10-0004 output
 
-For indicators not covered by the enhanced fetcher:
+# Simple single-series indicators
+Rscript r-tools/fetch_cansim_enhanced.R 14-10-0005 output --simple \
+  --filter "GEO=Canada" \
+  --filter "Type of claim=Initial and renewal claims, seasonally adjusted" \
+  --filter "Claim detail=Received" \
+  --name "ei_claims"
+```
 
-```r
-source("r-tools/save_verification_json.R")
-
-# One-step fetch and save
-fetch_and_save_verification(
-  series_name = "EI Claims",
-  table_number = "14-10-0005",
-  GEO == "Canada",
-  `Type of claim` == "Initial and renewal claims, seasonally adjusted",
-  `Claim detail` == "Received",
-  unit = "claims"
-)
-# → Creates output/ei_claims.json
+### Article Frontmatter
+
+```yaml
+---
+title: EI claims down 1.1% in October 2025
+verification_json: output/ei_claims.json
+toc: false
+---
 ```
 
-### JSON Structure
+### JSON Structure (Simple Mode)
 
 ```json
 {
-  "series": "EI Claims",
+  "series": "Ei claims",
   "ref_date": "2025-10",
   "value": 267280,
   "mom_pct": -1.1,
   "yoy_pct": 2.1,
   "time_series": [...],
   "provenance": {
     "table_number": "14-10-0005",
-    "fetched_at": "2026-01-10 14:30:00 EST",
+    "fetched_at": "2026-01-10 14:30:00 PST",
     "statcan_url": "https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410000501",
-    "filters_applied": {...},
-    "article_slug": "ei-claims-october-2025"
+    "filters_applied": {"GEO": "Canada", ...},
+    "r_version": "4.5.0",
+    "cansim_package_version": "0.4.4"
   }
 }
 ```
 
 ### Verification Workflow
 
-1. **Before generating**: Confirm JSON exists or create it
-2. **During generation**: Pull all values from JSON
-3. **After generation**: Record JSON path in article
-4. **During audit**: Compare article claims to JSON values
+1. **Fetch**: Run `fetch_cansim_enhanced.R` (with --simple if needed)
+2. **Declare**: Add `verification_json` to article frontmatter
+3. **Build**: Site validates JSON exists (fails if missing)
+4. **Audit**: Run `Rscript r-tools/check_verification_coverage.R`

package.json

@@ -5,7 +5,8 @@
   "type": "module",
   "scripts": {
     "dev": "observable preview",
-    "build": "observable build && node scripts/fix-paths.js",
+    "build": "node scripts/validate-verification.js && observable build && node scripts/fix-paths.js",
+    "validate": "node scripts/validate-verification.js",
     "clean": "rm -rf dist .observablehq",
     "test": "node --test scripts/*.test.js"
   },