feat(pipeline): harvest related legislation discovered during enrichment (#901)

* feat(pipeline): harvest related legislation discovered during enrichment

Regelingen reached only through the machine-readable model (source.regulation,
legal_basis, and open_terms/implements delegations like regeling_standaardpremie)
are never auto-harvested: the recursive harvester follows only <extref> BWB
hyperlinks in the source text, and those model links are a product of enrichment.

The enrichment agent now returns the related legislation it needs in a result
envelope (.enrichment-result.yaml) — deliberately kept OUT of the law YAML so the
law artifact stays schema-conformant. After a successful enrich the worker reads
the envelope, resolves each entry to a BWB id (agent bwb_id -> law_entries slug ->
single-hit SRU title search), and enqueues a follow-up harvest for each. Newly
harvested laws auto-enrich and return their own related legislation, so the
dependency graph fills itself by recursion.

- EnrichPayload.depth carries recursion depth (harvest -> enrich -> harvest);
  each related harvest is depth+1 at priority 40-(depth+1), so deeper nesting
  yields to shallower/interactive harvests.
- Opt-in: HARVEST_RELATED_LEGISLATION (off by default) + RELATED_HARVEST_MAX_DEPTH
  (default 2); reuses ENRICH_DAILY_LIMIT for spend and create_harvest_job_if_not_exists
  for dedup. Best-effort: nothing here can fail the already-committed enrichment.
- search_bwb_by_name extracted from the axum handler for reuse; find_bwb_id_by_slug
  made pub; slug hits re-validated as BWB (CVDR skipped). RFC-025 documents the
  pattern and its known limitations.

* refactor(pipeline): make related-legislation harvest always-on

Drop the HARVEST_RELATED_LEGISLATION opt-in flag. The follow-up harvest only
enqueues harvest jobs (no LLM cost); the expensive re-enrichment of those laws
stays gated by ENRICH_AUTO_ENQUEUE + ENRICH_DAILY_LIMIT, and the recursion is
bounded by RELATED_HARVEST_MAX_DEPTH. So there is nothing to protect behind a
separate flag.

* fix(dev): exclude enrichment sidecars from law YAML validation

The enrichment result envelope (.enrichment-result.yaml) and the existing
.enrichment.yaml metadata sidecar are written into a law directory but are not
law files. `find -name '*.yaml'` matches leading-dot names and the pre-commit
`files:` regex matches them too, so validating one fails (missing $id). Skip
dot-prefixed sidecars in both script/validate.sh and the validate-law-yaml hook.

* fix(pipeline): tighten related-legislation resolution; drop RFC-025

Address CI review findings:
- A CVDR slug hit no longer falls through to the SRU name search (the slug
  already identified the law; a title match could resolve a *different* national
  law). It now returns Unresolved.
- The harvest summary log separates already_queued and exhausted skips instead of
  conflating them in the resolved-but-not-enqueued gap.

Also drop RFC-025: the related-legislation harvest loop is an implementation
detail, not a cross-cutting design decision that warrants an RFC.

* fix(pipeline): address review nits on related-legislation resolution

- Validate the single-hit SRU result as a BWB id before resolving, so a
  malformed SRU id can't slip into a harvest payload (paths a/b already validate).
- Read the .enrichment-result.yaml sidecar via tokio::fs (was blocking std::fs)
  for consistency with the rest of execute_enrich_with_runner.
- Clarify the depth-inherit comment: the field is the shared extref-recursion
  counter, so deep-via-extref laws skip related discovery (roots/shallow laws,
  the intended case, are unaffected).

Author: tdjager

.claude/skills/law-generate/SKILL.md

@@ -763,6 +763,44 @@ The JSON payload format (written to the temp file):
     laws (>20 articles), this limit applies per batch — each batch of ~15 articles
     gets its own 3-iteration budget
 
+## Phase 4.5: Write the related-legislation result envelope
+
+After the `machine_readable` sections are final, write a **sibling result
+envelope** so the pipeline can auto-harvest the legislation this law depends on
+(delegated regelingen and cross-law references the extref-only harvester misses).
+
+Write it next to the law YAML as `.enrichment-result.yaml` (same directory,
+e.g. `corpus/regulation/nl/wet/wet_op_de_zorgtoeslag/.enrichment-result.yaml`).
+Use the `Write` tool — no new agent tools are needed.
+
+```yaml
+# .enrichment-result.yaml — result envelope, NOT part of the law schema
+law_id: wet_op_de_zorgtoeslag
+related_legislation:
+  - name: Regeling vaststelling standaardpremie en bestuursrechtelijke premie
+    relation: delegated_regeling      # source_regulation | legal_basis | delegated_regeling
+    bwb_id: BWBR0037841               # optional, best-effort
+    slug: regeling_standaardpremie    # optional, best-effort
+    open_term: standaardpremie        # optional, only for delegations
+  - name: Algemene wet inkomensafhankelijke regelingen
+    relation: source_regulation
+```
+
+Coverage: add one entry for **every** `source.regulation` you bound, every
+`legal_basis` you anchored on, and every `open_term` delegation you declared.
+Fields:
+
+- `name` — **required**; the human-readable law/regeling title (used for search
+  fallback when no id/slug is given).
+- `relation` — one of `source_regulation`, `legal_basis`, `delegated_regeling`.
+- `bwb_id`, `slug`, `open_term` — **optional**, best-effort. Supply what you know
+  (a known `bwb_id` resolves fastest); leave the rest out.
+
+**CRITICAL — this MUST NOT go in the law YAML.** The law file stays strictly
+schema-conformant (`just validate` must still pass). The related-legislation list
+lives only in the `.enrichment-result.yaml` sidecar, which the pipeline reads
+separately. Do not add a `related_legislation:` key anywhere inside the law YAML.
+
 ## Phase 5: Report
 
 Report to the user:

.pre-commit-config.yaml

@@ -59,6 +59,9 @@ repos:
         language: system
         pass_filenames: true
         files: ^corpus/regulation/.*\.yaml$
+        # Skip dot-prefixed sidecars (.enrichment.yaml, .enrichment-result.yaml):
+        # enrichment metadata/result envelopes, not law files.
+        exclude: (^|/)\.[^/]*\.yaml$
         types: [yaml]
 
       - id: skills-no-casus

packages/admin/src/handlers.rs

@@ -644,6 +644,8 @@ pub async fn create_enrich_jobs(
             law_id: law_id.clone(),
             yaml_path: yaml_path.clone(),
             provider: Some((*provider_name).to_string()),
+            // Admin-requested enrichments are roots of the related-harvest chain.
+            depth: None,
         };
 
         let payload_json = serde_json::to_value(&enrich_payload).map_err(|e| {

packages/pipeline/src/api/bwb_search.rs

@@ -29,9 +29,24 @@ pub async fn search_bwb(
     State(state): State<ApiState>,
     Query(params): Query<SearchParams>,
 ) -> Result<Json<Vec<BwbSearchResult>>, (StatusCode, String)> {
-    let q = params.q.trim();
-    if q.is_empty() || q.len() < 3 {
-        return Ok(Json(vec![]));
+    match search_bwb_by_name(&state.http_client, params.q.trim()).await {
+        Ok(results) => Ok(Json(results)),
+        Err(e) => Err((StatusCode::BAD_GATEWAY, e)),
+    }
+}
+
+/// Search wetten.overheid.nl via the SRU API for laws matching `q`.
+///
+/// The client-taking core shared by the axum handler and the enrich worker's
+/// related-legislation resolution. Queries shorter than 3 characters (after the
+/// same sanitize as the handler) return an empty list rather than an error.
+pub async fn search_bwb_by_name(
+    client: &reqwest::Client,
+    q: &str,
+) -> Result<Vec<BwbSearchResult>, String> {
+    let q = q.trim();
+    if q.len() < 3 {
+        return Ok(vec![]);
     }
 
     let sanitized: String = q
@@ -50,35 +65,20 @@ pub async fn search_bwb(
             ("maximumRecords", &MAX_RESULTS.to_string()),
         ],
     )
-    .map_err(|e| {
-        (
-            StatusCode::INTERNAL_SERVER_ERROR,
-            format!("URL build error: {e}"),
-        )
-    })?;
-
-    let response = state
-        .http_client
+    .map_err(|e| format!("URL build error: {e}"))?;
+
+    let response = client
         .get(url)
         .send()
         .await
-        .map_err(|e| (StatusCode::BAD_GATEWAY, format!("BWB search failed: {e}")))?;
-
-    let xml_text = response.text().await.map_err(|e| {
-        (
-            StatusCode::BAD_GATEWAY,
-            format!("BWB response read failed: {e}"),
-        )
-    })?;
-
-    let results = parse_sru_response(&xml_text).map_err(|e| {
-        (
-            StatusCode::INTERNAL_SERVER_ERROR,
-            format!("XML parse error: {e}"),
-        )
-    })?;
-
-    Ok(Json(results))
+        .map_err(|e| format!("BWB search failed: {e}"))?;
+
+    let xml_text = response
+        .text()
+        .await
+        .map_err(|e| format!("BWB response read failed: {e}"))?;
+
+    parse_sru_response(&xml_text).map_err(|e| format!("XML parse error: {e}"))
 }
 
 /// Parse SRU XML response and extract unique laws (deduplicated by BWBR ID).

packages/pipeline/src/api/harvest.rs

@@ -155,7 +155,7 @@ async fn resolve_identifiers(
 }
 
 /// Find a law's BWB ID by its slug in the law_entries table.
-async fn find_bwb_id_by_slug(
+pub async fn find_bwb_id_by_slug(
     pool: &sqlx::PgPool,
     slug: &str,
 ) -> Result<Option<String>, sqlx::Error> {

packages/pipeline/src/config.rs

@@ -97,6 +97,11 @@ pub struct WorkerConfig {
     /// Off by default; enrichment is otherwise requested explicitly via the admin
     /// API. Configurable via `ENRICH_AUTO_ENQUEUE`.
     pub auto_enrich_enqueue: bool,
+    /// Maximum recursion depth for related-legislation follow-up harvests.
+    /// A depth-0 enrichment may enqueue harvests at depth 1, whose enrichments
+    /// may enqueue at depth 2, etc., up to this cap. Default: 2. Configurable
+    /// via `RELATED_HARVEST_MAX_DEPTH`.
+    pub related_harvest_max_depth: u32,
 }
 
 impl std::fmt::Debug for WorkerConfig {
@@ -118,6 +123,7 @@ impl std::fmt::Debug for WorkerConfig {
             )
             .field("enrich_daily_limit", &self.enrich_daily_limit)
             .field("auto_enrich_enqueue", &self.auto_enrich_enqueue)
+            .field("related_harvest_max_depth", &self.related_harvest_max_depth)
             .finish()
     }
 }
@@ -193,6 +199,11 @@ impl WorkerConfig {
             })
             .unwrap_or(false);
 
+        let related_harvest_max_depth: u32 = std::env::var("RELATED_HARVEST_MAX_DEPTH")
+            .ok()
+            .and_then(|v| v.parse().ok())
+            .unwrap_or(2);
+
         Ok(Self {
             database_url,
             max_connections,
@@ -207,6 +218,7 @@ impl WorkerConfig {
             max_consecutive_resource_failures,
             enrich_daily_limit,
             auto_enrich_enqueue,
+            related_harvest_max_depth,
         })
     }