Skip to content

[Bug]: upstream-coverage matches only the bare slug literal per single file, so directory-path / split-across-files upstream citations read as unreferenced — a construction run logs hundreds of false SENSOR_FAILED #557

Description

@wowzoo

AI-DLC Version

v2 (v2 branch), 2.3.3. core/tools/aidlc-sensor-upstream-coverage.ts + the dispatcher core/tools/aidlc-sensor.ts (harness-neutral — lives in core/, so all harnesses).

AI-DLC Phase

Every phase whose stages declare consumes:; the volume concentrates in Construction's per-unit design stages (functional-design, nfr-requirements, nfr-design, infrastructure-design).


Summary

upstream-coverage checks that a stage output references each artifact in the stage's consumes: list. Its match is a case-insensitive regex anchored on the bare slug: \b<slug>\b OR [[<slug>]] (aidlc-sensor-upstream-coverage.ts, the per-slug loop). The dispatcher fires it per written file (aidlc-sensor.ts handleFire threads a single --output-path), and the per-slug loop tests that one file's body.

Two structural mismatches between what the sensor demands and how this framework's artifacts actually cite upstream make it report a legitimately-referenced upstream as unreferenced:

  1. Citation form. Artifacts cite upstream by directory path or backticked filename in their provenance header, not by bare slug. A representative nfr-design output header reads (paraphrased): "basis/trace: construction/api-service/nfr-requirements/, functional-design/, decisions.md …".

    That header covers performance-requirements, security-requirements, scalability-requirements, reliability-requirements (all produced under nfr-requirements/) by naming the producing-stage directory once. The slug performance-requirements never appears as a standalone token, so \bperformance-requirements\b misses it and the sensor reports it unreferenced. (Note: \b<slug>\b does already catch a backticked `<slug>.md` — the . is a word boundary — so filename citations of the exact artifact are fine; the gap is the directory-path form and item 2 below.)

  2. Per-file vs per-stage. A stage emits several artifacts and splits its upstream citations across them: in nfr-requirements, business-rules.md cites business-logic-model but tech-stack-decisions.md does not — and should not; a tech-stack decision doc has no reason to name the domain model. Because the sensor fires per file and demands the full consumes: set in each, every file that legitimately omits an upstream irrelevant to it fails. Coverage is a property of the stage's whole output, not of each file.

Neither is a bug in a specific artifact — they are how the framework's authoring conventions and the sensor's matching model diverge. The result is that the sensor's failures are dominated by false negatives.

Evidence (one real Construction run, 2.3.3)

A greenfield MVP run that reached Construction logged, from its audit shard:

  • 186 SENSOR_FAILED, of which 178 are upstream-coverage (the other 8: 7 linter, 1 required-sections). upstream-coverage fired-and-failed in 18 of the run's stages.
  • Re-scoring only the final deliverables (excluding *-questions.md and memory.md scaffolding) with the shipped regex: of the slugs the sensor marked unreferenced, ~97% are still unreferenced in the finished artifact — i.e. these are not mid-draft states that pass on the last save, they never pass.

I then re-scored the same deliverables against the two mismatches above (count a producing-stage directory path reference as coverage; evaluate coverage over the stage's sibling *.md union rather than the single fired file). The unreferenced slug count dropped ~81–89%, and the residue was dominated by genuinely-absent references (e.g. business-logic-model in nfr-design, which an NFR design legitimately may not cite) — i.e. the signal survives, the noise clears.

Why this is distinct from #529

#529 (debounce sensor firing) addresses the volume of fires — the same tuple re-firing on every save — and proposes firing document-shape sensors once at the stage boundary. That is real and complementary, but it assumes the terminal verdict is correct and characterises the failures as "half-written draft … only passes on the last one." The evidence above contradicts that assumption for this sensor: the final state still fails ~97% of the time. Firing once at the boundary would therefore still emit a large batch of false SENSOR_FAILED at each stage's gate — the noise moves to the gate, it does not clear. This report is about matching accuracy, not firing frequency; the two fixes are orthogonal.

Impact

SENSOR_FAILED is advisory (aidlc-sensor.ts header: "Sensor outcomes are advisory") so this does not block the workflow. The cost is:

  • Audit-ledger noise — hundreds of SENSOR_FAILED rows (+ a findings detail file each) that are mostly false, which buries the minority of real coverage gaps this sensor exists to surface.
  • Latent risk if [Feature]: Blocking sensor severity — make a sensor failure able to halt a gate #431 lands. If a future blocking severity lets a SENSOR_FAILED halt a gate, this sensor's current false-negative rate would block on citations that are in fact present. Accuracy should be fixed before (or with) making it able to gate.

Fix directions (two independent root causes)

These are separable — either helps, both compose:

  • A. Broaden the match to the citation forms the framework actually uses. In addition to \b<slug>\b / [[<slug>]], count a reference to the artifact's producing-stage directory (e.g. nfr-requirements/) as coverage. The dispatcher already resolves each consume's producer (producersOf in aidlc-sensor.ts presentConsumes), so the producing-stage slug can be threaded to the sensor alongside the artifact slug without new graph plumbing.
  • B. Evaluate coverage over the stage's output, not one file. Score the union of the stage directory's deliverables rather than the single fired file, so a citation in a sibling artifact counts. This can be done without changing firing granularity (the sensor reads sibling *.md at evaluation time), which keeps it independent of [Feature]: debounce sensor firing - the Write/Edit hook re-fires the same sensor + stage + path tuple on every save, so doc-heavy stages accumulate hundreds of advisory fires per run #529's firing-cadence change. Caveat worth calling out: the union should exclude scaffolding (memory.md, *-questions.md) — otherwise a slug that appears only in the Q&A file or the rule memory would falsely count as covered, trading a false negative for a false positive.

A design question underneath both: is a directory-path citation intended to satisfy coverage, or should artifacts be expected to name each consumed slug explicitly? That's a call for the sensor's owner — the two fixes assume the former (matching the current authoring convention), but if the latter is intended, the fix belongs in the stage authoring guidance / templates instead, and the sensor is behaving as designed (just louder than useful).

Environment

  • AI-DLC v2 2.3.3
  • Evidence from a single greenfield MVP run reaching Construction, on the Kiro IDE harness (IDE 1.0.116). The harness is incidental here: the sensor + dispatcher are harness-neutral core/ code, and the finding was reproduced by reading the run's audit shard and re-running the sensor over the on-disk artifacts, so the matching behaviour is identical on every harness and independent of the IDE version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions