test runner silently falls back to the FULL suite when directory= fails the allowlist (and reports green 0-bundle runs for single-file values)

[bpamiri]

Summary

Both test-runner endpoints accept directory= and validate it against an allowlist regex (core: ^(wheels\.tests|vendor\.<pkg>\.tests)(\.\w+)*$ in vendor/wheels/tests/runner.cfm:122-135; the app runner has an equivalent ^tests(\.\w+)*$ shape). When the value fails the regex, the runner silently substitutes the default root and runs the entire suite. When the value passes but names a spec FILE, TestBox discovers 0 bundles and reports green.

Repro (Lucee 7 + SQLite core endpoint, develop)

• GET /wheels/core/tests?db=sqlite&format=json&directory=tests.specs.model → 314 bundles / 4420 passes (full core suite) with no indication anything was rejected — this exact example appears in testing/index.mdx, where the reader asked for "just the model specs" and got a multi-minute full run that looks like a green model run.
• Correct value directory=wheels.tests.specs.model → 48 bundles / 893 passes.
• directory=wheels.tests.specs.model.callbacksSpec (single file) → 0 bundles, 0 specs, green — the inverse trap. testBundles= is the working single-file filter (→ 43 passes, just that bundle).

The CLI already works around the first trap client-side ($normalizeTestFilter, whose comment notes the runners "silently fall back to the default scope, running the entire suite"), but anyone driving the URL directly — which the docs recommend for the inner loop — gets no signal.

Proposed direction

When url.directory is present but rejected by the allowlist, or accepted but discovers 0 bundles:

1. include directoryRequested / directoryResolved / bundlesDiscovered fields in the JSON payload, and
2. either respond 400, or at minimum emit a warnings array entry — so humans and CI scripts can detect the mismatch instead of trusting a green total from the wrong scope.

Defense-in-depth allowlisting stays unchanged; this only adds visibility. Docs-side example fixes (correct directory= shapes; testBundles= for single files) ride the guide-audit docs wave.

Acceptance

• A rejected directory= is detectable from the response (status or explicit fields/warnings) on both core and app runners.
• A 0-bundle discovery is distinguishable from a passing run.
• Specs cover both shapes.

Not a duplicate of #3025 (application-scope mutation) — this is request-scoped input handling.

Reported by the guide-behavioral-audit P1 batch 2 (work item p1-15-testing, claims IDX-07/RUN-06; live on the Lucee 7 docker harness).

[wheels-bot]

Wheels Bot — Triage

Classified as bug.

The report describes observable wrong behavior with a clean repro: a directory= value that fails the allowlist regex is silently swapped for the default root and the entire suite runs reporting green ("directory=tests.specs.model → 314 bundles / 4420 passes … no indication anything was rejected"), and an allowlist-passing single-file value discovers 0 bundles yet still reports green ("directory=wheels.tests.specs.model.callbacksSpec (single file) → 0 bundles, 0 specs, green"). Both are reproducible in isolation, so this triages as bug despite the "Proposed direction" framing.

Suspected layer

controller — see .ai/wheels/controllers/ for request/response patterns. The defect is request-scoped input handling in the test-runner HTTP endpoints (the issue says as much: "this is request-scoped input handling"). The canonical visibility reference for the inner loop also lives under .ai/wheels/testing/, and the CLI's $normalizeTestFilter is prior art for the client-side workaround.

Fix sketch

The allowlist resolution in vendor/wheels/tests/runner.cfm:119-135 accepts url.directory, falls back to wheels.tests.specs on regex failure, and never records that a fallback happened — and testBox.getBundles() (line 155) returning an empty array isn't surfaced. The fix adds visibility without changing the defense-in-depth allowlist: thread directoryRequested / directoryResolved / bundlesDiscovered into the JSON payload, detect (a) a present-but-rejected directory= and (b) a 0-bundle discovery, and either return 400 or push a warnings[] entry (the 400-vs-warnings choice is the one design decision here). Mirror the same change in the app-side runner (equivalent ^tests(\.\w+)*$ allowlist). Specs must cover both shapes per the acceptance criteria.

Confidence: medium

The report is unusually precise (exact lines, exact bundle counts, both failure shapes), but the fix spans two runner files, carries a small 400-vs-warnings design choice, and rides on the response path whose $header()/$content() calls already carry an Adobe CF + Undertow "response already committed" caveat (lines 144-149) — that cross-engine concern fires the auto-downgrade rule, capping confidence at medium.

<!-- wheels-bot:triage:3083 -->
<!-- wheels-bot:triage-class:bug -->
<!-- wheels-bot:triage-confidence:medium -->

[wheels-bot]

Wheels Bot — Fix proposed

Draft PR: #3106

Spec: vendor/wheels/tests/specs/dispatch/TestScopeVisibilitySpec.cfc — failing → passing
Implementation:

• vendor/wheels/tests/_assets/dispatch/TestDirectoryResolver.cfc (resolveScope / scopeWarnings / injectScopeMetadata)
• vendor/wheels/tests/runner.cfm (core endpoint wiring)
• vendor/wheels/tests/app-runner.cfm (built-in app-test runner wiring)

The runners now thread directoryRequested / directoryResolved / directoryRejected / bundlesDiscovered / warnings[] into the JSON payload, so a rejected directory= (silently swapped for the full default suite) and an allowlist-passing 0-bundle single-file value are both detectable from the response — verified live on the Lucee 7 + SQLite core endpoint (failing-first capture, clean pass, and the 0-bundle repro all confirmed). The defense-in-depth allowlist is unchanged; this only adds visibility.

A human review is required before merge. The Reviewer will weigh in shortly. Doc updates (MDX guides, .ai/wheels/, CLAUDE.md) are handled separately by bot-update-docs.yml.