`wheels test` ignores `--filter` and `--db` arguments

[zainforbjs]

Describe the bug
The wheels test command does not appear to respect the --filter or --db flags.
Even when specifying a filter directory or a database engine, the test runner:

• Executes all tests instead of filtered tests
• Continues using SQLite regardless of the provided --db value

To Reproduce
Steps to reproduce the behavior:

1. Create test suites in multiple directories:

• models/PostSpec.cfc
• controllers/PostsControllerSpec.cfc
• functional/ExampleSpec.cfc

2. Run:

wheels test --filter=models

3. Observe that all tests execute instead of only model tests
4. Run:

wheels test --filter=models --db=mysql

5. Observe that:

• all tests still execute
• runner still reports SQLite

Actual output

Running app tests (sqlite)...
3 passed (0.02s)

This occurs even when --db=mysql is explicitly provided.

Expected behavior

--filter

The following command:

wheels test --filter=models

should only execute tests matching the models directory.
Expected example:

Running app tests (sqlite)...
models.PostSpec
1 passed

--db

The following command:

wheels test --db=mysql

should:

• run tests against MySQL
• report MySQL in output
• Initialize the selected database engine appropriately
  Expected example:

Running app tests (mysql)...

Actual behavior

• --filter is ignored
• --db is ignored
• All tests execute regardless of filter
• SQLite is always used

Screenshots

Desktop

• OS: macOS

[bpamiri]

Research findings + decisions

Researched how Rails, Laravel, Symfony, Django, Phoenix, PHPUnit, and pytest handle the two parts of this issue, then mapped the result back to Wheels.

--filter — already works

The filter side was fixed by PR #2394 (commit 4c5233b on develop). $normalizeTestFilter maps short names like models to the dotted path tests.specs.models that the app runner's regex accepts. If you saw wheels test --filter=models ignored, you were probably on a snapshot built before that landed — current develop correctly scopes to tests/specs/models/.

The convention this matches:

Framework	Mechanism
Rails	Positional paths (bin/rails test test/models), -n /regex/, file.rb:27
Laravel	php artisan test --filter=Pattern, --testsuite=Unit
PHPUnit	phpunit tests/Unit, --filter, --group
pytest	Positional, pytest tests/foo.py::test_method, -k expression
Phoenix	Positional, :line suffix, --only tag

Wheels' --filter, --directory alias, and bare positional all work consistently with the ecosystem.

--db — design correction, no spec change

Surveyed how every comparable framework selects the test database:

Framework	Mechanism
Rails	DATABASE_URL=postgres://... bin/rails test env var
Laravel	phpunit.xml <env name="DB_CONNECTION" value="testing"/>
PHPUnit	<env> blocks in phpunit.xml
Symfony	APP_ENV=test, DATABASE_URL=... env
Django	DATABASES settings dict, DJANGO_SETTINGS_MODULE env
Phoenix / Ecto	MIX_ENV=test, sandbox; matrix runs via env vars

No framework exposes per-invocation CLI database switching for app tests. The reason is structural: the framework's transactional test isolation needs to know the connection at boot, and switching mid-run breaks that contract. DB selection is a process-level decision; filtering is per-invocation.

So wheels test --db=mysql for app tests isn't a feature this issue should add. What was a real bug, though, is that the CLI accepted the flag, attached it to the test URL, and printed Running app tests (mysql)... while the framework's vendor/wheels/tests/app-runner.cfm silently ignored url.db — so the user saw a misleading echo of their own flag. The CLI was lying about which engine it ran against.

Wheels' --core matrix is the legitimate place for --db: the framework's self-test wires up wheelstestdb_<engine> per supported engine. App tests don't have that need.

Decisions shipped in #2517

The fixes target the lying-CLI bug and document the convention, without adding the per-invocation switch:

1. --db warns when passed to an app run. Yellow warning block: "--db only applies to --core tests; ignoring for the app suite." Points at the testing guide for the env-var pattern instead. No silent drop.

2. The preamble shows the real datasource for app tests. New $resolveAppTestDataSource helper reads .env DATASOURCE_NAME and falls back to config/settings.cfm's set(dataSourceName=...), applying _test when --useTestDB is on. So users see Running app tests (myapp_test)... instead of the misleading (sqlite).

3. Docs explain the env-var pattern.

   • command-line-tools/wheels-commands/testing.mdx — flag table now annotates --db as --core only; new "Testing against different engines" section walks through the env-var pattern and names the Rails / Laravel / Symfony / Django / Phoenix parity.
   • testing/running-tests-locally.mdx — Aside calling out the same boundary near the example block.

The research that informed the decisions is on the issue thread above. Closing as completed by #2517.

---

Generated by Claude Code

[zainforbjs]

Thanks for the detailed investigation and ecosystem comparison.

That clears things up — especially the distinction between app-test datasource configuration vs framework/core test matrix execution.

The clarification around --db only applying to --core tests makes sense, and the updated warning/output behavior should help avoid confusion going forward.

However, after testing again on the latest snapshot, it appears the --filter behavior may still not be working correctly for app tests.
Example:

wheels test --filter=models

still executes all test suites instead of only model tests.

Current test structure:

models.PostSpec
controllers.PostsControllerSpec
functional.ExampleSpec

but the runner still reports:

Running app tests (testappdb_test_test)...
3 passed (0.02s)

I also tested:

wheels test --filter=model
wheels test --filter=controller

with the same result — all tests continue running.

Screenshots

Additionally, there appears to be a small datasource display issue where set(coreTestDataSourceName="testappdb_test"); is being displayed as testappdb_test_test, which looks like an extra _test suffix is being appended in the output.

Appreciate the fixes and documentation updates overall — just wanted to report these remaining behaviors after retesting on the newer snapshot.

[wheels-bot]

Wheels Bot — Triage

Classified as bug.

Suspected layer

cli — the Wheels CLI's test command. The relevant code lives in cli/lucli/Module.cfc; see also the in-repo CLI guide at cli/CLAUDE.md and the framework-side runner at vendor/wheels/tests/app-runner.cfm.

Fix sketch

Two sub-bugs in cli/lucli/Module.cfc:

1. --filter still ignored after the snapshot built post-fix(cli): wheels test --db is --core only; surface real datasource for app suite #2517. The reporter retested on a current build and short filters (models, model, controller) still run every spec. $normalizeTestFilter (lines ~491–511) maps models → tests.specs.models, and runTests appends &directory=… to the test URL (line ~3752), so the CLI side looks correct on paper. The fix needs to verify three things end-to-end: (a) that the normalized directory= value actually arrives at vendor/wheels/tests/app-runner.cfm, (b) that the runner's directory regex matches the dotted form CLI emits, and (c) that a freshly-reloaded server picks up the change (Lucee inspectTemplate=once can serve a stale runner across reloads). Likely fix surface: app-runner.cfm plus a regression spec exercising the full wheels test --filter=models flow.

2. testappdb_test_test datasource doubling. $resolveAppTestDataSource (lines 5057–5085) unconditionally appends _test when useTestDB=true, even if the resolved base already ends in _test (the reporter's set(coreTestDataSourceName=\"testappdb_test\") shows up as testappdb_test_test). Mechanical fix: skip the suffix when base already ends in _test, and also consider reading coreTestDataSourceName from config/settings.cfm since that is the setting the reporter actually configured.

Confidence: `medium`

The datasource-doubling half is high confidence (obvious off-by-one in string append), but the `--filter` half needs end-to-end investigation across the CLI → URL → `app-runner.cfm` boundary to identify which side is silently dropping the directory parameter — that pushes the overall classification down to medium.

<!-- wheels-bot:triage:2489 -->
<!-- wheels-bot:triage-class:bug -->