Skip to content

Fix ctest reporting success when requested tests match nothing (#4915) - #5013

Open
Omotola wants to merge 1 commit into
mainfrom
dev/oakeredolu/4915-ctest-no-match-failure
Open

Fix ctest reporting success when requested tests match nothing (#4915)#5013
Omotola wants to merge 1 commit into
mainfrom
dev/oakeredolu/4915-ctest-no-match-failure

Conversation

@Omotola

@Omotola Omotola commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Fixes #4915

When specific tests are requested but none match a discovered test, ctest runs a filter that matches nothing and still exits 0 ("No tests were found!!!"). Callers such as the API/Copilot agent read that 0 as success, so an agent that passes a partial or non-existent test name gets told "all tests passed" when nothing ran.

This returns a failure (with a clear message) when tests exist but the requested ones weren't found. It's applied in both run paths:

  • runCTestHelper (Test Explorer integration, the default) — the zero-match case was already detected and logged, now it also returns a non-zero exit code.
  • runCTestDirectly (Test Explorer integration disabled) — added a guard before running ctest.

The message is also placed in the result's stderr so it can surface to the caller.

Per the issue discussion, this only fails when tests are available but the requested name isn't found — a project with no tests at all is not treated as an error.

Testing

  • Reproduced the original bug: agent passing ["ClassTest"] (real tests are ClassTest.Test1/2/3) reported success with nothing run.
  • Confirmed ctest -R <nonMatching> exits 0, while the extension now returns a failure for that case.
  • yarn compile clean; unit tests pass (only the pre-existing select-debugger fakebin failures remain, unrelated).

Before
image

After
image

When specific tests are requested but none match a discovered test, ctest
exits 0 ("No tests were found!!!"), which callers such as the API/Copilot
agent misread as success. Return a failure with a clear message when tests
exist but the requested ones weren't found, in both the Test Explorer
(runCTestHelper) and direct (runCTestDirectly) paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Copilot agent reports running 0 out of 0 tests as success when using the RunCtest_CMakeTools tool

1 participant