test-projects is silently ignored in solution mode, including non-existent paths

[ramedey]

Summary

When solution is set in stryker-config.json, test-projects has no effect and is never validated. A test-projects entry pointing at a file that does not exist produces no error, no warning, and no log line — Stryker silently discovers and runs every test project in the solution instead.

The operating-modes docs say solution mode analyses the whole solution, so the "runs everything" half may well be intended. The half that looks unambiguously wrong is that an invalid test-projects path is accepted in silence. That is what makes the first half undiscoverable: the config looks like it is scoping the run, and there is no signal anywhere that it is inert.

Closest existing report I could find is discussion #2815, which hits the same wall but does not cover the silent acceptance of a non-existent path.

Minimal repro

A solution with one source project and two test projects — UnitTests (2 tests) and OtherTests (3 tests), both referencing Calc:

Repro.slnx
src/Calc/Calc.csproj
tests/UnitTests/UnitTests.csproj    # 2 tests
tests/OtherTests/OtherTests.csproj  # 3 tests

Case A — scope to one test project:

{ "stryker-config": {
    "project": "Calc.csproj",
    "solution": "Repro.slnx",
    "test-projects": ["tests/UnitTests/UnitTests.csproj"]
} }

[INF] Number of tests found: 5 ...

Expected 2. Both test projects ran.

Case B — point test-projects at a path that does not exist:

{ "stryker-config": {
    "project": "Calc.csproj",
    "solution": "Repro.slnx",
    "test-projects": ["tests/DoesNotExist/Nope.csproj"]
} }

[INF] Number of tests found: 5 ...

No error. No warning. Nothing in the log mentions the missing project at all.

Case C — the documented workaround, run from the test project directory with no solution key:

[INF] Analyzing 1 test project(s).
[INF] Number of tests found: 2 ...

Correct. So the scoping itself works; it is specifically inert in solution mode.

I reproduced the same behaviour with both "test-runner": "vstest" and "mtp".

Actual vs expected

	actual	expected
valid test-projects + solution	silently ignored	honoured, or a warning that it is ignored in solution mode
non-existent test-projects path	silently ignored, exit 0	error, or at minimum a warning

Why this is worth fixing

On a real project this cost us roughly 20 minutes of CI wall-clock per run for weeks before anyone found the cause. test-projects named only the unit test suite, so the config read as correctly scoped. In fact the solution's integration test project was also being run — against every mutant — and each of those tests starts a PostgreSQL container via Testcontainers. That worked out to ~11s per mutant against a unit suite that runs in 209 ms, and it pushed the pipeline past its 30-minute job ceiling.

Nothing in the output suggested the config was being disregarded. The only reason we found it was noticing database containers churning during a mutation run. Switching to project mode dropped the step from 23m16s to about 30s.

Suggested fix

Either would have surfaced this immediately:

1. Validate test-projects regardless of mode — error on an entry that does not resolve to an existing project file. This seems worth doing independently of anything else.
2. Warn when test-projects is set but inert, e.g. test-projects is ignored in solution mode; all test projects in the solution will be run.

Honouring test-projects as a filter within solution mode would be the most useful outcome, but a diagnostic alone would have saved all of the above.

Environment

• Stryker.NET 4.16.0
• .NET SDK 10.0.400
• macOS (Darwin 25.5.0), also observed on ubuntu-24.04 GitHub-hosted runners
• xunit, both vstest and mtp test runners

[dupdob]

thanks for opening this issue. I am sure I clearly understand your problem here, so please forgive me if I misunderstood your points.
Here are a few general remarks/pointers that may help you:

1. Stryker can either run in single project mode or solution mode. Single project mode is usually achieved when using a test project folder as the current directory, but other options are available. Solution mode is only achieved when using the solution folder as the current directory
2. In Solution mode, Stryker discovers every projects and mutates everything it can. Scoping options such as test-projects or project has no effect on purpose (see below for explanation)

If you want to use solution mode, I suggest you add a specific solution level configuration that keeps only the projects you want to check with Stryker. Use then the --configuration option; then Stryker will only mutate enabled projects and used enabled test projects.

If you want fine grain control, you need to use Stryker in project mode and specify the configuration for each of your projects, disregarding solution mode which is not appropriate.

(*) scoping options are not interpreted in solution mode for these reasons:

• project: the purpose of solution mode is to test several projects. If you want to test a single project, you should use project mode. As a reminder, project option exists to disambiguate conflicts
• 'test-projects': it would be hard to apply this settings for multiple projects: should Stryker force-use this list for all projects? should it discard projects that are not tested against said test projects? should it raised warning on partial match?

In any case, we consider using a dedicated solution configuration more powerful and easier to use than going through these options.

That being said, you are right that Stryker silently accepts options it will not honor, and this is not good.

[dupdob]

@rouke-broersma , @richardwerkman : the problem is that Stryker silently accepts test-projects' and (probably) project` in solution mode, which may lead to surprising behavior for users.
I see two options and I would like your input on which one to pick:

1. raises a warning when ignoring settings
2. abort Stryker run when inconsistencies are detected in settings

I tend to favor 2, as warning may go unnoticed. The only reason I see to tolerate inconsistencies is when people want to perform experiment with options, but solution or project mode are significantly different and typically requires different settings.

WDYT?

[rouke-broersma]

Absolutely agree with 2 as a stopgap. Long term we need to design a better model for configuration. I think we have an existing issue discussing a potential design.

[richardwerkman]

In that previous issue we were thinking about loading config per project when running from solution. That way test-projects could still be configured, but just from config file in a project dir. But like Rouke said we need more design work on that.

For now I favor option 2 as well. If it doesn't work it shouldn't be possible.