Code coverage reporting using popular tools does not work with Aspire.Hosting.Testing

[hwoodiwiss]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

When using Aspire.Hosting.Testing to run integration tests, and using coverlet.msbuild we had no coverage reported for our tests, regardless of what code they exercised. This reduces the value of the tests that otherwise provide a great amount of value by exercising our system front-to-back.

Describe the solution you'd like

It would be useful if coverage tooling that already exists would just work ™, however I appreciate that given how different running an application in Aspire/DistributedApplicationFactory is from something like WebApplicationFactory, that may be unworkable.

I think whatever the solution, it will require some collaboration with the maintainers of coverage tooling.

Additional context

No response

[radical]

cc @ReubenBond

[ArcadeMode]

I've been going over the functionality that coverlet provides as-is and there seem to be opportunities for out of process coverage collection: dotnet-coverage in server mode.

Perhaps on the testing side of things Aspire could support a way to instead of build > run app > run tests to add some extra 'hooks' to build > instrument > run collector > run app > run tests > shutdown collector. It seems from the docs that we could just run these dotnet-coverage commands at the right time and get this working that way.

I am however unsure if that is a desirable implementation.

[TheRubble]

Did you ever get to the bottom of this, I've just hit the same issue.

[ArcadeMode]

@TheRubble I personally did not, but would love to see this resolved.

I'd guess that the intrumentation steps could still work as is, but the collector being the main problem due to the app running outside the aspire process (which is where the coverage collector is probably collecting from).

Will this issue get attention from the team sometime @radical / @ReubenBond? If a core contributor could point out a desired implementation that could be a starting point for contributions to be made.

I will also ask around in the coverlet repo to see if they have any suggestions

[deanro]

I'm hitting this as well. I wouldn't mind contributing to fix the issue if we had a defined plan of the best way to accomplish this.

[nuno-tiago-reis]

I'm also getting the same problem, the weird thing is that using the FineCodeCoverage extension and running the tests via VisualStudio, code coverage gets reported correctly. It's only when running it via the console that it shows the code as being uncovered.

[deanro]

Similarly I also need it to run from the command line. We have some some folks using Visual Studio on Windows, and some are using Rider on the Mac, we also have a GitHub workflow that builds all the code, runs all the tests and generates the code coverage report using reportgenerator, which is where this really needs to work. I tried assigning an issue to Copilot in our private repro and after about 7 attempts, it hopelessly failed to generate the code coverage for the services that are started via the apphost test - after using many of my credits 🙂

[ArcadeMode]

coverlet-coverage/coverlet#1760 (comment)

Sharing this reply in the coverlet repo. There's a suggested workaround with coverlet.console. Doesn't seem too hard to include in a simple script which runs dotnet test and sets up coverage collection.