No Unit Test Coverage or Mutation Coverage reported for Robolectric based Unit Tests

[im-not-josh]

Hello,

I am seeing an issue where, the Unit Test coverage and Mutation test coverage reported for each unit test using RobolectricTestRunner is 0.
Pitest runs to completion, there is no error. If i remove RobolectricTestRunner (and disable tests that need it) coverage is reported as expected.

I am using the latest version of pitest and robolectric, 1.7.3 and 4.7.3.

Setting excludeMockableAndroidJar = true or excludeMockableAndroidJar = false doesnt seem to make a difference here either.

Any ideas?

[koral--]

Could you share a project with a minimal reproducible example demonstrating this issue?

[im-not-josh]

Hey @koral--
Finally coming back to this, sorry for the delay.
https://github.com/im-not-josh/PlayGround
Clone this repo, run pitestDebug gradle task and you should see this:

The class NotCoveredLogic has the RobolectricTestrunner, which, if you comment out, then returns 100% mutation coverage, the same as CorrectlyCoveredLogic class

[im-not-josh]

Interestingly, if i add
excludeMockableAndroidJar = true
to the task config, i then see this

[koral--]

Thanks, will check it.

[im-not-josh]

Hey there @koral-- checking in with you, did you get a chance to check this issue over?

[koral--]

Hi, sorry for the delay but not yet.

[koral--]

It seems that robolectric perform some actions that cause pitest to not work correctly.
There is one issue reported related to Robolectric: hcoles/pitest#744 (comment)

I've reported this issue to upstream: hcoles/pitest#1065

[im-not-josh]

Thanks for the effort to date, hoping this can get some attention upstream

[hcoles]

I've taken a look at this and the issue looks to be that Roboelectric now loads the classes from a classloader other than the context one (this is a change from the last time I had to prod pitest to get the two working together some years ago).

This is issue is similar to one I fixed recently for QuarkusTest, but roboelectric appears to keep a pool of classloaders somewhere, which complicates things for pitest considerably.

I have some horrible hacking locally that looks to fix the issue for the simple case demonstrated in https://github.com/im-not-josh/PlayGround.

I'm concernted it may not work with something more complex that uses roboelectric for real. I don't do Android dev, so have no idea what that looks like.

If someone could create a slightly more complex project that makes more realistic use of roboeletric I'll see if the fix looks viable.

If there's anyway this can be made self contained (ie not rely on an android sdk being already configured on a machine) that would be helpful. At the moment I can't see how I could add an integration test to the build for this.