Issue running PITest with RobolectricTestRunner on Android project (False Positives)

[Kvarnefalk]

Hey and thanks for building an awesome product.

I've had some issues running PITest with RobolectricTestRunner on my Android Project. PITest is telling me that there are mutants surviving. However, when I manually perform the mutations and run the regular test suite, the mutants are killed.

I've created a simple Android project reproducing the error. You can find that here Repro.

Any clues on what could be wrong here?

I've opened the same issue on the PITest project, you can find that issue here Issue on repo

[koral--]

Thanks for the report I'll investigate it ASAP.

[Kvarnefalk]

@koral--

Hey, I think I may have found a solution for this problem. The issue seems to be related to how PITest is inserting mutations using the Java Instrumentation Api and how Robolectric loads the classes.

PITest mutations. Here is the relevant code where PITest is inserting its mutations

Robolectric loading. Here is the code where Robolectric is loading the classes. Since Robolectric is sandboxing each test case, it loads the class code from disc everytime it creates a new sandbox. This leads to the mutated code not being picked up by the Robolectric test suite.

I did a workaround for this where I, from the HotSwapAgent also mutates the class on disc. This leads to Robolectric properly picking up the mutations.

Does this sound like a reasonable approach?

[koral--]

@Kvarnefalk Sorry for the delay. Could you please make a PR with workaround you mentioned?

[Kvarnefalk]

No worries. I am not really sure that my solution is good enough to merge here though. I still have some errors with false positives. Fewer than I had before but it seems like the tooling are still letting some through.

@hcoles mentioned an approach of instrumenting how Robolectric is loading the classcode. It would be nice to look into that but I haven't really had the time to be able to figure out how that could be done. It doesn't seem that straightforward. But if I were to merge something, I guess that is a more viable approach then the one I have now of just writing the mutations to disc.

[im-not-josh]

@Kvarnefalk Are you able to post your code for the workaround for this issue? Wanting to see if i can use it