mutmut run bombs out during initial test run

[liquidpele]

Using "mutmut run" with 3.5.0 and config:

[mutmut]
paths_to_mutate=src/my_project
mutate_only_covered_lines=false
debug = true

does the following:

....
⠙ Running stats     python -m pytest  -vv -x -q --rootdir=.
....
platform linux -- Python 3.10.19, pytest-9.0.2, pluggy-1.6.0
cachedir: .pytest_cache
rootdir: <redacted>/mutants
configfile: setup.cfg
testpaths: test
......
<lots of passed tests until one fails>
E           AssertionError: xǁScanTriggerTypeǁis_rescan_trigger__mutmut_mutants must be either event or rescan, not both or neither. is_event=False, is_rescan=False
E           assert (False ^ False)

Normal "pytest test" runs just fine, the test does NOT have a bug in it and passes normally without mutmut.

This is triggering an area where in one of our tests, we assert that a value is one of a few options... but somehow mutmut is changing this value, seeing the test fail, and then just stopping? ScanTriggerType is an enum.

It seems like it's applying mutants to the initial run of pytest that's supposed to just verify the tests work, or it's failing to catch asserts on tests once it's validating mutants?

The biggest issue here though is that the output doesn't tell me anything. As a first time user, I have no idea why this is failing, what stage it's in, no tips for what might be wrong, no debug output about what mutmut is doing from step to step, etc...

[liquidpele]

So, playing with both mutmut and cosmic-ray, one major difference I'm seeing is that I can execute cosmic-ray against a package without it needing to have pytest in the same runtime, where mutmut seems to require pytest to be in it's runtime so it can run it natively... is there an option to have it "shell out" to pytest rather than trying to import it and using it like this?

The main reason is that having it required to be in the same runtime requires having to build the whole thing into our product dependencies, rather than something that's already there on the build system that can just be executed against the code after build.

[boxed]

Mutmut 2 did that. But that entire system was WAY slower. So there's a tradeoff here.

I don't understand the issue with installing mutmut and pytest in the environment. That's just a pip install?

[liquidpele]

Basically our corp build infrastructure makes it a pain to build it in everywhere. Imagine like 1,000 different builds I'd have to add it into. Not impossible, just a pain :) Yea, I figured it was an efficiency thing, if that's the decided arch that's fine I get it. Still not sure why it's bombing out on the mutated string with "__mutmut_mutants" added though.

[Otto-AA]

Hi @liquidpele , can you try it again with the current main branch? The PR #476 updated the enum support, so maybe that's fixed now. If not, some more steps to debug this would be necessary (though tbh I don't know when I have time for troubleshooting).

It seems like it's applying mutants to the initial run of pytest that's supposed to just verify the tests work, or it's failing to catch asserts on tests once it's validating mutants?

Mutmut instruments the code, such that we can switch between the original method an mutated methods via environment variables. In the initial pytest run we set the environment variable such that it runs the original methods. However, it seems that the code instrumentation is broken in your case somehow. Maybe because of the Enum.

[Otto-AA]

So, playing with both mutmut and cosmic-ray, one major difference I'm seeing is that I can execute cosmic-ray against a package without it needing to have pytest in the same runtime, where mutmut seems to require pytest to be in it's runtime so it can run it natively... is there an option to have it "shell out" to pytest rather than trying to import it and using it like this?

I guess it would be possible to implement, but as @boxed mentioned, the performance would be much worse. We import pytest and run the tests to get all modules loaded into memory, and then use fork() to quickly run (subsets of) tests for each mutant. Loading pytest + all your modules for each mutant would be a huge overhead.

[liquidpele]

Hi there, I can't pull latest to use until a new version is released but I can re-test then.

[boxed]

@liquidpele You can install from git head with pip

[liquidpele]

Sorry for not being clear, this is in a locked down corp environment build system, I have to request another team to pull a new version from pypi to get it.