Incremental Testing based on `git-diff`

[ASuciuX]

Hello @smoelius, @moodmosaic told me about this and found it quite intriguing and a great complement to the concept of mutants!

Currently, I'm focusing on incorporating mutation testing into Stacks. This led me to ponder how we could integrate this into our entire workflow. Similar to our approach with mutants, I see two primary applications for this in a large-scale project:

1. Ensuring that all new tests are thoroughly vetted before their integration into the production environment.
2. Identifying and improving existing tests that aren't up to the mark.

Given the extensive size of the stacks repository, we've created a specific process that runs with each pull request (PR) to indicate if the functions are inadequately tested, utilizing cargo-mutants. It's noteworthy that cargo-mutants already includes certain custom features for incremental runs, such as testing only those mutants associated with functions modified in a git-diff.

I'm curious if a similar approach could be applied to necessist, performing the operations only on modified tests based on a git-diff.

I'd greatly appreciate your thoughts on this and any feedback you might have. Also, thanks for your contributions to the field!

[ASuciuX]

The requirement can be divided into two distinct tasks:

1. Execute a specific subset of tests from a test file, identified by their unique test names.

necessist stackslib/src/chainstate/stacks/boot/pox_4_tests.rs \ 
-- --package stackslib chainstate::stacks::boot::pox_4_tests \
[list of test names]

2. Extract the list of test names from the changes highlighted in the git-diff file.

Are there any particular details or considerations related to these tasks?

[smoelius]

Hi, @ASuciuX.

I'm curious if a similar approach could be applied to necessist, performing the operations only on modified tests based on a git-diff.

Necessist doesn't currently have this functionality. But I agree it could be worthwhile to have something like this built in.

It sounds like you are considering scripting something together to do this. If you do, I'd be curious to hear about your experience.

Just to state the obvious: a change to a function called by a test could affect Necessist's results. The described approach wouldn't catch such changes. But I don't think that detracts from the idea's merits.

---

Sorry for being slow to reply. Your question got me thinking about how best to incorporate Necessist into a CI workflow, generally. I don't currently have a good answer to that. But that fact should not have delayed my response. Apologies.