Diff based cargo-mutants #910

benalleng · 2025-07-30T15:19:32Z

This is a PR to test adding diff based mutations for future changes in the source code moving forward.

Note: this does not add mutations if the mutants.toml or other non .rs code is changed

The last 2 commits are simply testing commits to show diff mutations are caught or missed or the diff is ignored alltogether.

the ohttp commit shows that while an .rs file was changed since it is in the test cfg it is ultimately ignored
my mock commit creates some code with intentionally missed mutants

coveralls · 2025-07-30T15:23:13Z

Pull Request Test Coverage Report for Build 16662599796

Details

15 of 15 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 86.023%

Totals
Change from base Build 16627870999:	0.02%
Covered Lines:	8007
Relevant Lines:	9308

💛 - Coveralls

benalleng · 2025-07-30T16:10:51Z

I allowed this to create comments but not sure if we want that long term as we can just look into the failing action check.
benalleng#5 (comment) I removed it already

benalleng · 2025-07-31T15:11:49Z

@DanGould certainly this can vary the CI test time based on how complex the diff is but it does seem on par with our integration tests for most changes

spacebear21

Concept ACK - 3min 30s for an isolated change sounds reasonable. I guess the worst-case would be around 40min (based on the weekly job time) if every possible mutant was touched?

spacebear21 · 2025-07-31T16:04:01Z

.github/workflows/cargo-mutants.yml

+      - uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: beta


Let's use the stable toolchain to keep cache usage efficient:

Suggested change

- uses: dtolnay/rust-toolchain@master

with:

toolchain: beta

- uses: dtolnay/rust-toolchain@stable

Did some additional cleanup to better match this workflow with the rest of our actions

How does this keep it efficient? Is it because we have a cache from using the stable toolchain already elsewhere?

~~Ok so It is intended to use the cache from the stable toolchain but looking closer at it I am a bit confused as to whether we are using it correctly, making a more pointed issue in #912~~

The cache here is specific to the Diff Mutants job so we should be able to save some time for the stable toolchain with subsequent runs of this job, stable is chosen so the hash doesn't change often

See my reply here: #912 (comment)

spacebear21 · 2025-07-31T16:04:29Z

.github/workflows/cargo-mutants.yml

+          save-if: false
+
+      - name: Run cargo-mutants on diff
+        run: >


Could we update the output of this command to only print missed mutants?

spacebear21 · 2025-07-31T16:10:28Z

.github/workflows/cargo-mutants.yml

I'd prefer keeping the workflows separated by function rather than by tool, i.e. leave this file as-is (or rename to just cron-weekly.yml) and put the diff mutants job in rust.yml.

Ok, I like moving the diff mutant into the rust.yml it feels more correct as we use that for PR CI

This workflow is a planned replacement for the current weekly mutants that run every sunday on the rust-payjoin codebase. This will hopefully allow more continuous coverage without a weekly delay.

I added one additional test for mutation coverage on ohttp logic with my plan to create more tests for full test coverage to follow.

spacebear21 · 2025-07-31T19:44:36Z

Looks good to me, pending removal of the DO NOT MERGE commit.

DanGould · 2025-08-01T15:14:33Z

I was confused at first that this might check the diff in mutants.out since we don't commit or cache a mutants.out for each commit on master after some reading, I realize that cargo mutants --in-diff actually checks for mutants in that actual diff and that this will work. neat! Seems very unlikely to change everything all at once except perhaps in some file moves or following formatting changes.

spacebear21

ACK aeba995

There was a rebasing mistake on a previous commit as I created some exclusions but their name was changed before payjoin#910 was merged and they were not caught.

There was a rebasing mistake on a previous commit as I created some exclusions but their name was changed before payjoin#910 was merged and they were not caught. There is one exclusion that is simply renamed but another I decided to change the logic such that the exclusions are no longer possible.

Closes #917 There were some stale exclusions before #910 was merged, hopefully that PR minimizes as many PRs like this one as possible. I simply renamed the exclusion for one but decided to try and change the logic to try and prevent a mutant in another

benalleng marked this pull request as draft July 30, 2025 15:22

benalleng force-pushed the diff-mutants branch 3 times, most recently from 1286515 to 3642002 Compare July 30, 2025 15:47

benalleng force-pushed the diff-mutants branch 4 times, most recently from 0ffb33b to 283f6cb Compare July 30, 2025 16:42

benalleng changed the title ~~Diff mutants~~ Diff based cargo-mutants Jul 30, 2025

benalleng force-pushed the diff-mutants branch 5 times, most recently from 3f97303 to 438cec4 Compare July 30, 2025 22:27

benalleng requested a review from DanGould July 31, 2025 15:10

benalleng marked this pull request as ready for review July 31, 2025 15:11

spacebear21 reviewed Jul 31, 2025

View reviewed changes

benalleng force-pushed the diff-mutants branch from 438cec4 to 8cca76d Compare July 31, 2025 17:00

benalleng added 2 commits July 31, 2025 13:03

Add diff based mutations

cd4f7af

This workflow is a planned replacement for the current weekly mutants that run every sunday on the rust-payjoin codebase. This will hopefully allow more continuous coverage without a weekly delay.

Add mutants covereage for ohttp logic

aeba995

I added one additional test for mutation coverage on ohttp logic with my plan to create more tests for full test coverage to follow.

benalleng force-pushed the diff-mutants branch from 8cca76d to 45711e3 Compare July 31, 2025 17:03

benalleng force-pushed the diff-mutants branch from 45711e3 to aeba995 Compare July 31, 2025 23:58

spacebear21 approved these changes Aug 1, 2025

View reviewed changes

spacebear21 merged commit a5d82d1 into payjoin:master Aug 1, 2025
10 checks passed

benalleng added a commit to benalleng/rust-payjoin that referenced this pull request Aug 3, 2025

Rename mutant exclusions

6fe6be5

There was a rebasing mistake on a previous commit as I created some exclusions but their name was changed before payjoin#910 was merged and they were not caught.

benalleng mentioned this pull request Aug 3, 2025

Stale mutant exclusions #918

Merged

benalleng mentioned this pull request Aug 6, 2025

Expanding Test Coverage #935

Open

Diff based cargo-mutants #910

Diff based cargo-mutants #910

Uh oh!

Conversation

benalleng commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 16662599796

Details

💛 - Coveralls

Uh oh!

benalleng commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benalleng commented Jul 31, 2025

Uh oh!

spacebear21 left a comment

Choose a reason for hiding this comment

Uh oh!

spacebear21 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

benalleng Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

DanGould Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

benalleng Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spacebear21 Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

spacebear21 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

spacebear21 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

benalleng Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

spacebear21 commented Jul 31, 2025

Uh oh!

DanGould commented Aug 1, 2025

Uh oh!

spacebear21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benalleng commented Jul 30, 2025 •

edited

Loading

coveralls commented Jul 30, 2025 •

edited

Loading

benalleng commented Jul 30, 2025 •

edited

Loading

benalleng Aug 1, 2025 •

edited

Loading