Skip to content

Remove witness exclude-vars from svcomp conf #1593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 22, 2024
Merged

Conversation

sim642
Copy link
Member

@sim642 sim642 commented Oct 4, 2024

Similarly to #1592, this is something where we appear to fingerprint tasks... Although these exclusion don't affect our verdict, only the generated witnesses.
But witness validation still affects SV-COMP scores, so it's not harmless.

I want to do some runs with witnesses to see what difference this really makes. Over time we've improved invariant generation to be smarter and avoid redundant data, so maybe this sort of witness size reduction is unnecessary now.

@sim642 sim642 added cleanup Refactoring, clean-up sv-comp SV-COMP (analyses, results), witnesses labels Oct 4, 2024
@sim642 sim642 added this to the SV-COMP 2025 milestone Oct 4, 2024
@sim642
Copy link
Member Author

sim642 commented Oct 8, 2024

I want to do some runs with witnesses to see what difference this really makes. Over time we've improved invariant generation to be smarter and avoid redundant data, so maybe this sort of witness size reduction is unnecessary now.

This doesn't seem to impact the size at all (with 60s timeout): before and after we produce 2.5GB of YAML witnesses.
In terms of results, Goblint can validate 30 fewer ldv-linux-3.4-simple/ task witnesses out of ~7000.

@sim642 sim642 marked this pull request as ready for review October 8, 2024 07:44
@michael-schwarz
Copy link
Member

Maybe we need to some mechanism to give up on witnesses once they become too large and produce the trivial witness so we can still get points?

I'm not sure I'm happy with the general direction, we currently seem to solve fewer tasks in each of the PRs from the last few weeks. Is this really a good strategy?

@sim642
Copy link
Member Author

sim642 commented Oct 8, 2024

This is just self-validation though. I sampled some of the differing tasks and we don't score points just from self-validation, but other validators also confirm our results. So I don't think there's much of a loss here. As I said, this doesn't make a notable size difference.

Solving fewer tasks is better than getting disqualified for using task-specific identifiers to score more points.

@sim642 sim642 mentioned this pull request Oct 17, 2024
1 task
@sim642 sim642 merged commit 913220d into master Oct 22, 2024
21 checks passed
@sim642 sim642 deleted the svcomp25-no-exclude-vars branch October 22, 2024 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cleanup Refactoring, clean-up sv-comp SV-COMP (analyses, results), witnesses
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants