CLI regression suite - #1
Open
HanifKoh wants to merge 1 commit into
Open
Conversation
HanifKoh
marked this pull request as draft
September 7, 2026 13:00
HanifKoh
force-pushed
the
cli-test-suite
branch
from
September 8, 2026 03:43
e2e49ea to
d15fe9d
Compare
HanifKoh
marked this pull request as ready for review
September 8, 2026 06:27
HanifKoh
force-pushed
the
cli-test-suite
branch
from
September 8, 2026 09:52
1221ffc to
8d475c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the single
run_test.pyscript with a pytest-based black-box suite for theorca-slicerCLI.run_test.pyis kept as the compatibility entry point OrcaSlicer's own CI already calls (build_orca.yml, "Run external slicer regression tests"), so that integration keeps working unchanged: it sets up a venv on first run and delegates to pytest.What the suite covers
cases/**/*.yaml, 63 cases): crash resistance under crafted arguments, CLI parsing, config precedence, bed/array bounds, settings import, CLI action/transform flags, custom G-code emission, per-object overrides. Each case is CLI args plus checks;status: opencases are strict xfails, so an upstream fix flips them loudly instead of drifting.test_projects/*.3mfmust slice; metrics (layers, max Z, filament, E sum, time, size) are compared tobaseline.jsonand drift is reported, not failed.PrintConfig.cpp, poisoned per type, daily-rotating sample by default and--fuzz-fullfor a sweep.--uptodatethrough every export path, with the three artifacts cross-checked.TESTING_STRATEGY.mddocuments each layer, the case schema and the status model.Reference build
Case statuses describe OrcaSlicer main, the build OrcaSlicer's CI runs the suite against. This repo's own workflow self-checks the suite against the nightly AppImage published from main (through
AppRun, with the matching source tree cloned for the option surface); a release tag can be tested on demand via theorca_refdispatch input.Verification
Against main
37e1582c4c(local Release build): 99 passed, 3 skipped, 9 xfailed in 2 m 17 s. The fullnilfuzz sweep over all 243 vector options passes. The suite also ran cleanly through the nightly AppImage'sAppRun. OrcaSlicer/OrcaSlicer#15453 runs this branch inside OrcaSlicer's own CI.The nine still-open cases on main:
load-defaultfila,cli-config-repair-clamping-parity,load-filaments-count-exceeds-project,mismatched-extruder-geometry-arrays,export-settings-into-missing-dir-silently-noop,partial-load-filaments-variant-key-zero-filled,sparse-user-filament-with-colour-segfault,vector-override-full-length-multi-filament,vector-override-single-value-multi-filament.