Skip to content

CLI regression suite - #1

Open
HanifKoh wants to merge 1 commit into
mainfrom
cli-test-suite
Open

CLI regression suite#1
HanifKoh wants to merge 1 commit into
mainfrom
cli-test-suite

Conversation

@HanifKoh

@HanifKoh HanifKoh commented Sep 7, 2026

Copy link
Copy Markdown

Replaces the single run_test.py script with a pytest-based black-box suite for the orca-slicer CLI. run_test.py is kept as the compatibility entry point OrcaSlicer's own CI already calls (build_orca.yml, "Run external slicer regression tests"), so that integration keeps working unchanged: it sets up a venv on first run and delegates to pytest.

What the suite covers

  • Declarative regression cases (cases/**/*.yaml, 63 cases): crash resistance under crafted arguments, CLI parsing, config precedence, bed/array bounds, settings import, CLI action/transform flags, custom G-code emission, per-object overrides. Each case is CLI args plus checks; status: open cases are strict xfails, so an upstream fix flips them loudly instead of drifting.
  • Golden slices: every test_projects/*.3mf must slice; metrics (layers, max Z, filament, E sum, time, size) are compared to baseline.json and drift is reported, not failed.
  • Config-surface fuzzing: the ~890 accepted options enumerated from PrintConfig.cpp, poisoned per type, daily-rotating sample by default and --fuzz-full for a sweep.
  • Settings import/export matrix: presets, filaments, overrides and --uptodate through every export path, with the three artifacts cross-checked.
  • Override sweep: every config option set on the command line must land in the merged config and in the sliced G-code's config block.

TESTING_STRATEGY.md documents each layer, the case schema and the status model.

Reference build

Case statuses describe OrcaSlicer main, the build OrcaSlicer's CI runs the suite against. This repo's own workflow self-checks the suite against the nightly AppImage published from main (through AppRun, with the matching source tree cloned for the option surface); a release tag can be tested on demand via the orca_ref dispatch input.

Verification

Against main 37e1582c4c (local Release build): 99 passed, 3 skipped, 9 xfailed in 2 m 17 s. The full nil fuzz sweep over all 243 vector options passes. The suite also ran cleanly through the nightly AppImage's AppRun. OrcaSlicer/OrcaSlicer#15453 runs this branch inside OrcaSlicer's own CI.

The nine still-open cases on main: load-defaultfila, cli-config-repair-clamping-parity, load-filaments-count-exceeds-project, mismatched-extruder-geometry-arrays, export-settings-into-missing-dir-silently-noop, partial-load-filaments-variant-key-zero-filled, sparse-user-filament-with-colour-segfault, vector-override-full-length-multi-filament, vector-override-single-value-multi-filament.

@HanifKoh
HanifKoh marked this pull request as draft September 7, 2026 13:00
@HanifKoh
HanifKoh marked this pull request as ready for review September 8, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant