Skip to content

REGRESSION_TESTING.md

Latest commit

 

History

History
88 lines (66 loc) · 4.5 KB

REGRESSION_TESTING.md

File metadata and controls

88 lines (66 loc) · 4.5 KB
Raw

Regression Testing System

Overview

Shimmy uses an automated regression testing system to prevent previously fixed bugs from returning. Every user-reported bug that gets fixed MUST have a corresponding regression test.

Test Structure (v2.2+)

As of v2.2, the old per-issue-file structure under tests/regression/ has been retired in favor of three consolidated, architecturally-organized test files:

tests/
├── core.rs             # 28 tests — CLI, registry, templates, discovery,
│                       #            safetensors, OpenAI/Ollama serde, versions,
│                       #            multi-part content, SSE chunk format, error format
├── handlers.rs         #  5 tests — HTTP server health, /v1/models,
│                       #            /v1/chat/completions, /api/tags, concurrency
└── compile_checks.rs   #  1 test  — include_str template file existence

Historical Issue Coverage

Every user-reported issue that previously had a dedicated regression file is now covered in the consolidated suite:

Issue Title New Test(s) Location
#12 Custom model directories not detected test_custom_model_directory_env_vars, test_cli_model_dirs_option core.rs:208-218, core.rs:56-63
#13 Qwen / VSCode wrong template test_template_auto_detection, test_registry_infer_template core.rs:557-579, core.rs:171-186
#53 SSE duplicate data: prefix test_sse_streaming_chunk_format core.rs:435-486
#63 Windows exe wrong version test_version_is_sane core.rs:123-132
#64 / #73 / #86 / #88 Missing template files test_template_files_exist compile_checks.rs:4-44
#65 404 error format test_error_response_json_shape, test_chat_completions_model_not_found core.rs:366-381, handlers.rs:100-126
#112 SafeTensors wrong engine test_safetensors_extension_detection core.rs:256-280
#113 OpenAI / Ollama frontend compat test_model_struct_completeness, test_models_response_serde, test_api_tags_response_structure, and handler endpoint tests core.rs:347-363, core.rs:330-344, core.rs:388-433, handlers.rs:78-147
#191 422 multi-part content array test_multi_part_content_array_deserialization core.rs:439-456

Retired Tests

The following user-reported issues did not carry forward as dedicated tests because their old tests covered code paths that no longer exist or only tested process-spawning behavior. Each is still noted here for transparency:

Issue Title Reason for Retirement
#68 MLX Apple Silicon support MLX engine is an empty stub; no runtime behavior to lock in
#72 GPU backend flag Test was a no-op compilation check; actual GPU wiring is tested by Airframe
#80 LLM-only filtering Spawned real binary via assert_cmd — tested flag parsing, not core logic
#87 Apple GPU info detection MLX engine is a stub
#101 High CPU / streaming perf Spawned real binary — tested --help, not streaming behavior
#106 Windows server crash Spawned real binary — tested serve --help, not crash handling
#108 Memory allocation flags Spawned real binary
#109 Anthropic API format Anthropic translation layer removed from codebase
#111 GPU metrics endpoint No-op test (empty body)
#114 MLX distribution features MLX engine is a stub
#127 / #128 MLX placeholder errors MLX engine is a stub

Running Tests

# Full suite (everything)
cargo test --features airframe,huggingface

# Core unit tests only
cargo test --test core --features airframe,huggingface

# HTTP handler tests only
cargo test --test handlers --features airframe,huggingface

# Template compilation check only
cargo test --test compile_checks

Adding a Regression Test for a New Issue

MANDATORY when fixing any user-reported bug:

  1. Identify which architectural area the bug belongs to (CLI, registry, templates, handlers, serde, etc.)
  2. Add the test to the appropriate file (core.rs, handlers.rs, or compile_checks.rs)
  3. Verify the test fails before the fix, passes after
  4. Add a cross-reference comment above the test in the format:
// Regression: Issue #XXX — one-line description

Policy

  • Every user-reported bug fix requires a regression test
  • Zero tolerance: all tests must pass before merge or release
  • Tests run automatically on every PR and every release via ci.yml and release.yml