Skip to content

Commit 6081c90

Browse files
Michael-A-KuykendallMichael-A-Kuykendall
committed
chore: purge regression tests unrelated to v2.x airframe functionality
Delete 3 files (274 lines) testing dead/transitional features: - issue_068: MLX is empty feature stub, no code - issue_110: crates.io publish disabled, tests used slow cargo subprocesses redundant with CI - issue_111: two no-op tests, compile-time check already enforced by codebase Keep 7 files testing current active functionality: - issue_012: model directory discovery - issue_013: Qwen template inference - issue_053: SSE streaming format - issue_063: CLI/version consistency - issue_064: template file inclusion - issue_112: SafeTensors routing - issue_113: OpenAI API structure All 24 tests pass locally (0.01s), format clean.
1 parent ebff993 commit 6081c90

5 files changed

Lines changed: 34 additions & 413 deletions

File tree

‎tests/regression.rs‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,18 +16,8 @@ mod issue_063_version_mismatch;
1616
#[path = "regression/issue_064_template_packaging.rs"]
1717
mod issue_064_template_packaging;
1818

19-
#[path = "regression/issue_068_mlx_support.rs"]
20-
mod issue_068_mlx_support;
21-
22-
#[path = "regression/issue_110_crates_io_build.rs"]
23-
mod issue_110_crates_io_build;
24-
25-
#[path = "regression/issue_111_gpu_metrics.rs"]
26-
mod issue_111_gpu_metrics;
27-
2819
#[path = "regression/issue_112_safetensors_engine.rs"]
2920
mod issue_112_safetensors_engine;
3021

3122
#[path = "regression/issue_113_openai_api.rs"]
3223
mod issue_113_openai_api;
33-

‎tests/regression/README.md‎

Lines changed: 34 additions & 129 deletions
Original file line numberDiff line numberDiff line change
@@ -1,129 +1,34 @@
1-
# Regression Test Directory
2-
3-
**Purpose**: Organized regression tests preventing user-reported bugs from recurring.
4-
5-
## Structure
6-
7-
Each issue gets its own test file following naming convention:
8-
- `issue_NNN_<short_description>.rs` - For specific GitHub issues
9-
- Tests must compile, run, and pass before closing related issues
10-
11-
## Complete Regression Test Inventory
12-
13-
**TOTAL: 20 test files covering 23+ user-reported issues**
14-
15-
### Organized Regression Tests (tests/regression/)
16-
17-
| Issue # | File | Description | Status |
18-
|---------|------|-------------|--------|
19-
| #12 | `issue_012_custom_model_dirs.rs` | Custom model directory environment variables | ✅ Active |
20-
| #13 | `issue_013_qwen_template.rs` | Qwen model ChatML template detection | ✅ Active |
21-
| #51 | `issue_051_lmstudio_discovery.rs` | LMStudio model auto-discovery | ✅ Active |
22-
| #53 | `issue_053_sse_duplicate_prefix.rs` | SSE streaming duplicate 'data:' prefix | ✅ Active |
23-
| #58, #59 | `issue_058_059_cuda_compilation.rs` | CUDA compilation errors, GPU prebuilt binaries | ✅ Active |
24-
| #63 | `issue_063_version_mismatch.rs` | Pre-built Windows exe version reporting | ✅ Active |
25-
| #64 | `issue_064_template_packaging.rs` | Template files missing from crates.io | ✅ Active |
26-
| #65 | `issue_065_error_handling.rs` | Better error handling for missing models | ✅ Active |
27-
| #68 | `issue_068_mlx_support.rs` | MLX Apple Silicon compilation support | ✅ Active |
28-
| #72 | `issue_072_gpu_backend_flag.rs` | GPU backend flag not wired to model loading | ✅ Active |
29-
| #87 | `issue_087_apple_gpu_info.rs` | Apple GPU info detection errors | ✅ Active |
30-
| #101 | `issue_101_performance_fixes.rs` | Threading, streaming, OLLAMA_MODELS support | ✅ Active |
31-
| #106 | `issue_106_windows_crash.rs` | Windows server crashes | ✅ Active |
32-
| #108 | `issue_108_memory_allocation.rs` | Memory allocation CLI flags broken | ✅ Active |
33-
| #109 | `issue_109_anthropic_api.rs` | Anthropic Claude API compatibility | ✅ Active |
34-
| #110 | `issue_110_crates_io_build.rs` | Build failure on cargo install shimmy | ✅ Active |
35-
| #111 | `issue_111_gpu_metrics.rs` | GPU metrics missing from /metrics endpoint | ✅ Active |
36-
| #112 | `issue_112_safetensors_engine.rs` | SafeTensors engine selection | ✅ Active |
37-
| #113 | `issue_113_openai_api.rs` | OpenAI API frontend compatibility | ✅ Active |
38-
| #114 | `issue_114_mlx_distribution.rs` | MLX distribution pipeline support | ✅ Active |
39-
| #127, #128 | `issue_127_128_mlx_placeholder.rs` | MLX placeholder instead of proper error | ✅ Active |
40-
| General | `issue_012_013_model_discovery.rs` | Combined model discovery tests (#12, #13) | ✅ Active |
41-
| General | `issue_packaging_general.rs` | General packaging regression tests | ✅ Active |
42-
| General | `issue_version_validation.rs` | Version validation across releases | ✅ Active |
43-
44-
### Additional Coverage in Integration Tests
45-
46-
| Issue # | Location | Notes |
47-
|---------|----------|-------|
48-
| #60 | `tests/release_gate_integration.rs` | Template inclusion gate tests |
49-
| #72 | `tests/gpu_backend_tests.rs`, `tests/gpu_layer_verification.rs` | Additional GPU integration tests |
50-
| #101 | `tests/cli_integration_tests.rs` | CLI-level integration testing |
51-
52-
## Adding New Regression Tests
53-
54-
**MANDATORY when fixing any user-reported bug:**
55-
56-
1. **Create test file**: `tests/regression/issue_NNN_<description>.rs`
57-
2. **Write failing test**: Reproduce the exact bug scenario
58-
3. **Fix the bug**: Make changes to src/
59-
4. **Verify test passes**: `cargo test --test issue_NNN_<description>`
60-
5. **Add to this README**: Update table above
61-
6. **Commit together**: Test file + fix in same commit
62-
63-
## Running Regression Tests
64-
65-
```bash
66-
# Run all regression tests
67-
cargo test --tests regression/
68-
69-
# Run specific issue test
70-
cargo test --test regression/issue_072_gpu_backend_flag
71-
72-
# Run with specific features
73-
cargo test --test regression/issue_127_128_mlx_placeholder --features mlx
74-
```
75-
76-
## CI/CD Integration
77-
78-
Regression tests run automatically in:
79-
- `.github/workflows/ci.yml` - On every PR
80-
- `.github/workflows/release.yml` - Before every release
81-
- Pre-commit hooks (future)
82-
83-
**Zero tolerance**: If ANY regression test fails, PR/release BLOCKED.
84-
85-
## Test Requirements
86-
87-
Each regression test MUST:
88-
1. **Reference issue number** in file name and doc comment
89-
2. **Link to GitHub issue** in module-level documentation
90-
3. **Describe exact bug** that was fixed
91-
4. **Test the fix** with clear assertions
92-
5. **Be reproducible** - no flaky tests allowed
93-
6. **Run in CI/CD** - verified automatically
94-
95-
## Example Test Structure
96-
97-
```rust
98-
/// Regression test for Issue #XXX: Bug description
99-
///
100-
/// GitHub: https://github.com/Michael-A-Kuykendall/shimmy/issues/XXX
101-
///
102-
/// **Bug**: What was broken
103-
/// **Fix**: What changed
104-
/// **This test**: How we verify it stays fixed
105-
106-
#[cfg(test)]
107-
mod issue_xxx_tests {
108-
use super::*;
109-
110-
#[test]
111-
fn test_issue_xxx_bug_scenario() {
112-
// Reproduce exact bug conditions
113-
// Assert fix works
114-
// Add helpful error messages
115-
}
116-
117-
#[test]
118-
fn test_issue_xxx_edge_cases() {
119-
// Test boundary conditions
120-
}
121-
}
122-
```
123-
124-
## Maintenance
125-
126-
- **NEVER delete** regression tests unless issue was invalid
127-
- **NEVER skip** failing regression tests - FIX THEM
128-
- **Update immediately** if APIs change (keep tests working)
129-
- **Review quarterly** for obsolete tests (rare)
1+
# Regression Test Directory
2+
3+
**Purpose**: Organized regression tests preventing user-reported bugs from recurring.
4+
5+
## Active Tests (7 files)
6+
7+
Each tests current shimmy v2.x functionality:
8+
9+
| Issue | File | What it tests |
10+
|-------|------|---------------|
11+
| #12 | `issue_012_custom_model_dirs.rs` | Custom model directory env vars and discovery |
12+
| #13 | `issue_013_qwen_template.rs` | Qwen model ChatML template inference |
13+
| #53 | `issue_053_sse_duplicate_prefix.rs` | SSE streaming `data:` prefix format |
14+
| #63 | `issue_063_version_mismatch.rs` | CLI commands and version consistency |
15+
| #64 | `issue_064_template_packaging.rs` | Template files included in package |
16+
| #112 | `issue_112_safetensors_engine.rs` | SafeTensors file extension routing |
17+
| #113 | `issue_113_openai_api.rs` | OpenAI API response structure |
18+
19+
## Removed Tests
20+
21+
The following were removed in the v2.0 airframe migration because they tested
22+
dead or transitional functionality:
23+
24+
| Issue | File | Reason |
25+
|-------|------|--------|
26+
| #68 | `issue_068_mlx_support.rs` | MLX is empty `[]` feature stub, no code |
27+
| #110 | `issue_110_crates_io_build.rs` | `publish = false`, binary-only dist; tests used slow `cargo` subprocesses |
28+
| #111 | `issue_111_gpu_metrics.rs` | Two no-op tests; compile-time check enforced by rest of codebase |
29+
30+
## Running
31+
32+
```bash
33+
cargo test --features airframe,huggingface --test regression
34+
```

‎tests/regression/issue_068_mlx_support.rs‎

Lines changed: 0 additions & 37 deletions
This file was deleted.

0 commit comments

Comments
 (0)