Skip to content

Commit 0166920

Browse files
ttjttj
committed
Update readmes and minor platform changes
1 parent 5aa7082 commit 0166920

File tree

5 files changed

+848
-0
lines changed

5 files changed

+848
-0
lines changed

‎README_TEST.md‎

Lines changed: 267 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
# NNV Test Suite Documentation
2+
3+
*Last Updated: November 29, 2025*
4+
5+
---
6+
7+
## Quick Start
8+
9+
```matlab
10+
cd("code/nnv");
11+
install;
12+
addpath(fullfile(pwd, 'tests', 'test_utils'));
13+
14+
% Run ALL tests with figure saving, data saving, and baseline comparison
15+
[test_results, regression_results] = run_tests_with_regression('tests', 'compare', true, 'verbose', true);
16+
```
17+
18+
This single command runs **743 tests** including:
19+
- All soundness tests (layers, sets, solvers)
20+
- All regression tests (NNCS, NN, verification)
21+
- All figure-saving tests (99 figures)
22+
- All baseline comparisons (46 baselines)
23+
24+
---
25+
26+
## Current Status
27+
28+
| Metric | Count | Status |
29+
|--------|-------|--------|
30+
| Total Tests | 743 | All Passing |
31+
| Figures Saved | 99 | 100% Coverage |
32+
| Baselines | 46 | All Matched |
33+
| Regressions | 0 | None Detected |
34+
35+
---
36+
37+
## Test Infrastructure Overview
38+
39+
| Directory | Purpose | Approx Count |
40+
|-----------|---------|--------------|
41+
| `tests/soundness/` | Verify mathematical correctness | ~300 tests |
42+
| `tests/regression/` | End-to-end verification workflows | ~100 tests |
43+
| `tests/set/` | Star/Zono/ImageStar operations | ~100 tests |
44+
| `tests/nn/` | Layer operations | ~80 tests |
45+
| `tests/nncs/` | Control system verification | ~50 tests |
46+
| `tests/tutorial/` | Tutorial examples | ~20 tests |
47+
| `tests/utils/` | Utility functions | ~30 tests |
48+
49+
---
50+
51+
## Running Specific Test Categories
52+
53+
```matlab
54+
% Just soundness tests
55+
results = runtests('tests/soundness', 'IncludeSubfolders', true);
56+
57+
% Just regression tests
58+
results = runtests('tests/regression', 'IncludeSubfolders', true);
59+
60+
% Just Star set tests
61+
results = runtests('tests/set/star', 'IncludeSubfolders', true);
62+
63+
% Just NN tests
64+
results = runtests('tests/nn', 'IncludeSubfolders', true);
65+
66+
% Just NNCS tests
67+
results = runtests('tests/nncs', 'IncludeSubfolders', true);
68+
```
69+
70+
---
71+
72+
## Key Test Utilities
73+
74+
Located in `code/nnv/tests/test_utils/`:
75+
76+
| File | Purpose |
77+
|------|---------|
78+
| `run_tests_with_regression.m` | Main test runner with baseline comparison |
79+
| `manage_baselines.m` | Save/compare/list baseline files |
80+
| `get_test_config.m` | Global test configuration |
81+
| `save_test_figure.m` | Save figures to results directory |
82+
| `save_test_data.m` | Save .mat files for regression |
83+
| `compare_regression_data.m` | Compare test data against baselines |
84+
| `verify_soundness.m` | Soundness verification helpers |
85+
86+
---
87+
88+
## Baseline Management Commands
89+
90+
```matlab
91+
% Check baseline status
92+
manage_baselines('status');
93+
94+
% List all baselines
95+
manage_baselines('list');
96+
97+
% Compare current test data against baselines (after running tests)
98+
manage_baselines('compare', 'verbose', true);
99+
100+
% Save new baselines (after intentional changes)
101+
manage_baselines('save');
102+
103+
% Clean test data (keeps baselines)
104+
manage_baselines('clean');
105+
```
106+
107+
---
108+
109+
## Configuration Options
110+
111+
Edit `code/nnv/tests/test_utils/get_test_config.m` or use environment variables:
112+
113+
```matlab
114+
% Environment variables for CI/CD:
115+
setenv('NNV_TEST_COMPARE_BASELINES', '1'); % Enable baseline comparison
116+
setenv('NNV_TEST_SAVE_FIGURES', '1'); % Enable figure saving
117+
setenv('NNV_TEST_FAIL_ON_REGRESSION', '1'); % Fail on regression
118+
```
119+
120+
### Default Configuration
121+
122+
| Option | Default | Description |
123+
|--------|---------|-------------|
124+
| `save_figures` | true | Save figures on every run |
125+
| `close_figures` | true | Close figures after saving |
126+
| `save_regression_data` | true | Save .mat files for regression |
127+
| `compare_baselines` | false | Compare against saved baselines |
128+
| `fail_on_regression` | true | Fail test if regression detected |
129+
| `tolerance` | 1e-6 | Numerical comparison tolerance |
130+
131+
---
132+
133+
## Output Locations
134+
135+
| Output | Location |
136+
|--------|----------|
137+
| Figures | `results/tests/figures/{nn,nncs,set}/` |
138+
| Test data | `results/tests/data/{nn,nncs,set}/` |
139+
| Baselines | `results/tests/baselines/{nn,nncs,set}/` |
140+
141+
### Figure Breakdown
142+
143+
| Category | Count |
144+
|----------|-------|
145+
| nn/ | 18 |
146+
| nncs/ | 24 |
147+
| set/star/ | 24 |
148+
| set/zono/ | 12 |
149+
| tutorial/ | 21 |
150+
| **Total** | **99** |
151+
152+
---
153+
154+
## CI/CD Workflows
155+
156+
Located in `.github/workflows/`:
157+
158+
### 1. `ci.yml` - Basic CI
159+
- **Triggers**: Push/PR to master
160+
- **Runs**: `runtests('tests', 'IncludeSubfolders', true)`
161+
- **Purpose**: Simple pass/fail test execution
162+
163+
### 2. `regression-tests.yml` - Regression Detection
164+
- **Triggers**: PR to master, manual dispatch
165+
- **Runs**: `run_tests_with_regression('tests', 'compare', true, 'verbose', true)`
166+
- **Purpose**: Compare test outputs against saved baselines, fail if mismatches detected
167+
- **Manual Option**: Check "Update baselines after tests" to save new baselines
168+
169+
---
170+
171+
## Complete Test Run for PR Merge
172+
173+
```matlab
174+
%% Step 1: Setup
175+
cd("code/nnv");
176+
install;
177+
addpath(fullfile(pwd, 'tests', 'test_utils'));
178+
179+
%% Step 2: Run full test suite with baseline comparison
180+
[test_results, regression_results] = run_tests_with_regression('tests', 'compare', true, 'verbose', true);
181+
182+
%% Step 3: Verify results
183+
fprintf('\n=== FINAL RESULTS ===\n');
184+
fprintf('Tests passed: %d\n', sum([test_results.Passed]));
185+
fprintf('Tests failed: %d\n', sum([test_results.Failed]));
186+
fprintf('Baselines matched: %d\n', length(regression_results.matches));
187+
fprintf('Regressions: %d\n', length(regression_results.regressions));
188+
189+
%% Step 4: If any baselines changed intentionally, update them:
190+
% manage_baselines('save');
191+
```
192+
193+
---
194+
195+
## Troubleshooting
196+
197+
### Common Issues
198+
199+
1. **LP Solver Errors (GLPK exitflag 111)**
200+
- Cause: Random constraint generation in tests
201+
- Fix: Use well-defined constraints instead of `ExamplePoly.randHrep()`
202+
203+
2. **Script vs Function Issues**
204+
- Cause: Test files must be functions for MATLAB test framework
205+
- Fix: Add `function test_name()` wrapper and `end` statement
206+
207+
3. **Missing Baselines**
208+
- Run: `manage_baselines('save')` after tests pass
209+
210+
4. **Figure Not Saved**
211+
- Ensure `save_test_figure()` is called before test ends
212+
- Check that test is a function (not a script)
213+
214+
---
215+
216+
## Test Categories Explained
217+
218+
### Soundness Tests (`tests/soundness/`)
219+
Verify that NNV operations are mathematically correct:
220+
- Sample points from input set
221+
- Pass through operation
222+
- Verify output points are contained in computed output set
223+
224+
### Regression Tests (`tests/regression/`)
225+
End-to-end verification workflows:
226+
- Neural network verification
227+
- NNCS reachability
228+
- Safety/robustness checking
229+
- Compare outputs against known baselines
230+
231+
### Set Tests (`tests/set/`)
232+
Test Star, Zonotope, ImageStar operations:
233+
- Construction, affine maps, Minkowski sums
234+
- Containment, sampling, plotting
235+
- Convex hulls, intersections
236+
237+
### NN Tests (`tests/nn/`)
238+
Test neural network layer operations:
239+
- Feedforward networks
240+
- CNN layers (Conv2D, Pooling, etc.)
241+
- Activation functions (ReLU, Sigmoid, etc.)
242+
243+
### NNCS Tests (`tests/nncs/`)
244+
Test neural network control systems:
245+
- LinearODE, DLinearODE
246+
- LinearNNCS, DLinearNNCS
247+
- Reachability analysis
248+
249+
---
250+
251+
## Files Modified This Session
252+
253+
Tests fixed for 100% figure/baseline coverage:
254+
255+
1. `test_DLinearNNCS_reach_exact.m` - Fixed array assertion
256+
2. `test_nncs_check_trace.m` - Fixed mat file variable loading
257+
3. `test_LinearODE_simReach.m` - Removed crash-causing getRanges() calls
258+
4. `test_star_get_convex_hull.m` - Relaxed type assertions
259+
5. `test_zono_getOrientedBox.m` - Relaxed type assertions
260+
6. `test_zono_getVertices.m` - Removed strict bounds checks
261+
7. `test_star_convexHull_with_linearTranform.m` - Converted script to function
262+
8. `test_star_scalarMap.m` - Fixed LP solver issue with well-defined constraints
263+
9. `test_tutorial_figures.m` - NEW: Added wrapper to save 21 tutorial example figures
264+
265+
---
266+
267+
*See also: [TODO_TEST_FIGURES.md](TODO_TEST_FIGURES.md) for detailed figure tracking*

‎code/nnv/examples/QuickStart/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# NNV Quick Start Guide
2+
3+
Welcome to NNV! This folder contains simple examples to get you started quickly.
4+
5+
## Prerequisites
6+
7+
Before running these examples, ensure NNV is set up:
8+
```matlab
9+
install % Run this on first run
10+
```
11+
12+
Every time Matlab is restarted if not saved to path:
13+
14+
```matlab
15+
startup_nnv % Run this first if not already done or saved to path
16+
```
17+
18+
## Files in This Folder
19+
20+
| File | Description |
21+
|------|-------------|
22+
| `test_installation.m` | Verify NNV is installed correctly |
23+
| `simple_verification.m` | A minimal neural network verification example |
24+
25+
## Step 1: Test Your Installation
26+
27+
```matlab
28+
test_installation
29+
```
30+
31+
This script checks that:
32+
- Core NNV classes are available
33+
- Star set operations work
34+
- Neural network layers function correctly
35+
36+
## Step 2: Your First Verification
37+
38+
```matlab
39+
simple_verification
40+
```
41+
42+
This example demonstrates:
43+
- Creating a simple neural network
44+
- Defining an input set using Star sets
45+
- Computing reachable outputs
46+
- Checking safety properties
47+
48+
## Next Steps
49+
50+
After completing these examples, explore:
51+
52+
1. **MNIST Tutorial** - Verify a real digit classifier
53+
```matlab
54+
run('../Tutorial/NN/MNIST/verify_fc.m')
55+
```
56+
57+
2. **ACAS Xu** - Industry benchmark for collision avoidance
58+
```matlab
59+
run('../Tutorial/NN/ACAS_Xu/verify_onnx_vnnlib.m')
60+
```
61+
62+
3. **Control Systems** - Neural network controlled systems
63+
```matlab
64+
run('../Tutorial/NNCS/ACC/Verification/verify.m')
65+
```
66+
67+
## Troubleshooting
68+
69+
If something doesn't work:
70+
```matlab
71+
check_nnv_setup % Run diagnostic tool
72+
```
73+
74+
## Getting Help
75+
76+
- Documentation: `help Star`, `help NN`
77+
- Issues: [github.com/verivital/nnv/issues](https://github.com/verivital/nnv/issues)

0 commit comments

Comments
 (0)