Skip to content

VALIDATION.md

Latest commit

 

History

History
144 lines (112 loc) · 7.17 KB

VALIDATION.md

File metadata and controls

144 lines (112 loc) · 7.17 KB
Raw

Validation Against Known Results

Date: June 12, 2026 Status: Tier 1 (analytic goldens) and Tier 2 (real measured channels, parser/math cross-check) complete

This document describes how sigil is validated against results that are known independently of sigil itself, and records the current results.

Methodology: three kinds of "known result"

  1. Analytic (exact): synthetic channels whose correct simulation results are closed-form formulas. Any deviation beyond numerical tolerance is a pipeline bug by definition.
  2. Cross-tool (independent implementation): the same input processed by an unrelated, widely-used implementation (scikit-rf); agreement validates parsing and S-parameter math.
  3. Real measured data: channels measured on lab VNAs and published by the IEEE P802.3ck task force; behavior must be physically sensible and the parsed data must match the independent reader exactly.

Still pending (Tier 3): end-to-end eye comparison against PyBERT / SignalIntegrity on identical inputs, and ibisami-built .so models for the FFI path. See the validation plan in the project README.

Tier 1 — Analytic golden fixtures

Generated by examples/generate_golden_channels.py (pure numpy), asserted end-to-end (parse → condition → convert → eye) by crates/lib-dsp/tests/golden_channels.rs.

Fixture Construction Known result asserted
golden/ideal_delay.s2p S21 = e^(−j2πfτ), τ = 1 ns, band edge Nyquist-aligned (32 GHz = 2/UI) Impulse peak at exactly 1.0 ns; ∫h dt = 1; pulse = 1.0 V + Gibbs (≤ ~9%); eye ≈ 2.0 V (zero ISI)
golden/rc_pole.s2p S21 = 1/(1 + jf/fc), fc = 8 GHz, 100 GHz grid Statistical eye equals the closed form 2(1 − 2e^(−φ/τc)) at the optimal phase
test_channel.s2p Causal skin (0.3 dB/√GHz) + dielectric (0.08 dB/GHz) line, τ = 2 ns Strictly passive as parsed; IL(16 GHz) = 2.48 dB by formula; ∫h dt = 1; peak at 2.0 ns
test_channel_4port.s4p Two uncoupled copies of the line as a differential pair SDD21 ≡ single-ended S21; SDC21 = SCD21 = 0; differential sim result identical to single-ended

Bugs these tests caught on first run (both fixed):

  1. Propagation delay double-counted — apply_group_delay() shifted the impulse data by τ and t_start was set to τ, so the ideal-delay fixture peaked at exactly 2.000 ns. The pre-existing unit test asserted only t_start ≈ τ (±50%) and never the peak position, locking the bug in. Delay now lives in-band only (t_start = 0).
  2. Mode-conversion warning fired on every differential channel — mode_conversion_ratio() returns linear ratios; the orchestrator formatted them as dB and compared to −40, so zero coupling displayed as "SDC=0.00 dB" and always tripped the warning. (Also the reason the old "high mode conversion" warning on the bundled example was nonsense.)

The previous hand-rolled test_channel.s2p was itself non-passive (max singular value 1.0547 — physically impossible for an interconnect), which is why every example run used to log a passivity-enforcement warning.

Tier 2 — Real measured channels (IEEE 802.3ck)

Fetched by examples/fetch_802p3ck_channels.sh from the public channel repository at https://www.ieee802.org/3/ck/public/tools/index.html (data is gitignored, not redistributed). Channels used:

  • kareti_3ck_01_1118_backplane — measured traditional backplane (Agilent N5247A PNA-X, 10 MHz – 67 GHz, 6701 points), loss ladder Bch1 (3.5″) … Bch4 (30″)
  • tracy_3ck_02_0319_OSFP1p5m — measured OSFP 1.5 m 28 AWG cable assemblies (IdEM export, DC – 50 GHz, 5001 points), Thru/NEXT/FEXT sets

Cross-check vs scikit-rf (independent parser + mixed-mode math)

examples/crosscheck_skrf.py loads the same .s4p with scikit-rf, computes SDD21 independently, and diffs against the channel_response.csv sigil writes for the same file:

Channel Points sigil vs skrf SDD21
Bch1_3p5_t (3.5″ backplane) 6701 max Δ = 0.0000 dB
Bch4_30_t (30″ backplane) 6701 max Δ = 0.0000 dB
Thru_Tx8 OSFP 1.5 m cable 5001 max Δ = 0.0000 dB

Exact agreement on all points: Touchstone parsing and the IEEE P370 mixed-mode conversion match an independent implementation bit-for-bit on real VNA data.

Physical sanity (raw NRZ at 32 GT/s, no equalization)

Channel IL @ 16 GHz Group delay Statistical eye Verdict
Bch1 3.5″ backplane 8.9 dB 1.96 ns 0.385 V / 0.34 UI open but marginal — plausible
Bch4 30″ backplane 23.3 dB 5.95 ns 0.0 (closed) correct: 23 dB at Nyquist closes a raw NRZ eye
OSFP 1.5 m 28 AWG cable ~13 dB (TP1→TP4) 10.8 ns 0.081 V / 0.12 UI near closed — correct for a PAM4-era channel run as raw NRZ

Closed eyes on the long channels are the expected result: these channels are operated with TX FFE + RX CTLE/DFE (and PAM4 for 802.3ck); the raw channel eye is exactly what sigil currently models. The mode-conversion warnings on this data report real measured imbalance (SDC −19 to −27 dB), not artifacts.

Parser gaps found by real data (all fixed)

  1. Wrapped multi-line 4-port exports (rows split 9/8/8/8 values per line) were mis-inferred as 2-port — the 9-value first line is genuinely ambiguous. The .sNp extension is now the authoritative port count (parse_touchstone_file / parse_touchstone_with_ports), and shape inference recognizes the wrapped-4-port continuation signature.
  2. Comment lines interleaved inside the data section (IdEM separates every frequency point with a bare !) terminated parsing after the first point. Comments/blank lines are now skippable anywhere.
  3. Touchstone 1.0 2-port column order is S11 S21 S12 S22; the parser used row-major everywhere, silently swapping S21/S12 on 2-port files (invisible for reciprocal channels, wrong per spec). Fixed, with an asymmetric-data regression test.

Reproducing

# Tier 1: regenerate fixtures and run golden tests
python3 examples/generate_golden_channels.py
cargo test -p lib-dsp --test golden_channels

# Tier 2: fetch channels (~200 MB) and cross-check
examples/fetch_802p3ck_channels.sh
python3 -m venv /tmp/skrf-venv && /tmp/skrf-venv/bin/pip install scikit-rf
cargo build --release
target/release/si-kernel simulate --config <differential config for an .s4p> --output /tmp/out
/tmp/skrf-venv/bin/python examples/crosscheck_skrf.py <file.s4p> /tmp/out/channel_response.csv

Known limitations affecting validation

  • MixedModeSParameters::from_single_ended hardcodes the standard port convention (1+,3−)→(2+,4−); the config accepts other mappings but only warns. All channels validated here use the standard convention (verified via the cross-check script, which detects the convention from the data).
  • Raw-channel analysis only: results cannot yet be compared against published COM values, which include reference TX/RX equalization.
  • The bit-by-bit EyeAnalyzer width metric assumes a centered eye and is unreliable; statistical eye metrics were used throughout.