Skip to content

Sensor Fusion & Noise Cleanup#27

Merged
jctoledo merged 30 commits intomainfrom
noise-cleanup
Jan 19, 2026
Merged

Sensor Fusion & Noise Cleanup#27
jctoledo merged 30 commits intomainfrom
noise-cleanup

Conversation

@jctoledo
Copy link
Copy Markdown
Owner

Major overhaul of the acceleration processing pipeline for reliable mode detection.

Core Changes

GPS-Corrected Orientation (ArduPilot-style)

  • OrientationCorrector learns pitch/roll errors by comparing IMU vs GPS acceleration
  • Fixes WT901 AHRS errors during acceleration/braking (can't distinguish linear accel from tilt)
  • Enables accurate 200Hz IMU data instead of being limited to GPS rate

Vibration Filtering

  • 2nd-order Butterworth low-pass at 10Hz removes engine vibration (30-100Hz)
  • Preserves driving dynamics (0-5Hz) for responsive mode detection

Centripetal Lateral

  • lat = speed × yaw_rate - mount-angle independent, doesn't "stick" after turns
  • Yaw rate calibration using GPS heading during highway driving

Architecture

  • Moved pure algorithm modules (fusion.rs, mode.rs, filter.rs) to framework/ crate
  • Enables host-side unit testing without ESP32 hardware
  • 63 comprehensive tests covering all sensor fusion logic

Bug Fixes

  • GPS processing: process_gps() now called reliably (was blocking OrientationCorrector learning)
  • Filter sample rate: Fixed mismatch between config (33ms) and filter (was 200Hz, now 30Hz)
  • Settings reset: Was resetting EMA alpha to 0.25 on every settings change
  • Lateral sticking: Fixed G-meter showing old values after turn ends

Diagnostics & UI

  • Color-coded health indicators on diagnostics page
  • OrientationCorrector status: pitch/roll correction, confidence %
  • Driving presets with tuned thresholds (Track, Canyon, City, Highway)

Tooling

  • analyze_telemetry.py: Correlates recorded data with ground truth, computes precision/recall

jctoledo and others added 30 commits January 17, 2026 15:56
- Add Biquad IIR low-pass filter (2Hz cutoff) for vibration removal
- Add GPS-derived longitudinal acceleration from velocity changes
- Add configurable GPS/IMU blending (70/50/30% based on GPS rate)
- Add tilt estimation for mounting angle correction when stationary
- Add gravity estimation for continuous correction during steady driving
- Add hybrid mode detection using blended lon accel and filtered lat accel
- Update diagnostics with gps_fix_count() for fusion rate tracking
- Display corrected G-meter values instead of raw IMU data
- Tilt learning now visible on dashboard (G-meter returns to zero)
- Fix clippy warnings (doc comments, unused methods)
- Dashboard now shows blended longitudinal acceleration (same as mode classifier)
- Add unit tests verifying display values match classification inputs
- Fix CLAUDE.md blending ratios (70/50/30 not 90/70/30)
- Update README.md with noise cleanup features
- Add data flow diagram showing dashboard receives same values as mode.rs

Co-Authored-By: Claude <noreply@anthropic.com>
Root cause: Filtering lateral acceleration in earth frame before
transforming to vehicle frame caused the filter to smooth a rotating
vector. When the vehicle yaws, the filtered earth-frame values
get transformed with the new heading, causing lateral G to "stick"
at incorrect values until the vehicle stops.

Changes:
- Remove Biquad filter for lateral in earth frame entirely
- Use centripetal formula (speed * yaw_rate) for mode detection
  This is what pro racing systems (VBOX, Motec) use - mount-independent
- Filter longitudinal in vehicle frame (safe, no rotation issues)
- Increase filter cutoff from 2Hz to 5Hz for faster response
- Rename config fields: imu_sample_rate -> sample_rate,
  accel_filter_cutoff -> lon_filter_cutoff

Display continues to show accelerometer-based lateral with tilt/gravity
correction. Mode detection uses centripetal lateral which responds
immediately when yaw rate changes.

Added smart unit tests that simulate turn scenarios and verify
lateral returns to zero immediately when turn ends.
Added rule to CLAUDE.md: dead code must be removed or used, not silenced.

Fixed fusion.rs: removed get_lat_centripetal() getter, updated test to
use return value from process_imu() instead.
- README: Update filter description from 2Hz to 5Hz
- mode.rs: Fix update_hybrid docs - receives centripetal lateral, not filtered
- filter.rs: Add test for actual 5Hz config used in fusion.rs
PROBLEM: Mode detection had ~300ms latency due to double filtering
- Biquad (5Hz) in fusion.rs: ~70ms delay
- EMA (α=0.35) in mode.rs: ~116ms time constant
- Combined: 90% response in ~250-300ms

SOLUTION:
1. Remove Biquad filter for longitudinal acceleration
   - GPS blend already provides smooth signal (no vibration)
   - 70% GPS / 30% IMU means 70% of signal is already clean
   - mode.rs EMA handles residual noise

2. Increase EMA α from 0.35 to 0.50
   - Time constant reduced from 116ms to 72ms
   - 90% response: 165ms (was 267ms)

RESULT:
- Estimated 90% response: ~120-150ms (was ~300ms)
- Mode detection should feel significantly more responsive
- Corner detection unchanged (uses centripetal, already instant)

OTHER CHANGES:
- filter.rs moved to #[cfg(test)] - only compiled for tests
- Removed unused FusionConfig fields (lon_filter_cutoff, sample_rate)
- Updated README to reflect new architecture
When user changed mode thresholds via web UI, alpha was hardcoded to 0.25
instead of using the optimized 0.50. This would undo the latency improvement
after any settings change.

Also fixed:
- Outdated comment mentioning removed Biquad filter
- Misleading comment about lateral (display uses accelerometer, mode uses centripetal)
The bug where settings handler used alpha=0.25 while default used 0.50
was caused by the same value being hardcoded in two places.

Fix:
- Add DEFAULT_MODE_ALPHA constant in mode.rs
- Use constant in both ModeConfig::default() and main.rs settings handler
- Add unit test to verify default config uses the constant

This ensures future changes to alpha only need to update one place.
…ting

Moved mode.rs, fusion.rs, and filter.rs from the embedded blackbox crate
to the sensor-fusion framework crate. This enables running 52 unit tests
on the host machine without requiring ESP32 hardware.

Changes:
- framework/src/mode.rs: Mode classifier with 10 tests
- framework/src/fusion.rs: GPS/IMU blending with 9 tests
- framework/src/filter.rs: Biquad filter with 6 tests (unused but verified)
- sensors/blackbox/src/*.rs: Thin re-exports from framework
- README.md: Updated file structure, added test command

Tests now runnable with: cargo test -p sensor-fusion -p wt901 -p ublox-gps
- Add 5Hz Butterworth low-pass filter to IMU longitudinal acceleration
  to remove engine vibration (20-100Hz) while preserving driving dynamics
- Based on ArduPilot research: accelerometer outer loop uses 10Hz filter
- Add YawRateCalibrator to learn gyro bias during straight-line driving
- Switch lateral G to centripetal calculation (speed × yaw_rate)
  for mount-independent, instant response without sticking
- Add tests for vibration attenuation and dynamics passthrough
- Update CLAUDE.md with filter theory and data flow diagrams
Critical bug fix:
- lon_sample_rate was 20Hz but filter runs at 200Hz (IMU rate)
- This made filter ~10x more aggressive than intended, killing real dynamics

Filter improvements:
- Increase cutoff from 5Hz to 15Hz (ArduPilot uses 20Hz)
- Sharp braking events (up to 10Hz) now preserved

GPS blending improvements:
- Reduce GPS weight from 70% to 40% (trust filtered IMU more)
- Add GPS accel validity check: when GPS=0 but IMU has signal, use 100% IMU
- Use lon_blended for display (was GPS-only, often showing 0)

Analysis showed:
- 57% of samples had lon_g=0 due to GPS-only display
- Mode detection was 38% accurate for ACCEL, 43% for BRAKE
- 604 samples with |GT|>0.15g but reported 0

New tests:
- test_default_config_has_correct_filter_settings
- test_lon_display_equals_lon_blended
- test_sharp_braking_preserved_with_15hz_filter
- test_gps_accel_validity_check
The GravityEstimator tried to learn gravity offset during "steady state"
driving (constant speed, low yaw). This was fundamentally flawed because:
- Aerodynamic drag, rolling resistance, and road grade create real accelerations
- These were incorrectly learned as "gravity offsets"
- Result: -2.6 m/s² constant offset causing 76% false BRAKE detections

Changes:
- Remove GravityEstimator struct and all references
- Remove gravity fields from FusionDiagnostics and dashboard
- Add 3 new unit tests to catch similar regressions:
  - test_cruising_at_constant_speed_produces_zero_lon
  - test_tilt_only_updates_when_stationary
  - test_no_drift_during_extended_cruise
- Update CLAUDE.md documentation
- Fix stale comment (5Hz -> 15Hz filter)

The TiltEstimator (learns only when stationary) is sufficient for
mounting offset correction. This matches ArduPilot's approach: only
learn corrections when you KNOW the truth (stationary = zero velocity).
The WT901 AHRS cannot distinguish linear acceleration from tilt - during
forward acceleration it reports false pitch, corrupting gravity removal.

Solution: OrientationCorrector learns pitch/roll errors by comparing
IMU-derived acceleration with GPS-derived acceleration (ground truth).
The innovation reveals orientation error: pitch_error ≈ (ax_imu - ax_gps) / G

Key changes:
- Add OrientationCorrector to fusion.rs (learns pitch/roll from GPS comparison)
- New process_imu() API takes raw body-frame IMU + AHRS angles
- Confidence-based GPS/IMU blending (80% GPS initially → 30% when learned)
- Add drive_sim.rs example demonstrating AHRS error correction
- Update diagnostics with orientation correction fields
- Update CLAUDE.md, README.md, docs/index.html

Architecture: This is NOT a 9D EKF - OrientationCorrector runs separately
from the 7-state EKF as a modular correction layer. This approach is simpler,
testable, and appropriate for embedded constraints.

Benefits:
- Device can be mounted at any angle within ±15° (auto-corrected)
- Enables accurate 200 Hz IMU instead of being limited to 25 Hz GPS
- Fixes false mode detections caused by AHRS pitch error during acceleration
- Add Open Graph and Twitter Card meta tags for social sharing
- Add JSON-LD structured data for search engines
- Add "How It Works" section with:
  - Explanation of all 7 EKF states (x, y, psi, vx, vy, bax, bay)
  - Glossary defining: EKF, IMU, GPS, AHRS, ZUPT, Sensor Fusion, G-Force, WiFi AP
- Update feature cards with plain-language descriptions
- Add tooltips to hero stats
- Update navigation with How It Works link

Makes technical content accessible to anyone without prior knowledge.
- Add analyze_telemetry.py: analyzes CSV exports from dashboard
  - Computes timing, speed, acceleration statistics
  - Compares lon_g with GPS-derived acceleration (ground truth)
  - Calculates mode detection precision/recall
  - Reports correlation coefficient and bias detection
  - Provides diagnostic summary of potential issues

- Document drive_sim example in build commands
- Document analyze_telemetry.py in Python tools section
- Move plain English project description to the very top (before TOC)
- Add table of contents with 30+ links for easy navigation
- Add dedicated 'How It Works' section explaining sensor fusion in plain terms
- Consolidate 'Why build this' into intro section
- Document transforms, AHRS correction, and motion models for non-experts
Display GPS-corrected orientation metrics on diagnostics page:
- Pitch Correction (degrees) - learned AHRS pitch error
- Roll Correction (degrees) - learned AHRS roll error
- Orient Conf (pitch% / roll%) - confidence in corrections

Color coding:
- Green: correction < 10deg, confidence > 50%
- Yellow: moderate values
- Red: confidence < 10% (still learning)

These metrics help validate that GPS-corrected orientation is working
during test drives.
Color coding (green/yellow/red) added for:
- Loop Rate: >500 ok, >200 warn, <200 err
- Position σ: <5m ok, <10m warn, >10m err
- Velocity σ: <0.5 ok, <1.0 warn, >1.0 err
- Yaw σ: <5° ok, <10° warn, >10° err
- Bias X/Y: <0.3 ok, <0.5 warn, >0.5 err (absolute value)
- Heap: >40KB ok, >20KB warn, <20KB err
- Yaw Bias: <10 mrad/s ok, <50 warn, >50 err
- Tilt X/Y: <0.3 ok, <0.5 warn, >0.5 err (max of both)

Makes it easy to see at a glance if values are within spec.
- Fix Butterworth filter sample rate mismatch (200Hz -> 30Hz)
  The filter was misconfigured for 200Hz but called at ~30Hz telemetry rate,
  causing effective cutoff of ~1.5Hz instead of 10Hz and removing valid signals

- Add cruise bias learning to OrientationCorrector
  Learns mounting offset during constant-speed driving when GPS shows ~0 accel
  New fields: cruise_bias, cruise_bias_sum, cruise_bias_count, cruise_time

- Lower min_accel threshold from 0.5 to 0.3 m/s² for more learning opportunities

- Add dt parameter to OrientationCorrector::update() for cruise timing

- Add new tests for cruise bias learning behavior

- Update filter tests to use correct 30Hz sample rate

- Expand CSV export with fusion diagnostics columns

- Enhance analyze_telemetry.py with fusion diagnostics analysis
City preset changes:
- acc: unchanged at 0.10g (cruise bias learning handles offset)
- brake: 0.18 -> 0.25g (less aggressive, reduce false braking)
- lat: 0.12 -> 0.10g (more sensitive corner detection)
- yaw: 0.05 -> 0.04 rad/s (more sensitive corner detection)

Also updates:
- mode.rs default ModeConfig to match new city preset
- Slider default values in UI to match
- Summary display defaults
- Test values for combined brake+corner
- Fix critical bug: process_gps() now called with scalar speed on every
  new fix, not just when velocity_enu available (enables OrientationCorrector)
- Fix lon_sample_rate: 20Hz → 30Hz to match TelemetryConfig default (33ms)
- Fix gps_max_age: 0.2s → 0.4s for 25Hz GPS with processing margin
- Lower brake thresholds across all presets to account for mounting bias
- Sync ModeSettings::default() and HTML slider defaults with mode.rs
- Update CLAUDE.md preset documentation
- Enhance docs/index.html JSON-LD metadata for LLM discovery
- Rename ambiguous variable 'l' to 'lon'/'true'
- Add whitespace around ** operators
- Break long lines for readability
- Use TRUE_ACCEL_THRESH/TRUE_BRAKE_THRESH constants
- Add whitespace around * operators in f-strings
- Remove unused lon_g variable
- Break long lines (<120 chars)
@jctoledo jctoledo marked this pull request as ready for review January 19, 2026 20:30
@jctoledo jctoledo merged commit 4dd2849 into main Jan 19, 2026
5 checks passed
@jctoledo jctoledo deleted the noise-cleanup branch January 19, 2026 20:32
@jctoledo jctoledo restored the noise-cleanup branch January 19, 2026 21:27
@georgkloeck georgkloeck deleted the noise-cleanup branch January 21, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant