Advanced Multi-Modal Sensor Fusion System for Detecting Falling Humans

Authors: Dr. Nick Barua · Prof. Masahito Hitosugi
Department of Legal Medicine, Shiga University of Medical Science, Otsu, Shiga, Japan
Published: Vehicles, Vol. 7, No. 4, p. 149 (2025) · Part 1 of 4 in the AFODS Research Program
Patent Filed: Japanese Patent Application No. 2025-167440 (Filed: 3 October 2025)

Collisions with fallen pedestrians pose a lethal challenge to current Advanced Driver Assistance Systems (ADAS). Standard visible-spectrum architectures yield a True Positive Rate (TPR) as low as 21.4% at night (0 lux) for prone individuals — a critical and life-threatening classification gap.

This repository contains the software framework and quantitative evaluation for the Advanced Falling Object Detection System (AFODS). AFODS architecturally integrates LWIR Thermal, NIR Stereo, and Ultrasonic sensors, processed through a custom AI pipeline combining YOLOv7-Tiny for object detection and a GRU-based RNN for proactive threat assessment. Validated across 320 controlled trials, AFODS achieved a 99.5% TPR in clear daylight and a 98.2% TPR at night (0 lux) — a condition where the baseline system collapsed to 21.4%.

Block diagram showing the flow from perception sensors (LWIR, NIR, Ultrasonic, Acoustic, IMU) through the NVIDIA Jetson AGX Orin central processing unit to the vehicle's CAN bus, emergency braking system, and driver alerts.

(A) Standard camera: fallen object nearly invisible at 0 lux. (B) NIR: high-resolution imagery for depth and classification. (C) LWIR: distinct thermal signature of the human body. (D) Final fused system output: TARGET DETECTED.

Five-stage advanced pipeline: Acquisition & Preprocessing → Detection & Classification (YOLOv7-Tiny + Hypothermia Mode) → Depth Estimation & Motion Analysis → Predictive Threat Assessment (GRU/RNN) → Decision & Action (TTC threshold: 1.2 s, confidence threshold: >95%).

Composite image illustrating the three critical phases: (A) Initial state, (B) Predictive detection escalating to CRITICAL threat, (C) Final state — THREAT AVERTED.

Seven-stage experimental procedure: Objective → Experimental Design → Test Subjects & Simulation → Independent Variables → Testing Procedure (20 repetitions per condition) → Data Collection → Results Visualisation. Total: 320 trials.

AFODS vs. baseline system True Positive Rate across all four environmental conditions. AFODS maintains >95% TPR throughout; the baseline collapses to 21.4% at night and 32.1% in fog.

📝 Note on Published PDF: Due to a layout rendering glitch by the publisher in the final printed PDF of the manuscript, the chart axes and numeric labels in the journal's article layout were shifted out of alignment. The underlying experimental data is completely and correctly preserved in Table 1 of the published paper and is accurately displayed in the corrected chart above.

Resource	DOI
Key Figures & Methodology
Operational Sequence Video
Archived Source Code

AFODS is designed to target ASIL B compliance under ISO 26262, leveraging redundant sensor modalities to mitigate single-point failures:

• Spatial Detection: YOLOv7-Tiny trained on 15,000+ images (70/15/15% split) — identifies human shapes and thermal signatures including a dynamic Hypothermia Detection Mode
• Predictive Kinematics: GRU-based RNN analyses pose sequences over a 1–2 s window to identify pre-fall indicators (staggering, loss of balance) before collapse is complete
• Depth & Motion: SGM stereo algorithm for disparity mapping + Lucas–Kanade optical flow for vertical motion tracking + lightweight human pose estimation
• Acoustic Verification: MFCC-based classification (CNN/RNN) distinguishes fall acoustic signatures from ambient road noise — used as a final corroborating factor
• Processing: NVIDIA Jetson AGX Orin — target pipeline latency under 50 ms
• Vehicle Integration: CAN bus interface → emergency braking command + driver alerts

The combined detection probability $P_d$ is dynamically adjusted based on environmental conditions (e.g., in heavy rain, LWIR and ultrasonic weights are increased as NIR degrades):

$$P_d = w_{LWIR} \cdot C_{thermal} + w_{NIR} \cdot C_{visual}$$

where $w$ represents the confidence weight and $C$ represents the classifier's confidence score.

The braking decision is triggered when TTC < 1.2 s with detection confidence > 95%:

$$TTC = \frac{D}{v_{vehicle}}$$

where $D$ is the depth to the object from the SGM algorithm (m) and $v_{vehicle}$ is host vehicle speed (m/s).

Using NIR stereo disparity, the system calculates vertical velocity ($V_z$) to distinguish a fall from standard pedestrian motion:

$$V_z = \frac{\Delta d}{\Delta t} \cdot \frac{B \cdot f}{Z^2}$$

where $B$ is the stereo baseline (25 cm) and $f$ is the focal length of the NIR stereo pair.

All metrics are drawn directly from the peer-reviewed publication (Tables 1 & 2). The validation used standardised adult and child ATDs (50th percentile male; 6-year-old child) deployed via pneumatic rig at 20 m, tested at 30 and 50 km/h. Each condition was repeated 20 times.

Mean System Latency: 46.3 ms (SD = 4.1 ms) across all 320 trials
Mean Detection Range: AFODS 41.5 m (SD = 4.8 m) vs. Baseline 22.3 m (SD = 12.5 m) — t(158) = 15.72, p < 0.001

AFODS targets ASIL B compliance. The hazard classification for the detection of fallen pedestrians is:

• Severity (S3): Life-threatening or fatal injuries
• Exposure (E3): Significant occurrence in urban and nighttime environments
• Controllability (C0): Near-zero ability for a driver to avoid the hazard once within the detection gap

The decision logic requires corroboration between thermal signatures, visual object classification, and motion analysis before any braking action is triggered, directly addressing ISO 26262 requirements for fault tolerance and system reliability. ANOVA confirmed that environmental condition had minimal effect on AFODS performance (F(3,156) = 2.8, p = 0.042) versus a highly significant effect on the baseline (F(3,156) = 112.4, p < 0.001).

This repository is Part 1 of a unified 4-paper road safety research program:

• AFODS-Sensor-Fusion-Code — YOLOv7 and GRU model scripts
• AFODS-Operational-Sequence — Five-stage pipeline diagrams and graphical abstract
• From-Post-Mortem-to-Prevention-AFODS — ISO 26262-aligned conceptual framework
• Leveraging-Multi-Modal-AI-and-ISO-26262-to-Protect-Vulnerable-Road-Users — Safety framework

@article{vehicles7040149,
  author    = {Barua, Nick and Hitosugi, Masahito},
  title     = {Advanced Multi-Modal Sensor Fusion System for Detecting Falling Humans:
               Quantitative Evaluation for Enhanced Vehicle Safety},
  journal   = {Vehicles},
  volume    = {7},
  number    = {4},
  pages     = {149},
  year      = {2025},
  doi       = {10.3390/vehicles7040149},
  url       = {https://doi.org/10.3390/vehicles7040149}
}

This project is licensed under the Apache 2.0 License — see the LICENSE file for details.

The system described in this repository is subject to Japanese Patent Application No. 2025-167440 (Filed: 3 October 2025).