HDFMambaVAD: A Hybrid Dual-Branch Mamba Framework for Unsupervised Video Anomaly Detection

HDF-MambaVAD

A Hybrid Dual-Branch Mamba Framework for Unsupervised Video Anomaly Detection

HDF-MambaVAD is a two-stage, object-centric framework for unsupervised video anomaly detection. The model is trained only on normal videos and detects anomalous events using optical-flow reconstruction and future-frame prediction errors.

Status: Manuscript in preparation. Results and implementation details may be updated during revision.

Method Overview

HDF-MambaVAD consists of two stages:

Stage 1: Motion Reconstruction

• Decomposes optical flow into magnitude and direction components.
• Encodes the two components using separate branches.
• Applies magnitude-guided feature fusion.
• Uses an EMA-based vector-quantised bottleneck to learn compact normal-motion representations.
• Reconstructs optical flow through a progressive decoder.

Stage 2: Future-Frame Prediction

• Processes RGB frames and reconstructed optical flow using two VMamba branches.
• Fuses appearance and motion features using iAFF.
• Uses Lite CONASkip to dynamically control decoder skip information.
• Predicts the next RGB frame from the historical RGB and motion sequences.

The final anomaly score combines normalised optical-flow reconstruction error and future-frame prediction error.

Main Contributions

• Two-stage object-centric reconstruction-prediction framework.
• Disentangled magnitude-direction optical-flow modelling.
• Magnitude-guided motion feature fusion.
• EMA-based vector-quantised motion bottleneck.
• Dual-branch VMamba prediction network.
• iAFF-based appearance-motion fusion.
• Lite CONASkip with reconstruction-error-based pseudo labels.

Results

Frame-level AUROC with temporal history length L = 16:

Dataset	AUROC
UCSD Ped2	98.57%
CUHK Avenue	92.50%
ShanghaiTech	74.07%

These results are based on the current manuscript experiments.

Datasets and Preprocessing

The framework is evaluated on:

• UCSD Ped2
• CUHK Avenue
• ShanghaiTech

The preprocessing pipeline:

1. Detects foreground objects using Cascade R-CNN.
2. Extracts optical flow using FlowNet2.0.
3. Crops RGB and optical-flow regions around detected objects.
4. Resizes object crops to 32 × 32.
5. Constructs object-level spatio-temporal cubes.

Preprocessing scripts are located in:

pre_process/
├── extract_bboxes.py
├── extract_flows.py
└── extract_samples.py

The datasets are not included in this repository.

Configuration

Main configuration files:

cfgs/ml_memAE_sc_cfg.yaml
cfgs/cfg.yaml

Update the dataset and device settings before training:

dataset_base_dir: ./data
dataset_name: avenue
device: cuda:0

Supported dataset names:

ped2
avenue
shanghaitech

Training

Stage 1

Configure:

cfgs/ml_memAE_sc_cfg.yaml

Run:

python ml_memAE_sc_train_edited.py

The best checkpoint is saved to:

ckpt/<stage1_experiment>/best.pth

Stage 2

Set the Stage-1 checkpoint in cfgs/cfg.yaml:

ML_MemAE_SC_pretrained: ./ckpt/<stage1_experiment>/best.pth

Run:

python train_edited.py

The Stage-1 reconstructor remains frozen while the VMamba predictor and Lite CONASkip classifier are trained.

Evaluation

python eval_edited.py \
    --model_save_path ./ckpt/<stage2_experiment>/best.pth \
    --cfg_file ./log/<stage2_experiment>/cfg.yaml

The evaluation pipeline computes reconstruction and prediction errors, normalises them using normal-training statistics, aggregates object-level scores through frame-level max pooling, and reports frame-level AUROC.

Requirements

Main dependencies include:

• Python 3.10+
• PyTorch
• CUDA
• OpenCV
• NumPy
• SciPy
• scikit-learn
• PyYAML
• tqdm
• TensorBoardX
• joblib
• VMamba selective-scan dependencies

A complete environment specification will be added later.

Code Attribution

HDF-MambaVAD was developed from and substantially modifies the official HF2VAD implementation.

The main modifications include:

• Magnitude-direction optical-flow reconstruction.
• Magnitude-guided fusion.
• Vector quantisation.
• Dual-branch VMamba prediction.
• iAFF feature fusion.
• Lite CONASkip.
• Revised training, evaluation and anomaly-scoring procedures.

Third-party components retain their original licenses and copyright notices.

Acknowledgements

This project builds upon the open-source implementations of HF2VAD and VADMamba.

Third-party components remain subject to their original licenses and copyright notices.

Citation

The manuscript is currently in preparation. Citation information will be added after a public preprint or publication becomes available.