VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking of Highly Similar Objects

Official implementation of VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects.

In scenes crowded with highly similar objects (e.g., honeybees), instance-level re-ID embeddings lose their discriminative power: every candidate looks alike, and appearance cues collapse into the component shared by the whole frame. VLA-ReID defines the identity evidence of each trajectory relative to the candidate set of the current frame: Frame-Common Appearance Estimation (FCAE) estimates the appearance component shared by all candidates in a frame, and Common Appearance Suppression (CAS) removes it, so association is driven by the residual, discriminative part of appearance. The module is trained with video-level episodes that mirror the online association task, and plugs into TrackTrack by replacing only its appearance term — detector, motion model, matching procedure, and trajectory management are untouched.

Highlights

• Set-relative appearance: identity evidence is computed against the current frame's candidate set, not in isolation — designed for objects that are nearly indistinguishable one-by-one.
• Video-level episode training: the encoder and the FCAE+CAS modules are trained on trajectory-history vs. frame-candidate-set episodes, consistent with how the tracker uses them at inference.
• Drop-in: only the appearance term of the association cost changes; everything else in the tracker is stock.
• Byte-exact reproducibility: the released weights, detections, and commands reproduce the reported numbers exactly (verified end-to-end on a fresh clone).

BEE24 test (official 5-sequence split, shared YOLOX detections):

Method	HOTA↑	MOTA↑	IDF1↑	AssA↑	AssR↑	IDs↓
TrackTrack (baseline)	48.05	65.69	64.12	45.91	54.90	606
VLA-ReID (ours)	49.22	67.02	64.93	46.64	61.72	437

Repository layout

Dir	Stage	Role
detector/	1	YOLOX-X bee detector: training + detection export (dual-NMS pickles)
preprocessing/	2a	Detection-crop preprocessing: SAM3 foreground removal + orientation alignment
reid/	2b	FastReID (AGW-ResNeSt50) encoder: stage-1 training + per-detection feature extraction
srie/	3	Set-relative module (FCAE + CAS): model code + stage-2 video-level episode training
tracker/	4	TrackTrack association with FCAE+CAS injection + AFLink + TrackEval evaluation

Pipeline: BEE24 frames → detector → detection pickles → (preprocessing + reid) → feature pickles → tracker → HOTA / MOTA / IDF1

1. Installation

Tested with Python 3.11, PyTorch 2.1.2, CUDA 11.8, on Linux.

conda create -n vla-reid python=3.11 -y
conda activate vla-reid
pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Feature extraction additionally needs the SAM3 stack (install SAM3 per the official Meta AI instructions):

pip install timm==1.0.27 transformers==4.32.1
# + the sam3 package (0.1.4)

reid/bee_ext_feats.py includes a small compatibility shim so SAM3 also runs on torch 2.1.2 (SAM3 itself calls torch.compiler.is_dynamo_compiling, which newer torch versions provide natively).

2. Data preparation

Download BEE24 from the official page and unpack it next to the code:

VLA-ReID/
├── dataset/
│   └── BEE24/
│       ├── train/BEE24-01/{img1,gt,seqinfo.ini} ...
│       └── test/BEE24-12 BEE24-13 BEE24-16 BEE24-34 BEE24-36

The official test split is BEE24-12/13/16/34/36 (tracker/trackeval/seqmap/bee24/test.txt).

Only needed to re-train the detector — convert GT to COCO json:

cd detector
python tools/convert_bee24_topictrack.py   # writes detector/jsons/annotations/{train,test}.json

3. Weights

Download all release assets from huggingface.co/holmescao/VLA-ReID and place them as listed in WEIGHTS.md (SHA-256 manifest). In short: most files go to weights/, the orientation classifier goes to preprocessing/models/, the detection pickles go to outputs/1. det/.

4. Reproduce the main result

4.1 Detections

Use the released detection pickles (recommended — they are the exact "shared detections" protocol input): place bee24_test_0.80.pickle and bee24_test_0.95.pickle in outputs/1. det/.

Or regenerate them with the released detector weights (GPU):

cd detector
bash get_bee24_det.sh   # runs both NMS 0.80 and 0.95 exports

4.2 Per-detection re-ID features

Extract features for both NMS thresholds with the VLA-ReID encoder. SAM3 preprocessing is compute-heavy (several hours for the full test set on one GPU; shard with --shard_id/--num_shards across GPUs to speed up — shards are dicts, merge by union):

cd reid
for NMS in 0.80 0.95; do
python bee_ext_feats.py \
  --dataset BEE24 --data_path ../dataset/BEE24/test/ \
  --pickle_path "../outputs/1. det/bee24_test_${NMS}.pickle" \
  --output_path "../outputs/2. det_feat/vla/bee24_test_${NMS}.pickle" \
  --config_path configs/bee/AGW_S50.yml \
  --weight_path ../weights/vla_reid_encoder.pth \
  --crop_preprocess_mode foreground_aligned \
  --crop_preprocess_root ../preprocessing \
  --foreground_provider sam3 --foreground_device cuda \
  --sam3_checkpoint ../weights/sam3_bee24_finetuned.pt \
  --sam3_prompt bee --sam3_strategy center_prior \
  --sam3_center_frac 0.40 --sam3_center_overlap_threshold 0.01 \
  --sam3_close_px 3 --sam3_dilate_px 1 \
  --crop_preprocess_batch_size 1 --background_value 0 \
  --crop_preprocess_orientation_border_value 0 --orientation_target head-left \
  --reid_crop_width 384 --reid_crop_height 384 --reid_resize_mode resize \
  --seed 10000
done

Notes:

• The orientation classifier preprocessing/models/bee_direction_resnet18_triclass.joblib must be in place (it is picked up automatically); the documented command fails with FileNotFoundError if it is missing.
• Every extraction log must report sam3_failed_masks: 0 in the "Bee crop preprocessing stats" line; a non-zero value means SAM3 silently failed on some crops and the features are invalid.
• Alternatively, download the precomputed feature pickles (see WEIGHTS.md) and skip this step.

4.3 Tracking + evaluation

cd tracker
python run.py --dataset BEE24 --mode test \
  --pickle_dir "../outputs/2. det_feat/vla/" \
  --output_dir "../outputs/3. track/vla/" \
  --data_dir ../dataset/ \
  --seqmap_file trackeval/seqmap/bee24/test.txt \
  --aflink_weight ../weights/aflink.pth \
  --det_thr_override 0.75 --init_thr_override 0.80 --match_thr_override 0.70 \
  --seed 10000 --force_max_time_lost 3 \
  --srie_ckpt ../weights/vla_reid_cas.pth --srie_mode replace --srie_norm minmax

Expected result summary (written to ../outputs/3. track/vla/bee24_test_0.80_post/metrics.json; the IDs column of the paper corresponds to the IDSW key):

HOTA 49.22   MOTA 67.02   IDF1 64.93   AssA 46.64   IDSW 437

Notes:

• --force_max_time_lost 3 (short track buffer) and the three threshold overrides are part of the reported configuration — without them the tracker silently falls back to different defaults and the numbers will not match.
• The tracker is deterministic: repeated runs produce identical results.

4.4 Baseline (TrackTrack, same detections)

Extract features with the stage-1 encoder — same command as 4.2 with --weight_path ../weights/reid_stage1_agw_s50.pth and --output_path "../outputs/2. det_feat/baseline/..." — then run 4.3 with --pickle_dir "../outputs/2. det_feat/baseline/" and without the three --srie_* arguments → HOTA 48.05.

5. Training from scratch (optional)

5.1 Detector (YOLOX-X, 80 epochs)

cd detector
# weights/yolox_x.pth = COCO-pretrained YOLOX-X from the official YOLOX release
python tools/convert_bee24_topictrack.py
python tools/train.py -f exps/yolox_x_bee24_train.py -d 1 -b 4 --fp16 -o -c weights/yolox_x.pth

5.2 Stage-1 re-ID encoder (AGW-ResNeSt50, from scratch, 120 epochs, 8 GPUs)

Training uses BEE24 re-ID crops (from the BEE24 release) preprocessed with the same SAM3 foreground alignment (preprocessing/scripts/preprocess_bee_reid_crops.py --mode foreground_aligned with the SAM3 provider). Point BEE24_REID_DATA_DIR at the preprocessed crop root (train/query/test dirs):

cd reid
BEE24_REID_DATA_DIR=<path-to-preprocessed-reid-crops> python train_net.py --num-gpus 8 \
  --config-file configs/bee/AGW_S50.yml \
  MODEL.WEIGHTS "" \
  MODEL.BACKBONE.PRETRAIN False \
  SOLVER.AMP.ENABLED False \
  SOLVER.MAX_EPOCH 120 \
  SOLVER.IMS_PER_BATCH 256 \
  SOLVER.BASE_LR 0.0014 \
  SOLVER.WARMUP_ITERS 1000 \
  INPUT.FLIP.ENABLED False \
  INPUT.REA.ENABLED False \
  INPUT.RESIZE_MODE resize \
  SEED 10000 \
  OUTPUT_DIR outputs/stage1

outputs/stage1/model_final.pth is the baseline encoder; the epoch-60 checkpoint (model_0059.pth) is the init for stage 2.

5.3 Stage-2 video-level episode training (FCAE + CAS, 3 epochs, 8 GPUs)

Build the episode dataset and index from the BEE24 train split (uses the same SAM3-preprocessed crops):

python srie/tools/build_srie_video_dataset.py --dataset-root dataset/BEE24 --split train \
  --output-root dataset/episodes
python srie/tools/prepare_video_reid_episode_index.py --episode-root dataset/episodes \
  --output-dir outputs/episode_index --max-episodes 800 --val-ratio 0.15

The released checkpoint was trained on an 800-episode index (680 train / 120 val, tail-by-seq split, seed 10000). Then:

torchrun --standalone --nnodes=1 --nproc_per_node=8 srie/tools/train_srie_episode.py \
  --index-dir outputs/episode_index \
  --config-file reid/configs/bee/AGW_S50.yml \
  --init-weight weights/reid_stage1_agw_s50_ep60.pth \
  --use-srie --srie-components cas \
  --epochs 3 --max-train-episodes 0 --max-val-episodes 120 \
  --tracklet-batch-size 6 --history-sample-stride 4 --history-chunk-size 8 --det-chunk-size 8 \
  --image-workers 24 --lr 1e-4 --srie-lr 1e-3 --temperature 0.1 --seed 10000 --log-every 50 \
  --output-dir outputs/stage2 \
  --opts MODEL.BACKBONE.PRETRAIN False

outputs/stage2/model_epoch003.pth is the tracker-injection checkpoint (equivalent to vla_reid_cas.pth). Export the plain encoder for feature extraction:

import torch
ck = torch.load("outputs/stage2/model_epoch003.pth", map_location="cpu")
torch.save(ck["model"], "outputs/stage2/encoder_for_eval.pth")

Citation

If you find VLA-ReID useful, please cite:

@misc{qin2026vlareidvideolevelassociationreidentification,
      title={VLA-ReID: Video-Level Association for Re-Identification in Multi-Object Tracking with Highly Similar Objects},
      author={Yanrong Qin and Xiaoyan Cao and Yao Yao},
      year={2026},
      eprint={2607.17157},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.17157},
}

If you use the BEE24 benchmark or the shared-detections protocol, please also cite:

@ARTICLE{10851814,
  author={Cao, Xiaoyan and Zheng, Yiyao and Yao, Yao and Qin, Huapeng and Cao, Xiaoyu and Guo, Shihui},
  journal={IEEE Transactions on Image Processing},
  title={TOPIC: A Parallel Association Paradigm for Multi-Object Tracking Under Complex Motions and Diverse Scenes},
  year={2025},
  volume={34},
  pages={743-758},
  doi={10.1109/TIP.2025.3526066}}

Acknowledgements

This repository builds on TrackTrack (MIT), YOLOX (Apache-2.0), fast-reid (Apache-2.0), TrackEval (MIT), and SAM3 (Meta AI Research). BEE24 is introduced by TOPIC (IEEE TIP 2025). See NOTICE.md.

License

MIT for the code in this repository; vendored components keep their upstream licenses (see NOTICE.md).