Skip to content

feat(streampetr): harden the streaming training infrastructure - #3

Draft
vividf wants to merge 1 commit into
feat/streampetr-base-configsfrom
feat/streampetr-training-infra
Draft

feat(streampetr): harden the streaming training infrastructure#3
vividf wants to merge 1 commit into
feat/streampetr-base-configsfrom
feat/streampetr-training-infra

Conversation

@vividf

@vividf vividf commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

Hardens the training infrastructure that the production StreamPETR recipes rely on. Four independent pieces, grouped because they are all training-loop plumbing.

Changes

  • Partial-annotation status plumbing (datamodule/common/multiview_detection3d.py + subclasses): a configurable annotation_status_field names the per-frame annotation-completeness field in the info files; samples expose it under the generic annotation_status key. Nothing dataset-specific is hardcoded in the shared layer, and datasets without the field emit no key at all.
  • Scene-contiguity validation: prev_exists is derived from file-order adjacency, so an annotation file that interleaves scenes would silently reset temporal memory mid-scene — dataset construction now fails loudly instead.
  • Checkpoint loading (utils/checkpoints.py): shared-tensor alias detection (identity by storage) so modules registered twice — e.g. a backbone reachable both directly and through a feature extractor — pass full-coverage checks; shape mismatches still refuse to load with a per-key report.
  • Epoch-end validation loop (utils/lightning_loops.py): Lightning derives val_check_batch once from the dataloader length at setup; variable-length samplers (the streaming sampler trims differently per epoch) make shorter epochs silently skip validation, feeding stale metrics to the checkpoint monitor. The override triggers validation on the true last batch of a due epoch. Step-based validation configs (check_val_every_n_epoch=None) are left untouched. A canary test signals when the upstream fix lands so the workaround can be retired.
  • IterWarmupEpochCosineLR: 500-iteration linear warmup multiplied by per-epoch cosine decay (the mmengine chained-scheduler recipe). Closed-form over the step counter, so checkpoint resume replays the original schedule even when the framework re-estimates total_steps; covered by a resume-trajectory test.

Testing

Unit tests for every piece (serialization dtype round-trips, contiguity guard, alias loading, loop behavior incl. the upstream canary, scheduler shape/boundary/resume). Full suite green at this commit.

Stack 5/8 — base: feat/streampetr-base-configs.

@vividf
vividf force-pushed the feat/streampetr-training-infra branch from ce301f5 to 6579398 Compare September 2, 2026 08:43
@vividf
vividf changed the base branch from feat/streampetr-core to feat/streampetr-base-configs September 2, 2026 08:43
@vividf
vividf force-pushed the feat/streampetr-base-configs branch from 76d282b to e74d99d Compare September 2, 2026 08:57
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 6579398 to 57c4562 Compare September 2, 2026 08:57
@vividf
vividf force-pushed the feat/streampetr-base-configs branch from e74d99d to edddb29 Compare September 2, 2026 09:31
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 57c4562 to 428e760 Compare September 2, 2026 09:31
@vividf
vividf force-pushed the feat/streampetr-base-configs branch from edddb29 to 45c00fa Compare September 2, 2026 21:02
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 428e760 to 2228774 Compare September 2, 2026 21:02
@vividf
vividf marked this pull request as draft September 3, 2026 17:24
@vividf
vividf force-pushed the feat/streampetr-base-configs branch from 45c00fa to a326226 Compare September 4, 2026 06:24
@vividf
vividf force-pushed the feat/streampetr-training-infra branch 2 times, most recently from 4a2e1bf to 5af526a Compare September 4, 2026 07:29
@vividf
vividf force-pushed the feat/streampetr-base-configs branch from a326226 to 16e5c4b Compare September 4, 2026 07:29
Partial-annotation status plumbed through the multiview datasets, scene
contiguity validated at load time, shared-tensor-aware checkpoint loading,
epoch-end validation for variable-length streaming samplers, and the
iteration-warmup epoch-cosine LR schedule.

Signed-off-by: vividf <yihsiang.fang@tier4.jp>
@vividf
vividf force-pushed the feat/streampetr-base-configs branch from 16e5c4b to c53556a Compare September 4, 2026 08:00
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 5af526a to 7a02cde Compare September 4, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant