Skip to content

feat(streampetr): project 2D annotations and fold augmentations into ego poses - #4

Draft
vividf wants to merge 1 commit into
feat/streampetr-training-infrafrom
feat/streampetr-2d-annotations
Draft

feat(streampetr): project 2D annotations and fold augmentations into ego poses#4
vividf wants to merge 1 commit into
feat/streampetr-training-infrafrom
feat/streampetr-2d-annotations

Conversation

@vividf

@vividf vividf commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

Everything the auxiliary 2D head needs from the data side, plus augmentation-consistent ego poses for the temporal memory.

Changes

  • LoadAnnotations2DFromBoxes3D (transforms/camera/annotations2d.py): projects the (already augmented) 3D gravity-center boxes onto every camera and emits per-camera gt_bboxes_2d / centers_2d / gt_labels_2d. Vectorized over boxes; corners behind the camera are masked out of the 2D extent; a box is kept only when its projected center lands inside the image (matching the reference recipe — a clamped center could fall outside its own clipped box and distort the center-based assignment); boxes are clipped to the image and zero-area slivers dropped. Runs after all geometric augmentations so the projection matrices and pixels agree.
  • Ego-pose folding (transforms/geometry3d.py + camera / camera-lidar geometry): a scene augmentation A updates ego_pose' = ego_pose @ A⁻¹ (and the inverse accordingly), keeping lidar→global consistent — exactly what StreamPETR's memory warp (ego_pose_inv(t) @ ego_pose(t-1)) needs under train-time augmentation.
  • Camera-order shuffling (transforms/camera/loading.py): per-frame train-time shuffle following the reference shuffle_cameras=True; images, intrinsics, and extrinsics are reordered together by construction.

Testing

Projective edge cases are unit-tested (box behind the camera, center leaving the image, partial visibility/clipping, multi-camera assignment, empty and malformed GT), plus ego-pose round-trip tests and a velocity-scaling regression test.

Stack 6/8 — base: feat/streampetr-training-infra.

@vividf
vividf force-pushed the feat/streampetr-training-infra branch from ce301f5 to 6579398 Compare September 2, 2026 08:43
@vividf
vividf force-pushed the feat/streampetr-2d-annotations branch from 6fc9075 to f94c6cb Compare September 2, 2026 08:43
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 6579398 to 57c4562 Compare September 2, 2026 08:57
@vividf
vividf force-pushed the feat/streampetr-2d-annotations branch 2 times, most recently from 262f7ee to 4e78b19 Compare September 2, 2026 09:31
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 57c4562 to 428e760 Compare September 2, 2026 09:31
@vividf
vividf force-pushed the feat/streampetr-2d-annotations branch from 4e78b19 to d6579bd Compare September 2, 2026 21:02
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 428e760 to 2228774 Compare September 2, 2026 21:02
@vividf
vividf marked this pull request as draft September 3, 2026 17:24
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 2228774 to 4a2e1bf Compare September 4, 2026 06:24
@vividf
vividf force-pushed the feat/streampetr-2d-annotations branch from d6579bd to 7ae5816 Compare September 4, 2026 06:24
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 4a2e1bf to 5af526a Compare September 4, 2026 07:29
@vividf
vividf force-pushed the feat/streampetr-2d-annotations branch from 7ae5816 to a750d87 Compare September 4, 2026 07:29
…ego poses

LoadAnnotations2DFromBoxes3D projects the augmented 3D boxes onto every
camera for the auxiliary 2D head; the camera geometric augmentations fold
their transforms into the ego poses so the temporal memory warp stays
consistent; train-time camera-order shuffling follows the reference recipe.

Signed-off-by: vividf <yihsiang.fang@tier4.jp>
@vividf
vividf force-pushed the feat/streampetr-training-infra branch from 5af526a to 7a02cde Compare September 4, 2026 08:00
@vividf
vividf force-pushed the feat/streampetr-2d-annotations branch from a750d87 to 2bdd538 Compare September 4, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant