scene: accuracy vs. real-time trade-off in the ConceptGraphs perception pipeline, and this week's tuning plan

[enkerewpo]

cc @ForeverYolo

Context

The perception entry point of system/scene, scene_service/ingest/perception_concept_graphs.py, builds an open-vocabulary object-level scene representation on top of ConceptGraphs (Gu et al., ICRA 2024; see References). Mapping accuracy is currently not good enough: the same physical object is registered more than once, bounding boxes inflate, and objects of different classes get merged.

Subproject 3 will port this pipeline to RK3588. Before that starts we have to separate "problems in our own implementation" from "problems introduced by the port", otherwise nothing that goes wrong on the board can be attributed. This issue records the differences we have located against upstream, the analysis, and the tuning plan for this week.

Sequencing: model/parameter work first, RK3588 adaptation after

This week @enkerewpo will revisit model selection and the association parameters, and produce results inside the test framework. This work is deliberately not run concurrently with the RK3588 adaptation.

The reason is that the model choice and the ML framework underneath it are themselves still under test. If the model tier, the runtime and the association parameters are all moving while the port is in progress, a regression on the board cannot be attributed to any one of them. So the intended order is: settle the model tier and the parameters on the development machine against a fixed benchmark, freeze them, and only then adapt the frozen configuration to RKNN on RK3588.

@ForeverYolo — please treat sections 2 and 3 below as "known issues in the current implementation, not something you introduced". Section 6 is the acceptance criterion we need in place before the port starts.

1. We benchmarked against the wrong upstream pipeline

The upstream repository ships two different pipelines:

• conceptgraph/slam/cfslam_pipeline_batch.py — the offline two-stage pipeline used for the paper's evaluation. Detections are produced ahead of time by scripts/generate_gsa_results.py, pickled into a gsa_detections_*/ directory, and the mapping stage simply reads them back from disk. Node captioning runs an LVLM plus GPT-4 after the entire sequence has been processed. This pipeline was never designed to run in real time; Tables I and II of the paper come from it.
• conceptgraph/slam/realtime_mapping.py — upstream's own online variant, with the models loaded in-process and frames handled one at a time.

Our implementation corresponds to the second pipeline, but our parameter baseline has been taken from the first one's configs/slam_pipeline/base.yaml. The two make very different trade-offs, and mixing them is the origin of most of what follows.

2. Model selection: only one of our substitutions was wrong

Stage	Upstream realtime_mapping.py	Current dev	Verdict
Detection	YOLO('yolov8l-world.pt') (YOLO-World)	same	Identical, no change needed
Segmentation	SAM('sam_l.pt') (SAM), with SAM('mobile_sam.pt') (MobileSAM) given as an alternative on the same line	MobileSAM	An alternative upstream itself offers; acceptable, but the cost has to be accounted for
Visual features	open_clip ViT-H-14 / laion2b_s32b_b79k (CLIP / OpenCLIP, LAION-2B checkpoint)	ViT-B-32	The one wrong cut

To go online, upstream already dropped the detector to yolov8l-world and kept segmentation at sam_l — but even in the real-time variant it did not touch CLIP. The reason is structural: ConceptGraphs puts instance discrimination entirely on the CLIP cosine similarity. compute_visual_similarities in conceptgraph/slam/mapping.py returns F.cosine_similarity(det_ft, obj_ft) directly, with no re-training and no calibration.

ViT-B-32 (image tower ≈ 88M parameters) produces a heavily compressed cosine distribution over in-domain indoor crops. "Different instances of the same class" and "the same instance from a different viewpoint" become nearly indistinguishable. Once the semantic term stops carrying information, the whole association chain is left with geometry alone.

3. What the weaker feature forced downstream

With the semantic term unusable, the implementation was pushed into changing the structure of association, not just its thresholds:

Item	Upstream	Current dev	Effect
match_method	sep_thresh (semantic ≥ 0.5 and geometric ≥ 0.5)	sim_sum (addition)	A conjunction became a sum, so one strong term alone can carry a match
Association threshold	1.1 in the paper	0.85	Loosened further
obj_min_detections	3	1	Upstream's most effective false-positive filter is switched off; a single-frame misdetection stays in the map forever
Per-detection DBSCAN (Ester et al., 1996)	On — Section II-A of the paper states each masked region is denoised after back-projection and before fusion	run_dbscan=False	Depth-boundary fliers enter the object point cloud, boxes inflate, and the geometric term is corrupted as well
merge_overlap_thresh	0.7	0.5	Loosened
merge_visual_sim_thresh	0.7	0.65	Loosened
min_points_threshold	16	50	Tightened; thin objects are dropped

On top of that we added six hand-written gates that upstream does not have: max_merge_dist_m, the same-class gate plus SCENE_CG_MERGE_CLASS_GROUPS, cross_class_centroid_max_m, the periodic forced merge driven by cross_class_iou_thresh / cross_class_overlap_thresh, the same-class proximity collapse driven by same_class_merge_dist_m, and a per-class floor-height filter.

Every threshold in that layer was tuned by eye on a single scene, and they are coupled to each other. This is where the accuracy is actually being lost — not in any single number.

There is one further structural cost. Upstream's Seg(·) is class-agnostic: the class label is metadata and discrimination is left to CLIP. We use a closed 56-class YOLO-World vocabulary and promoted the class label into a hard gate on merging. Detector label flicker is therefore amplified directly into mapping errors — one wall cabinet labelled picture_frame / shelf / cabinet across frames becomes three records. Objects outside the vocabulary have identically zero recall, which no amount of parameter tuning can recover.

4. The compute budget needs to be recomputed

The earlier reasoning behind "shrink CLIP so it runs in real time" does not hold up:

• SAM produces masks over the full image every frame. Its cost scales with image resolution and is the fixed dominant term per tick.
• CLIP runs once per detection crop at a fixed 224×224 input. Its cost is per-crop model cost × number of crops per tick, and is independent of image resolution. max_detections is currently capped at 30.

CLIP's cost therefore has two factors, and we have been cutting the first one — model quality, which is exactly what the association depends on — while leaving the second untouched. Ways to cut the second factor: take the top-K crops by confidence or area, compute features only for detections that did not match this tick, and reuse the stored clip_ft for objects that did.

The constraint on RK3588 is genuinely tighter (a 6 TOPS INT8 NPU, shared LPDDR bandwidth, and limited attention-operator coverage in RKNN — see the RKNN Toolkit 2 and RKNN Model Zoo links below), and ViT-H-14 most likely will not fit. What is needed here is a measurement rather than another guess: run ViT-B-32 / ViT-L-14 / ViT-H-14 over the same recorded sequence, record per-tick latency and association quality for each, and pick the tier from that curve instead of defaulting to the smallest one.

5. Tuning plan for this week

Ordered as "restore upstream behaviour first, optimise second". Each step is validated on its own and reported on its own:

1. obj_min_detections 1 → 3. A one-number change that restores upstream's most effective false-positive filter.
2. run_dbscan=True at back-projection, restoring the per-detection denoising of Section II-A. Once an association is wrong, the periodic denoise_objects pass cannot undo it.
3. Raise CLIP to ViT-L-14 and measure per-tick latency separately on Jetson and on RK3588.
4. After step 3 is in effect, try match_method back at sep_thresh and remove the six hand-written gates from section 3 one at a time, checking which are no longer needed once the feature is stronger. The goal is for the number of gates to decrease monotonically.
5. Move CLIP-side cost reduction from "lower the model tier" to "reduce the crop count" (top-K, skip already-matched objects, reuse clip_ft).
6. Return the class label to a metadata role rather than a hard merge gate; at minimum make the same-class gate switchable so it can be A/B tested.

6. Prerequisite: a benchmark we can actually decide on

There is no ground truth today, so none of the steps above can be shown to be an improvement. This is also the single largest risk for the RK3588 port: on the board it would be impossible to tell "the port is wrong" from "it was always like this".

The paper evaluates on the Replica dataset with human raters on Amazon Mechanical Turk, and reports node precision 0.71, edge precision 0.88, and 0–5 duplicate objects per scene. We need an equivalent:

• A fixed recorded sequence from the lab (RGB-D plus poses), checked in as the regression input.
• A hand-annotated list of the objects that should appear in that sequence, with approximate positions.
• Three metrics: object precision, duplication rate (how many records one physical object is split into), and recall.
• Every parameter or model change reruns the same sequence and reports the same three numbers, and that is the acceptance criterion.

This has to be in place before work on the RK3588 port starts.

References

Papers

1. Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, et al. ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. ICRA 2024. arXiv:2309.16650 — https://arxiv.org/abs/2309.16650
2. A. Kirillov, E. Mintun, N. Ravi, et al. Segment Anything. ICCV 2023. arXiv:2304.02643 — https://arxiv.org/abs/2304.02643
3. C. Zhang, D. Han, Y. Qiao, et al. Faster Segment Anything: Towards Lightweight SAM for Mobile Applications. arXiv:2306.14289 — https://arxiv.org/abs/2306.14289
4. T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang, Y. Shan. YOLO-World: Real-Time Open-Vocabulary Object Detection. CVPR 2024. arXiv:2401.17270 — https://arxiv.org/abs/2401.17270
5. A. Radford, J. W. Kim, C. Hallacy, et al. Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. arXiv:2103.00020 — https://arxiv.org/abs/2103.00020
6. M. Cherti, R. Beaumont, R. Wightman, et al. Reproducible Scaling Laws for Contrastive Language-Image Learning. CVPR 2023. arXiv:2212.07143 — https://arxiv.org/abs/2212.07143 (source of the laion2b_s32b_b79k checkpoint)
7. M. Ester, H.-P. Kriegel, J. Sander, X. Xu. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. KDD 1996 (DBSCAN)
8. J. Straub, T. Whelan, L. Ma, et al. The Replica Dataset: A Digital Replica of Indoor Spaces. arXiv:1906.05797 — https://arxiv.org/abs/1906.05797
9. Q.-Y. Zhou, J. Park, V. Koltun. Open3D: A Modern Library for 3D Data Processing. arXiv:1801.09847 — https://arxiv.org/abs/1801.09847

Repositories

• ConceptGraphs — https://github.com/concept-graphs/concept-graphs (relevant files: conceptgraph/slam/realtime_mapping.py, conceptgraph/slam/cfslam_pipeline_batch.py, conceptgraph/slam/mapping.py, conceptgraph/slam/utils.py, conceptgraph/configs/slam_pipeline/base.yaml)
• OpenCLIP — https://github.com/mlfoundations/open_clip
• Ultralytics (the YOLO and SAM wrappers we call) — https://github.com/ultralytics/ultralytics
• MobileSAM — https://github.com/ChaoningZhang/MobileSAM
• YOLO-World — https://github.com/AILab-CVC/YOLO-World
• PyTorch3D (ops.box3d_overlap, the upstream overlap spatial-similarity path we replaced) — https://github.com/facebookresearch/pytorch3d
• RKNN Toolkit 2 — https://github.com/airockchip/rknn-toolkit2
• RKNN Model Zoo — https://github.com/airockchip/rknn_model_zoo
• This file — system/scene/scene_service/ingest/perception_concept_graphs.py

Sites

• ConceptGraphs project page — https://concept-graphs.github.io/
• Ultralytics YOLO-World documentation — https://docs.ultralytics.com/models/yolo-world/
• Ultralytics SAM / MobileSAM documentation — https://docs.ultralytics.com/models/mobile-sam/
• OpenCLIP pretrained checkpoint list — https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv

[enkerewpo]

@Bunnycxk