Distilling Back to Image Encoder

[niklas030]

Hey, I'm currently writing my master's thesis. And I probably will investigate the possibility of distilling the features from a point cloud encoder to the image encoder for the AD domain. I believe they have nice properties such as view independence and strong geometric priors compared to the SOTA image FM.

Two questions: intuition about which input features should be active. Best performance with all, but the image FM was DINOv2. Probably going to take DINOv3, questioning myself if the colors are available to the model, if it will be a stronger prior to the DINOv2 features which are worse than DINOv3. Should I densify the results on nuScenes using models which take in images and point clouds to densify the PC, or are the sparse LiDAR PCs from nuScenes enough and they already encapsulate most of the information of this scene?

Thanks for your amazing work, I think your work will leverage a lot of things in the future. I'm also happy for any type of tips or problems which I might encounter if I'm distilling back to an image encoder???

[Yujia-Zhang0913]

Hi, thanks for your interest in our work!
Based on the others' feedback, distilling from our previous point encoder Concerto and Sonata to other backbones in AD helps.
For your questions, I think this choice depends on the experiments' results. The features from DINOv3 and DINOv2 are in quite different distributions, which may cause differences in training recipes, learning rate, and loss curves. And if you want better results, I think the image information is necessary.
Like the reply in #22 , I recommend using multiple-frame point clouds as input for distillation, and better with colors and normals.

[niklas030]

Yes, thanks for your answer Yujia! I also tried stacking the pointclouds from multiple frames. The results look better. Also, thanks for the vis in the other issue. I just have one concern about stacking the pointclouds. Don't you think that for dynamic objects the features might be noise? because diffrent parts of same object are at the same space.