Cotracker usage during training process

[Tomy20Wang]

Wonderful work you guys did!
I have read your paper, as you mentioned in the paper "Then, we employ Co-Tracker [23] to track these control points to generate corresponding point trajectories as the ground truth"
and I want to know how to use Co-Tracker? did you use it before training or during training?

[Tomy20Wang]

if you were using Co-Tracker during training, what was the training-time cost?
THANKS!

[pPetrichor]

Thank you for your question. We did not use CoTracker during the training process. We just used the trajectories extracted by CoTracker during evaluation as the ground truth for calculating the ObjMC metric.

[Tomy20Wang]

Oh! Maybe I didn't get it right
So during the training process, how did you get the control points(with K-means method) for the entire video. If you just use K-means to get the control points in the first frame, how did you track them through the video.
THANKS FOR YOUR EXPLAINATION!

[pPetrichor]

Hello, during the training process, since we used the SAM2 dataset which includes mask annotations for videos, we directly performed clustering on the masks of each video frame to obtain KMeans clustering points as control conditions for video generation.

[Tomy20Wang]

So there is no connection among these control points from different frames?
Then how did you locate the number of masks and their coordinates for each mask image? Does the SA-V dataset annotate the coordinates for each mask in each image? Or did you directly use the K-means method for the mask regions without considering how many masks there are in one video
sorry for so much questions…… but I want to know how it worked

[Tomy20Wang]

I reviewed the scheme in gradio_run.py in the code. Did you go through this process once for all the data in the dataset

[pPetrichor]

Hello, as I recall, the SAM2 dataset's sav file should contain mask information for different objects arranged in order. Therefore, the annotations inherently include object mask ID information. Our data preprocessing simply reads this information and applies k-means clustering on the masks of each frame to obtain control points. The ID information of these control points still originates from the annotations in the sav data, which allows us to establish correspondences of control points across different frames.