[McByte part 4] Add Cutie mask propagator#452
Open
tstanczyk95 wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds a Cutie-based mask propagation component for McByte.
The new
CutieMaskPropagatorinitializes Cutie memory from masks generated on a reference frame and propagates those masks to subsequent frames. The implementation integrates Cutie as an external dependency and follows the existing mask interfaces used by McByte.Main additions:
Type of Change
Testing
Test details:
Added unit tests covering:
Manual validation:
Checklist
Additional Context
This PR is stacked on top of the SAM mask generator PR #441 and uses the masks produced by SAMBoxMaskGenerator for Cutie initialization.
It requires installation of torch, torch vision and SAM as specified in #441 (section Additional Context).
Furthermore, Cutie must be installed in the following manner (adapted steps from the original repo):
trackers.git clone https://github.com/hkchengrex/Cutie.gitcd CutieCutie/pyproject.toml. Remove:'cchardet >= 2.1.7',(whole line 43).pip install -e .Tested on CPU and on GPU (CUDA 12.6) - separate Anaconda environments.
Visual test performed on SoccerNet-tracking:
Example test run for the visual test (you can also run it just with a single bounding box):
Analogously to
SAMBoxMaskGeneratorin #441, whenCutieMaskPropagatoris instantiated, it checks if the pre-trained Cutie weights file is already saved in the pointed location. If the model not saved, then it downloaded. Default location for the pre-trained model:trackers_root/models/cutie/cutie-base-mega.pth. Weighs from here.