Skip to content

[McByte part 4] Add Cutie mask propagator#452

Open
tstanczyk95 wants to merge 2 commits into
feat/mcbyte-sam-mask-generatorfrom
feat/mcbyte-cutie-mask-propagator
Open

[McByte part 4] Add Cutie mask propagator#452
tstanczyk95 wants to merge 2 commits into
feat/mcbyte-sam-mask-generatorfrom
feat/mcbyte-cutie-mask-propagator

Conversation

@tstanczyk95

Copy link
Copy Markdown
Collaborator

What does this PR do?

This PR adds a Cutie-based mask propagation component for McByte.

The new CutieMaskPropagator initializes Cutie memory from masks generated on a reference frame and propagates those masks to subsequent frames. The implementation integrates Cutie as an external dependency and follows the existing mask interfaces used by McByte.

Main additions:

  • Added CutieMaskPropagator
  • Added automatic Cutie weight download with checksum verification
  • Added utilities for:
    • mask format conversion between McByte and Cutie
    • tracklet-to-object ID mapping
    • mask confidence computation
    • Cutie package/config discovery
  • Added validation checks for mask shapes and tracklet-mask mappings
  • Added unit tests covering helper utilities and propagator behavior
  • Added a visual integration script for manual validation of SAM initialization and Cutie mask propagation

Type of Change

  • New feature (non-breaking change that adds functionality)

Testing

  • I have tested this change locally
  • I have added/updated tests for this change

Test details:
Added unit tests covering:

  • RGB image to Cutie tensor conversion
  • Cutie probability tensor conversion to indexed masks
  • Conversion between indexed and binary mask formats
  • Non-overlapping mask generation
  • Tracklet-to-object mapping construction
  • Mask confidence computation
  • Propagator state reset behavior
  • Input validation during initialization

Manual validation:

  • Tested CutieMaskPropagator initialization on CPU
  • Tested CutieMaskPropagator initialization on CUDA
  • Verified automatic checkpoint download
  • Verified Hydra configuration loading
  • Verified end-to-end SAM to Cutie propagation using the visualization script on a multi-frame sequence

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in hard-to-understand areas
  • My changes generate no new warnings or errors
  • [McByte as a whole is still in the development phase] I have updated the documentation accordingly (if applicable)

Additional Context

This PR is stacked on top of the SAM mask generator PR #441 and uses the masks produced by SAMBoxMaskGenerator for Cutie initialization.

It requires installation of torch, torch vision and SAM as specified in #441 (section Additional Context).

Furthermore, Cutie must be installed in the following manner (adapted steps from the original repo):

  • Go to your parent folder of trackers.
  • git clone https://github.com/hkchengrex/Cutie.git
  • cd Cutie
  • Open Cutie/pyproject.toml. Remove: 'cchardet >= 2.1.7', (whole line 43).
  • pip install -e .

Tested on CPU and on GPU (CUDA 12.6) - separate Anaconda environments.

Visual test performed on SoccerNet-tracking:

  • sequence SNMOT-116
  • frames from from 000548.jpg to 000750.jpg (inclusive)
  • oracle detections converted to xyxy format

Example test run for the visual test (you can also run it just with a single bounding box):

python visual_tests/visualize_cutie_mask_propagator.py \
  --image-path your_dataset_path/SoccerNet/tracking/test/SNMOT-116/img1/000548.jpg \
  --start-file 000548.jpg \
  --end-file 000750.jpg \
  --box 177,482,249,631 \
  --box 1241,569,1297,716 \
  --box 191,344,240,459 \
  --box 686,298,751,426 \
  --box 625,469,686,623 \
  --box 1382,378,1440,497 \
  --box 1220,480,1240,498 \
  --device cuda 

Analogously to SAMBoxMaskGenerator in #441, when CutieMaskPropagator is instantiated, it checks if the pre-trained Cutie weights file is already saved in the pointed location. If the model not saved, then it downloaded. Default location for the pre-trained model: trackers_root/models/cutie/cutie-base-mega.pth. Weighs from here.

@tstanczyk95 tstanczyk95 requested a review from SkalskiP as a code owner June 8, 2026 16:06
@tstanczyk95 tstanczyk95 requested a review from Borda June 8, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant