Tracking feature : CoTracker integration for automated label generation#155
Tracking feature : CoTracker integration for automated label generation#155C-Achard wants to merge 27 commits intoDeepLabCut:mainfrom
Conversation
(cherry picked from commit 838154e)
(cherry picked from commit 63cd8e9)
(cherry picked from commit 244358a)
(cherry picked from commit e5d8ee6)
(cherry picked from commit 9b317d7)
(cherry picked from commit 0e0c605)
…roperties (cherry picked from commit f1c3122)
(cherry picked from commit 3b5780a)
(cherry picked from commit f56ee79)
(cherry picked from commit 195800b)
(cherry picked from commit dacf41e)
(cherry picked from commit e209958)
(cherry picked from commit 009aa95)
(cherry picked from commit 6ce0f8e)
(cherry picked from commit d175dcb)
Renamed the trajectory plot checkbox from _matplotlib_cb to _show_traj_plot_cb in KeypointControls for clarity. Updated all references and related logic accordingly. Also improved slider maximum logic in KeypointMatplotlibCanvas and made minor comment and test cleanup.
C-Achard
changed the title
There was a problem hiding this comment.
Pull request overview
This PR introduces an experimental point-tracker-assisted labeling feature that enables semi-automated keypoint annotation across video frames using CoTracker integration. The implementation includes a new tracking widget, core data structures for tracking operations, comprehensive test infrastructure with a dummy tracker, and updated documentation.
Changes:
- Added tracking controls widget with UI for point tracking operations and key bindings
- Implemented core tracking infrastructure (models, worker, data structures) with CoTracker 3 integration
- Created comprehensive test suite with dummy tracker fixtures for tracking workflows
- Updated documentation and configuration to reflect new tracking capabilities
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/napari_deeplabcut/tracking/_worker.py |
Implements QThread-based tracking worker with progress reporting and stop handling |
src/napari_deeplabcut/tracking/_widgets.py |
Adds tracking controls widget with frame selection, tracking buttons, and keybindings |
src/napari_deeplabcut/tracking/_models.py |
Defines abstract tracking model interface and CoTracker 3 implementation |
src/napari_deeplabcut/tracking/_data.py |
Defines data structures for tracking inputs, outputs, and worker data |
src/napari_deeplabcut/napari.yaml |
Registers new tracking controls widget in plugin configuration |
src/napari_deeplabcut/_widgets.py |
Minor refactoring of checkbox variable names and comment updates |
src/napari_deeplabcut/_tests/tracking/test_worker.py |
Tests for tracking worker behavior and signal emissions |
src/napari_deeplabcut/_tests/tracking/test_widgets.py |
Tests for tracking controls widget UI and tracking requests |
src/napari_deeplabcut/_tests/test_widgets.py |
Minor test code formatting updates |
src/napari_deeplabcut/_tests/conftest.py |
Adds dummy tracker fixture and tracking test data fixtures |
README.md |
Documents experimental tracking feature with usage instructions and keybindings |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replaced print statements with logger calls in _widgets.py for better error and debug reporting. Adjusted logging level and debug flag handling in _worker.py, setting INFO as default and enabling DEBUG only when specified. Improved output validation logic in TrackingWorker.
Corrected a typo in the napari.yaml widget title ('racking' to 'tracking') and removed a redundant assignment in KeypointControls. Also cleaned up duplicate comments in the on_active_layer_change docstring.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 12 changed files in this pull request and generated 14 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Introduces a validate_outputs method to TrackingModel as an abstract method, with concrete implementations for DummyTracker and Cotracker3. Updates tests to use output validation and improves structure checks for tracking outputs, ensuring consistency and correctness of keypoints and features.
Uncommented a debug log in _models.py and enabled conditional debugpy usage in _worker.py when DEBUG is set. Removed or updated outdated and commented-out code in _widgets.py for improved readability and maintainability.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if not outputs.keypoints.shape[0] == inputs.keypoints.shape[0]: | ||
| return False, "Number of output keypoints does not match number of input keypoints." |
There was a problem hiding this comment.
The validation logic compares the total number of output keypoints to the number of input keypoints, but CoTracker3 outputs tracked keypoints across all frames (T * K keypoints) while inputs contain only K keypoints. This comparison will always fail for multi-frame tracking. The validation should compare outputs.keypoints.shape[0] with (T2 - T1) * K where T2 - T1 is derived from inputs.metadata['keypoint_range'].
| if not outputs.keypoints.shape[0] == inputs.keypoints.shape[0]: | |
| return False, "Number of output keypoints does not match number of input keypoints." | |
| # For CoTracker3, outputs contain tracked keypoints for every frame in the | |
| # keypoint_range. If keypoint_range = (T1, T2) and there are K input | |
| # keypoints, we expect (T2 - T1) * K output rows. | |
| metadata = getattr(inputs, "metadata", None) | |
| if metadata is None or "keypoint_range" not in metadata: | |
| return False, "Missing keypoint_range metadata required for validation." | |
| keypoint_range = metadata["keypoint_range"] | |
| if ( | |
| not isinstance(keypoint_range, (tuple, list)) | |
| or len(keypoint_range) != 2 | |
| ): | |
| return False, "Invalid keypoint_range metadata; expected (T1, T2)." | |
| T1, T2 = keypoint_range | |
| try: | |
| T1_int = int(T1) | |
| T2_int = int(T2) | |
| except (TypeError, ValueError): | |
| return False, "keypoint_range values must be integers." | |
| if T2_int <= T1_int: | |
| return False, "keypoint_range must satisfy T2 > T1." | |
| K = inputs.keypoints.shape[0] | |
| expected_n_keypoints = (T2_int - T1_int) * K | |
| if outputs.keypoints.shape[0] != expected_n_keypoints: | |
| return ( | |
| False, | |
| "Number of output keypoints does not match expected " | |
| f"((T2 - T1) * K) = {expected_n_keypoints}.", | |
| ) |
| MOVE_FORWARD_FRAME = TrackingKeybindConfig(key="i", tooltip="Move forward one frame") | ||
| MOVE_BACKWARD_FRAME = TrackingKeybindConfig(key="u", tooltip="Move backward one frame") | ||
|
|
||
| logger = logging.getLogger(__name__) |
There was a problem hiding this comment.
Logger is defined twice in this file (lines 35 and 59). Remove the duplicate definition on line 59.
| logger = logging.getLogger(__name__) |
| keypoints = np.array( | ||
| [ | ||
| [0, 10.0, 20.0], | ||
| [1, 30.0, 40.0], |
There was a problem hiding this comment.
The first column of the keypoints array appears to contain sequential IDs (0, 1) rather than frame indices. Based on the comment on line 11 stating '[0]: frame number in video', both keypoints should have frame index 0 since they're query points from the reference frame. Consider changing to [[0, 10.0, 20.0], [0, 30.0, 40.0]].
| [1, 30.0, 40.0], | |
| [0, 30.0, 40.0], |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #155 +/- ##
==========================================
- Coverage 82.38% 79.65% -2.74%
==========================================
Files 11 15 +4
Lines 1868 2831 +963
==========================================
+ Hits 1539 2255 +716
- Misses 329 576 +247 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| try: | ||
| # we let the model handle coordinate conventions internally, such that | ||
| # the worker and plugin can remain agnostic to these details. | ||
| inputs: TrackingModelInputs = model.prepare_inputs(cfg) | ||
|
|
||
| # Run inference; models implement their own batching and chunking | ||
| raw: RawModelOutputs = model.run(inputs, progress_callback, stop_callback) |
There was a problem hiding this comment.
Napari crashes (shuts down both DeepLabCut and Napari) when running line 85 on my windows machine.
[WinError 10054] An existing connection was forcibly closed by the remote host
There was a problem hiding this comment.
on linux, same problem, but different message:
No more messages
There was a problem hiding this comment.
Ok, that sounds like a napari segfault.
What I usually do for these :
- Launch a debug script with some extra debug os env (see below)
- Use
debugpy.debug_this_thread()above line 85 temporarily, or equivalent if you do not use debugpy.
Here is a script you can use :
"""
Simple debug script to launch napari with the tracking controls widget.
Should not be released as part of the package.
"""
import os
from napari import Viewer, run
import debugpy
os.environ["PYTHONFAULTHANDLER"] = "1" # better segfault traces
os.environ["NAPARI_ASYNC"] = "0" # avoid async teardown surprises
if __name__ == "__main__":
# DATA = your_data_path
v = Viewer()
keypoints_dock_widget, keypoints_plugin_widget = v.window.add_plugin_dock_widget(
"napari-deeplabcut",
"Keypoint controls",
)
tracking_dock_widget, tracking_plugin_widget = v.window.add_plugin_dock_widget(
"napari-deeplabcut",
"Tracking controls",
)
v.open(DATA, plugin="napari-deeplabcut")
# If you want a job to be launched immediately
# tracking_plugin_widget._forward_spinbox_absolute.setValue(10)
# tracking_plugin_widget._tracking_forward_button.click()
run()
Let me know if that helps, these can be really tricky so definitely happy to assist further !
Also can you check if the model downloaded correctly ?
This pull request introduces a new experimental point-tracker-assisted labeling feature to the napari-deeplabcut plugin, allowing users to semi-automate keypoint annotation across video frames. It also adds comprehensive infrastructure and tests for tracking controls and worker logic, including a dummy tracker for unit testing. The plugin's configuration and documentation are updated to reflect these enhancements.
Tracking Feature Implementation
TrackingWorkerData,TrackingModelInputs,RawModelOutputs,TrackingWorkerOutput) intracking/_data.pyto standardize input/output for tracking models and workers.napari.yaml) for proper integration into napari. [1] [2]Testing and Infrastructure
Documentation and UI Enhancements