Skip to content

Commit 911eb52

Browse files
Merge pull request #249 from SenteraLLC/feature/bitmask
Feature/bitmask
2 parents 11b5075 + f6fa160 commit 911eb52

23 files changed

Lines changed: 2493 additions & 68 deletions

‎.github/tasks.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,12 @@
11
## Tasks
22

3+
### Bitmask segmentation annotation mode
4+
Per-annotation binary masks, COCO-style RLE serialization, brush + erase interaction.
5+
6+
- [x] Phase 1: Data model + RLE utils + tests (`src/mask_utils.ts`, `ULabelSpatialType`, `SPATIAL_TYPE_SET`)
7+
- [x] Phase 2: Bitmask rendering layer (`draw_bitmask`, dispatch, redraw/clear)
8+
- [x] Phase 3: Brush/erase paints pixels (begin/continue/finish for bitmask)
9+
- [x] Phase 4: Undo/redo patch diffs for brush strokes (single-stroke `bitmask_stroke` action)
10+
- [x] Phase 5: Toolbox + mode registration (mode button, brush enable/disable, keybinds)
11+
- [ ] Phase 6: Export/import round-trip + tests + demo page
12+

‎CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,20 @@ All notable changes to this project will be documented here.
44

55
## [unreleased]
66

7+
## [0.25.0] - Aug 5th, 2026
8+
- Add a `bitmask` annotation mode for raster (per-pixel) segmentation, selectable via `allowed_modes: ["bitmask", ...]`.
9+
- Painted with the brush (toggle with `toggle_brush_mode_keybind`, default `g`); erase with `toggle_erase_mode_keybind` (default `e`); resize the brush with `increase_brush_size_keybind` / `decrease_brush_size_keybind` (defaults `]` / `[`) or `alt+scroll`. The brush/erase toggles now apply to both `polygon` and `bitmask` modes.
10+
- A paint stroke only joins an existing mask of the currently-selected class; painting a different class (or over empty space) starts a new mask of the selected class. Erase is class-agnostic. This differs from the `polygon` brush, which joins any polygon under the brush.
11+
- Hover a mask (with the brush off) to change its class via the ID dialog, or move/delete it like any other spatial annotation.
12+
- Each bitmask annotation stores a single binary mask. On export, `spatial_payload` is a COCO-style run-length-encoded object: `{ "counts": <number[]>, "size": [<height>, <width>] }` (column-major, starting with a background run). Fully-erased masks are deprecated (ULabel's delete semantics).
13+
- Configurable render opacity for bitmask classes via `mask_annotation_opacity`.
14+
- Add brush **overlap modes** for bitmask painting, controlled globally and persisted to localStorage: `none` (default), `exclude`, and `overwrite`.
15+
- `exclude`: newly-painted pixels never cover pixels owned by other undeprecated bitmask annotations (existing masks win).
16+
- `overwrite`: newly-painted pixels are removed from any other bitmask annotation that owned them (the new mask wins); a mask fully carved away is deprecated.
17+
- Resolution is deferred to the end of a stroke (you see the active mask cover others, then it snaps to the resolved result on release) and only affects the pixels a stroke adds; erase is unaffected.
18+
- Selectable via the Brush toolbox item (shown in bitmask mode) and keybinds `set_brush_overlap_none_keybind` / `set_brush_overlap_exclude_keybind` / `set_brush_overlap_overwrite_keybind` (defaults `shift+n` / `shift+e` / `shift+o`), which also appear in the `Keybinds` toolbox item. Initial value configurable via `default_brush_overlap_mode`.
19+
- Brush toolbox buttons have hover tooltips.
20+
721
## [0.24.0] - July 22nd, 2026
822
- Add `ConfidenceSlider` toolbox item (`AllowedToolboxItem.ConfidenceSlider`) that deprecates (hides) or shows spatial annotations based on their confidence values. Unlike the now-deprecated `KeypointSlider`, it works with all spatial annotation types that have a confidence payload (`bbox`, `bbox3`, `polygon`, `polyline`, `contour`, `tbar`, and `point`), across every subtask.
923
- Supports a single global "all" slider and/or per-class sliders, controlled by `class_filter_mode` (`"toggle"`, `"all-only"`, or `"class-only"`).

‎api_spec.md‎

Lines changed: 67 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,11 @@ class ULabel({
6666
toggle_erase_mode_keybind: string,
6767
increase_brush_size_keybind: string,
6868
decrease_brush_size_keybind: string,
69+
mask_annotation_opacity: number,
70+
default_brush_overlap_mode: BrushOverlapMode,
71+
set_brush_overlap_none_keybind: string,
72+
set_brush_overlap_exclude_keybind: string,
73+
set_brush_overlap_overwrite_keybind: string,
6974
fly_to_next_annotation_keybind: string,
7075
fly_to_previous_annotation_keybind: string,
7176
annotation_size_small_keybind: string,
@@ -178,7 +183,9 @@ As you can see, each subtask will have a corresponding list of annotation object
178183
"spatial_type": "<string>",
179184

180185
// (nullable) e.g. [[x1, y1], [x2, y2], ...]
181-
"spatial_payload": "<array>",
186+
// For "bitmask" annotations this is instead a run-length-encoded object:
187+
// { "counts": <number[]>, "size": [<height>, <width>] }. See Bitmask annotations.
188+
"spatial_payload": "<array | object>",
182189

183190
// The class associated with the annotation
184191
"classification_payloads": [
@@ -281,9 +288,49 @@ The full list of `"allowed_modes"` that are currently supported is:
281288
- `"whole-image"`: A label to be applied to an entire frame
282289
- `"global"`: A label to be applied to the entire series of frames
283290
- `"point"`: A keypoint within a single frame
291+
- `"bitmask"`: A raster (per-pixel) segmentation mask, painted with the brush. See [Bitmask annotations](#bitmask-annotations).
284292
- `"delete_polygon"`: Allows drawing a polygon around an area, and all annotations within that area will be deleted
285293
- `"delete_bbox"`: Allows drawing a bounding box around an area, and all annotations within that area will be deleted
286294
295+
#### Bitmask annotations
296+
297+
The `"bitmask"` mode enables raster (per-pixel) segmentation. Each bitmask annotation stores a single binary mask the size of the image.
298+
299+
**Interaction**
300+
301+
- Painting uses the brush, shared with the `polygon` brush. Toggle the brush with `toggle_brush_mode_keybind` (default `g`) or the Brush toolbox item, erase with `toggle_erase_mode_keybind` (default `e`), and resize the brush with `increase_brush_size_keybind` / `decrease_brush_size_keybind` (defaults `]` / `[`) or `alt+scroll`.
302+
- Starting a paint stroke over an existing bitmask of the **currently-selected class** adds to that mask; otherwise (a different class is selected, or you start over empty space) a new bitmask annotation of the selected class is created. Erasing is class-agnostic — it removes from whichever mask is under the brush. (This class-aware joining differs from the `polygon` brush, which joins any polygon under the brush.)
303+
- With the brush off, hovering a mask surfaces the usual edit dialogs: change its class via the ID dialog, or move/delete it like any other spatial annotation. Erasing a mask entirely deprecates the annotation (ULabel's delete semantics).
304+
- Requires the `Brush` toolbox item (`AllowedToolboxItem.Brush`) to be present.
305+
306+
**Overlap modes**
307+
308+
When painting, the brush can enforce mutual exclusivity with *other* undeprecated bitmask annotations. The mode is a single **global** value, persisted to localStorage, and is chosen via the Brush toolbox item (shown in bitmask mode) or the overlap keybinds. Its initial value comes from [`default_brush_overlap_mode`](#default_brush_overlap_mode).
309+
310+
- `"none"` (default): painting only adds to the active mask; other masks are untouched (pixels may be owned by multiple annotations).
311+
- `"exclude"`: newly-painted pixels never cover pixels owned by other bitmask annotations (existing masks win).
312+
- `"overwrite"`: newly-painted pixels are removed from any other bitmask annotation that owned them (the new mask wins); a mask fully carved away is deprecated.
313+
314+
Resolution is **deferred to the end of a stroke** and only affects the pixels the stroke adds (pre-existing overlaps are left alone). Erase strokes are unaffected. These modes govern *new strokes only* — they do not retroactively de-overlap already-imported masks.
315+
316+
**Serialization**
317+
318+
A bitmask's `spatial_payload` is a COCO-style, uncompressed run-length encoding:
319+
320+
```javascript
321+
{
322+
// Alternating run lengths in column-major (Fortran) order, always starting
323+
// with a background (0) run. A leading foreground pixel is a leading 0.
324+
"counts": [<number>, ...],
325+
// [height, width] of the mask (matches COCO's size convention)
326+
"size": [<height>, <width>]
327+
}
328+
```
329+
330+
Note this is the *uncompressed* form (`counts` as an integer array), not the LEB128-packed string used by `pycocotools`. Masks import from and export to this same object shape.
331+
332+
The render opacity of bitmask annotations is configurable via [`mask_annotation_opacity`](#mask_annotation_opacity).
333+
287334
The `resume_from` attributes are used to import existing annotations into the annotation session for each subtask, respectively. Existing annotations must be provided as a list of annotations of the form specified above.
288335
289336
### `task_meta` and `annotation_meta`
@@ -506,16 +553,31 @@ Keybind to toggle between annotation and selection modes. Default is `u`.
506553
Keybind to create a bounding box annotation around the `initial_crop`. Default is `f`. Requires the active subtask to have a `bbox` mode.
507554
508555
### `toggle_brush_mode_keybind`
509-
Keybind to toggle brush mode for polygon annotations. Default is `g`. Requires the active subtask to have a `polygon` mode.
556+
Keybind to toggle brush mode for `polygon` and `bitmask` annotations. Default is `g`. Requires the active subtask to have a `polygon` or `bitmask` mode.
510557
511558
### `toggle_erase_mode_keybind`
512-
Keybind to toggle erase mode for polygon annotations. Default is `e`. Requires the active subtask to have a `polygon` mode.
559+
Keybind to toggle erase mode for `polygon` and `bitmask` annotations. Default is `e`. Requires the active subtask to have a `polygon` or `bitmask` mode.
513560
514561
### `increase_brush_size_keybind`
515-
Keybind to increase the brush size. Default is `]`. Requires the active subtask to have a `polygon` mode.
562+
Keybind to increase the brush size. Default is `]`. Requires the active subtask to have a `polygon` or `bitmask` mode.
516563
517564
### `decrease_brush_size_keybind`
518-
Keybind to decrease the brush size. Default is `[`. Requires the active subtask to have a `polygon` mode.
565+
Keybind to decrease the brush size. Default is `[`. Requires the active subtask to have a `polygon` or `bitmask` mode.
566+
567+
### `mask_annotation_opacity`
568+
The fill opacity (`0`-`1`) used when rendering `bitmask` (raster segmentation) annotations. Default is `0.4`.
569+
570+
### `default_brush_overlap_mode`
571+
The initial [brush overlap mode](#overlap-modes) for bitmask painting: `"none"` (default), `"exclude"`, or `"overwrite"`. The live value is global and persisted to localStorage, so a user's last choice takes precedence over this default on subsequent sessions.
572+
573+
### `set_brush_overlap_none_keybind`
574+
Keybind to set the brush overlap mode to `none`. Default is `shift+n`.
575+
576+
### `set_brush_overlap_exclude_keybind`
577+
Keybind to set the brush overlap mode to `exclude`. Default is `shift+e`.
578+
579+
### `set_brush_overlap_overwrite_keybind`
580+
Keybind to set the brush overlap mode to `overwrite`. Default is `shift+o`.
519581
520582
### `fly_to_next_annotation_keybind`
521583
Keybind to set the zoom to focus on the next annotation. Default is `Tab`, which also will disable any default browser behavior for `Tab`.

‎demo.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,5 @@ console.log(`http://localhost:${port}/frames.html`);
1313
console.log(`http://localhost:${port}/box-roi.html`);
1414
console.log(`http://localhost:${port}/resume-from.html`);
1515
console.log(`http://localhost:${port}/row-filtering-example.html`);
16+
console.log(`http://localhost:${port}/bitmask-example.html`);
1617
console.log(`http://localhost:${port}/live_demo.html`);

‎demo/bitmask-example.html‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<title>ULabel - Bitmask Segmentation</title>
5+
6+
<!-- ULabel Library -->
7+
<script src="/ulabel.js"></script>
8+
9+
<!-- JQuery Library -->
10+
<script src="https://code.jquery.com/jquery-3.5.1.min.js" integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0=" crossorigin="anonymous"></script>
11+
12+
<!-- ULabel Usage -->
13+
<script>
14+
/* global $ */
15+
/* global ULabel */
16+
17+
$(window).on("load", function() {
18+
19+
function on_submit(annotations) {
20+
var element = document.createElement('a');
21+
element.setAttribute(
22+
"href", ('data:text/plain;charset=utf-8,' +
23+
encodeURIComponent(JSON.stringify(annotations, null, 2)))
24+
);
25+
element.setAttribute("download", "annotations.json");
26+
element.style.display = 'none';
27+
document.body.appendChild(element);
28+
element.click();
29+
document.body.removeChild(element);
30+
}
31+
32+
let subtasks = {
33+
"segmentation": {
34+
"display_name": "Segmentation",
35+
"classes": [
36+
{
37+
"name": "Vehicle",
38+
"color": "orange",
39+
"id": 10
40+
},
41+
{
42+
"name": "Obstacle",
43+
"color": "green",
44+
"id": 11
45+
}
46+
],
47+
"allowed_modes": ["bitmask", "polygon", "delete_bbox"],
48+
"resume_from": null,
49+
"task_meta": null,
50+
"annotation_meta": null
51+
}
52+
};
53+
54+
// Initial ULabel configuration
55+
let ulabel = new ULabel({
56+
"container_id": "container",
57+
"image_data": "https://ulabel.s3.us-east-2.amazonaws.com/cs-demo-0.png",
58+
"username": "DemoUser",
59+
"submit_buttons": on_submit,
60+
"subtasks": subtasks
61+
});
62+
// Wait for ULabel instance to finish initialization
63+
ulabel.init(function() {
64+
// ULabel is now ready for use
65+
});
66+
67+
});
68+
</script>
69+
</head>
70+
<body>
71+
<div id="container" style="width: 100%; height: 100vh; position: absolute; top: 0; left: 0;"></div>
72+
</body>
73+
</html>

0 commit comments

Comments
 (0)