Feature/voxel annotation

[briossant]

This Draft Pull Request introduces an interactive voxel annotation feature, allowing users to perform manual segmentation by painting directly onto volumetric layers. This implementation is based on the proposal in Issue #851 and incorporates the feedback from @jbms.

Here is a live demo to try the feature, watch out, there is persistent storage, so your annotations will be saved and will override the ones already present: OPEN DEMO VIEWER

Key Changes & Architectural Overview

Following the discussion, this implementation has been significantly revised from the initial prototype:

1. Integration with Existing Layers: Instead of a new vox layer type, the voxel editing functionality is now integrated directly into ImageUserLayer and SegmentationUserLayer via a UserLayerWithVoxelEditingMixin. This mixin adds a new "Draw" tab in the UI.

2. New Tool System: The Brush and Flood Fill tools are implemented as toggleable LayerTools, while the Picker tool is a one-shot tool. All integrate with Neuroglancer's new tool system. The drawing action is bound to Ctrl + Left Click.

3. Optimistic Preview for Compressed Chunks: To provide immediate visual feedback and solve the performance problem with compressed chunks, edits are now rendered through an optimistic preview layer.

• When a user paints, edits are first applied to an InMemoryVolumeChunkSource.
• This preview source is rendered by a second instance of the layer's primary RenderLayer (e.g., ImageRenderLayer or SegmentationRenderLayer). This ensures the preview perfectly matches the user's existing shader and display settings.
• The base data chunk is not modified on the frontend, avoiding the need to decompress/recompress it.

Data-flow

sequenceDiagram
participant User
participant Tool as VoxelBrushTool
participant ControllerFE as VoxelEditController (FE)
participant EditSourceFE as OverlayChunkSource (FE)
participant BaseSourceFE as VolumeChunkSource (FE)
participant ControllerBE as VoxelEditController (BE)
participant BaseSourceBE as VolumeChunkSource (BE)

    User->>Tool: Mouse Down/Drag
    Tool->>ControllerFE: paintBrushWithShape(mouse, ...)
    ControllerFE->>ControllerFE: Calculates affected voxels and chunks

    ControllerFE->>EditSourceFE: applyLocalEdits(chunkKeys, ...)
    activate EditSourceFE
    EditSourceFE->>EditSourceFE: Modifies its own in-memory chunk data
    note over EditSourceFE: This chunk's texture is re-uploaded to the GPU
    deactivate EditSourceFE

    ControllerFE->>ControllerBE: commitEdits(edits, ...) [RPC]

    activate ControllerBE
    ControllerBE->>ControllerBE: Debounces and batches edits
    ControllerBE->>BaseSourceBE: applyEdits(chunkKeys, ...)
    activate BaseSourceBE
    BaseSourceBE-->>ControllerBE: Returns VoxelChange (for undo stack)
    deactivate BaseSourceBE
    ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
    activate ControllerFE
    ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
    note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
    ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
    deactivate ControllerFE

    ControllerBE->>ControllerBE: Pushes change to Undo Stack & enqueues for downsampling
    deactivate ControllerBE

    loop Downsampling & Reload Cascade
        ControllerBE->>ControllerBE: downsampleStep(chunkKeys)
        ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
        activate ControllerFE
        ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
        note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
        ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
        deactivate ControllerFE
    end

Loading

4. Writable source selection Aside every activated volume sub-sources of a writable datasource, an additional checkbox lets the user mark the sub-source as writable, then neuroglancer will try to write in it.

5. Dataset creation To complete Neuroglancer's writing capabilities, a dataset metadata creation/initialization feature was introduced.

The workflow is triggered when a user provides a URL to a data source that does not resolve:

Neuroglancer recognizes the potential intent to create a new dataset and prompts the user:

Finally, the user is able to access dataset creation form:

Data sources & Kvstores

Currently, there is a very limited set of supported data sources and kvstores, which are:

• datasources:
  • zarr v2 and v3 with codecs: raw, gzip, blosc
• kvstores:
  • s3+http(s): can be used with a local s3 bucket (e.g. minio) or with anonymous s3 urls
  • opfs: in-browser storage, also used for local development at some point, the relevancy can be discussed.
  • ssa+https: a kvstore linked to an in development project, which is a stateless (thanks to OAuth 2.0) worker providing signed urls to read/write in s3 stores

Limitations

• Only the rank 3 volumes are supported
• Float32 dataset are not supported
• Unaligned chunk hierarchy/resolutions (e.g. child chunks that may have multiple parents) is not supported

Open Questions & Future Work

This PR focuses on establishing the core architecture. Several larger topics from the original discussion are noted here as future work:

• Efficient Low-Resolution Drawing: As discussed, efficient, multi-resolution drawing with upsampling is a complex challenge that requires a new data format.
• 3D Drawing Tools: As suggested by @fcollman, 3D-specific tools like interpolation between slices are out of scope for this initial PR but could be a valuable direction for future work.

---

Checklist

• Completed the todo list found in src/voxel_annotations/TODOs.md
• [ ] Added support to more (every?) datasources and kvstores
• Signed the CLA.

---

Edits

• updated zarr support
• added "5. Dataset creation" section
• strikethrough what's no longer part of this PR
• added the Limitations section
• added live demo

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[jbms]

Can you complete the CLA?

[jbms]

The brush hover outline (circle where the mouse pointer is) seems to go away in some cases when changing the zoom level.

[briossant]

I need to rewrite history, my commits are signed with the wrong email, I will open a new PR I made a mistake with this force push nevermind its fixed

[briossant]

I've updated the demo to the latest commit, this includes:

• addition of a value-specific eraser mode
• a fix of the "double-opacity" artifact and the missing preview for the eraser in the segmentation layer, I used the sentinel strategy, as explained in previous comments. You can see the intentionally added artifact using the paint value 18446744073709551614.
• addition of a bottom bar for the brush and flood fill, with their respective settings moved to it. I did not change the settings binding strategy; as suggested by @chrisj, pre-defined bindings (e.g. shift+wheel for the brush size instead of shift+key+wheel for example) that only become active along the tool would be beneficial for ergonomics, at the cost of configurability; if this is the path we want to take, I would like some feedback on what bindings should we choose, my initial proposal is: shift+wheel for the brush size and the max flood fill area, shift+keys to cycle brush shapes.

[chrisj]

@briossant It feels better!

These controls are draggable right now which prevents dragging the slider

Have you thought about having ctrl+mousedown+shift trigger erasing and then have the erase setting just be a toggle between everything and selected?

[briossant]

@briossant It feels better!

These controls are draggable right now which prevents dragging the slider

Have you thought about having ctrl+mousedown+shift trigger erasing and then have the erase setting just be a toggle between everything and selected?

Yeah erasing feels better like this, I've updated the demo with the refactored erasing.

[briossant]

Hello,

I've cleared my current todo list and am standing by for review, or further feedback.

One question regarding scope: @jbms mentioned keeping new datasources and kvstores separate, but would you recommand to add write support for existing datasources (e.g. precomputed and n5) in this PR?

[chrisj]

I would remove this then because when we later add it, it will be more clear that it that it was a requirement for that functionality.

[chrisj]

I think it is best to use a block statement with if statements that aren't single line

[chrisj]

From some performance testing of the paint tool, it seems you can get a significant speedup for the UI thread if you only check getEnsuredValue if filterValue !== undefined and ignore the v === value check. Though I believe that would cause some unnecessary writes so perhaps that logic could be moved to the backend.

I'm wondering how much of this logic could be moved to the backend, if the only thing the frontend did when receiving input was to forward that user input to the backend and have the backend do all the voxel calculations.

[briossant]

Yes, the v === value check can be removed. This would indeed speed up the function, and I can add the check back in the backend to avoid unnecessary writes. As for moving the whole logic to the backend: for the brush, I believe this would make the preview delay longer, resulting in a less responsive user experience; but for the flood fill it has less downside as the preview is already slow, and it would also fix the slowdown on big regions.

[chrisj]

Performing a large sphere brush stroke is the main performance issue I've noticed

[briossant]

We could cheat by drawing a disk in the preview and calculating the sphere in the backend; this would be invisible to the user as long as they do not scroll through the slices, or rotate the slice, right after drawing. Would leaving those artifacts be acceptable for the performance boost?

[chrisj]

Which artifact? Using disk for preview sounds like a good solution

[briossant]

I've updated the demo with what we talked (floodfill and brush sphere moved to the backend). I also temporarily added the different ways to render the sphere preview as brush shapes (see image). Note that I have not thoroughly tested the refactor yet.

Now that drawing big sphere is fast, it is easy to overwhelm the backend if you don't have a really fast computer (it is my case) ; to solve this I am thinking of adding a mechanism to temporarily halt drawing and message the user when the backend job queue is too big.

[briossant]

Here is how the compute load indicator may look:

[jbms]

Here is an example of artifact I have in mind:


Note: this particular one could be solve by drawing 3 disks (one per sliceview). The other artifact I have in mind involves user motions during the preview phase (scrolling through slices or rotating the sliceview).

One idea is to keep a list of pending draw operations on each preview chunk, and then only apply it when the chunk actually becomes visible and is about to be drawn. Additionally, for the sphere brush there could be optimized handling for the case that the brush covers the entire chunk, to make it constant time.

[jbms]

Here is an example of artifact I have in mind:


Note: this particular one could be solve by drawing 3 disks (one per sliceview). The other artifact I have in mind involves user motions during the preview phase (scrolling through slices or rotating the sliceview).

One idea is to keep a list of pending draw operations on each preview chunk, and then only apply it when the chunk actually becomes visible and is about to be drawn. Additionally, for the sphere brush there could be optimized handling for the case that the brush covers the entire chunk, to make it constant time.

[briossant]

I see how this would work, but this require to change the chunk shape to 8x8x8 or 16x16x16, which I guess is fine? With the current 64x64x64 chunk shape, the modification wouldn't be perceptible as the we already feel the performance issues with spheres of radius 32.

One other thing eating performances is the amount of paintBrushWithShape calls, each one recalculates the full sphere/disk even if it overlap 90% of the voxels from the previous call. I could reduce a bit the amount of calls but reducing too much would make it impossible to draw nice curves. I think calculating and drawing the difference between the new sphere/disk and the sphere/disk of the previous call would be the solution.

[chrisj]

I was showing @ceesem the state of this PR and he had a few points to make. We are very excited to see the progress made

• we think the most common painting strategy will be encircle and fill, we think it would be great if it was possible to fill without switching between the paint and fill tools. It seems heavyhanded to try to squeeze it all in one tool, we like how easy it is to swap between the paint and erase.
• it would be desirably to be able to paint a single voxel.
• it is not obvious what voxels are going to be painted going from a circle to the cross that actually gets painted. I'm not sure if it is feasible to give a real time indication to the user of the affected voxels while hovering

• it is not clear where is the origin/target point for the flood fill cursor
• we should indicate to the user when edits are actively being written to prevent users refreshing and losing an edit

Also, is off-axis painting supported/planned? It tries paint but it there are visual artifacts and it doesn't save, it causes TypeError: Cannot convert undefined to a BigInt in readLocal

[briossant]

I was showing @ceesem the state of this PR and he had a few points to make. We are very excited to see the progress made

• we think the most common painting strategy will be encircle and fill, we think it would be great if it was possible to fill without switching between the paint and fill tools. It seems heavyhanded to try to squeeze it all in one tool, we like how easy it is to swap between the paint and erase.

Maybe with ctrl + middle click while in the brush tool? And we keep the current flood fill tool as a second option, so we can keep the tool settings separated in their respective bottom bar.

• it would be desirably to be able to paint a single voxel.

yes

• it is not obvious what voxels are going to be painted going from a circle to the cross that actually gets painted. I'm not sure if it is feasible to give a real time indication to the user of the affected voxels while hovering

This may be doable thanks to a sentinel value in the preview layer, similar to how the eraser preview is working. But I would personally not put this as the most urgent feature to implement?

• it is not clear where is the origin/target point for the flood fill cursor

true

• we should indicate to the user when edits are actively being written to prevent users refreshing and losing an edit

I think this joins the "compute load indicator" I talked about earlier. However, instead of a bar that fills up when you draw, it would be a bar that fills up when you stop drawing until it disappears and reappears / resets when you draw again.

Also, is off-axis painting supported/planned? It tries paint but it there are visual artifacts and it doesn't save, it causes TypeError: Cannot convert undefined to a BigInt in readLocal

It is supposed to be working, I have introduced this error while doing the last refactor, this is fixed. There are artifacts in the preview, but the fillPlaneAliasingGaps function get rid of most of them in the backend.

[chrisj]

This may be doable thanks to a sentinel value in the preview layer, similar to how the eraser preview is working. But I would personally not put this as the most urgent feature to implement?

@briossant agreed. It might not even be the right solution. I think it really only applies to doing small radius painting and it might be solved by making it possible for the origin to be the center of a voxel or between voxels. In the image above, a 2x2 square would probably map best to the cursor.