Add brush-based editing for mask and polygon annotations

[kimchioverfit]

Summary

I would like to propose brush-based editing for existing annotations.

The main use case is correcting generated or manually created segmentation annotations when only a small part needs to be removed or added. Currently, polygon editing is mostly vertex-based, so removing a local region can require manually adding/moving/deleting multiple points.

Proposed behavior

Add brush tools in edit mode:

• Brush Add
• Brush Erase
• Circle Brush
• Rect Brush

For shape_type: "mask":

• Brush Add paints pixels into the mask.
• Brush Erase removes pixels from the mask.
• Circle and rect brush footprints are supported.

For shape_type: "polygon":

• Brush Erase removes polygon vertices that fall inside the brush footprint.
• The existing minimum polygon constraint is preserved, so polygons are not reduced below 3 points.
• Brush Add is disabled for polygons, since adding polygon geometry from a brush footprint is ambiguous.

Why this helps

This would make it easier to clean up annotations produced by AI segmentation or dense manual labeling. For example:

• remove small false-positive regions from a mask
• clean up rough segmentation boundaries
• quickly delete unwanted polygon points in a local area
• use a rectangular brush for grid-like/pixel-precise cleanup
• use a circular brush for more natural local edits

Implementation idea

A possible implementation is:

• Add brush state to Canvas
  • mode: add / erase
  • shape: circle / rect
  • size: configurable integer radius
• Enable brush actions only when a compatible shape is selected.
• For mask shapes, update the boolean mask array directly.
• For polygon shapes, erase vertices inside the brush footprint while keeping at least 3 points.
• Reuse the existing undo/backup path by backing up shapes after a brush stroke.

Open questions

• Should brush size be exposed as a toolbar slider/spinbox?
• Should polygon brush erase remove only vertices, or should it eventually support real polygon boolean subtraction?
• Should mask brush editing be available only for selected masks, or also for the topmost mask under the cursor?
• Should this feature support undo per stroke or per mouse movement?

[kimchioverfit]

#2110

[wkentaro]

brush tool for manipulating mask and polygon shapes sounds very interesting. let me try and see how it feels.