Skip to content

Feature: Implement Fine-Tuned SAM 3 Spleen Models #5

Description

@blayyyyyk

Description

As an ML engineer, I want to load our team's custom fine-tuned SAM 3 spleen models into the browser pipeline so that the segmentation accurately and exclusively targets the spleen.

Acceptance Criteria

  • Convert the highest-performing fine-tuned spleen SAM 3 PyTorch model to the required web-compatible format (ONNX).
  • Update the inference script to fetch and initialize these custom weights instead of the base SAM 3 model.
  • Map NiiVue canvas click/drag events to the SAM 3 prompt inputs (transforming canvas coordinates to spatial model dimensions).
  • Render the resulting binary mask returned by the model back onto the NiiVue volume as a colored overlay.

Implementation Notes & Resources

  • Managing the coordinate space transformation between the NiiVue canvas and the SAM 3 image encoder is the critical path here.
  • Use NiiVue's built-in overlay methods (e.g., nv.addVolume where the volume is designated as a label map) to display the returned mask.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions