Skip to content

[Feature]: Evaluate Roboflow Supervision for offline perception benchmarking and visual debugging #919

Description

@ll7

Feature Description

PAF already uses local Ultralytics-based perception in the VisionNode and already has perception evaluation documents, but it lacks a lightweight, reusable offline toolkit for inspecting detections, masks, tracks, and derived metrics on recorded frames or generated datasets.

roboflow/supervision looks like a good fit for this exact gap:

  • model-agnostic CV utilities
  • detection and segmentation overlays
  • dataset helpers and format conversions
  • basic tracking / zone / counting style primitives for analysis

The goal is not to add a new runtime dependency into the ROS2 driving stack immediately.
The goal is to evaluate whether supervision should become an experiments-only or testing-only dependency for offline perception debugging and benchmarking around the current PAF outputs.

Definition of Done

  • Build a small offline prototype that consumes representative PAF perception outputs or recorded CARLA frames.
  • Verify that current Ultralytics/PAF detections can be converted into a useful supervision-based analysis flow.
  • Demonstrate at least one concrete debugging or benchmarking benefit, for example better visualization of false positives, traffic-light misses, or object-track stability.
  • Decide whether adoption should stay limited to experiments/testing or whether a broader integration is justified.
  • Record the adoption decision and integration boundary in doc/reasoning/ or the relevant perception docs.

Testability

Run the prototype on a small representative sample from the current perception stack and verify that it produces useful overlays or metrics that are not already easy to obtain in the existing workflow.

Dependencies

Related to #918, which is already tracking stronger dataset generation and replay foundations. This issue should stay focused on offline perception evaluation tooling rather than dataset capture itself.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PAF26All opened bugs, issues and ideas which can't be implemented by the PAF24 team. Have funperceptionresearchResearching or collection informationtesting

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions