Overview
Our primary users are running test and evaluation on computer vision models and datasets. FiftyOne covers viewing of pre-labeled datasets. We need a tool that will enable users to label new datasets or edit existing ones. The two most popular open source tools are CVAT and LabelStudio. These tools help users:
FiftyOne integrates seamlessly with both CVAT and LabelStudio
Important: They are down selecting vendors for the Data Labeling Challenge this week. The tooling that the vendors require may dictate the choice we make here.
Requirements
| |
User Requirement |
| 1 |
Must be deployed on AWS GovCloud (COSMOS) |
| 2 |
Must support multi-user (user identities tied back to keycloak) and be performant at scale |
| 3 |
Must support transitive permission control - restricted access to datasets based on user or user group (should be able to rely on the RBAC from the data catalog, and the request to data catalog must come from the user’s claim, not a “global/admin” FiftyOne credential) |
| 4 |
Must be able to access datasets from data catalog |
| 5 |
Nice to have: access datasets from JupyterHub storage |
Decision Guidance on CVAT and LabelStudio
We have not yet made a decision on which to build. Kim is leaning toward CVAT but if several DLC vendors are going to require LabelStudio, that will affect this decision.
CVAT (Computer Vision Annotation Tool)
CVAT is designed for velocity and precision on visual tasks. If your team is annotating
thousands of dense images or long video clips, CVAT is built to make that process as fast as
possible.
Pros:
built-in frame interpolation means you only annotate keyframes, and the software
automatically calculates the bounding box movements for the frames in between.
skeletons (pose estimation), 3D cuboids, AI-assisted masking (Segment Anything), and
intelligent scissors.
allowing experienced annotators to fly through datasets without moving their mouse.
annotation features completely free.
Cons:
Vision Only: It does not support text (NLP), audio, or time-series data.
Rigid Interface: The UI is what it is; you cannot customize the layout to fit a niche
workflow or display conditional logic.
number of tools and settings can be overwhelming.
Label Studio
Label Studio takes a framework approach. Instead of giving you a fixed annotation interface,
it gives you a blank canvas and XML-like tags to build the exact UI your project requires.
Pros:
types. You can use it for text classification, audio transcription, named entity recognition
(NER), and image segmentation—all under the same platform.
workflow that shows an image, asks the annotator to transcribe text found in the image,
and provides a dropdown to classify the sentiment of that text.
pipelines, handling cloud-native storage easily, and its Enterprise version has highly
polished role-based access control (RBAC), SSO, and reviewer workflows.
Cons:
rudimentary. It treats video mostly as a sequence of extracted frames and lacks the
smooth, native interpolation required for heavy video tracking tasks.
declarative XML configuration before you can start annotating.
vision tools can feel slightly clunky compared to CVAT when annotating highly complex,
dense visual scenes.
The Final Verdict
Choose CVAT if: Your core focus is evaluating and testing dedicated computer vision systems. If you need to rigorously review dense visual data, validate bounding box accuracy, or scrutinize video tracking performance frame-by-frame, CVAT’s specialized tools will save your testing teams hundreds of hours.
Choose Label Studio if: Your evaluation workflows span multiple modalities beyond just pure vision. If you need a single platform to review OCR text extraction, audit audio transcriptions, and edit computer vision labels, Label Studio’s flexible interfaces are unbeatable
Overview
Our primary users are running test and evaluation on computer vision models and datasets. FiftyOne covers viewing of pre-labeled datasets. We need a tool that will enable users to label new datasets or edit existing ones. The two most popular open source tools are CVAT and LabelStudio. These tools help users:
Add annotations/labels to image and video datasets
Modify existing labels
Export various common annotation formats in support of different models
FiftyOne integrates seamlessly with both CVAT and LabelStudio
Important: They are down selecting vendors for the Data Labeling Challenge this week. The tooling that the vendors require may dictate the choice we make here.
Requirements
Decision Guidance on CVAT and LabelStudio
We have not yet made a decision on which to build. Kim is leaning toward CVAT but if several DLC vendors are going to require LabelStudio, that will affect this decision.
CVAT (Computer Vision Annotation Tool)
CVAT is designed for velocity and precision on visual tasks. If your team is annotating
thousands of dense images or long video clips, CVAT is built to make that process as fast as
possible.
Pros:
Unmatched Video Tooling: CVAT is arguably the best open-source tool for video. Its
built-in frame interpolation means you only annotate keyframes, and the software
automatically calculates the bounding box movements for the frames in between.
Deep Vision Features: It offers highly refined tools for complex vision tasks, including
skeletons (pose estimation), 3D cuboids, AI-assisted masking (Segment Anything), and
intelligent scissors.
Speed for Power Users: The interface is heavily optimized for keyboard shortcuts,
allowing experienced annotators to fly through datasets without moving their mouse.
Generous Open-Source Version: The self-hosted version includes almost all of the core
annotation features completely free.
Cons:
Vision Only: It does not support text (NLP), audio, or time-series data.
Rigid Interface: The UI is what it is; you cannot customize the layout to fit a niche
workflow or display conditional logic.
Steep Initial Learning Curve: For a beginner just trying to draw a few boxes, the sheer
number of tools and settings can be overwhelming.
Label Studio
Label Studio takes a framework approach. Instead of giving you a fixed annotation interface,
it gives you a blank canvas and XML-like tags to build the exact UI your project requires.
Pros:
True Multi-Modality: Label Studio shines when your AI pipelines involve multiple data
types. You can use it for text classification, audio transcription, named entity recognition
(NER), and image segmentation—all under the same platform.
Endless UI Flexibility: You can design custom layouts. For example, you can create a
workflow that shows an image, asks the annotator to transcribe text found in the image,
and provides a dropdown to classify the sentiment of that text.
Strong Enterprise & MLOps Features: It offers robust integration into modern ML
pipelines, handling cloud-native storage easily, and its Enterprise version has highly
polished role-based access control (RBAC), SSO, and reviewer workflows.
Cons:
Weak Video Support: Compared to CVAT, Label Studio’s video capabilities are
rudimentary. It treats video mostly as a sequence of extracted frames and lacks the
smooth, native interpolation required for heavy video tracking tasks.
Setup Overhead: You have to build (or configure) your labeling interface using their
declarative XML configuration before you can start annotating.
Less Refined for Dense Vision: While it handles bounding boxes and polygons fine, its
vision tools can feel slightly clunky compared to CVAT when annotating highly complex,
dense visual scenes.
The Final Verdict
Choose CVAT if: Your core focus is evaluating and testing dedicated computer vision systems. If you need to rigorously review dense visual data, validate bounding box accuracy, or scrutinize video tracking performance frame-by-frame, CVAT’s specialized tools will save your testing teams hundreds of hours.
Choose Label Studio if: Your evaluation workflows span multiple modalities beyond just pure vision. If you need a single platform to review OCR text extraction, audit audio transcriptions, and edit computer vision labels, Label Studio’s flexible interfaces are unbeatable