Research spike to investigate the following:
Overview
We have several terabytes of compute vision datasets (image and full motion video) that will live on our deployment. We need a way to organize those datasets and also control who gets access.
Important: We are not tied to any specific data catalog solution. This effort should include a review of available options. However, our client is considering commercial Databricks for data holdings and other services, so OSS Databricks Unity Catalog should be strongly considered.
Requirements
| |
User Requirement |
| 1 |
Must be deployed on AWS GovCloud (COSMOS) |
| 2 |
Must support multi-user (user identities tied back to keycloak) and be performant at scale |
| 3 |
Must support permission control - restricted access to datasets based on user or user group |
| 4 |
Other software packs must be able to access the data holdings (e.g. JupyterHub, Ray, FiftyOne) |
| 5 |
Regular users on Nebari must be able to add/remove data from the catalog (no k8s access) |
| 6 |
Supports both datasets and computer vision models |
| 7 |
Admin users must be able to set permissions on shared datasets |
| 8 |
Data compliance = guaranteed access control (for MVP - we can/will expand later) |
| 9 |
Internal backup system or ability to connect to Nebari’s backup system |
| 10 |
This catalog becomes the gate through which all data access must flow. It must be able to control access even when other software packs are accessing the data. |
Research spike to investigate the following:
Overview
We have several terabytes of compute vision datasets (image and full motion video) that will live on our deployment. We need a way to organize those datasets and also control who gets access.
Important: We are not tied to any specific data catalog solution. This effort should include a review of available options. However, our client is considering commercial Databricks for data holdings and other services, so OSS Databricks Unity Catalog should be strongly considered.
Requirements