Summary
Query live Unity Catalog storage metrics on scan so dead-asset ranking does not depend on a fresh dbt docs generate (like Snowflake and Redshift today).
Context
In #3, table_storage() for Databricks is deferred and returns {}. Ranking uses catalog.json bytes from dbt docs generate — same fallback pattern as BigQuery when live INFORMATION_SCHEMA.TABLE_STORAGE is unavailable due to permissions.
Snowflake and Redshift overwrite catalog sizes each scan; Databricks does not yet.
Open questions
- Best source:
system.information_schema.tables, table properties, or another UC API
- Permissions, query cost, and latency at project scale.
- Whether live bytes should include only active storage or a breakdown (if UC exposes one).
Proposed approach
- Spike live storage query against a real catalog/schema.
- Implement
table_storage() + SQL builder + parser reuse from jobs.parse_table_storage_rows if column shape matches.
- Update
cli.py to fetch live storage for Databricks when implemented.
- Tests, docs, and note any permission requirements in USAGE.md.
Related
Summary
Query live Unity Catalog storage metrics on scan so dead-asset ranking does not depend on a fresh
dbt docs generate(like Snowflake and Redshift today).Context
In #3,
table_storage()for Databricks is deferred and returns{}. Ranking usescatalog.jsonbytesfromdbt docs generate— same fallback pattern as BigQuery when liveINFORMATION_SCHEMA.TABLE_STORAGEis unavailable due to permissions.Snowflake and Redshift overwrite catalog sizes each scan; Databricks does not yet.
Open questions
system.information_schema.tables, table properties, or another UC APIProposed approach
table_storage()+ SQL builder + parser reuse fromjobs.parse_table_storage_rowsif column shape matches.cli.pyto fetch live storage for Databricks when implemented.Related