fix(fusion): dbt State run_cache no longer falls back to per-node queries for legitimately-empty Databricks schemas#15602
Open
DipakMandlik wants to merge 2 commits into
Conversation
… legitimately-empty Databricks schemas crates/dbt-tasks-core/src/run_cache/run_cache_service.rs's bulk_prefetch_last_modified_by_schema treated any empty result from freshness_all_in_schema as evidence of the same problem: Snowflake's INFORMATION_SCHEMA propagation lag, which the fallback + StateServiceWarn exist specifically to work around. That's correct for Snowflake and BigQuery (both issue a single INFORMATION_SCHEMA-style bulk query that can genuinely lag behind a just-created table), and correct for any adapter that hasn't overridden freshness_all_in_schema at all (the trait's default impl deliberately returns empty to signal "fall back", per its own doc comment). It's wrong for Databricks: its freshness_all_in_schema issues a real per-relation DESCRIBE HISTORY (or INFORMATION_SCHEMA.last_altered for views) lookup directly, with no intermediate eventually-consistent catalog snapshot to lag behind. An empty result there means every relation in the group genuinely doesn't exist yet -- the normal state for a freshly created CI schema before any model has run -- so the fallback just repeats the same (still-empty) per-relation lookups at higher cost while emitting a misleading warning on every CI run (reported: 60s-5min added per run, 100% of schema groups affected). Add schema_dump_empty_requires_fallback(), gating the fallback+warning to adapters that actually need it, verified directly against the current Snowflake, BigQuery, Databricks, and default-trait implementations rather than assumed from the bug report alone. Fixes dbt-labs#15594
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #15594
Problem
dbt State's run_cache prefetch (
run_cache_service.rs) batches a schema-level freshness dump per (database, schema) group viafreshness_all_in_schema, and falls back to slower per-node queries — with aStateServiceWarn— whenever that dump comes back empty. The fallback exists specifically to work around Snowflake'sINFORMATION_SCHEMAeventual consistency: Fusion's global prefetch fires eagerly at run start, before dbt-core's plugin-style lazy prefetch would have given the catalog a few seconds to catch up on a table from the immediately preceding run.That reasoning doesn't hold for every adapter. On Databricks, CI jobs that build into fresh per-PR ephemeral schemas hit
dump.is_empty()on 100% of schema groups, on every run, adding 60s–5min of per-node fallback queries before the build even starts — because the schema is genuinely brand new, not because of a race.The root cause is that
dump.is_empty()alone is the wrong signal. It conflates "the adapter's bulk lookup mechanism can lag behind a real table" with "there's genuinely nothing here yet," and which one applies depends entirely on how each adapter implementsfreshness_all_in_schema.Solution
Read all three currently-implemented
freshness_all_in_schemamethods plus the trait's default, rather than assuming the bug report's framing, and gate the fallback on which category each adapter actually falls into:INFORMATION_SCHEMA-style bulk query against a metadata view that's documented to lag behind DDL — the fallback is genuinely necessary here, unchanged.DESCRIBE HISTORY(orINFORMATION_SCHEMA.last_alteredfor views) lookup directly, with no intermediate eventually-consistent catalog snapshot. An empty result there unambiguously means every relation in the group doesn't exist yet — exactly the state of a freshly created CI schema before any model has run. No longer falls back or warns.New private helper added:
schema_dump_empty_requires_fallback(adapter_type, dump_is_empty)— a small, pure, directly-testable gate function, following the same allow/deny-list pattern already established in this file byheuristic_clock_enabled_for_adapterfor an analogous Databricks-specific issue.Checklist
Automated checks
cargo build -p dbt-tasks-corecargo clippy -p dbt-tasks-core --all-targetscargo test -p dbt-tasks-core --lib