Skip to content

fix(fusion): dbt State run_cache no longer falls back to per-node queries for legitimately-empty Databricks schemas#15602

Open
DipakMandlik wants to merge 2 commits into
dbt-labs:mainfrom
DipakMandlik:fix/run-cache-databricks-schema-dump-fallback
Open

fix(fusion): dbt State run_cache no longer falls back to per-node queries for legitimately-empty Databricks schemas#15602
DipakMandlik wants to merge 2 commits into
dbt-labs:mainfrom
DipakMandlik:fix/run-cache-databricks-schema-dump-fallback

Conversation

@DipakMandlik

Copy link
Copy Markdown

Resolves #15594

Problem

dbt State's run_cache prefetch (run_cache_service.rs) batches a schema-level freshness dump per (database, schema) group via freshness_all_in_schema, and falls back to slower per-node queries — with a StateServiceWarn — whenever that dump comes back empty. The fallback exists specifically to work around Snowflake's INFORMATION_SCHEMA eventual consistency: Fusion's global prefetch fires eagerly at run start, before dbt-core's plugin-style lazy prefetch would have given the catalog a few seconds to catch up on a table from the immediately preceding run.

That reasoning doesn't hold for every adapter. On Databricks, CI jobs that build into fresh per-PR ephemeral schemas hit dump.is_empty() on 100% of schema groups, on every run, adding 60s–5min of per-node fallback queries before the build even starts — because the schema is genuinely brand new, not because of a race.

The root cause is that dump.is_empty() alone is the wrong signal. It conflates "the adapter's bulk lookup mechanism can lag behind a real table" with "there's genuinely nothing here yet," and which one applies depends entirely on how each adapter implements freshness_all_in_schema.

Solution

Read all three currently-implemented freshness_all_in_schema methods plus the trait's default, rather than assuming the bug report's framing, and gate the fallback on which category each adapter actually falls into:

  • Snowflake / BigQuery: each issues a single INFORMATION_SCHEMA-style bulk query against a metadata view that's documented to lag behind DDL — the fallback is genuinely necessary here, unchanged.
  • Unimplemented adapters: fall through to the trait's default implementation, which deliberately returns an empty map to signal "fall back" (per its own doc comment) — the fallback is the only way those adapters get real freshness data, unchanged.
  • Databricks: issues a real per-relation DESCRIBE HISTORY (or INFORMATION_SCHEMA.last_altered for views) lookup directly, with no intermediate eventually-consistent catalog snapshot. An empty result there unambiguously means every relation in the group doesn't exist yet — exactly the state of a freshly created CI schema before any model has run. No longer falls back or warns.

New private helper added: schema_dump_empty_requires_fallback(adapter_type, dump_is_empty) — a small, pure, directly-testable gate function, following the same allow/deny-list pattern already established in this file by heuristic_clock_enabled_for_adapter for an analogous Databricks-specific issue.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
  • This PR includes type annotations for new and modified functions.

Automated checks

Check Command Result
Build cargo build -p dbt-tasks-core exit 0
Lint cargo clippy -p dbt-tasks-core --all-targets clean
Unit tests cargo test -p dbt-tasks-core --lib 90 passed, 0 failed (5 new)

… legitimately-empty Databricks schemas

crates/dbt-tasks-core/src/run_cache/run_cache_service.rs's
bulk_prefetch_last_modified_by_schema treated any empty result from
freshness_all_in_schema as evidence of the same problem: Snowflake's
INFORMATION_SCHEMA propagation lag, which the fallback + StateServiceWarn
exist specifically to work around.

That's correct for Snowflake and BigQuery (both issue a single
INFORMATION_SCHEMA-style bulk query that can genuinely lag behind a
just-created table), and correct for any adapter that hasn't overridden
freshness_all_in_schema at all (the trait's default impl deliberately
returns empty to signal "fall back", per its own doc comment).

It's wrong for Databricks: its freshness_all_in_schema issues a real
per-relation DESCRIBE HISTORY (or INFORMATION_SCHEMA.last_altered for
views) lookup directly, with no intermediate eventually-consistent catalog
snapshot to lag behind. An empty result there means every relation in the
group genuinely doesn't exist yet -- the normal state for a freshly
created CI schema before any model has run -- so the fallback just repeats
the same (still-empty) per-relation lookups at higher cost while emitting
a misleading warning on every CI run (reported: 60s-5min added per run,
100% of schema groups affected).

Add schema_dump_empty_requires_fallback(), gating the fallback+warning to
adapters that actually need it, verified directly against the current
Snowflake, BigQuery, Databricks, and default-trait implementations rather
than assumed from the bug report alone.

Fixes dbt-labs#15594
@DipakMandlik
DipakMandlik requested a review from a team as a code owner July 19, 2026 03:41
@cla-bot cla-bot Bot added the cla:yes label Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v2 Bug] dbt State schema-level freshness dump fallback fires on all adapters when schema is legitimately empty, causing excessive pre-build delays

1 participant