From 0cfbb2069d518cd54be715fb0c14e57d0778bab4 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Tue, 1 Sep 2026 14:07:00 -0600 Subject: [PATCH 01/11] docs: add the Bakehouse SDP project guide Replace the generic ETL page with an executable batch pipeline path so agents can build and verify the shared Bakehouse foundation for later project resources. --- .../02-databricks-projects/etl-pipelines.md | 363 +++++++++++++----- .../content/02-databricks-projects/index.md | 5 +- docs/agentic-starter-journey/lib/nav.ts | 2 +- 3 files changed, 269 insertions(+), 101 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 580f124..c2ec153 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -1,49 +1,44 @@ --- -description: Build the medallion bronze, silver, gold ETL pipeline as a bundle resource with databricks-pipelines. Layer mapping, run steps, verification, and failure modes. +description: bakehouse_e2e_pipeline reads samples.bakehouse and publishes governed bronze and silver materialized views with enforced expectations. --- -# ETL Pipelines +# Spark Declarative Pipelines ## Mental Model -ETL is the transformation layer on top of ingested data. A Lakeflow Spark Declarative Pipeline (SDP) holds the bronze, silver, and gold datasets in one pipeline resource, with expectations for data quality. -Bronze is raw, exactly as it arrived. Silver is cleaned and conformed. Gold is aggregates over the full dataset. -The dataset type per layer is load-bearing: a streaming table is append-only and will not recompute an aggregate when source rows change; a materialized view does. +`samples.bakehouse` is a batch source. +Bronze preserves source rows. +Silver applies quality rules. +Batch inputs use materialized views and `spark.read.table`. +The existing project DABs bundle owns and deploys the pipeline. ## Goal -A bronze, silver, gold SDP defined as a bundle resource, deployed and running against the dev target. +Successfully update the pipeline named `bakehouse_e2e_pipeline`. ## Prerequisites - Auth surface: `workspace`. -- [Project repo](/docs/02-databricks-projects/project-repo/) set up: a Git repo with a `databricks.yml` bundle and a dev target. -- A configured Databricks CLI profile that reaches the dev workspace. -- [Ingestion Pipelines](/docs/02-databricks-projects/ingestion-pipelines/) complete, so bronze has data in it. +- Complete the [Project repo](/docs/02-databricks-projects/project-repo/) outcome. +- Use a target catalog that contains `bakehouse_bronze` and `bakehouse_silver`. +- Confirm the deployment principal can read `samples.bakehouse`. +- Use a SQL warehouse for verification. ## Skill -`databricks-pipelines` (databricks-agent-skills). Invoke it before writing any pipeline code, not after. `databricks-dabs` for the resource YAML. +Invoke `databricks-pipelines` before writing the pipeline code. +Invoke `databricks-dabs` before writing the bundle resource. ## Inputs -Collect all of these before writing code. The first four decide the pipeline's shape; get them wrong and the fix is dropping tables, not editing YAML. - | Input | Source | How to obtain | |---|---|---| -| Source table | Human | The bronze table from Ingestion Pipelines, `._bronze.` | -| Update semantics | Human | Append only, or upserts and change tracking. Upserts mean Auto CDC and need a key plus a sequence column. | -| Expectations | Human | The actual rules: which columns are never null, which ranges are valid, what a duplicate means. Push for specifics. | -| Language | Human | SQL or Python. Sinks, ForEachBatch sinks, CDC from snapshots, and custom data sources are Python-only. | -| Target catalog | You derive | The bundle's `catalog` variable | -| Target schemas | You derive | `_bronze`, `_silver`, `_gold` | -| Serverless? | You derive | Default yes. Automatic incremental refresh of aggregating materialized views needs serverless plus Delta row tracking on the source. | - -:::warning -Ask the arrival pattern before choosing a dataset type. -A streaming source needs a streaming table; a batch source needs a materialized view. -You cannot change a streaming table into a materialized view in place, and a full refresh does not help: the table has to be dropped or the dataset renamed. -::: +| Existing project repository path | Human-provided | Use the repository completed on the Project repo page | +| Target catalog | Agent-derived | Read the active bundle target's `catalog` variable | +| Workspace profile | Human-provided | Use the named Databricks CLI profile for the target workspace | +| Workspace host and workspace ID | Agent-derived | Read the named profile and `databricks metastores current` | +| SQL warehouse ID | Agent-derived | Select a running warehouse available to the deployment principal | +| Language | Agent-derived, fixed runbook decision | Use Python | ## Run @@ -87,125 +82,297 @@ Do not invoke skills, run `bundle validate`, or deploy until auth is green. | `workspace_id` mismatch on `metastores current` | `databricks account workspaces list --profile -o json` and align id with the named host | | `current-user me` fails after profile is Valid | Workspace admin assigns the user or SP to the workspace | -### 1. The layer mapping +### 1. Inspect the batch source -| Layer | Dataset type | Why | -|---|---|---| -| Bronze | Streaming table with Auto Loader (or the ingested table from Ingestion Pipelines) | Raw, exactly as it arrived. Nothing filtered or deduplicated, so a broken downstream transformation can be reprocessed from here. | -| Silver | Streaming table, or a streaming table populated by Auto CDC for upserts | Cleaned, deduplicated, conformed. The source of truth for ad-hoc queries. | -| Gold | Materialized view | Aggregates over the full dataset. A streaming table is append-only and will not recompute an aggregate when source rows change; a materialized view does. | +Invoke `databricks-pipelines` and `databricks-dabs`. +Inspect all four source schemas before using source columns in expectations. + +```bash +statement=$(cat <<'SQL' +SELECT table_name, column_name, full_data_type +FROM samples.information_schema.columns +WHERE table_schema = 'bakehouse' + AND table_name IN ( + 'sales_transactions', + 'sales_customers', + 'sales_franchises', + 'sales_suppliers' + ) +ORDER BY table_name, column_name +SQL +) + +databricks api post /api/2.0/sql/statements \ + --profile \ + --json "$(jq -n \ + --arg warehouse_id '' \ + --arg statement "$statement" \ + '{warehouse_id: $warehouse_id, statement: $statement, wait_timeout: "50s"}')" \ + | jq -e ' + .result.data_array as $rows + | { + tables: ([$rows[][0]] | unique), + transaction_columns: ([ + $rows[] + | select( + .[0] == "sales_transactions" + and (.[1] == "customerID" or .[1] == "quantity" or .[1] == "franchiseID") + ) + | .[1] + ] | sort) + } + | select( + (.tables | length) == 4 + and .transaction_columns == ["customerID", "franchiseID", "quantity"] + )' +``` -Gold reading from a streaming table needs a **batch** read, `spark.read.table` in Python or `SELECT ... FROM
` without `STREAM` in SQL. Using a streaming read for an aggregation is the most common mistake in this layer. +Expected: four table names and all three required transaction columns. +Stop if `customerID`, `quantity`, or `franchiseID` is absent. ### 2. Write the pipeline source -Invoke `databricks-pipelines` with the collected inputs. Its decision tree picks the dataset types and features; its reference file per feature and language has the exact API. Read the reference file for the feature before writing the code. +Create `src/bakehouse_pipeline/transformations.py`: + +```python +from pyspark import pipelines as dp + +silver_schema = spark.conf.get("silver_schema") + +@dp.materialized_view(name="sales_transactions_raw") +def sales_transactions_raw(): + return spark.read.table("samples.bakehouse.sales_transactions") + +@dp.materialized_view(name="sales_customers_raw") +def sales_customers_raw(): + return spark.read.table("samples.bakehouse.sales_customers") -Shape of a silver streaming table with Auto CDC, in SQL: +@dp.materialized_view(name="sales_franchises_raw") +def sales_franchises_raw(): + return spark.read.table("samples.bakehouse.sales_franchises") -```sql -CREATE OR REFRESH STREAMING TABLE orders_silver - (CONSTRAINT valid_id EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW) -AS SELECT * FROM STREAM read_kafka(....); +@dp.materialized_view(name="sales_suppliers_raw") +def sales_suppliers_raw(): + return spark.read.table("samples.bakehouse.sales_suppliers") + +@dp.materialized_view(name=f"{silver_schema}.transactions_clean") +@dp.expect_or_drop("valid_customer", "customerID IS NOT NULL") +@dp.expect_or_drop("valid_quantity", "quantity > 0") +@dp.expect_or_drop("valid_franchise", "franchiseID IS NOT NULL") +def transactions_clean(): + return spark.read.table("sales_transactions_raw") + +@dp.materialized_view(name=f"{silver_schema}.customers_clean") +def customers_clean(): + return spark.read.table("sales_customers_raw") + +@dp.materialized_view(name=f"{silver_schema}.franchises_clean") +def franchises_clean(): + return spark.read.table("sales_franchises_raw") + +@dp.materialized_view(name=f"{silver_schema}.suppliers_clean") +def suppliers_clean(): + return spark.read.table("sales_suppliers_raw") ``` -Shape of a gold materialized view, in SQL: +The default target publishes these bronze tables: -```sql -CREATE OR REFRESH MATERIALIZED VIEW orders_daily -AS SELECT order_date, count(*) AS orders, sum(amount) AS revenue - FROM ._silver.orders_silver - GROUP BY order_date; +```text +bakehouse_bronze.sales_transactions_raw +bakehouse_bronze.sales_customers_raw +bakehouse_bronze.sales_franchises_raw +bakehouse_bronze.sales_suppliers_raw ``` -Note `CREATE OR REFRESH`, not `CREATE OR REPLACE`. The latter is standard SQL and not valid for SDP datasets. +The configured silver schema publishes these tables: -### 3. Add the resource to the bundle +```text +bakehouse_silver.transactions_clean +bakehouse_silver.customers_clean +bakehouse_silver.franchises_clean +bakehouse_silver.suppliers_clean +``` + +### 3. Add the pipeline resource + +Create `resources/bakehouse_e2e_pipeline.pipeline.yml`: ```yaml -# resources/.pipeline.yml resources: pipelines: - _medallion: - name: ${bundle.name}-medallion + bakehouse_e2e_pipeline: + name: bakehouse_e2e_pipeline catalog: ${var.catalog} - schema: ${var.schema_prefix}_bronze - serverless: true + target: bakehouse_bronze + root_path: ../src/bakehouse_pipeline libraries: - glob: - include: ../src/pipelines/** + include: ../src/bakehouse_pipeline/** + serverless: true + continuous: false + development: true + photon: true + channel: current configuration: - source_path: ${var.source_path} + silver_schema: bakehouse_silver ``` -Targets other than bronze come from fully-qualified dataset names in the source, `${var.catalog}.${var.schema_prefix}_gold.orders_daily`, since the pipeline has one default schema. +Both paths resolve relative to the YAML file under `resources/`. +`${var.catalog}` prevents a hardcoded workspace catalog. -Declare `source_path` as a variable in `databricks.yml` with a per-target value. A dev pipeline reading the production landing path is the failure this prevents. +### 4. Deploy and run -### 4. Validate and deploy to dev +Run from the existing project repository: ```bash -databricks bundle validate --strict --target dev --profile -databricks bundle deploy --target dev --profile -databricks bundle run _medallion --target dev --profile +databricks bundle validate --strict --target dev --profile +databricks bundle deploy --target dev --profile +databricks bundle run bakehouse_e2e_pipeline --target dev --profile ``` -Deploy to dev only. Staging and production go through CI/CD, which is a later section not in this seed. - ## Verify +Prove the bundle update completed and all eight governed tables exist: + ```bash -# Pipeline exists and the last update succeeded -databricks bundle run _medallion --target dev --profile -o json \ - | jq -r '.state, .cause' - -# All three layers materialized -for layer in bronze silver gold; do - echo "== $layer" - databricks tables list --catalog --schema _$layer --profile -o json \ - | jq -r '.[] | .name' +databricks bundle run bakehouse_e2e_pipeline \ + --target dev \ + --profile \ + -o json \ + | jq -e 'select(.state == "COMPLETED")' + +for table in \ + .bakehouse_bronze.sales_transactions_raw \ + .bakehouse_bronze.sales_customers_raw \ + .bakehouse_bronze.sales_franchises_raw \ + .bakehouse_bronze.sales_suppliers_raw \ + .bakehouse_silver.transactions_clean \ + .bakehouse_silver.customers_clean \ + .bakehouse_silver.franchises_clean \ + .bakehouse_silver.suppliers_clean +do + databricks tables get "$table" --profile -o json \ + | jq -er '.full_name' done +``` + +Expected: the run state is `COMPLETED` and all eight exact table names print. + +Define a helper for the SQL Statement Execution API: + +```bash +run_sql() { + local statement=$1 + databricks api post /api/2.0/sql/statements \ + --profile \ + --json "$(jq -n \ + --arg warehouse_id '' \ + --arg statement "$statement" \ + '{warehouse_id: $warehouse_id, statement: $statement, wait_timeout: "50s"}')" +} +``` + +Use one query to prove every silver table has rows: + +```bash +statement=$(cat <<'SQL' +SELECT 'transactions_clean' AS table_name, count(*) AS row_count +FROM .bakehouse_silver.transactions_clean +UNION ALL +SELECT 'customers_clean', count(*) +FROM .bakehouse_silver.customers_clean +UNION ALL +SELECT 'franchises_clean', count(*) +FROM .bakehouse_silver.franchises_clean +UNION ALL +SELECT 'suppliers_clean', count(*) +FROM .bakehouse_silver.suppliers_clean +SQL +) + +run_sql "$statement" \ + | jq -e ' + [.result.data_array[] | {table: .[0], rows: (.[1] | tonumber)}] + | select(length == 4 and all(.[]; .rows > 0))' +``` + +Expected: four rows with `rows` greater than zero. -# Gold has rows, and they came from the source -databricks api post /api/2.0/sql/statements --profile --json '{ - "warehouse_id": "", - "statement": "SELECT count(*) AS rows FROM ._gold.
", - "wait_timeout": "50s" -}' | jq -r '.result.data_array[0][0]' +Prove no invalid transaction rows survived: + +```bash +statement=$(cat <<'SQL' +SELECT + count_if(customerID IS NULL) AS null_customer_ids, + count_if(quantity IS NULL OR quantity <= 0) AS invalid_quantities, + count_if(franchiseID IS NULL) AS null_franchise_ids +FROM .bakehouse_silver.transactions_clean +SQL +) + +run_sql "$statement" \ + | jq -e ' + .result.data_array[0] + | map(tonumber) + | select(. == [0, 0, 0])' ``` -Expected: `COMPLETED`, tables listed in all three schemas, and a non-zero row count in gold. +Expected: `[0, 0, 0]`. -Check the expectations actually fired rather than assuming they are wired: +Prove all expectations emitted numeric counters: ```bash -databricks api post /api/2.0/sql/statements --profile --json '{ - "warehouse_id": "", - "statement": "SELECT explode(from_json(get_json_object(details, \"$.flow_progress.data_quality.expectations\"), \"array>\")) AS e FROM event_log(TABLE(._silver.orders_silver)) WHERE event_type = \"flow_progress\"", - "wait_timeout": "50s" -}' | jq -r '.result.data_array' +statement=$(cat <<'SQL' +SELECT + expectation.name, + expectation.passed_records, + expectation.failed_records +FROM event_log(TABLE(.bakehouse_silver.transactions_clean)) +LATERAL VIEW explode( + from_json( + get_json_object(details, '$.flow_progress.data_quality.expectations'), + 'array>' + ) +) exploded AS expectation +WHERE event_type = 'flow_progress' +QUALIFY row_number() OVER ( + PARTITION BY expectation.name + ORDER BY timestamp DESC +) = 1 +ORDER BY expectation.name +SQL +) + +run_sql "$statement" \ + | jq -e ' + [.result.data_array[] | { + name: .[0], + passed_records: (.[1] | tonumber), + failed_records: (.[2] | tonumber) + }] + | select( + map(.name) == ["valid_customer", "valid_franchise", "valid_quantity"] + and all(.[]; (.passed_records | type) == "number") + and all(.[]; (.failed_records | type) == "number") + )' ``` -Expected: a row per expectation with pass and fail counts. An empty result means the expectations are not attached to the dataset. +Expected: `valid_customer`, `valid_franchise`, and `valid_quantity`, each with numeric pass and fail counters. +An empty result fails the check. ## Where this fails | Symptom | Cause | Fix | |---|---|---| -| Auth precheck blocked: missing named targets | Brief only has a workspace URL or display name | Collect Databricks account id, workspace id, workspace host, and workspace profile name before Run | -| Profile `Valid=NO` | Expired OAuth or SP secret | `databricks auth login --host --profile ` or rotate the SP secret | -| Account id or host mismatch on `auth describe` | Profile points at the wrong account or workspace | Re-login against the named host; confirm account id in the account console | -| `workspace_id` mismatch on `metastores current` | Wrong profile or wrong workspace in the brief | List workspaces and align id, host, and profile | -| `current-user me` fails with Valid profile | Principal not on the workspace | Workspace admin assigns the user or SP | -| `Cannot create streaming table from batch query` | `FROM read_files(...)` instead of `FROM STREAM read_files(...)` | Add `STREAM` | -| `CREATE OR REPLACE` rejected | Not valid for SDP datasets | Use `CREATE OR REFRESH` | -| `Column not found` at ingest | `schemaHints` disagree with the files | Sample the source with `read_files` and align the hints | -| Pipeline stuck `INITIALIZING` on serverless | Cold start | Normal, takes a few minutes. Do not kill it. | -| Gold aggregate never updates when source rows change | Gold is a streaming table, which is append-only | Make it a materialized view with a batch read | -| Materialized view falls back to full recompute | No serverless, or no Delta row tracking on the source | Serverless plus `delta.enableRowTracking = true` | -| SCD2 query returns nothing on `START_AT` | Columns are `__START_AT` and `__END_AT`, double underscore | `WHERE __END_AT IS NULL` for current rows | -| Real error missing from the events output | Reading `.message`, which only says the update failed | Read `error.exceptions[0].message` | -| `databricks fs ls /Volumes/...` errors | Volume paths still need the `dbfs:` prefix | `databricks fs ls dbfs:/Volumes/...` | +| Auth precheck is blocked or targets mismatch | Required auth value is missing or the profile reaches another workspace | Apply the auth remediation table and stop until all checks pass | +| Source inspection returns permission denied | The deployment principal cannot read `samples.bakehouse` | Grant `USE CATALOG`, `USE SCHEMA`, and `SELECT` on the source | +| Deploy reports a missing catalog or schema | The target catalog, `bakehouse_bronze`, or `bakehouse_silver` does not exist | Create the missing governed namespace before deploying | +| Source inspection omits a required column | Bakehouse source column spelling drifted | Update expectations only after confirming the replacement column with the human | +| Pipeline code contains legacy decorators | The source imports the legacy `dlt` module | Migrate to `from pyspark import pipelines as dp` | +| A batch source fails validation as a stream | The source uses a streaming read | Use materialized views with `spark.read.table` | +| Pipeline remains `INITIALIZING` for several minutes | Normal serverless cold start | Wait for the update and do not cancel it | +| Polling reports an idle pipeline before work completes | The check polls top-level pipeline state | Poll the active update or use the blocking bundle run | +| Expectation query returns no rows | Expectations did not attach or no flow progress event contains metrics | Inspect the update event log and fix the decorators before continuing | ## Next diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/index.md b/docs/agentic-starter-journey/content/02-databricks-projects/index.md index 6927f1b..aab2f7f 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/index.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/index.md @@ -13,13 +13,14 @@ One repo, one bundle, one owning team. ## Run in this order -The project repo and bundle must exist before any pipeline lands in it. Ingestion must land data before ETL can transform it. +The project repo and bundle must exist before any pipeline lands in it. +The third page reads the shared Bakehouse batch source and publishes governed bronze and silver materialized views. | Order | Page | Skill | Status | |---|---|---|---| | 1 | [Project repo](/docs/02-databricks-projects/project-repo/) | `databricks-dabs` | Done | | 2 | [Ingestion Pipelines](/docs/02-databricks-projects/ingestion-pipelines/) | `databricks-lakeflow-connect`, `databricks-zerobus-ingest` | Done | -| 3 | [ETL Pipelines](/docs/02-databricks-projects/etl-pipelines/) | `databricks-pipelines` | Done | +| 3 | [Spark Declarative Pipelines](/docs/02-databricks-projects/etl-pipelines/) | `databricks-pipelines` | Done | Pages are linked as they are added. diff --git a/docs/agentic-starter-journey/lib/nav.ts b/docs/agentic-starter-journey/lib/nav.ts index b5e96da..5843480 100644 --- a/docs/agentic-starter-journey/lib/nav.ts +++ b/docs/agentic-starter-journey/lib/nav.ts @@ -38,7 +38,7 @@ export const SECTIONS: Section[] = [ children: [ { label: "Project repo", slug: "02-databricks-projects/project-repo" }, { label: "Ingestion Pipelines", slug: "02-databricks-projects/ingestion-pipelines" }, - { label: "ETL Pipelines", slug: "02-databricks-projects/etl-pipelines" }, + { label: "Spark Declarative Pipelines", slug: "02-databricks-projects/etl-pipelines" }, ], }, ]; From 6c289c773d59dc5b14dc102b3d359a24d650e332 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Tue, 1 Sep 2026 14:12:09 -0600 Subject: [PATCH 02/11] docs: fix Bakehouse SDP polling Poll exact pipeline updates and SQL statements so verification cannot pass on incomplete work or stale expectation metrics. --- .../02-databricks-projects/etl-pipelines.md | 103 +++++++++++++----- 1 file changed, 76 insertions(+), 27 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index c2ec153..00ede98 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -85,6 +85,44 @@ Do not invoke skills, run `bundle validate`, or deploy until auth is green. ### 1. Inspect the batch source Invoke `databricks-pipelines` and `databricks-dabs`. +Define one helper that waits for SQL Statement Execution to finish: + +```bash +run_sql() { + local statement=$1 response statement_id state + + response=$(databricks api post /api/2.0/sql/statements \ + --profile \ + --json "$(jq -n \ + --arg warehouse_id '' \ + --arg statement "$statement" \ + '{warehouse_id: $warehouse_id, statement: $statement, wait_timeout: "0s"}')") \ + || return + statement_id=$(jq -er '.statement_id' <<<"$response") || return + + while :; do + state=$(jq -er '.status.state' <<<"$response") || return + case "$state" in + SUCCEEDED) + printf '%s\n' "$response" + return 0 + ;; + PENDING|RUNNING) + sleep 5 + response=$(databricks api get "/api/2.0/sql/statements/$statement_id" \ + --profile ) || return + ;; + *) + jq -c '.status.error // .status' <<<"$response" >&2 + return 1 + ;; + esac + done +} +``` + +The helper prints results only for `SUCCEEDED`. +It prints the API error and fails for every terminal failure state. Inspect all four source schemas before using source columns in expectations. ```bash @@ -102,12 +140,7 @@ ORDER BY table_name, column_name SQL ) -databricks api post /api/2.0/sql/statements \ - --profile \ - --json "$(jq -n \ - --arg warehouse_id '' \ - --arg statement "$statement" \ - '{warehouse_id: $warehouse_id, statement: $statement, wait_timeout: "50s"}')" \ +run_sql "$statement" \ | jq -e ' .result.data_array as $rows | { @@ -227,7 +260,19 @@ Run from the existing project repository: ```bash databricks bundle validate --strict --target dev --profile databricks bundle deploy --target dev --profile -databricks bundle run bakehouse_e2e_pipeline --target dev --profile + +pipeline_id=$(databricks bundle summary \ + --target dev \ + --profile \ + -o json \ + | jq -er '.resources.pipelines.bakehouse_e2e_pipeline.id') + +update_id=$(databricks bundle run bakehouse_e2e_pipeline \ + --target dev \ + --profile \ + --no-wait \ + -o json \ + | jq -er '.update_id') ``` ## Verify @@ -235,11 +280,28 @@ databricks bundle run bakehouse_e2e_pipeline --target dev --profile \ - -o json \ - | jq -e 'select(.state == "COMPLETED")' +while :; do + update=$(databricks pipelines get-update \ + "$pipeline_id" \ + "$update_id" \ + --profile \ + -o json) || exit 1 + state=$(jq -er '.update.state' <<<"$update") || exit 1 + printf 'update=%s state=%s\n' "$update_id" "$state" + + case "$state" in + COMPLETED) + break + ;; + FAILED|CANCELED) + jq '.update' <<<"$update" >&2 + exit 1 + ;; + *) + sleep 30 + ;; + esac +done for table in \ .bakehouse_bronze.sales_transactions_raw \ @@ -258,20 +320,6 @@ done Expected: the run state is `COMPLETED` and all eight exact table names print. -Define a helper for the SQL Statement Execution API: - -```bash -run_sql() { - local statement=$1 - databricks api post /api/2.0/sql/statements \ - --profile \ - --json "$(jq -n \ - --arg warehouse_id '' \ - --arg statement "$statement" \ - '{warehouse_id: $warehouse_id, statement: $statement, wait_timeout: "50s"}')" -} -``` - Use one query to prove every silver table has rows: ```bash @@ -322,7 +370,7 @@ Expected: `[0, 0, 0]`. Prove all expectations emitted numeric counters: ```bash -statement=$(cat <<'SQL' +statement=$(cat < Date: Tue, 1 Sep 2026 16:53:44 -0600 Subject: [PATCH 03/11] docs: fix SDP auth profile parsing Support the current nested CLI response so healthy workspace credentials do not trigger a false preflight blocker. --- .../02-databricks-projects/etl-pipelines.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 00ede98..2776408 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -26,8 +26,7 @@ Successfully update the pipeline named `bakehouse_e2e_pipeline`. ## Skill -Invoke `databricks-pipelines` before writing the pipeline code. -Invoke `databricks-dabs` before writing the bundle resource. +Read `databricks-pipelines` and `databricks-dabs`, but invoke them only after the auth precheck passes. ## Inputs @@ -55,7 +54,11 @@ Refuse to continue if the brief or prior pages do not name all of these: databricks auth profiles databricks auth describe --profile -o json \ - | jq '{host, account_id}' + | jq '{ + host: (.host // .details.host), + account_id: (.account_id // .details.configuration.account_id.value), + workspace_id: (.workspace_id // .details.configuration.workspace_id.value) + }' databricks current-user me --profile -o json \ | jq '{id, userName}' @@ -67,10 +70,12 @@ databricks metastores current --profile -o json \ Expected: - `` shows `Valid` = `YES` in `auth profiles`. -- `auth describe` `host` equals the named workspace host, and `account_id` equals the named Databricks account id. +- `auth describe` prints `{"host":"https://.cloud.databricks.com","account_id":"","workspace_id":""}` with no null values. +- `auth describe` `host`, `account_id`, and `workspace_id` equal the named values. - `current-user me` succeeds with no auth error. - `metastores current` `workspace_id` equals the named workspace id. +If any projected `auth describe` value is null, inspect the raw `databricks auth describe --profile -o json` response before prescribing re-login. On any failure: print **blocked: auth preflight failed**, list the failing check, give the human the remediation below, and stop. Do not invoke skills, run `bundle validate`, or deploy until auth is green. @@ -78,7 +83,7 @@ Do not invoke skills, run `bundle validate`, or deploy until auth is green. |---|---| | Missing named account id, workspace id, host, or profile | Ask the human for all four before continuing | | Profile `Valid=NO` or auth error on describe | `databricks auth login --host --profile ` (or refresh the SP OAuth secret on the profile) | -| Host or account id mismatch on `auth describe` | Re-login the profile against the named host; confirm the Databricks account id in the account console | +| Host, account id, or workspace id mismatch on `auth describe` | Inspect the raw response, then re-login the profile against the named host if the configured values are wrong; confirm the Databricks account id in the account console | | `workspace_id` mismatch on `metastores current` | `databricks account workspaces list --profile -o json` and align id with the named host | | `current-user me` fails after profile is Valid | Workspace admin assigns the user or SP to the workspace | From 6ca5cbdfb56f76f191b0953f992cf3321bda8536 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Tue, 1 Sep 2026 16:55:44 -0600 Subject: [PATCH 04/11] docs: compact SDP auth output Match the auth projection output to the documented single-line expected JSON. --- .../content/02-databricks-projects/etl-pipelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 2776408..e995f9b 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -54,7 +54,7 @@ Refuse to continue if the brief or prior pages do not name all of these: databricks auth profiles databricks auth describe --profile -o json \ - | jq '{ + | jq -c '{ host: (.host // .details.host), account_id: (.account_id // .details.configuration.account_id.value), workspace_id: (.workspace_id // .details.configuration.workspace_id.value) From 8a0aeb64c9cb44b449774a481166ea4bd8ead71b Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Tue, 1 Sep 2026 17:08:30 -0600 Subject: [PATCH 05/11] docs: align SDP layout with pipeline skill Use one focused source file per dataset so the runbook follows the upstream pipeline structure guidance. --- .../02-databricks-projects/etl-pipelines.md | 71 ++++++++++++++++--- 1 file changed, 61 insertions(+), 10 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index e995f9b..94a5952 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -128,7 +128,7 @@ run_sql() { The helper prints results only for `SUCCEEDED`. It prints the API error and fails for every terminal failure state. -Inspect all four source schemas before using source columns in expectations. +Confirm all four source tables exist, then confirm the three expectation columns on `sales_transactions`. ```bash statement=$(cat <<'SQL' @@ -165,33 +165,59 @@ run_sql "$statement" \ )' ``` -Expected: four table names and all three required transaction columns. +Expected: all four source table names and all three required `sales_transactions` columns. Stop if `customerID`, `quantity`, or `franchiseID` is absent. ### 2. Write the pipeline source -Create `src/bakehouse_pipeline/transformations.py`: +Create one file per dataset. + +Create `src/bakehouse_pipeline/bronze/sales_transactions_raw.py`: ```python from pyspark import pipelines as dp -silver_schema = spark.conf.get("silver_schema") - @dp.materialized_view(name="sales_transactions_raw") def sales_transactions_raw(): return spark.read.table("samples.bakehouse.sales_transactions") +``` + +Create `src/bakehouse_pipeline/bronze/sales_customers_raw.py`: + +```python +from pyspark import pipelines as dp @dp.materialized_view(name="sales_customers_raw") def sales_customers_raw(): return spark.read.table("samples.bakehouse.sales_customers") +``` + +Create `src/bakehouse_pipeline/bronze/sales_franchises_raw.py`: + +```python +from pyspark import pipelines as dp @dp.materialized_view(name="sales_franchises_raw") def sales_franchises_raw(): return spark.read.table("samples.bakehouse.sales_franchises") +``` + +Create `src/bakehouse_pipeline/bronze/sales_suppliers_raw.py`: + +```python +from pyspark import pipelines as dp @dp.materialized_view(name="sales_suppliers_raw") def sales_suppliers_raw(): return spark.read.table("samples.bakehouse.sales_suppliers") +``` + +Create `src/bakehouse_pipeline/silver/transactions_clean.py`: + +```python +from pyspark import pipelines as dp + +silver_schema = spark.conf.get("silver_schema") @dp.materialized_view(name=f"{silver_schema}.transactions_clean") @dp.expect_or_drop("valid_customer", "customerID IS NOT NULL") @@ -199,21 +225,45 @@ def sales_suppliers_raw(): @dp.expect_or_drop("valid_franchise", "franchiseID IS NOT NULL") def transactions_clean(): return spark.read.table("sales_transactions_raw") +``` + +Create `src/bakehouse_pipeline/silver/customers_clean.py`: + +```python +from pyspark import pipelines as dp + +silver_schema = spark.conf.get("silver_schema") @dp.materialized_view(name=f"{silver_schema}.customers_clean") def customers_clean(): return spark.read.table("sales_customers_raw") +``` + +Create `src/bakehouse_pipeline/silver/franchises_clean.py`: + +```python +from pyspark import pipelines as dp + +silver_schema = spark.conf.get("silver_schema") @dp.materialized_view(name=f"{silver_schema}.franchises_clean") def franchises_clean(): return spark.read.table("sales_franchises_raw") +``` + +Create `src/bakehouse_pipeline/silver/suppliers_clean.py`: + +```python +from pyspark import pipelines as dp + +silver_schema = spark.conf.get("silver_schema") @dp.materialized_view(name=f"{silver_schema}.suppliers_clean") def suppliers_clean(): return spark.read.table("sales_suppliers_raw") ``` -The default target publishes these bronze tables: +The default target publishes these bronze materialized views: ```text bakehouse_bronze.sales_transactions_raw @@ -222,7 +272,7 @@ bakehouse_bronze.sales_franchises_raw bakehouse_bronze.sales_suppliers_raw ``` -The configured silver schema publishes these tables: +The configured silver schema publishes these materialized views: ```text bakehouse_silver.transactions_clean @@ -257,6 +307,7 @@ resources: Both paths resolve relative to the YAML file under `resources/`. `${var.catalog}` prevents a hardcoded workspace catalog. +Development mode may prefix the displayed workspace pipeline name, while the resource key `bakehouse_e2e_pipeline` remains the bundle command target. ### 4. Deploy and run @@ -282,7 +333,7 @@ update_id=$(databricks bundle run bakehouse_e2e_pipeline \ ## Verify -Prove the bundle update completed and all eight governed tables exist: +Prove the bundle update completed and all eight governed materialized views exist with the `tables get` API command: ```bash while :; do @@ -323,9 +374,9 @@ do done ``` -Expected: the run state is `COMPLETED` and all eight exact table names print. +Expected: the run state is `COMPLETED` and all eight exact materialized view names print. -Use one query to prove every silver table has rows: +Use one query to prove every silver materialized view has rows: ```bash statement=$(cat <<'SQL' From 0a3c196752c594eaa6bebf02593fbaf801cdee99 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Wed, 2 Sep 2026 09:02:13 -0600 Subject: [PATCH 06/11] docs: group SDP quality expectations Use the grouped same-action decorator so the runbook matches upstream pipeline guidance without changing quality semantics. --- .../content/02-databricks-projects/etl-pipelines.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 94a5952..753864c 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -214,15 +214,19 @@ def sales_suppliers_raw(): Create `src/bakehouse_pipeline/silver/transactions_clean.py`: +Group the three same-action quality rules in one `expect_all_or_drop` decorator. + ```python from pyspark import pipelines as dp silver_schema = spark.conf.get("silver_schema") @dp.materialized_view(name=f"{silver_schema}.transactions_clean") -@dp.expect_or_drop("valid_customer", "customerID IS NOT NULL") -@dp.expect_or_drop("valid_quantity", "quantity > 0") -@dp.expect_or_drop("valid_franchise", "franchiseID IS NOT NULL") +@dp.expect_all_or_drop({ + "valid_customer": "customerID IS NOT NULL", + "valid_quantity": "quantity > 0", + "valid_franchise": "franchiseID IS NOT NULL", +}) def transactions_clean(): return spark.read.table("sales_transactions_raw") ``` @@ -477,7 +481,7 @@ An empty result fails the check. | A batch source fails validation as a stream | The source uses a streaming read | Use materialized views with `spark.read.table` | | Pipeline remains `INITIALIZING` for several minutes | Normal serverless cold start | Wait for the update and do not cancel it | | Polling reports an idle pipeline before work completes | The check polls top-level pipeline state | Poll the active update or use the blocking bundle run | -| Expectation query returns no rows | Expectations did not attach or no flow progress event contains metrics | Inspect the update event log and fix the decorators before continuing | +| Expectation query returns no rows | Expectations did not attach or no flow progress event contains metrics | Inspect the update event log and fix the grouped decorator before continuing | ## Next From 3b1279ea5452e82577092fa59073d398b979db1a Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Wed, 2 Sep 2026 09:11:48 -0600 Subject: [PATCH 07/11] docs: load the pipeline parent skill Load the core CLI guidance first so pipeline work follows the required upstream skill hierarchy after authentication succeeds. --- .../content/02-databricks-projects/etl-pipelines.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 753864c..3858deb 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -26,7 +26,11 @@ Successfully update the pipeline named `bakehouse_e2e_pipeline`. ## Skill -Read `databricks-pipelines` and `databricks-dabs`, but invoke them only after the auth precheck passes. +Read these skill names, but invoke them only after the auth precheck passes: + +1. `databricks-core`, the parent skill used first. +2. `databricks-pipelines`. +3. `databricks-dabs`. ## Inputs @@ -89,7 +93,7 @@ Do not invoke skills, run `bundle validate`, or deploy until auth is green. ### 1. Inspect the batch source -Invoke `databricks-pipelines` and `databricks-dabs`. +Invoke `databricks-core`, then `databricks-pipelines`, then `databricks-dabs`. Define one helper that waits for SQL Statement Execution to finish: ```bash From 08f18cb0a2974c7559220060220fe8af8ed82892 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Wed, 2 Sep 2026 09:14:51 -0600 Subject: [PATCH 08/11] docs: link SDP skill references Link the ordered skills to their verified upstream definitions so agents can load the cited guidance directly. --- .../content/02-databricks-projects/etl-pipelines.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 3858deb..d7d1e60 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -28,9 +28,9 @@ Successfully update the pipeline named `bakehouse_e2e_pipeline`. Read these skill names, but invoke them only after the auth precheck passes: -1. `databricks-core`, the parent skill used first. -2. `databricks-pipelines`. -3. `databricks-dabs`. +1. [`databricks-core`](https://github.com/databricks/databricks-agent-skills/tree/main/plugins/databricks/claude/skills/databricks-core), the parent skill used first. +2. [`databricks-pipelines`](https://github.com/databricks/databricks-agent-skills/tree/main/plugins/databricks/claude/skills/databricks-pipelines). +3. [`databricks-dabs`](https://github.com/databricks/databricks-agent-skills/tree/main/plugins/databricks/claude/skills/databricks-dabs). ## Inputs From 5244c7cd2222fb8f1218ad17d7aa27fbd646c929 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Wed, 2 Sep 2026 09:24:12 -0600 Subject: [PATCH 09/11] docs: verify SDP materialized view types Require the tables API response to prove each governed object is a materialized view instead of accepting names alone. --- .../content/02-databricks-projects/etl-pipelines.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index d7d1e60..83f094b 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -378,11 +378,12 @@ for table in \ .bakehouse_silver.suppliers_clean do databricks tables get "$table" --profile -o json \ - | jq -er '.full_name' + | jq -er 'select(.table_type == "MATERIALIZED_VIEW") | .full_name' done ``` -Expected: the run state is `COMPLETED` and all eight exact materialized view names print. +Expected: the run state is `COMPLETED`, every response has `table_type` equal to `MATERIALIZED_VIEW`, and all eight exact materialized view names print. +Any other object type fails the check. Use one query to prove every silver materialized view has rows: From 52eab71534cf6e1e9a30c5a3336c2f437f9571be Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Wed, 2 Sep 2026 09:33:22 -0600 Subject: [PATCH 10/11] docs: fix SDP workspace mismatch recovery Recover workspace mismatches with the profile data already collected so the runbook does not require an undefined account profile. --- .../content/02-databricks-projects/etl-pipelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 83f094b..6b2bcfb 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -88,7 +88,7 @@ Do not invoke skills, run `bundle validate`, or deploy until auth is green. | Missing named account id, workspace id, host, or profile | Ask the human for all four before continuing | | Profile `Valid=NO` or auth error on describe | `databricks auth login --host --profile ` (or refresh the SP OAuth secret on the profile) | | Host, account id, or workspace id mismatch on `auth describe` | Inspect the raw response, then re-login the profile against the named host if the configured values are wrong; confirm the Databricks account id in the account console | -| `workspace_id` mismatch on `metastores current` | `databricks account workspaces list --profile -o json` and align id with the named host | +| `workspace_id` mismatch on `metastores current` | Inspect the `workspace_id` and `host` projected by `auth describe`, align the human-named workspace ID and host with the profile, and run `databricks auth login --host --profile ` if they differ | | `current-user me` fails after profile is Valid | Workspace admin assigns the user or SP to the workspace | ### 1. Inspect the batch source From e3c8563f96f4ce7c8f29370236b1cd56d0b45998 Mon Sep 17 00:00:00 2001 From: ivancalvo-dbxs Date: Wed, 2 Sep 2026 09:45:35 -0600 Subject: [PATCH 11/11] docs: close SDP verification gaps Fail immediately when any governed object has the wrong type, and keep the ingestion handoff aligned with the renamed SDP page. --- .../content/02-databricks-projects/etl-pipelines.md | 4 ++-- .../content/02-databricks-projects/ingestion-pipelines.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md index 6b2bcfb..1933568 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/etl-pipelines.md @@ -377,8 +377,8 @@ for table in \ .bakehouse_silver.franchises_clean \ .bakehouse_silver.suppliers_clean do - databricks tables get "$table" --profile -o json \ - | jq -er 'select(.table_type == "MATERIALIZED_VIEW") | .full_name' + (databricks tables get "$table" --profile -o json \ + | jq -er 'select(.table_type == "MATERIALIZED_VIEW") | .full_name') || exit 1 done ``` diff --git a/docs/agentic-starter-journey/content/02-databricks-projects/ingestion-pipelines.md b/docs/agentic-starter-journey/content/02-databricks-projects/ingestion-pipelines.md index e7372ec..5a71184 100644 --- a/docs/agentic-starter-journey/content/02-databricks-projects/ingestion-pipelines.md +++ b/docs/agentic-starter-journey/content/02-databricks-projects/ingestion-pipelines.md @@ -175,5 +175,5 @@ For Zerobus, also confirm the client received acknowledgments: ## Next -- **Do next:** [ETL Pipelines](/docs/02-databricks-projects/etl-pipelines/) +- **Do next:** [Spark Declarative Pipelines](/docs/02-databricks-projects/etl-pipelines/) - **Reference:** [Lakeflow Connect](https://docs.databricks.com/aws/en/data-ingestion/ingest/), [Zerobus Ingest](https://docs.databricks.com/ingestion/zerobus-ingest)