You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Changes
<!-- Summary of your changes that are easy to understand. Add
screenshots when necessary -->
This PR is based on #22 and merges internal changes into the flowx repo:
* Airflow support (@peterpark-db)
* Fix missing notebook headers in bridge notebooks (@lorenzorubi-db )
* Emit `sync.include` in generated `databricks.yml` (@alexnastetsky )
* Fixes to documentation (@zanitarahimi)
* Consolidation and conflict resolution (@matthewmoorcroft )
* Updates to GitHub actions (@matthewmoorcroft and @ghanse )
### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes,
fixed, resolve, resolves, resolved. See
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
-->
Resolves#18
### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->
- [x] manually tested
- [x] added unit tests
- [x] added integration tests
---------
Co-authored-by: matthewmoorcroft <31916486+matthewmoorcroft@users.noreply.github.com>
Copy file name to clipboardExpand all lines: README.md
+67-26Lines changed: 67 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,34 +1,42 @@
1
1
# flowx
2
2
3
-
ADF to Databricks Lakeflow Jobs translator, delivered as agent skills.
3
+
Orchestrator-to-Databricks Lakeflow Jobs translator, delivered as agent skills.
4
4
5
-
flowx converts Azure Data Factory (ADF) pipeline definitions into Databricks Lakeflow Jobs packaged as Declarative Automation Bundles (DABs). It deterministically translates known activity types and falls back to agentic (LLM-assisted) translation for complex or rare types. flowx runs as a set of [agent skills](skills/) usable from Databricks Genie Code, Claude Code, or any tool that supports the Agent Skills standard.
5
+
flowx converts a source orchestrator's pipelines — **Azure Data Factory (ADF)** or **Apache
6
+
Airflow** — into Databricks Lakeflow Jobs packaged as Declarative Automation Bundles (DABs). It
7
+
deterministically translates known activity/operator types and falls back to agentic (LLM-assisted)
8
+
translation for complex or rare types. flowx runs as a set of [agent skills](skills/) usable from
9
+
Databricks Genie Code, Claude Code, or any tool that supports the Agent Skills standard.
10
+
11
+
Both sources emit the same source-neutral Pipeline IR, so the convert-configuration and package
12
+
phases are shared; only discovery and translation are source-specific. Pick the source with
13
+
`--source {adf,airflow}` (required for discover/convert; package is source-independent).
Operators without a deterministic mapping become a failing placeholder and are recorded in
191
+
`gaps.json` for review. Eligible leaf gaps can use the fingerprint-bound resolver backed by the pinned [`airflow-to-dabs`](https://github.com/park-peter/airflow-to-dabs/tree/main/providers/flowx-gap-resolver) provider profile; flowx retains ownership of parsing, graph identity, policy, IR, and packaging. Full matrix:
Airflow discovery independently audits DAG declarations, task candidates, dependency declarations,
195
+
DAG settings, mapped calls, and operator arguments before comparing them with captured IR. An
196
+
included DAG is `verified` when every audited construct has a proven translation,
197
+
`verified_with_gaps` when every unsupported construct is linked to a runnable-failure placeholder,
198
+
or `failed` when reconciliation finds unexplained loss. Failed reconciliation exits nonzero and
199
+
blocks package writes. `--exclude-dag <dag_id>` is repeatable; excluded DAGs emit no Job but remain
200
+
visible with zero translated activities in inventory and coverage reporting. This guarantee applies
201
+
to the supported static subset; flowx never imports or executes DAG modules.
202
+
163
203
## How It Works
164
204
165
205
### Phase 1: Discover
166
-
Reads ADF JSON definitions from Unity Catalog volumes (or a `/Workspace` Git folder), normalizes ARM template format, parses into typed AST nodes, and classifies each activity as deterministic, agentic, or unsupported. Produces `metadata/inventory.json` and a per-pipeline complexity report at `metadata/profile_report.csv`.
206
+
Parses the source into typed nodes and classifies each activity/operator as deterministic, agentic, or unsupported — ADF JSON from Unity Catalog volumes (or a `/Workspace` Git folder, normalizing ARM template format), or Airflow DAG `.py` modules read statically with `ast`. Airflow inventory includes audited/deterministic/agentic/failed/excluded counts, reconciliation status, stable finding fingerprints, translation-path coverage, and deterministic coverage. Produces `metadata/inventory.json` and a per-pipeline complexity report at `metadata/profile_report.csv`.
167
207
168
208
### Phase 2: Convert
169
-
Applies deterministic translators via registry dispatch, resolves dependencies through topological sort, and threads immutable `TranslationContext` through control-flow visitors. Agentic gaps are flagged for LLM-assisted translation. Produces Pipeline IR.
209
+
Applies deterministic translators (ADF activity registry / Airflow operator mapping), resolves dependencies, and records unresolved gaps. ADF supports its guided agentic translation workflow. Airflow supports a fingerprint-bound, explicitly reviewed leaf-gap workflow whose constrained provider output is replayed against an immutable deterministic baseline before packaging. Produces the shared Pipeline IR consumed unchanged by the package phase.
170
210
171
211
### Phase 3: Package
172
212
Converts Pipeline IR into a deployable DABs project: `databricks.yml`, per-job YAML resource files, generated Python notebooks, and setup scripts for UC volumes, secrets, and connections.
@@ -180,7 +220,7 @@ flowx_output/
180
220
databricks.yml # Bundle configuration (package)
181
221
resources/
182
222
jobs/
183
-
<pipeline_name>.yml # One job per ADF pipeline
223
+
<pipeline_name>.yml # One Job per included ADF pipeline or Airflow DAG
184
224
src/
185
225
notebooks/
186
226
<pipeline_name>/
@@ -209,11 +249,12 @@ for deployment (SDK notebook or CLI script) and Genie Code registration.
209
249
## Development
210
250
211
251
```bash
212
-
make dev # Install dependencies (uses uv)
213
-
make test# Run unit tests
214
-
make integration # Run integration tests
215
-
make fmt # Format + lint (ruff + mypy)
216
-
make clean # Remove build artifacts
252
+
make dev # Install dependencies (uses uv)
253
+
make test# Run unit tests
254
+
make integration # Run integration tests (excludes the live-Azure suite; gates CI)
255
+
make integration-live # Also run tests needing live ADF access (az login + factory access)
Copy file name to clipboardExpand all lines: docs/content/docs/architecture.mdx
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,7 +28,7 @@ Each activity is classified with a `TranslationStrategy`:
28
28
*`AGENTIC` (LLM-assisted gaps)
29
29
*`UNSUPPORTED`
30
30
31
-
The reporting layer (`reporting/`) can write per-run coverage to a Unity Catalog table and publish an AI/BI dashboard.
31
+
The reporting layer (`reporting/`) can write per-run coverage to a Unity Catalog table and publish an AI/BI dashboard. Airflow rows use independently audited candidates as the denominator and persist reconciliation status, failed/excluded counts, stable finding fingerprints, translation-path coverage, deterministic coverage, reviewed agentic outcomes, and mechanically validated code-attached coverage. Provider-authored code remains distinct from deterministic translation and requires human review.
32
32
33
33
## Two surfaces over one core
34
34
@@ -59,7 +59,7 @@ The unified `flowx.adapter` CLI is the single contract. Both surfaces go through
59
59
|`mcp/runner.py`| Subprocess bridge to `flowx.adapter` with artifact summarizers (for running translation without the `mcp` dependency) |
60
60
|`mcp/__main__.py`|`python -m flowx.mcp` entry point (stdio default, `--http` for hosting) |
61
61
62
-
The `flowx` tool's `command` selects the adapter operation: `inputs`, `discover`, `convert`, `merge_agentic`, `inspect`, `apply_answers`, `materialize_lookup`, `workspace_paths`, `package`, `migrate`, `record_results`, and `install_dashboard` (with `parameters` carrying that command's arguments).
62
+
The `flowx` tool's `command` selects the adapter operation: `inputs`, `discover`, `convert`, `merge_agentic` (ADF only), `resolve_agentic` (Airflow only), `inspect`, `apply_answers`, `materialize_lookup`, `workspace_paths`, `package`, `migrate`, `record_results`, and `install_dashboard` (with `parameters` carrying that command's arguments).
63
63
64
64
## Deployment topology
65
65
@@ -78,7 +78,7 @@ The MCP server runs in whichever transport fits the calling tool. This is chosen
78
78
own service principal
79
79
```
80
80
81
-
See [Installation](/docs/installation#running-flowx-as-an-mcp-server) for the exact commands and the [app README](https://github.com/databricks-solutions/flowx/tree/main/app) for deployment details.
81
+
See [Installation](/flowx/docs/installation#running-flowx-as-an-mcp-server) for the exact commands and the [app README](https://github.com/databricks-solutions/flowx/tree/main/app) for deployment details.
82
82
83
83
<Callouttype="info"title="Inputs and outputs on a hosted app">
84
84
A Databricks App can't read the user's workspace / UC Volume files (`/Volumes/...` is **not** auto-mounted). Two ways to get data in/out of the `flowx` tool:
-**`record-results`** writes one row **per pipeline per run** to the supplied Unity Catalog
151
151
table (`catalog.schema.table`), combining the complexity columns above with the
152
-
deterministic/agentic/unsupported coverage breakdown. Every row is stamped with a shared
152
+
audited/deterministic/agentic/failed/excluded coverage breakdown, reconciliation and migration
153
+
status, finding fingerprints, translation-path coverage, deterministic coverage, unresolved agentic count, reviewed-resolution outcomes/provider version, and code-attached coverage. The corresponding result columns are `resolved_agentic_count`, `unresolved_agentic_count`, and `code_attached_coverage_pct`. Airflow's audited count remains the denominator even for failed or excluded candidates. Code-attached coverage counts deterministic tasks plus accepted `resolved` provider candidates; it means the generated code passed mechanical contract validation, not that its semantics were certified. Every row is stamped with a shared
153
154
**`run_id`** (UUID), **`run_date`** (`CURRENT_TIMESTAMP()`), and **`run_by`**
154
155
(`CURRENT_USER()`), so coverage is trackable across runs and users.
155
156
-**`install-dashboard`** creates and publishes an AI/BI (Lakeview) dashboard over that table —
156
-
KPI counters (pipelines, coverage %, deterministic/agentic/unsupported activity totals), a
157
-
pipelines-by-complexity bar chart, a coverage-over-runs line, and a per-pipeline coverage
158
-
table.
157
+
KPI counters (pipelines, audited activities, and mechanically validated code-attached coverage), failed/excluded totals, a pipelines-by-complexity bar chart, a code-attached-coverage trend, and a per-pipeline table that retains translation-path and deterministic coverage.
159
158
160
159
The SQL warehouse is auto-detected (preferring a running serverless warehouse) when
161
160
`results_warehouse_id` is left blank. Both run via the Databricks SDK and degrade gracefully
Copy file name to clipboardExpand all lines: docs/content/docs/guide.mdx
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -94,6 +94,15 @@ The bundle contains:
94
94
Connection strings, credentials, and other protected configuration parameters are emitted as `SecretInstruction` steps that require [Databricks Secrets](https://docs.databricks.com/aws/en/security/secrets/).
95
95
Run the setup scripts to add any required secret values before deploying and running pipelines in your workspace.
96
96
</Callout>
97
+
98
+
When running with workspace auth (e.g. Genie Code), `package` can optionally persist this run's
99
+
coverage to a Unity Catalog table — one row per pipeline stamped with a UUID `run_id`, `run_date`,
100
+
and `run_by` (`record-results`) — and install a published AI/BI coverage dashboard over that table
101
+
(`install-dashboard`). See [Configuration options](/flowx/docs/options) for details.
102
+
103
+
For Airflow, `activities` is the independent source-audit count rather than the number of tasks the
104
+
translator happened to emit. Reporting distinguishes deterministic, agentic, failed, and excluded
105
+
candidates and carries reconciliation status, translation-path coverage, deterministic coverage, unresolved agentic outcomes, and mechanically validated code-attached coverage. Code attachment is not a certification that provider-authored code is semantically correct.
If calling a skill raises a `ModuleNotFoundError`, the virtual environment is missing or incomplete. Ensure Python is installed
164
-
in your environment and that you have access to a Python package registry for installing dependencies, then re-run `/flowx:flowx-setup`.
165
-
</Callout>
162
+
If you hit a `ModuleNotFoundError` while running a phase, the venv is missing or incomplete — re-run `/flowx:flowx-setup`. Every Python command the skills run uses the interpreter recorded in `<plugin_dir>/.migration-venv`, with `src/` on `PYTHONPATH`:
0 commit comments