PolyFed

A governed query-federation gateway. You send one SQL query and a role, and it runs across PostgreSQL, MySQL and a Parquet file without copying anything into a warehouse. Column masking, row-level security and column-level lineage are applied server-side on every result, and predicates and projections are pushed down to the sources.

DuckDB does the federation. The governance layer is mine, and the hard part was making it hold however the SQL is shaped.

Why the governance cannot be bypassed

A role never queries the raw sources. For each (role, table) pair, polyfed/govern.py compiles the YAML policy into a governed view that projects only the granted columns, replaces masked columns with masking expressions, and appends the role's row filter as the WHERE clause. The user's query runs with search_path set to that role's view schema, so every bare table name binds to a governed view. SELECT *, aliases, subqueries, UNION and CTEs all see only masked columns and permitted rows, because that is all the view has.

The other half is polyfed/sqlguard.py. Before anything executes, the SQL is parsed into DuckDB's AST with json_serialize_sql and checked: one statement, read-only SELECT, and every base-table reference must be an unqualified logical name. Catalog-qualified names like pg.public.customers and table functions like read_parquet are rejected with a 403. Lineage comes from the same AST plus the static catalog.

Four roles live in polyfed/policies.yaml: admin (ungoverned, the correctness oracle), analyst_a (tenant A rows only, ssn and email masked), auditor (all rows, PII masked), and ops_no_pii (tenant B orders and line items, no grant on customers at all).

Results

All numbers are from a 2026-08-11 run against live docker-compose sources with DuckDB 1.3.2, on synthetic seeded data (sources/gen_data.py, seed 42): 22,000 rows across 3 sources and 3 tables.

Federation correctness: 12 queries (single-source, two- and three-source joins, aggregations, HAVING, DISTINCT, subquery) run federated and then against one consolidated in-memory copy. 12 of 12 are byte-identical.

Push-down, measured with source-side counters (pg_stat_statements.rows, MySQL ROWS_SENT, the Parquet scan operator's cardinality) against a pull-everything baseline:

source	baseline rows	pushed-down rows	reduction
PostgreSQL (customers)	2,000	170	91.50%
MySQL (orders)	5,000	325	93.50%
Parquet (line_items)	15,000	4,933	67.11%
total	22,000	5,428	75.33%

Governance: 18 adversarial exfiltration attempts (qualified source access, table functions, information_schema probes, multi-statement injection, masked-column aliasing and filtering, cross-tenant predicates, UNION, CTEs), 0 leaks. Masking precision and recall are both 1.0 over 5,332 PII cells. RLS row counts are exact for all three tables. Lineage: 30 of 30 output columns trace to a source column.

Latency for /query, 360 requests after 20 warm-ups, mixed roles: p50 9.77 ms, p95 21.40 ms, p99 22.35 ms, measured in-process through FastAPI's TestClient, so excluding the network socket.

One caveat on push-down: for MySQL, ROWS_EXAMINED stays at 5,000 either way because the demo table has no index. The win is rows transferred, not rows scanned. RESULTS.md has the methodology and the verbatim SQL captured at each source.

Getting it running

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

docker compose up -d postgres mysql      # the two database sources
python -m sources.seed                    # seed all three sources, writes the Parquet file
pytest -q                                 # 50 passed
python -m bench.run_all                   # regenerate results/*.json

Or docker compose up -d --build for the whole stack with the gateway on port 8000. Then:

# tenant-A analyst: ssn and email come back masked, only tenant_a rows
curl -s -X POST localhost:8000/query -H 'content-type: application/json' \
  -d '{"sql":"SELECT customer_id, ssn, email, tenant FROM customers LIMIT 3","role":"analyst_a"}'

Other endpoints: POST /lineage, GET /policy, /policy/{role}, /roles, /health.

Project structure

polyfed/
  policies.yaml   roles, column grants, masks, row filters
  policy.py       policy model and YAML loader
  govern.py       policy to governed-view SQL
  sqlguard.py     AST validation before execution
  lineage.py      column-level lineage from the AST
  engine.py       DuckDB: ATTACH sources, build views, execute
  api.py          FastAPI routes
sources/          deterministic generator and seeder
bench/            correctness, push-down, governance, latency, run_all
tests/            50 tests
results/          committed benchmark JSON

Things that broke

DuckDB is pinned to 1.3.2. The 1.5.x line has a binder bug (Failed to bind column reference in ColumnBindingResolver) that intermittently crashes on aggregation over a view wrapping a Postgres or MySQL scan, which is exactly the governed-view plus GROUP BY pattern everything here depends on.

Caveats

The gateway serializes DuckDB access behind a lock, since the per-role search_path is connection state. Fine for a demo, not for throughput. Parquet push-down is reported as scan-output rows rather than bytes because the profiler does not expose bytes reliably. Not built: result caching, an /explain endpoint, deployment manifests.