From 11a027588d51d30b78d926cd842946e9bf42a6b0 Mon Sep 17 00:00:00 2001 From: Uday Bhan <158012869+udaybhan05@users.noreply.github.com> Date: Thu, 9 Jul 2026 12:52:04 +0530 Subject: [PATCH 1/2] test(dla): add 125-table Postgres stress fixture and large-scale e2e configs - new fixture at apps/dla/tests/fixtures/postgres_large (port 55433, container dla_fixture_postgres_large) with 5 schemas: star/snowflake regions, conformed dims, 9 junction tables, a no-FK staging zone, 25 generic distractor tables, text-heavy tables, and structural edge cases (reserved-word/quoted/60-char identifiers, composite and multi-column FKs, self-referencing FKs, 110-column wide table, 100k-row tall table, jsonb/uuid/array/enum/range types) - seeded data-quality ground truth: empty tables, all-null and constant columns, 70% null rate, inferred and declared-NOT-VALID broken FKs, orphaned staging joins, mixed-case status values - example configs: postgres_large.yaml (all schemas) and postgres_large_staging_only.yaml (inference-only slice) - fixture README documents every region and expected detection --- apps/dla/config/examples/postgres_large.yaml | 32 ++ .../examples/postgres_large_staging_only.yaml | 27 ++ .../tests/fixtures/postgres_large/README.md | 71 ++++ .../postgres_large/docker-compose.yaml | 18 + .../postgres_large/seed/00_schemas.sql | 16 + .../fixtures/postgres_large/seed/01_sales.sql | 367 ++++++++++++++++++ .../postgres_large/seed/02_finance.sql | 311 +++++++++++++++ .../fixtures/postgres_large/seed/03_hr.sql | 242 ++++++++++++ .../postgres_large/seed/04_staging_nofk.sql | 207 ++++++++++ .../postgres_large/seed/05_analytics_edge.sql | 223 +++++++++++ .../postgres_large/seed/06_distractors.sql | 34 ++ .../postgres_large/seed/07_quality_issues.sql | 93 +++++ 12 files changed, 1641 insertions(+) create mode 100644 apps/dla/config/examples/postgres_large.yaml create mode 100644 apps/dla/config/examples/postgres_large_staging_only.yaml create mode 100644 apps/dla/tests/fixtures/postgres_large/README.md create mode 100644 apps/dla/tests/fixtures/postgres_large/docker-compose.yaml create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/00_schemas.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/01_sales.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/02_finance.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/03_hr.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/04_staging_nofk.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/05_analytics_edge.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/06_distractors.sql create mode 100644 apps/dla/tests/fixtures/postgres_large/seed/07_quality_issues.sql diff --git a/apps/dla/config/examples/postgres_large.yaml b/apps/dla/config/examples/postgres_large.yaml new file mode 100644 index 0000000..e3b9a27 --- /dev/null +++ b/apps/dla/config/examples/postgres_large.yaml @@ -0,0 +1,32 @@ +# Large stress-test fixture (125 tables, 5 schemas) — see +# tests/fixtures/postgres_large/README.md. Bring it up with: +# docker compose -f apps/dla/tests/fixtures/postgres_large/docker-compose.yaml up -d +# export DLA_DB_PASSWORD=dla_dev_password + +source: + source_id: fixture_postgres_large + display_name: Fixture Postgres Large (5-schema stress test) + provider: postgres + postgres: + host: localhost + port: 55433 + database: dla_fixture_large + username: dla + password_env_var: DLA_DB_PASSWORD + schemas: + - sales + - finance + - hr + - staging + - analytics + +runtime: + bundle_dir: ./bundle_large + log_format: console + +thresholds: + name_match_min_score: 0.85 + value_overlap_min_ratio: 0.5 + high_null_rate: 0.5 + high_null_rate_critical: 0.9 + sample_budget_rows: 10000 diff --git a/apps/dla/config/examples/postgres_large_staging_only.yaml b/apps/dla/config/examples/postgres_large_staging_only.yaml new file mode 100644 index 0000000..9721512 --- /dev/null +++ b/apps/dla/config/examples/postgres_large_staging_only.yaml @@ -0,0 +1,27 @@ +# Staging-schema-only slice of the large fixture: ZERO declared foreign keys +# (cloud-warehouse dump simulation). Every relationship must be inferred, which +# is the interesting case for the strategy recommender's signals. + +source: + source_id: fixture_postgres_large_staging + display_name: Fixture Postgres Large (no-FK staging zone only) + provider: postgres + postgres: + host: localhost + port: 55433 + database: dla_fixture_large + username: dla + password_env_var: DLA_DB_PASSWORD + schemas: + - staging + +runtime: + bundle_dir: ./bundle_staging + log_format: console + +thresholds: + name_match_min_score: 0.85 + value_overlap_min_ratio: 0.5 + high_null_rate: 0.5 + high_null_rate_critical: 0.9 + sample_budget_rows: 10000 diff --git a/apps/dla/tests/fixtures/postgres_large/README.md b/apps/dla/tests/fixtures/postgres_large/README.md new file mode 100644 index 0000000..da3182e --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/README.md @@ -0,0 +1,71 @@ +# Large synthetic Postgres fixture (125 tables, 5 schemas) + +A stress-test fixture for exercising every documented `dla` capability at +realistic scale. It complements — and never replaces — the small demo fixture in +`../postgres/` (which stays on port 55432). + +```bash +docker compose -f apps/dla/tests/fixtures/postgres_large/docker-compose.yaml up -d +docker exec dla_fixture_postgres_large pg_isready -U dla -d dla_fixture_large +export DLA_DB_PASSWORD=dla_dev_password +``` + +- Host port: **55433** (container `dla_fixture_postgres_large`) +- Database: `dla_fixture_large`, user `dla`, password `dla_dev_password` +- Total: **125 tables**, ~130k rows (one deliberately tall 100k-row table) + +## Regions + +| Schema | Tables | What it stresses | +| ------ | ------ | ---------------- | +| `sales` | 26 | Star #1: 4 facts sharing conformed dims (`dim_date`, `dim_products`, `dim_stores`); two snowflake chains (`dim_products→dim_subcategories→dim_categories→dim_departments`, `dim_stores→dim_regions→dim_countries→dim_continents`); 3 bridge tables; text-heavy `product_reviews` / `customer_notes`; composite-PK `fact_inventory_snapshots` and `sales_targets`. | +| `finance` | 21 | Star #2; self-referencing `dim_accounts.parent_account_id`; composite PKs (`dim_fiscal_periods`, `fact_invoice_lines`, `purchase_order_lines`, `exchange_rates`); **multi-column FK** `fact_ledger_entries(fiscal_year, fiscal_month) → dim_fiscal_periods`; cross-schema FK `expense_reports.employee_id → hr.employees`; junctions `invoice_payments`, `vendor_contract_links`; text-heavy `audit_journal`, `vendor_contracts`. | +| `hr` | 16 | Self-referencing `employees.manager_id`; 3 junctions; enum column (`hr.employment_status`); **110-column wide table** `employee_survey_wide`; composite-PK `job_history`; text-heavy `performance_reviews`. | +| `staging` | 16 | **The no-FK zone**: zero declared FKs (cloud-warehouse dump). Obvious name/type/value-overlap joins for inference (`stg_orders.stg_customer_id → stg_customers.id`, etc.), one type-mismatch join (`stg_shipments.stg_order_id` VARCHAR), one orphaned join (`stg_returns.stg_order_id` values 900000+), one realistically named column the engine cannot match (`stg_invoices.customer_id`), and two tables with no PK at all (`stg_inventory`, `stg_exchange_rates`). | +| `analytics` | 46 | Edge cases: reserved-word table `"order"` (columns `"select"`, `"group"`), quoted mixed-case `"CamelCaseEvents"`, 60-char identifiers, `typed_showcase` (uuid / jsonb / numeric[] / text[] / enum / interval / bytea / inet / daterange), 100k-row `events_tall`, zero-row `zero_rows_events`; **25 generated distractor tables** all shaped `(id, name, status, created_at)`; 6 seeded quality-issue tables (below). | + +## Seeded quality issues (readiness ground truth) + +| # | Issue | Where | Expected | +| - | ----- | ----- | -------- | +| Q1 | empty table | `analytics.quality_empty_orders` (also `analytics.zero_rows_events`) | Critical | +| Q2 | all-null column | `analytics.quality_users.middle_name`, `analytics.quality_sensor_dump.calibration_note` | Critical | +| Q3 | constant column | `analytics.quality_users.country_code` ('IN'), `analytics.quality_sensor_dump.firmware` ('1.0.0') | Info | +| Q4 | high null rate (~70%) | `analytics.quality_users.referral_code` | Warning | +| Q5 | broken FK (inferred) | `analytics.quality_invoices.dim_customer_id` → `sales.dim_customers.id`, 20% orphans | Critical | +| Q6 | broken FK (declared, `NOT VALID`) | `analytics.quality_orders_notvalid.customer_ref` → `sales.dim_customers.id`, 25% orphans | Critical | +| Q7 | mixed-case categorical | `analytics.quality_status_mix.status` ('active'/'Active'/'ACTEVE'…) | **Known gap** — no M2 check; type_mismatch is deferred | +| Q8 | orphaned inferred join | `staging.stg_returns.stg_order_id` (values 900000+) | Critical broken_fk on an inferred relationship | + +## Pattern-detection ground truth (declared-FK graph only) + +- **Star facts** (≥2 FK targets + own measures): sales `fact_sales`, `fact_returns`, + `fact_shipments`, `fact_inventory_snapshots`, `product_reviews`*, `customer_notes`†; + finance `fact_invoices`, `fact_payments`, `fact_ledger_entries`, `fact_budgets`, + and other multi-FK tables that satisfy the shape (`hr.employees`, + `hr.payroll_items`, `hr.performance_reviews`, …). *Text tables that reference two + dims legitimately satisfy the detector's structural definition. +- **Junctions** (≥2 FK targets, ≤2 non-FK columns): `bridge_product_suppliers`, + `bridge_customer_segments`, `bridge_promotion_channels`, `invoice_payments`, + `vendor_contract_links`, `employee_skills`, `employee_benefits`, + `employee_training`, `job_history`. +- **Snowflakes**: every star fact whose dim itself references onward + (product/store chains in sales; `fact_payments`/`fact_ledger_entries` via + self-referencing `dim_accounts`). +- The `staging` schema should contribute **no declared** relationships — only + inferred ones. + +## Files + +| File | Contents | +| ---- | -------- | +| `seed/00_schemas.sql` | Schemas + enum types | +| `seed/01_sales.sql` | Star/snowflake region #1 (26 tables) | +| `seed/02_finance.sql` | Star region #2, composite/multi-col keys (21 tables) | +| `seed/03_hr.sql` | Self-ref, junctions, 110-col wide table (16 tables) | +| `seed/04_staging_nofk.sql` | No-FK zone (16 tables) | +| `seed/05_analytics_edge.sql` | Structural edge cases (15 tables) | +| `seed/06_distractors.sql` | 25 generated generic-shape distractors | +| `seed/07_quality_issues.sql` | Seeded quality issues (6 tables) | + +Tear down: `docker compose -f apps/dla/tests/fixtures/postgres_large/docker-compose.yaml down -v` diff --git a/apps/dla/tests/fixtures/postgres_large/docker-compose.yaml b/apps/dla/tests/fixtures/postgres_large/docker-compose.yaml new file mode 100644 index 0000000..958365c --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/docker-compose.yaml @@ -0,0 +1,18 @@ +services: + postgres: + image: postgres:16-alpine + container_name: dla_fixture_postgres_large + environment: + POSTGRES_DB: dla_fixture_large + POSTGRES_USER: dla + POSTGRES_PASSWORD: dla_dev_password + PGDATA: /var/lib/postgresql/data/pgdata + ports: + - "55433:5432" + volumes: + - ./seed:/docker-entrypoint-initdb.d:ro + healthcheck: + test: ["CMD-SHELL", "pg_isready -U dla -d dla_fixture_large"] + interval: 2s + timeout: 5s + retries: 30 diff --git a/apps/dla/tests/fixtures/postgres_large/seed/00_schemas.sql b/apps/dla/tests/fixtures/postgres_large/seed/00_schemas.sql new file mode 100644 index 0000000..2c975de --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/00_schemas.sql @@ -0,0 +1,16 @@ +-- Large-scale synthetic fixture (120+ tables across 5 schemas). +-- File 00: schemas + shared enum types. See README.md in this directory for +-- the full map of regions and deliberately seeded issues. + +SET client_min_messages = WARNING; + +CREATE SCHEMA sales; -- star + snowflake region #1 (retail) +CREATE SCHEMA finance; -- star region #2, composite PKs, multi-column FK, self-ref +CREATE SCHEMA hr; -- self-referencing FK, junctions, 110-column wide table +CREATE SCHEMA staging; -- NO declared foreign keys (cloud-warehouse dump simulation) +CREATE SCHEMA analytics; -- structural edge cases, distractors, quality issues + +-- Enum types (unusual-type coverage for discovery/profiling) +CREATE TYPE hr.employment_status AS ENUM ('full_time', 'part_time', 'contract', 'terminated'); +CREATE TYPE analytics.mood_type AS ENUM ('happy', 'neutral', 'sad'); +CREATE TYPE sales.return_reason AS ENUM ('damaged', 'wrong_item', 'too_late', 'changed_mind'); diff --git a/apps/dla/tests/fixtures/postgres_large/seed/01_sales.sql b/apps/dla/tests/fixtures/postgres_large/seed/01_sales.sql new file mode 100644 index 0000000..bb12f24 --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/01_sales.sql @@ -0,0 +1,367 @@ +-- File 01: sales schema — star + snowflake region #1 (26 tables). +-- Ground truth: +-- * Star facts: fact_sales, fact_returns, fact_shipments, fact_inventory_snapshots +-- (each references >= 2 dims and carries its own measures). product_reviews and +-- customer_notes also satisfy the detector's fact shape (>=2 FK targets + extra +-- columns) even though they are semantically text tables. +-- * Snowflake chains: dim_products -> dim_subcategories -> dim_categories -> +-- dim_departments; dim_stores -> dim_regions -> dim_countries -> dim_continents. +-- * Junctions: bridge_product_suppliers, bridge_customer_segments, +-- bridge_promotion_channels. +-- * Conformed dims shared across all four facts: dim_date, dim_stores, dim_products. +-- * Text-heavy (vector signal): product_reviews.review_text, customer_notes.body. + +SET client_min_messages = WARNING; +SET search_path = sales; + +-- ============ snowflake outer layers (created first for FK ordering) ============ + +CREATE TABLE dim_departments ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL UNIQUE +); + +CREATE TABLE dim_categories ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL, + department_id INTEGER NOT NULL REFERENCES dim_departments(id) +); + +CREATE TABLE dim_subcategories ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL, + category_id INTEGER NOT NULL REFERENCES dim_categories(id) +); + +CREATE TABLE dim_continents ( + id SERIAL PRIMARY KEY, + name VARCHAR(40) NOT NULL UNIQUE +); + +CREATE TABLE dim_countries ( + id SERIAL PRIMARY KEY, + iso_code CHAR(2) NOT NULL UNIQUE, + name VARCHAR(80) NOT NULL, + continent_id INTEGER NOT NULL REFERENCES dim_continents(id) +); + +CREATE TABLE dim_regions ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL, + country_id INTEGER NOT NULL REFERENCES dim_countries(id) +); + +-- ============ conformed dimensions ============ + +CREATE TABLE dim_date ( + date_key INTEGER PRIMARY KEY, -- yyyymmdd + full_date DATE NOT NULL UNIQUE, + year SMALLINT NOT NULL, + quarter SMALLINT NOT NULL, + month SMALLINT NOT NULL, + day_of_week SMALLINT NOT NULL, + is_weekend BOOLEAN NOT NULL +); + +CREATE TABLE dim_customers ( + id SERIAL PRIMARY KEY, + customer_code VARCHAR(24) NOT NULL UNIQUE, + full_name VARCHAR(160) NOT NULL, + email VARCHAR(255), + signed_up_on DATE NOT NULL, + loyalty_tier VARCHAR(16) NOT NULL DEFAULT 'bronze', + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE dim_products ( + id SERIAL PRIMARY KEY, + sku VARCHAR(40) NOT NULL UNIQUE, + name VARCHAR(160) NOT NULL, + subcategory_id INTEGER NOT NULL REFERENCES dim_subcategories(id), + list_price NUMERIC(10,2) NOT NULL, + is_active BOOLEAN NOT NULL DEFAULT TRUE, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE dim_stores ( + id SERIAL PRIMARY KEY, + store_code VARCHAR(16) NOT NULL UNIQUE, + name VARCHAR(120) NOT NULL, + region_id INTEGER NOT NULL REFERENCES dim_regions(id), + opened_on DATE NOT NULL, + sq_meters INTEGER +); + +CREATE TABLE dim_channels ( + id SERIAL PRIMARY KEY, + name VARCHAR(40) NOT NULL UNIQUE +); + +CREATE TABLE dim_promotions ( + id SERIAL PRIMARY KEY, + code VARCHAR(40) NOT NULL UNIQUE, + description TEXT, + discount_pct NUMERIC(5,2) NOT NULL, + valid_from DATE NOT NULL, + valid_to DATE NOT NULL +); + +CREATE TABLE dim_payment_methods ( + id SERIAL PRIMARY KEY, + name VARCHAR(40) NOT NULL UNIQUE +); + +CREATE TABLE dim_currencies ( + code CHAR(3) PRIMARY KEY, + name VARCHAR(40) NOT NULL, + symbol VARCHAR(4) +); + +-- ============ support tables ============ + +CREATE TABLE suppliers ( + id SERIAL PRIMARY KEY, + name VARCHAR(120) NOT NULL, + country_id INTEGER REFERENCES dim_countries(id), + rating NUMERIC(3,1), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE segments ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL UNIQUE, + description TEXT +); + +-- ============ junction / bridge tables ============ + +CREATE TABLE bridge_product_suppliers ( + product_id INTEGER NOT NULL REFERENCES dim_products(id), + supplier_id INTEGER NOT NULL REFERENCES suppliers(id), + since DATE, + PRIMARY KEY (product_id, supplier_id) +); + +CREATE TABLE bridge_customer_segments ( + customer_id INTEGER NOT NULL REFERENCES dim_customers(id), + segment_id INTEGER NOT NULL REFERENCES segments(id), + PRIMARY KEY (customer_id, segment_id) +); + +CREATE TABLE bridge_promotion_channels ( + promotion_id INTEGER NOT NULL REFERENCES dim_promotions(id), + channel_id INTEGER NOT NULL REFERENCES dim_channels(id), + PRIMARY KEY (promotion_id, channel_id) +); + +-- ============ facts ============ + +CREATE TABLE fact_sales ( + id BIGSERIAL PRIMARY KEY, + date_key INTEGER NOT NULL REFERENCES dim_date(date_key), + customer_id INTEGER NOT NULL REFERENCES dim_customers(id), + product_id INTEGER NOT NULL REFERENCES dim_products(id), + store_id INTEGER NOT NULL REFERENCES dim_stores(id), + channel_id INTEGER NOT NULL REFERENCES dim_channels(id), + promotion_id INTEGER REFERENCES dim_promotions(id), + payment_method_id INTEGER NOT NULL REFERENCES dim_payment_methods(id), + currency_code CHAR(3) NOT NULL REFERENCES dim_currencies(code), + quantity INTEGER NOT NULL CHECK (quantity > 0), + unit_price NUMERIC(10,2) NOT NULL, + discount_amount NUMERIC(10,2) NOT NULL DEFAULT 0, + tax_amount NUMERIC(10,2) NOT NULL DEFAULT 0, + net_amount NUMERIC(12,2) NOT NULL +); +CREATE INDEX idx_fact_sales_date ON fact_sales(date_key); +CREATE INDEX idx_fact_sales_customer ON fact_sales(customer_id); +CREATE INDEX idx_fact_sales_product ON fact_sales(product_id); + +CREATE TABLE fact_returns ( + id BIGSERIAL PRIMARY KEY, + date_key INTEGER NOT NULL REFERENCES dim_date(date_key), + customer_id INTEGER NOT NULL REFERENCES dim_customers(id), + product_id INTEGER NOT NULL REFERENCES dim_products(id), + store_id INTEGER NOT NULL REFERENCES dim_stores(id), + reason sales.return_reason NOT NULL, + quantity INTEGER NOT NULL, + amount NUMERIC(12,2) NOT NULL +); + +CREATE TABLE fact_shipments ( + id BIGSERIAL PRIMARY KEY, + date_key INTEGER NOT NULL REFERENCES dim_date(date_key), + store_id INTEGER NOT NULL REFERENCES dim_stores(id), + product_id INTEGER NOT NULL REFERENCES dim_products(id), + cartons INTEGER NOT NULL, + weight_kg NUMERIC(10,3), + freight_cost NUMERIC(12,2) +); + +CREATE TABLE fact_inventory_snapshots ( + date_key INTEGER NOT NULL REFERENCES dim_date(date_key), + product_id INTEGER NOT NULL REFERENCES dim_products(id), + store_id INTEGER NOT NULL REFERENCES dim_stores(id), + on_hand INTEGER NOT NULL, + reserved INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (date_key, product_id, store_id) -- composite PK, no surrogate +); + +-- ============ text-heavy tables (vector signal) ============ + +CREATE TABLE product_reviews ( + id SERIAL PRIMARY KEY, + product_id INTEGER NOT NULL REFERENCES dim_products(id), + customer_id INTEGER NOT NULL REFERENCES dim_customers(id), + rating SMALLINT NOT NULL CHECK (rating BETWEEN 1 AND 5), + review_text TEXT NOT NULL, + reviewed_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE customer_notes ( + id SERIAL PRIMARY KEY, + customer_id INTEGER NOT NULL REFERENCES dim_customers(id), + author VARCHAR(80) NOT NULL, + body TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Composite-PK planning table (no surrogate key) +CREATE TABLE sales_targets ( + store_id INTEGER NOT NULL REFERENCES dim_stores(id), + fiscal_year SMALLINT NOT NULL, + fiscal_quarter SMALLINT NOT NULL, + target_amount NUMERIC(14,2) NOT NULL, + PRIMARY KEY (store_id, fiscal_year, fiscal_quarter) +); + +-- ============ seed data ============ + +INSERT INTO dim_departments (name) VALUES ('Softlines'), ('Hardlines'), ('Grocery'); +INSERT INTO dim_categories (name, department_id) VALUES + ('Apparel', 1), ('Footwear', 1), ('Electronics', 2), ('Home', 2), ('Snacks', 3), ('Beverages', 3); +INSERT INTO dim_subcategories (name, category_id) +SELECT c.name || ' - sub ' || s, c.id FROM dim_categories c, generate_series(1, 3) s; + +INSERT INTO dim_continents (name) VALUES ('Europe'), ('North America'), ('Asia'); +INSERT INTO dim_countries (iso_code, name, continent_id) VALUES + ('GB', 'United Kingdom', 1), ('DE', 'Germany', 1), ('US', 'United States', 2), + ('CA', 'Canada', 2), ('IN', 'India', 3), ('JP', 'Japan', 3); +INSERT INTO dim_regions (name, country_id) +SELECT c.name || ' region ' || s, c.id FROM dim_countries c, generate_series(1, 2) s; + +INSERT INTO dim_date (date_key, full_date, year, quarter, month, day_of_week, is_weekend) +SELECT to_char(d, 'YYYYMMDD')::int, d::date, + EXTRACT(year FROM d)::smallint, EXTRACT(quarter FROM d)::smallint, + EXTRACT(month FROM d)::smallint, EXTRACT(isodow FROM d)::smallint, + EXTRACT(isodow FROM d) IN (6, 7) +FROM generate_series('2025-01-01'::date, '2026-06-30'::date, interval '1 day') d; + +INSERT INTO dim_customers (customer_code, full_name, email, signed_up_on, loyalty_tier) +SELECT 'CUST-' || lpad(i::text, 5, '0'), + 'Customer ' || i, + 'customer' || i || '@example.com', + DATE '2025-01-01' + (i % 500), + (ARRAY['bronze','silver','gold'])[1 + i % 3] +FROM generate_series(1, 400) i; + +INSERT INTO dim_products (sku, name, subcategory_id, list_price) +SELECT 'SKU-' || lpad(i::text, 5, '0'), + 'Product ' || i, + 1 + (i % 18), + round((5 + random() * 195)::numeric, 2) +FROM generate_series(1, 250) i; + +INSERT INTO dim_stores (store_code, name, region_id, opened_on, sq_meters) +SELECT 'ST-' || lpad(i::text, 3, '0'), + 'Store ' || i, + 1 + (i % 12), + DATE '2020-01-01' + (i * 30), + 400 + (i * 25) +FROM generate_series(1, 40) i; + +INSERT INTO dim_channels (name) VALUES ('in_store'), ('web'), ('mobile_app'), ('marketplace'); +INSERT INTO dim_promotions (code, description, discount_pct, valid_from, valid_to) +SELECT 'PROMO-' || i, 'Promotion number ' || i, 5 + (i % 4) * 5, + DATE '2025-01-01' + i * 10, DATE '2025-01-01' + i * 10 + 30 +FROM generate_series(1, 20) i; +INSERT INTO dim_payment_methods (name) VALUES ('card'), ('cash'), ('wallet'), ('bank_transfer'); +INSERT INTO dim_currencies (code, name, symbol) VALUES + ('USD', 'US Dollar', '$'), ('EUR', 'Euro', E'€'), ('GBP', 'Pound Sterling', E'£'), ('INR', 'Indian Rupee', E'₹'); + +INSERT INTO suppliers (name, country_id, rating) +SELECT 'Supplier ' || i, 1 + (i % 6), round((1 + random() * 4)::numeric, 1) +FROM generate_series(1, 30) i; + +INSERT INTO segments (name, description) VALUES + ('high_value', 'Top decile of lifetime spend'), + ('lapsed', 'No purchase in 180 days'), + ('new', 'First purchase within 30 days'), + ('promo_hunter', 'Purchases predominantly on promotion'); + +INSERT INTO bridge_product_suppliers (product_id, supplier_id, since) +SELECT i, 1 + (i % 30), DATE '2024-01-01' + i FROM generate_series(1, 250) i; +INSERT INTO bridge_product_suppliers (product_id, supplier_id, since) +SELECT i, 1 + ((i + 7) % 30), DATE '2024-06-01' + i FROM generate_series(1, 120) i; + +INSERT INTO bridge_customer_segments (customer_id, segment_id) +SELECT i, 1 + (i % 4) FROM generate_series(1, 400) i; + +INSERT INTO bridge_promotion_channels (promotion_id, channel_id) +SELECT p, c FROM generate_series(1, 20) p, generate_series(1, 4) c WHERE (p + c) % 2 = 0; + +INSERT INTO fact_sales (date_key, customer_id, product_id, store_id, channel_id, + promotion_id, payment_method_id, currency_code, + quantity, unit_price, discount_amount, tax_amount, net_amount) +SELECT (SELECT date_key FROM dim_date ORDER BY date_key OFFSET (i % 540) LIMIT 1), + 1 + (i % 400), + 1 + (i % 250), + 1 + (i % 40), + 1 + (i % 4), + CASE WHEN i % 5 = 0 THEN 1 + (i % 20) END, + 1 + (i % 4), + (ARRAY['USD','EUR','GBP','INR'])[1 + i % 4], + 1 + (i % 5), + round((5 + (i % 200))::numeric, 2), + CASE WHEN i % 5 = 0 THEN 2.50 ELSE 0 END, + round(((5 + (i % 200)) * 0.08)::numeric, 2), + round(((5 + (i % 200)) * 1.08 * (1 + i % 5))::numeric, 2) +FROM generate_series(1, 5000) i; + +INSERT INTO fact_returns (date_key, customer_id, product_id, store_id, reason, quantity, amount) +SELECT (SELECT date_key FROM dim_date ORDER BY date_key OFFSET (i % 540) LIMIT 1), + 1 + (i % 400), 1 + (i % 250), 1 + (i % 40), + (ARRAY['damaged','wrong_item','too_late','changed_mind'])[1 + i % 4]::sales.return_reason, + 1, round((5 + (i % 120))::numeric, 2) +FROM generate_series(1, 300) i; + +INSERT INTO fact_shipments (date_key, store_id, product_id, cartons, weight_kg, freight_cost) +SELECT (SELECT date_key FROM dim_date ORDER BY date_key OFFSET (i % 540) LIMIT 1), + 1 + (i % 40), 1 + (i % 250), 1 + (i % 12), + round((0.5 + (i % 90))::numeric, 3), round((10 + (i % 300))::numeric, 2) +FROM generate_series(1, 800) i; + +INSERT INTO fact_inventory_snapshots (date_key, product_id, store_id, on_hand, reserved) +SELECT (SELECT date_key FROM dim_date ORDER BY date_key OFFSET (i % 30) LIMIT 1), + 1 + (i % 250), 1 + ((i / 250) % 40), (i * 7) % 500, (i * 3) % 40 +FROM generate_series(0, 1999) i; + +INSERT INTO product_reviews (product_id, customer_id, rating, review_text) +SELECT 1 + (i % 250), 1 + (i % 400), 1 + (i % 5), + 'I have been using this product for ' || (1 + i % 11) || ' weeks now and the build quality ' + || 'continues to impress me. The finish feels premium, delivery was quick, and the ' + || 'packaging was thoughtful. My only complaint is that the instruction booklet is vague ' + || 'about maintenance, so I had to search online forums for cleaning advice. Overall I ' + || 'would recommend it to a friend who cares about durability more than price. Review #' || i +FROM generate_series(1, 350) i; + +INSERT INTO customer_notes (customer_id, author, body) +SELECT 1 + (i % 400), 'agent_' || (1 + i % 12), + 'Spoke with the customer about their recent delivery delay. They were understanding but ' + || 'asked to be notified proactively next time a shipment slips. Flagged the account for ' + || 'the loyalty win-back campaign and promised a follow-up call within five business days. ' + || 'Customer prefers email over phone for routine updates. Interaction log entry ' || i +FROM generate_series(1, 200) i; + +INSERT INTO sales_targets (store_id, fiscal_year, fiscal_quarter, target_amount) +SELECT s, y, q, 100000 + s * 1000 + q * 500 +FROM generate_series(1, 40) s, generate_series(2025, 2026) y, generate_series(1, 4) q; diff --git a/apps/dla/tests/fixtures/postgres_large/seed/02_finance.sql b/apps/dla/tests/fixtures/postgres_large/seed/02_finance.sql new file mode 100644 index 0000000..5670392 --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/02_finance.sql @@ -0,0 +1,311 @@ +-- File 02: finance schema — star region #2 (20 tables). +-- Ground truth: +-- * Self-referencing FK: dim_accounts.parent_account_id -> dim_accounts.id. +-- * Composite PKs: dim_fiscal_periods, fact_invoice_lines, purchase_order_lines, +-- exchange_rates. +-- * Multi-column FK: fact_ledger_entries(fiscal_year, fiscal_month) -> +-- dim_fiscal_periods(fiscal_year, fiscal_month). +-- * Cross-schema declared FK: expense_reports.employee_id -> hr.employees(id) +-- (added in 03_hr.sql after hr.employees exists). +-- * Junctions: invoice_payments, vendor_contract_links. +-- * Star facts: fact_invoices, fact_payments, fact_ledger_entries, fact_budgets. +-- * Text-heavy: audit_journal.narrative. + +SET client_min_messages = WARNING; +SET search_path = finance; + +CREATE TABLE dim_accounts ( + id SERIAL PRIMARY KEY, + account_code VARCHAR(20) NOT NULL UNIQUE, + name VARCHAR(120) NOT NULL, + account_type VARCHAR(20) NOT NULL, + parent_account_id INTEGER REFERENCES dim_accounts(id) -- self-referencing FK +); + +CREATE TABLE dim_cost_centers ( + id SERIAL PRIMARY KEY, + code VARCHAR(16) NOT NULL UNIQUE, + name VARCHAR(120) NOT NULL, + manager VARCHAR(120) +); + +CREATE TABLE dim_vendors ( + id SERIAL PRIMARY KEY, + name VARCHAR(160) NOT NULL, + tax_id VARCHAR(32), + country CHAR(2), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE dim_gl_codes ( + id SERIAL PRIMARY KEY, + gl_code VARCHAR(12) NOT NULL UNIQUE, + description VARCHAR(200) +); + +CREATE TABLE dim_fiscal_periods ( + fiscal_year SMALLINT NOT NULL, + fiscal_month SMALLINT NOT NULL, + starts_on DATE NOT NULL, + ends_on DATE NOT NULL, + is_closed BOOLEAN NOT NULL DEFAULT FALSE, + PRIMARY KEY (fiscal_year, fiscal_month) -- composite PK +); + +CREATE TABLE fact_invoices ( + id SERIAL PRIMARY KEY, + invoice_number VARCHAR(32) NOT NULL UNIQUE, + vendor_id INTEGER NOT NULL REFERENCES dim_vendors(id), + cost_center_id INTEGER NOT NULL REFERENCES dim_cost_centers(id), + issued_on DATE NOT NULL, + due_on DATE NOT NULL, + status VARCHAR(24) NOT NULL, + subtotal NUMERIC(14,2) NOT NULL, + tax NUMERIC(14,2) NOT NULL DEFAULT 0, + total NUMERIC(14,2) NOT NULL +); + +CREATE TABLE fact_invoice_lines ( + invoice_id INTEGER NOT NULL REFERENCES fact_invoices(id) ON DELETE CASCADE, + line_no SMALLINT NOT NULL, + gl_code_id INTEGER NOT NULL REFERENCES dim_gl_codes(id), + description VARCHAR(240), + quantity NUMERIC(10,2) NOT NULL DEFAULT 1, + unit_cost NUMERIC(12,2) NOT NULL, + amount NUMERIC(14,2) NOT NULL, + PRIMARY KEY (invoice_id, line_no) -- composite PK +); + +CREATE TABLE fact_payments ( + id SERIAL PRIMARY KEY, + payment_ref VARCHAR(32) NOT NULL UNIQUE, + vendor_id INTEGER NOT NULL REFERENCES dim_vendors(id), + account_id INTEGER NOT NULL REFERENCES dim_accounts(id), + paid_on DATE NOT NULL, + amount NUMERIC(14,2) NOT NULL, + method VARCHAR(24) NOT NULL +); + +-- Junction: invoices <-> payments (partial payments, batch payments) +CREATE TABLE invoice_payments ( + invoice_id INTEGER NOT NULL REFERENCES fact_invoices(id), + payment_id INTEGER NOT NULL REFERENCES fact_payments(id), + applied NUMERIC(14,2), + PRIMARY KEY (invoice_id, payment_id) +); + +CREATE TABLE fact_ledger_entries ( + id BIGSERIAL PRIMARY KEY, + account_id INTEGER NOT NULL REFERENCES dim_accounts(id), + cost_center_id INTEGER NOT NULL REFERENCES dim_cost_centers(id), + fiscal_year SMALLINT NOT NULL, + fiscal_month SMALLINT NOT NULL, + entry_date DATE NOT NULL, + debit NUMERIC(14,2) NOT NULL DEFAULT 0, + credit NUMERIC(14,2) NOT NULL DEFAULT 0, + memo VARCHAR(240), + FOREIGN KEY (fiscal_year, fiscal_month) -- multi-column FK + REFERENCES dim_fiscal_periods(fiscal_year, fiscal_month) +); + +CREATE TABLE fact_budgets ( + id SERIAL PRIMARY KEY, + cost_center_id INTEGER NOT NULL REFERENCES dim_cost_centers(id), + account_id INTEGER NOT NULL REFERENCES dim_accounts(id), + fiscal_year SMALLINT NOT NULL, + amount NUMERIC(14,2) NOT NULL +); + +CREATE TABLE tax_rates ( + id SERIAL PRIMARY KEY, + jurisdiction VARCHAR(80) NOT NULL, + rate_pct NUMERIC(6,3) NOT NULL, + valid_from DATE NOT NULL +); + +CREATE TABLE exchange_rates ( + currency_code CHAR(3) NOT NULL, + rate_date DATE NOT NULL, + usd_rate NUMERIC(14,6) NOT NULL, + PRIMARY KEY (currency_code, rate_date) -- composite PK +); + +CREATE TABLE purchase_orders ( + id SERIAL PRIMARY KEY, + po_number VARCHAR(24) NOT NULL UNIQUE, + vendor_id INTEGER NOT NULL REFERENCES dim_vendors(id), + ordered_on DATE NOT NULL, + status VARCHAR(24) NOT NULL, + total NUMERIC(14,2) +); + +CREATE TABLE purchase_order_lines ( + po_id INTEGER NOT NULL REFERENCES purchase_orders(id) ON DELETE CASCADE, + line_no SMALLINT NOT NULL, + item_desc VARCHAR(240) NOT NULL, + quantity NUMERIC(10,2) NOT NULL, + unit_cost NUMERIC(12,2) NOT NULL, + PRIMARY KEY (po_id, line_no) +); + +CREATE TABLE vendor_contracts ( + id SERIAL PRIMARY KEY, + vendor_id INTEGER NOT NULL REFERENCES dim_vendors(id), + starts_on DATE NOT NULL, + ends_on DATE, + terms TEXT +); + +-- Junction: contracts <-> cost centers that draw on them +CREATE TABLE vendor_contract_links ( + contract_id INTEGER NOT NULL REFERENCES vendor_contracts(id), + cost_center_id INTEGER NOT NULL REFERENCES dim_cost_centers(id), + PRIMARY KEY (contract_id, cost_center_id) +); + +CREATE TABLE expense_reports ( + id SERIAL PRIMARY KEY, + employee_id INTEGER NOT NULL, -- FK to hr.employees added in 03_hr.sql + submitted_on DATE NOT NULL, + status VARCHAR(24) NOT NULL, + total NUMERIC(12,2) NOT NULL +); + +CREATE TABLE expense_lines ( + id SERIAL PRIMARY KEY, + report_id INTEGER NOT NULL REFERENCES expense_reports(id) ON DELETE CASCADE, + category VARCHAR(40) NOT NULL, + amount NUMERIC(12,2) NOT NULL, + receipt_url VARCHAR(240) +); + +CREATE TABLE payment_batches ( + id SERIAL PRIMARY KEY, + batch_ref VARCHAR(32) NOT NULL UNIQUE, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + row_count INTEGER NOT NULL DEFAULT 0 +); + +CREATE TABLE audit_journal ( + id BIGSERIAL PRIMARY KEY, + occurred_at TIMESTAMPTZ NOT NULL DEFAULT now(), + actor VARCHAR(120) NOT NULL, + action VARCHAR(80) NOT NULL, + narrative TEXT NOT NULL, + details JSONB +); + +-- ============ seed data ============ + +INSERT INTO dim_accounts (account_code, name, account_type, parent_account_id) VALUES + ('1000', 'Assets', 'asset', NULL), + ('1100', 'Cash', 'asset', 1), + ('1200', 'Receivables', 'asset', 1), + ('2000', 'Liabilities', 'liability', NULL), + ('2100', 'Payables', 'liability', 4), + ('4000', 'Revenue', 'revenue', NULL), + ('4100', 'Product revenue', 'revenue', 6), + ('5000', 'Expenses', 'expense', NULL), + ('5100', 'Freight', 'expense', 8), + ('5200', 'Payroll', 'expense', 8); + +INSERT INTO dim_cost_centers (code, name, manager) +SELECT 'CC-' || lpad(i::text, 3, '0'), 'Cost center ' || i, 'Manager ' || i +FROM generate_series(1, 12) i; + +INSERT INTO dim_vendors (name, tax_id, country) +SELECT 'Vendor ' || i, 'TAX-' || lpad(i::text, 6, '0'), (ARRAY['US','GB','DE','IN'])[1 + i % 4] +FROM generate_series(1, 60) i; + +INSERT INTO dim_gl_codes (gl_code, description) +SELECT 'GL-' || lpad(i::text, 4, '0'), 'General ledger code ' || i +FROM generate_series(1, 25) i; + +INSERT INTO dim_fiscal_periods (fiscal_year, fiscal_month, starts_on, ends_on, is_closed) +SELECT y, m, make_date(y, m, 1), (make_date(y, m, 1) + interval '1 month - 1 day')::date, y = 2025 +FROM generate_series(2025, 2026) y, generate_series(1, 12) m; + +INSERT INTO fact_invoices (invoice_number, vendor_id, cost_center_id, issued_on, due_on, status, subtotal, tax, total) +SELECT 'INV-' || lpad(i::text, 6, '0'), 1 + (i % 60), 1 + (i % 12), + DATE '2025-01-05' + i, DATE '2025-02-05' + i, + (ARRAY['draft','approved','paid','void'])[1 + i % 4], + round((100 + i * 3)::numeric, 2), round(((100 + i * 3) * 0.18)::numeric, 2), + round(((100 + i * 3) * 1.18)::numeric, 2) +FROM generate_series(1, 400) i; + +INSERT INTO fact_invoice_lines (invoice_id, line_no, gl_code_id, description, quantity, unit_cost, amount) +SELECT 1 + (i / 3), 1 + (i % 3), 1 + (i % 25), 'Line item ' || i, 1 + (i % 4), + round((20 + i % 300)::numeric, 2), round(((20 + i % 300) * (1 + i % 4))::numeric, 2) +FROM generate_series(0, 1100) i; + +INSERT INTO fact_payments (payment_ref, vendor_id, account_id, paid_on, amount, method) +SELECT 'PAY-' || lpad(i::text, 6, '0'), 1 + (i % 60), 2, DATE '2025-02-01' + i, + round((100 + i * 2.7)::numeric, 2), (ARRAY['ach','wire','check'])[1 + i % 3] +FROM generate_series(1, 300) i; + +INSERT INTO invoice_payments (invoice_id, payment_id, applied) +SELECT 1 + (i % 400), 1 + (i % 300), round((50 + i)::numeric, 2) +FROM generate_series(1, 350) i +ON CONFLICT DO NOTHING; + +INSERT INTO fact_ledger_entries (account_id, cost_center_id, fiscal_year, fiscal_month, entry_date, debit, credit, memo) +SELECT 1 + (i % 10), 1 + (i % 12), 2025 + (i % 2), 1 + (i % 12), + make_date(2025 + (i % 2), 1 + (i % 12), 1 + (i % 28)), + CASE WHEN i % 2 = 0 THEN round((10 + i % 900)::numeric, 2) ELSE 0 END, + CASE WHEN i % 2 = 1 THEN round((10 + i % 900)::numeric, 2) ELSE 0 END, + 'Journal memo ' || i +FROM generate_series(1, 2000) i; + +INSERT INTO fact_budgets (cost_center_id, account_id, fiscal_year, amount) +SELECT c, a, y, 50000 + c * 100 + a * 10 +FROM generate_series(1, 12) c, generate_series(1, 10) a, generate_series(2025, 2026) y; + +INSERT INTO tax_rates (jurisdiction, rate_pct, valid_from) +SELECT 'Jurisdiction ' || i, round((5 + i % 15)::numeric, 3), DATE '2024-01-01' + i * 7 +FROM generate_series(1, 15) i; + +INSERT INTO exchange_rates (currency_code, rate_date, usd_rate) +SELECT c, DATE '2026-01-01' + d, round((0.5 + random() * 90)::numeric, 6) +FROM unnest(ARRAY['EUR','GBP','INR','JPY']) c, generate_series(0, 120) d; + +INSERT INTO purchase_orders (po_number, vendor_id, ordered_on, status, total) +SELECT 'PO-' || lpad(i::text, 5, '0'), 1 + (i % 60), DATE '2025-03-01' + i, + (ARRAY['open','received','closed'])[1 + i % 3], round((500 + i * 11)::numeric, 2) +FROM generate_series(1, 150) i; + +INSERT INTO purchase_order_lines (po_id, line_no, item_desc, quantity, unit_cost) +SELECT 1 + (i / 2), 1 + (i % 2), 'PO line ' || i, 1 + (i % 9), round((15 + i % 400)::numeric, 2) +FROM generate_series(0, 280) i; + +INSERT INTO vendor_contracts (vendor_id, starts_on, ends_on, terms) +SELECT 1 + (i % 60), DATE '2024-01-01' + i * 5, DATE '2026-01-01' + i * 5, + 'Net 45 payment terms with a two percent early-settlement discount when paid within ten ' + || 'days. Renewal is automatic unless either party gives ninety days written notice. ' + || 'Service credits accrue when monthly uptime falls below the agreed threshold. Contract ' || i +FROM generate_series(1, 45) i; + +INSERT INTO vendor_contract_links (contract_id, cost_center_id) +SELECT 1 + (i % 45), 1 + (i % 12) FROM generate_series(1, 60) i +ON CONFLICT DO NOTHING; + +INSERT INTO expense_reports (employee_id, submitted_on, status, total) +SELECT 1 + (i % 120), DATE '2026-01-10' + i, (ARRAY['submitted','approved','reimbursed'])[1 + i % 3], + round((40 + i * 3)::numeric, 2) +FROM generate_series(1, 90) i; + +INSERT INTO expense_lines (report_id, category, amount, receipt_url) +SELECT 1 + (i % 90), (ARRAY['travel','meals','lodging','supplies'])[1 + i % 4], + round((10 + i % 250)::numeric, 2), 'https://receipts.example.com/' || i +FROM generate_series(1, 200) i; + +INSERT INTO payment_batches (batch_ref, row_count) +SELECT 'BATCH-' || lpad(i::text, 4, '0'), 10 + i FROM generate_series(1, 25) i; + +INSERT INTO audit_journal (actor, action, narrative, details) +SELECT 'clerk_' || (1 + i % 8), (ARRAY['post','reverse','approve'])[1 + i % 3], + 'Period-end adjustment posted after reconciling the vendor statement against open ' + || 'payables. Two invoices required manual matching because the vendor combined several ' + || 'purchase orders into a single statement line. Supporting evidence is attached to the ' + || 'workflow ticket and the reviewer signed off on the variance explanation. Entry ' || i, + jsonb_build_object('ticket', 'FIN-' || i, 'variance', (i % 50)) +FROM generate_series(1, 120) i; diff --git a/apps/dla/tests/fixtures/postgres_large/seed/03_hr.sql b/apps/dla/tests/fixtures/postgres_large/seed/03_hr.sql new file mode 100644 index 0000000..7af90d2 --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/03_hr.sql @@ -0,0 +1,242 @@ +-- File 03: hr schema (16 tables). +-- Ground truth: +-- * Self-referencing FK: employees.manager_id -> employees.id. +-- * Junctions: employee_skills, employee_benefits, employee_training. +-- * Composite PK: job_history (employee_id, started_on). +-- * Wide table: employee_survey_wide has 110 columns (q001..q100 + 10 base +-- columns) — generated via a DO block for maintainability. +-- * Enum column: employees.status uses hr.employment_status. +-- * Text-heavy: performance_reviews.summary_text. +-- * Cross-schema declared FK (added at the end): finance.expense_reports.employee_id +-- -> hr.employees(id). + +SET client_min_messages = WARNING; +SET search_path = hr; + +CREATE TABLE locations ( + id SERIAL PRIMARY KEY, + city VARCHAR(80) NOT NULL, + country CHAR(2) NOT NULL, + timezone VARCHAR(40) +); + +CREATE TABLE departments ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL UNIQUE, + budget NUMERIC(14,2) +); + +CREATE TABLE positions ( + id SERIAL PRIMARY KEY, + title VARCHAR(120) NOT NULL, + level SMALLINT NOT NULL, + band VARCHAR(8) +); + +CREATE TABLE employees ( + id SERIAL PRIMARY KEY, + employee_code VARCHAR(16) NOT NULL UNIQUE, + full_name VARCHAR(160) NOT NULL, + email VARCHAR(255) NOT NULL, + department_id INTEGER NOT NULL REFERENCES departments(id), + position_id INTEGER NOT NULL REFERENCES positions(id), + location_id INTEGER REFERENCES locations(id), + manager_id INTEGER REFERENCES employees(id), -- self-referencing FK + status hr.employment_status NOT NULL DEFAULT 'full_time', + hired_on DATE NOT NULL, + salary NUMERIC(12,2), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX idx_employees_manager ON employees(manager_id); + +CREATE TABLE skills ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL UNIQUE +); + +CREATE TABLE employee_skills ( + employee_id INTEGER NOT NULL REFERENCES employees(id), + skill_id INTEGER NOT NULL REFERENCES skills(id), + proficiency SMALLINT, + PRIMARY KEY (employee_id, skill_id) +); + +CREATE TABLE benefits ( + id SERIAL PRIMARY KEY, + name VARCHAR(120) NOT NULL, + annual_cost NUMERIC(10,2) +); + +CREATE TABLE employee_benefits ( + employee_id INTEGER NOT NULL REFERENCES employees(id), + benefit_id INTEGER NOT NULL REFERENCES benefits(id), + enrolled_on DATE, + PRIMARY KEY (employee_id, benefit_id) +); + +CREATE TABLE training_courses ( + id SERIAL PRIMARY KEY, + title VARCHAR(160) NOT NULL, + hours SMALLINT +); + +CREATE TABLE employee_training ( + employee_id INTEGER NOT NULL REFERENCES employees(id), + course_id INTEGER NOT NULL REFERENCES training_courses(id), + completed_at TIMESTAMPTZ, + PRIMARY KEY (employee_id, course_id) +); + +CREATE TABLE payroll_runs ( + id SERIAL PRIMARY KEY, + run_date DATE NOT NULL, + period VARCHAR(16) NOT NULL, + status VARCHAR(16) NOT NULL +); + +CREATE TABLE payroll_items ( + id SERIAL PRIMARY KEY, + run_id INTEGER NOT NULL REFERENCES payroll_runs(id), + employee_id INTEGER NOT NULL REFERENCES employees(id), + gross NUMERIC(12,2) NOT NULL, + net NUMERIC(12,2) NOT NULL, + deductions NUMERIC(12,2) NOT NULL DEFAULT 0 +); + +CREATE TABLE performance_reviews ( + id SERIAL PRIMARY KEY, + employee_id INTEGER NOT NULL REFERENCES employees(id), + reviewer_id INTEGER REFERENCES employees(id), + period VARCHAR(16) NOT NULL, + rating SMALLINT, + summary_text TEXT NOT NULL +); + +CREATE TABLE job_history ( + employee_id INTEGER NOT NULL REFERENCES employees(id), + started_on DATE NOT NULL, + position_id INTEGER NOT NULL REFERENCES positions(id), + ended_on DATE, + PRIMARY KEY (employee_id, started_on) -- composite PK +); + +CREATE TABLE emergency_contacts ( + id SERIAL PRIMARY KEY, + employee_id INTEGER NOT NULL REFERENCES employees(id), + name VARCHAR(160) NOT NULL, + relationship VARCHAR(40), + phone VARCHAR(32) +); + +-- Wide table: 10 base columns + q001..q100 -> 110 columns total. +DO $$ +DECLARE + cols text := ''; +BEGIN + FOR i IN 1..100 LOOP + cols := cols || format(', q%s SMALLINT', lpad(i::text, 3, '0')); + END LOOP; + EXECUTE 'CREATE TABLE hr.employee_survey_wide (' + || 'id SERIAL PRIMARY KEY, employee_id INTEGER NOT NULL REFERENCES hr.employees(id), ' + || 'survey_year SMALLINT NOT NULL, submitted_at TIMESTAMPTZ NOT NULL DEFAULT now(), ' + || 'is_anonymous BOOLEAN NOT NULL DEFAULT FALSE, engagement_score NUMERIC(5,2), ' + || 'nps SMALLINT, tenure_bucket VARCHAR(16), comments TEXT, locale VARCHAR(8)' + || cols || ')'; +END $$; + +-- ============ seed data ============ + +INSERT INTO locations (city, country, timezone) VALUES + ('London', 'GB', 'Europe/London'), ('Berlin', 'DE', 'Europe/Berlin'), + ('Austin', 'US', 'America/Chicago'), ('Hyderabad', 'IN', 'Asia/Kolkata'); + +INSERT INTO departments (name, budget) +SELECT 'Department ' || i, 1000000 + i * 50000 FROM generate_series(1, 8) i; + +INSERT INTO positions (title, level, band) +SELECT 'Position ' || i, 1 + (i % 6), 'B' || (1 + i % 6) FROM generate_series(1, 20) i; + +INSERT INTO employees (employee_code, full_name, email, department_id, position_id, + location_id, manager_id, status, hired_on, salary) +SELECT 'EMP-' || lpad(i::text, 4, '0'), 'Employee ' || i, 'emp' || i || '@example.com', + 1 + (i % 8), 1 + (i % 20), 1 + (i % 4), + CASE WHEN i <= 8 THEN NULL ELSE 1 + (i % 8) END, -- first 8 are top-level managers + (ARRAY['full_time','part_time','contract','terminated'])[1 + i % 4]::hr.employment_status, + DATE '2019-01-01' + i * 9, 40000 + i * 350 +FROM generate_series(1, 120) i; + +INSERT INTO skills (name) SELECT 'Skill ' || i FROM generate_series(1, 25) i; + +INSERT INTO employee_skills (employee_id, skill_id, proficiency) +SELECT 1 + (i % 120), 1 + (i % 25), 1 + (i % 5) FROM generate_series(1, 300) i +ON CONFLICT DO NOTHING; + +INSERT INTO benefits (name, annual_cost) +SELECT 'Benefit ' || i, 500 + i * 120 FROM generate_series(1, 10) i; + +INSERT INTO employee_benefits (employee_id, benefit_id, enrolled_on) +SELECT 1 + (i % 120), 1 + (i % 10), DATE '2024-01-01' + i FROM generate_series(1, 240) i +ON CONFLICT DO NOTHING; + +INSERT INTO training_courses (title, hours) +SELECT 'Course ' || i, 2 + (i % 40) FROM generate_series(1, 18) i; + +INSERT INTO employee_training (employee_id, course_id, completed_at) +SELECT 1 + (i % 120), 1 + (i % 18), now() - (i || ' days')::interval +FROM generate_series(1, 260) i +ON CONFLICT DO NOTHING; + +INSERT INTO payroll_runs (run_date, period, status) +SELECT DATE '2025-01-31' + i * 30, '2025-M' || lpad((1 + i % 12)::text, 2, '0'), + CASE WHEN i < 14 THEN 'posted' ELSE 'draft' END +FROM generate_series(0, 17) i; + +INSERT INTO payroll_items (run_id, employee_id, gross, net, deductions) +SELECT 1 + (i % 18), 1 + (i % 120), round((3000 + i % 4000)::numeric, 2), + round(((3000 + i % 4000) * 0.72)::numeric, 2), round(((3000 + i % 4000) * 0.28)::numeric, 2) +FROM generate_series(1, 1500) i; + +INSERT INTO performance_reviews (employee_id, reviewer_id, period, rating, summary_text) +SELECT 1 + (i % 120), 1 + (i % 8), '2025-H' || (1 + i % 2), 1 + (i % 5), + 'Consistently delivered against the quarterly objectives and took ownership of the ' + || 'incident review process without being asked. Communication with partner teams has ' + || 'improved markedly since the last cycle, though estimation on larger projects still ' + || 'trends optimistic. Recommend pairing with a senior mentor next half and revisiting ' + || 'the promotion conversation once the reliability workstream lands. Review ' || i +FROM generate_series(1, 180) i; + +INSERT INTO job_history (employee_id, started_on, position_id, ended_on) +SELECT 1 + (i % 120), DATE '2019-01-01' + i * 13, 1 + (i % 20), + CASE WHEN i % 3 = 0 THEN DATE '2021-01-01' + i * 13 END +FROM generate_series(1, 200) i +ON CONFLICT DO NOTHING; + +INSERT INTO emergency_contacts (employee_id, name, relationship, phone) +SELECT 1 + (i % 120), 'Contact ' || i, (ARRAY['spouse','parent','sibling','friend'])[1 + i % 4], + '+1-555-' || lpad(i::text, 4, '0') +FROM generate_series(1, 150) i; + +INSERT INTO hr.employee_survey_wide (employee_id, survey_year, engagement_score, nps, tenure_bucket, comments, locale, + q001, q002, q003, q004, q005) +SELECT 1 + (i % 120), 2025 + (i % 2), round((1 + random() * 4)::numeric, 2), (i % 21) - 10, + (ARRAY['<1y','1-3y','3-5y','5y+'])[1 + i % 4], + 'Mostly satisfied with the tooling budget but the onboarding wiki is badly out of date.', + 'en-US', 1 + (i % 5), 1 + (i % 5), 1 + (i % 5), 1 + (i % 5), 1 + (i % 5) +FROM generate_series(1, 80) i; + +-- Fill the remaining q006..q100 with values for realism. +DO $$ +DECLARE + stmt text := ''; +BEGIN + FOR i IN 6..100 LOOP + stmt := stmt || format('q%s = 1 + (id %% 5), ', lpad(i::text, 3, '0')); + END LOOP; + stmt := left(stmt, length(stmt) - 2); + EXECUTE 'UPDATE hr.employee_survey_wide SET ' || stmt; +END $$; + +-- Cross-schema declared FK (finance -> hr). +ALTER TABLE finance.expense_reports + ADD CONSTRAINT fk_expense_reports_employee + FOREIGN KEY (employee_id) REFERENCES hr.employees(id); diff --git a/apps/dla/tests/fixtures/postgres_large/seed/04_staging_nofk.sql b/apps/dla/tests/fixtures/postgres_large/seed/04_staging_nofk.sql new file mode 100644 index 0000000..9fe6737 --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/04_staging_nofk.sql @@ -0,0 +1,207 @@ +-- File 04: staging schema — the NO-FK zone (16 tables). +-- Simulates a cloud-warehouse dump: primary keys survive, foreign keys do not. +-- Zero declared FK constraints in this schema. Join paths must be INFERRED. +-- +-- Inference ground truth (given the engine matches "_id" / +-- "_id" against another table's single-column PK): +-- * stg_orders.stg_customer_id -> stg_customers.id name+type+overlap => Strong +-- * stg_order_items.stg_order_id -> stg_orders.id name+type+overlap => Strong +-- * stg_order_items.stg_product_id -> stg_products.id name+type+overlap => Strong +-- * stg_products.stg_category_id -> stg_categories.id name+type+overlap => Strong +-- * stg_payments.stg_invoice_id -> stg_invoices.id name+type+overlap => Strong +-- * stg_inventory.stg_product_id / stg_store_id => Strong (table has no PK itself) +-- * stg_web_events.stg_customer_id -> stg_customers.id => Strong +-- * stg_shipments.stg_order_id (VARCHAR) -> stg_orders.id name only (type mismatch) => Weak +-- * stg_returns.stg_order_id values are ORPHANS (900000+) name+type, no overlap +-- -> inferred rel exists AND readiness should flag broken_fk on it +-- * stg_invoices.customer_id: realistic warehouse naming that does NOT match the +-- "stg_customers" pattern -> expected inference MISS (documents the engine's +-- naming-convention limitation) +-- * stg_web_events.store_id / stg_returns.store_id: matches the DISTRACTOR table +-- analytics.stores (not sales.dim_stores) -> deliberate false-positive bait + +SET client_min_messages = WARNING; +SET search_path = staging; + +CREATE TABLE stg_customers ( + id INTEGER PRIMARY KEY, + name VARCHAR(160), + email VARCHAR(255), + status VARCHAR(24), + created_at TIMESTAMPTZ +); + +CREATE TABLE stg_categories ( + id INTEGER PRIMARY KEY, + name VARCHAR(80) +); + +CREATE TABLE stg_products ( + id INTEGER PRIMARY KEY, + name VARCHAR(160), + stg_category_id INTEGER, + price NUMERIC(10,2), + status VARCHAR(24) +); + +CREATE TABLE stg_orders ( + id INTEGER PRIMARY KEY, + stg_customer_id INTEGER, + status VARCHAR(24), + order_total NUMERIC(12,2), + created_at TIMESTAMPTZ +); + +CREATE TABLE stg_order_items ( + id INTEGER PRIMARY KEY, + stg_order_id INTEGER, + stg_product_id INTEGER, + quantity INTEGER, + unit_price NUMERIC(10,2) +); + +CREATE TABLE stg_stores ( + id INTEGER PRIMARY KEY, + name VARCHAR(120), + city VARCHAR(80) +); + +CREATE TABLE stg_invoices ( + id INTEGER PRIMARY KEY, + customer_id INTEGER, -- realistic naming; will NOT match stg_customers + amount NUMERIC(12,2), + status VARCHAR(24), + issued_on DATE +); + +CREATE TABLE stg_payments ( + id INTEGER PRIMARY KEY, + stg_invoice_id INTEGER, + amount NUMERIC(12,2), + paid_on DATE +); + +CREATE TABLE stg_shipments ( + id INTEGER PRIMARY KEY, + stg_order_id VARCHAR(24), -- TYPE MISMATCH with stg_orders.id (INTEGER) + carrier VARCHAR(80), + shipped_on DATE +); + +CREATE TABLE stg_returns ( + id INTEGER PRIMARY KEY, + stg_order_id INTEGER, -- ORPHAN values (900000+): no overlap with stg_orders.id + store_id INTEGER, -- distractor bait: matches analytics.stores + reason VARCHAR(80), + amount NUMERIC(12,2) +); + +CREATE TABLE stg_suppliers ( + id INTEGER PRIMARY KEY, + name VARCHAR(120), + status VARCHAR(24) +); + +CREATE TABLE stg_inventory ( + stg_product_id INTEGER, -- NO primary key on this table at all + stg_store_id INTEGER, + on_hand INTEGER, + counted_at TIMESTAMPTZ +); + +CREATE TABLE stg_employees ( + id INTEGER PRIMARY KEY, + name VARCHAR(160), + email VARCHAR(255), + status VARCHAR(24) +); + +CREATE TABLE stg_web_events ( + id BIGINT PRIMARY KEY, + stg_customer_id INTEGER, + store_id INTEGER, -- distractor bait: matches analytics.stores + event_type VARCHAR(40), + url VARCHAR(240), + occurred_at TIMESTAMPTZ +); + +CREATE TABLE stg_promotions ( + id INTEGER PRIMARY KEY, + code VARCHAR(40), + status VARCHAR(24) +); + +CREATE TABLE stg_exchange_rates ( + currency_code CHAR(3), -- no PK + rate_date DATE, + usd_rate NUMERIC(14,6) +); + +-- ============ seed data (value overlap engineered for inference) ============ + +INSERT INTO stg_customers (id, name, email, status, created_at) +SELECT i, 'Staged Customer ' || i, 'sc' || i || '@example.com', + (ARRAY['active','inactive'])[1 + i % 2], now() - (i || ' hours')::interval +FROM generate_series(1, 300) i; + +INSERT INTO stg_categories (id, name) +SELECT i, 'Staged category ' || i FROM generate_series(1, 12) i; + +INSERT INTO stg_products (id, name, stg_category_id, price, status) +SELECT i, 'Staged product ' || i, 1 + (i % 12), round((3 + i % 90)::numeric, 2), + (ARRAY['active','discontinued'])[1 + i % 2] +FROM generate_series(1, 150) i; + +INSERT INTO stg_orders (id, stg_customer_id, status, order_total, created_at) +SELECT i, 1 + (i % 300), (ARRAY['pending','shipped','cancelled'])[1 + i % 3], + round((10 + i % 500)::numeric, 2), now() - (i || ' hours')::interval +FROM generate_series(1, 600) i; + +INSERT INTO stg_order_items (id, stg_order_id, stg_product_id, quantity, unit_price) +SELECT i, 1 + (i % 600), 1 + (i % 150), 1 + (i % 4), round((3 + i % 90)::numeric, 2) +FROM generate_series(1, 1400) i; + +INSERT INTO stg_stores (id, name, city) +SELECT i, 'Staged store ' || i, 'City ' || i FROM generate_series(1, 25) i; + +INSERT INTO stg_invoices (id, customer_id, amount, status, issued_on) +SELECT i, 1 + (i % 300), round((25 + i % 400)::numeric, 2), + (ARRAY['open','paid'])[1 + i % 2], DATE '2026-01-01' + (i % 150) +FROM generate_series(1, 250) i; + +INSERT INTO stg_payments (id, stg_invoice_id, amount, paid_on) +SELECT i, 1 + (i % 250), round((25 + i % 400)::numeric, 2), DATE '2026-02-01' + (i % 120) +FROM generate_series(1, 200) i; + +INSERT INTO stg_shipments (id, stg_order_id, carrier, shipped_on) +SELECT i, (1 + (i % 600))::text, (ARRAY['FedEx','UPS','DHL'])[1 + i % 3], DATE '2026-01-01' + (i % 150) +FROM generate_series(1, 350) i; + +INSERT INTO stg_returns (id, stg_order_id, store_id, reason, amount) +SELECT i, 900000 + i, 1 + (i % 25), 'reason ' || (i % 6), round((5 + i % 120)::numeric, 2) +FROM generate_series(1, 120) i; + +INSERT INTO stg_suppliers (id, name, status) +SELECT i, 'Staged supplier ' || i, 'active' FROM generate_series(1, 40) i; + +INSERT INTO stg_inventory (stg_product_id, stg_store_id, on_hand, counted_at) +SELECT 1 + (i % 150), 1 + (i % 25), (i * 3) % 400, now() - (i || ' minutes')::interval +FROM generate_series(1, 800) i; + +INSERT INTO stg_employees (id, name, email, status) +SELECT i, 'Staged employee ' || i, 'se' || i || '@example.com', + (ARRAY['active','terminated'])[1 + i % 2] +FROM generate_series(1, 90) i; + +INSERT INTO stg_web_events (id, stg_customer_id, store_id, event_type, url, occurred_at) +SELECT i, 1 + (i % 300), 1 + (i % 25), + (ARRAY['page_view','add_to_cart','checkout','search'])[1 + i % 4], + 'https://shop.example.com/p/' || (i % 150), now() - (i || ' minutes')::interval +FROM generate_series(1, 3000) i; + +INSERT INTO stg_promotions (id, code, status) +SELECT i, 'SPROMO-' || i, (ARRAY['live','expired'])[1 + i % 2] FROM generate_series(1, 15) i; + +INSERT INTO stg_exchange_rates (currency_code, rate_date, usd_rate) +SELECT (ARRAY['EUR','GBP','INR'])[1 + i % 3], DATE '2026-01-01' + (i / 3), round((0.4 + i * 0.01)::numeric, 6) +FROM generate_series(1, 90) i; diff --git a/apps/dla/tests/fixtures/postgres_large/seed/05_analytics_edge.sql b/apps/dla/tests/fixtures/postgres_large/seed/05_analytics_edge.sql new file mode 100644 index 0000000..95b6ae5 --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/05_analytics_edge.sql @@ -0,0 +1,223 @@ +-- File 05: analytics schema — structural edge cases (15 tables). +-- Ground truth: +-- * Reserved-word identifiers: table "order" with columns "select" and "group". +-- * Mixed-case quoted identifiers: "CamelCaseEvents" with "eventId", "eventName". +-- * Long identifiers (60 chars, near the 63-char Postgres limit): +-- tbl_customer_lifetime_value_rolling_window_aggregation_v2025 +-- with column cumulative_gross_merchandise_value_net_of_returns_and_promos. +-- * Unusual types: typed_showcase (uuid PK, jsonb, numeric[], text[], enum, +-- timestamptz, time, interval, bytea, inet, daterange). +-- * Tall table: events_tall with 100,000 rows (sampling behavior). +-- * Zero-row table with a full schema: zero_rows_events. + +SET client_min_messages = WARNING; +SET search_path = analytics; + +-- Reserved-word table + columns (all quoted) +CREATE TABLE "order" ( + id SERIAL PRIMARY KEY, + "select" VARCHAR(40), + "group" VARCHAR(40), + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Mixed-case quoted identifiers +CREATE TABLE "CamelCaseEvents" ( + "eventId" SERIAL PRIMARY KEY, + "eventName" VARCHAR(80) NOT NULL, + "occurredAt" TIMESTAMPTZ NOT NULL DEFAULT now(), + "payloadJson" JSONB +); + +-- 60-character identifiers (Postgres limit is 63) +CREATE TABLE tbl_customer_lifetime_value_rolling_window_aggregation_v2025 ( + id SERIAL PRIMARY KEY, + cumulative_gross_merchandise_value_net_of_returns_and_promos NUMERIC(16,2), + as_of DATE +); + +-- Unusual-type showcase +CREATE TABLE typed_showcase ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + payload JSONB, + price_points NUMERIC(12,4)[], + tags TEXT[], + mood analytics.mood_type, + seen_at TIMESTAMPTZ, + daily_at TIME, + lifetime INTERVAL, + blob BYTEA, + client_ip INET, + active_range DATERANGE +); + +-- Tall table: ~100k rows for sampling behavior +CREATE TABLE events_tall ( + id BIGSERIAL PRIMARY KEY, + occurred_at TIMESTAMPTZ NOT NULL, + event_type VARCHAR(40) NOT NULL, + user_ref INTEGER, + duration_ms INTEGER, + payload JSONB +); + +-- Zero rows, full schema +CREATE TABLE zero_rows_events ( + id BIGSERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL, + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Rollup / product-analytics tables (generic column names on purpose) +CREATE TABLE daily_kpi_rollup ( + id SERIAL PRIMARY KEY, + day DATE NOT NULL, + name VARCHAR(80) NOT NULL, + value NUMERIC(16,4), + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE monthly_kpi_rollup ( + id SERIAL PRIMARY KEY, + month DATE NOT NULL, + name VARCHAR(80) NOT NULL, + value NUMERIC(16,4), + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE funnel_steps ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL, + step_order SMALLINT NOT NULL, + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE cohort_retention ( + id SERIAL PRIMARY KEY, + cohort_month DATE NOT NULL, + period_no SMALLINT NOT NULL, + retained_pct NUMERIC(6,3), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE ab_test_results ( + id SERIAL PRIMARY KEY, + name VARCHAR(120) NOT NULL, + variant VARCHAR(24) NOT NULL, + metric NUMERIC(12,6), + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE page_views ( + id BIGSERIAL PRIMARY KEY, + url VARCHAR(240) NOT NULL, + viewed_at TIMESTAMPTZ NOT NULL, + user_ref INTEGER, + status VARCHAR(24) +); + +CREATE TABLE sessions ( + id BIGSERIAL PRIMARY KEY, + started_at TIMESTAMPTZ NOT NULL, + ended_at TIMESTAMPTZ, + device VARCHAR(40), + status VARCHAR(24) +); + +CREATE TABLE search_queries ( + id BIGSERIAL PRIMARY KEY, + query_text TEXT NOT NULL, + results INTEGER, + searched_at TIMESTAMPTZ NOT NULL +); + +CREATE TABLE feature_flags ( + id SERIAL PRIMARY KEY, + name VARCHAR(80) NOT NULL UNIQUE, + enabled BOOLEAN NOT NULL DEFAULT FALSE, + status VARCHAR(24), + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- ============ seed data ============ + +INSERT INTO "order" ("select", "group", status) +SELECT 'choice ' || i, 'bucket ' || (i % 5), (ARRAY['open','closed'])[1 + i % 2] +FROM generate_series(1, 40) i; + +INSERT INTO "CamelCaseEvents" ("eventName", "payloadJson") +SELECT 'Event' || i, jsonb_build_object('seq', i, 'ok', i % 2 = 0) +FROM generate_series(1, 60) i; + +INSERT INTO tbl_customer_lifetime_value_rolling_window_aggregation_v2025 + (cumulative_gross_merchandise_value_net_of_returns_and_promos, as_of) +SELECT round((1000 + i * 37.5)::numeric, 2), DATE '2025-01-01' + i +FROM generate_series(1, 90) i; + +INSERT INTO typed_showcase (payload, price_points, tags, mood, seen_at, daily_at, lifetime, blob, client_ip, active_range) +SELECT jsonb_build_object('k', i, 'nested', jsonb_build_object('deep', i * 2)), + ARRAY[round((i * 1.1)::numeric, 4), round((i * 2.2)::numeric, 4)], + ARRAY['tag' || i, 'tag' || (i + 1)], + (ARRAY['happy','neutral','sad'])[1 + i % 3]::analytics.mood_type, + now() - (i || ' hours')::interval, + make_time(i % 24, i % 60, 0), + (i || ' days')::interval, + decode(md5(i::text), 'hex'), + ('10.0.' || (i % 255) || '.' || (1 + i % 254))::inet, + daterange(DATE '2026-01-01', DATE '2026-01-01' + i) +FROM generate_series(1, 50) i; + +INSERT INTO events_tall (occurred_at, event_type, user_ref, duration_ms, payload) +SELECT now() - (i || ' seconds')::interval, + (ARRAY['click','view','scroll','hover','submit'])[1 + i % 5], + 1 + (i % 5000), (i * 13) % 30000, + jsonb_build_object('n', i % 100) +FROM generate_series(1, 100000) i; + +INSERT INTO daily_kpi_rollup (day, name, value, status) +SELECT DATE '2026-01-01' + (i % 180), (ARRAY['revenue','orders','aov','traffic'])[1 + i % 4], + round((100 + i * 1.7)::numeric, 4), 'final' +FROM generate_series(1, 720) i; + +INSERT INTO monthly_kpi_rollup (month, name, value, status) +SELECT date_trunc('month', DATE '2025-01-01' + (i % 18) * 31)::date, + (ARRAY['revenue','orders','aov','traffic'])[1 + i % 4], + round((3000 + i * 21)::numeric, 4), 'final' +FROM generate_series(1, 72) i; + +INSERT INTO funnel_steps (name, step_order, status) +SELECT 'Step ' || i, i, 'live' FROM generate_series(1, 8) i; + +INSERT INTO cohort_retention (cohort_month, period_no, retained_pct) +SELECT date_trunc('month', DATE '2025-01-01' + m * 31)::date, p, round((90 - p * 6.5)::numeric, 3) +FROM generate_series(0, 11) m, generate_series(0, 8) p; + +INSERT INTO ab_test_results (name, variant, metric, status) +SELECT 'Experiment ' || (1 + i / 2), (ARRAY['control','treatment'])[1 + i % 2], + round((0.01 + i * 0.001)::numeric, 6), (ARRAY['running','done'])[1 + i % 2] +FROM generate_series(1, 60) i; + +INSERT INTO page_views (url, viewed_at, user_ref, status) +SELECT 'https://shop.example.com/' || (i % 200), now() - (i || ' minutes')::interval, + 1 + (i % 2000), 'ok' +FROM generate_series(1, 5000) i; + +INSERT INTO sessions (started_at, ended_at, device, status) +SELECT now() - (i || ' hours')::interval, now() - (i || ' hours')::interval + interval '25 minutes', + (ARRAY['ios','android','web'])[1 + i % 3], (ARRAY['closed','abandoned'])[1 + i % 2] +FROM generate_series(1, 1200) i; + +INSERT INTO search_queries (query_text, results, searched_at) +SELECT 'where can I find ' || (ARRAY['red shoes','wireless earbuds','linen napkins','gift cards'])[1 + i % 4] + || ' with next day delivery in size ' || (i % 12), + i % 40, now() - (i || ' minutes')::interval +FROM generate_series(1, 900) i; + +INSERT INTO feature_flags (name, enabled, status) +SELECT 'flag_' || i, i % 2 = 0, 'live' FROM generate_series(1, 30) i; diff --git a/apps/dla/tests/fixtures/postgres_large/seed/06_distractors.sql b/apps/dla/tests/fixtures/postgres_large/seed/06_distractors.sql new file mode 100644 index 0000000..3a0adfe --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/06_distractors.sql @@ -0,0 +1,34 @@ +-- File 06: 25 distractor tables in analytics, generated by one DO block. +-- Every table has the SAME generic shape: id SERIAL PK, name, status, created_at, +-- plus 20 rows of near-identical data — so schema linking / glossary extraction / +-- term recurrence get genuinely hard, and generic column names recur heavily. +-- +-- One deliberate trap: the distractor "stores" gives any bare "store_id" column +-- elsewhere (staging.stg_returns, staging.stg_web_events) an inference target that +-- is NOT the real dimension (sales.dim_stores). + +SET client_min_messages = WARNING; + +DO $$ +DECLARE + t text; +BEGIN + FOREACH t IN ARRAY ARRAY[ + 'assets', 'tags', 'labels', 'batches', 'buckets', + 'snapshots', 'tokens', 'widgets', 'gadgets', 'portals', + 'queues', 'topics', 'threads', 'messages', 'alerts', + 'notices', 'tasks', 'jobs', 'runs', 'stages', + 'states', 'phases', 'marks', 'zones', 'stores' + ] LOOP + EXECUTE format( + 'CREATE TABLE analytics.%I (' + || 'id SERIAL PRIMARY KEY, ' + || 'name VARCHAR(80) NOT NULL, ' + || 'status VARCHAR(24), ' + || 'created_at TIMESTAMPTZ NOT NULL DEFAULT now())', t); + EXECUTE format( + 'INSERT INTO analytics.%I (name, status) ' + || 'SELECT %L || '' '' || i, (ARRAY[''active'',''archived''])[1 + i %% 2] ' + || 'FROM generate_series(1, 20) i', t, t); + END LOOP; +END $$; diff --git a/apps/dla/tests/fixtures/postgres_large/seed/07_quality_issues.sql b/apps/dla/tests/fixtures/postgres_large/seed/07_quality_issues.sql new file mode 100644 index 0000000..ecc0893 --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/seed/07_quality_issues.sql @@ -0,0 +1,93 @@ +-- File 07: deliberately seeded data-quality issues (6 tables, analytics schema). +-- Expected readiness detections: +-- Q1. empty_table analytics.quality_empty_orders (zero rows) -> Critical +-- Q2. all_null_column analytics.quality_users.middle_name -> Critical +-- Q3. constant_column analytics.quality_users.country_code ('IN') -> Info +-- Q4. high_null_rate analytics.quality_users.referral_code (~70%) -> Warning +-- Q5. broken_fk (inferred) analytics.quality_invoices.dim_customer_id has +-- orphans vs sales.dim_customers.id (name matches "dim_customer_id") +-- Q6. broken_fk (DECLARED, NOT VALID) analytics.quality_orders_notvalid.customer_ref +-- -> sales.dim_customers(id) added NOT VALID after orphan rows were inserted, +-- so a *declared* FK carries orphan values. +-- Q7. NOT expected to be caught (known gaps, for the report): +-- - analytics.quality_status_mix.status mixes 'active'/'Active'/'ACTIVE' +-- (case-inconsistent categorical) — no such check exists in M2. +-- - type_mismatch is documented as planned/deferred. +-- Q8. constant + all-null combo: analytics.quality_sensor_dump.firmware ('1.0.0' +-- constant) and calibration_note (all NULL). + +SET client_min_messages = WARNING; +SET search_path = analytics; + +CREATE TABLE quality_empty_orders ( + id SERIAL PRIMARY KEY, + placed_at TIMESTAMPTZ NOT NULL, + customer_id INTEGER NOT NULL, + status VARCHAR(24) +); + +CREATE TABLE quality_users ( + id SERIAL PRIMARY KEY, + email VARCHAR(255) NOT NULL, + middle_name VARCHAR(80), -- always NULL (Q2) + country_code CHAR(2) NOT NULL, -- always 'IN' (Q3) + referral_code VARCHAR(40) -- ~70% NULL (Q4) +); + +INSERT INTO quality_users (email, middle_name, country_code, referral_code) +SELECT 'qu' || i || '@example.com', NULL, 'IN', + CASE WHEN i % 10 < 3 THEN 'REF-' || i END +FROM generate_series(1, 200) i; + +-- Q5: inferred relationship with orphans. "dim_customer_id" name-matches +-- sales.dim_customers (single-col PK), but 40 of 200 values do not exist there. +CREATE TABLE quality_invoices ( + id SERIAL PRIMARY KEY, + dim_customer_id INTEGER NOT NULL, + amount NUMERIC(10,2) NOT NULL, + issued_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +INSERT INTO quality_invoices (dim_customer_id, amount) +SELECT CASE WHEN i % 5 = 0 THEN 90000 + i ELSE 1 + (i % 400) END, + round((10 + i)::numeric, 2) +FROM generate_series(1, 200) i; + +-- Q6: DECLARED broken FK via NOT VALID (constraint exists; orphans persist). +CREATE TABLE quality_orders_notvalid ( + id SERIAL PRIMARY KEY, + customer_ref INTEGER NOT NULL, + total NUMERIC(12,2) +); + +INSERT INTO quality_orders_notvalid (customer_ref, total) +SELECT CASE WHEN i % 4 = 0 THEN 77000 + i ELSE 1 + (i % 400) END, + round((20 + i)::numeric, 2) +FROM generate_series(1, 100) i; + +ALTER TABLE quality_orders_notvalid + ADD CONSTRAINT fk_qonv_customer FOREIGN KEY (customer_ref) + REFERENCES sales.dim_customers(id) NOT VALID; + +-- Q7: mixed-case categorical (known detection gap — expect NO readiness issue) +CREATE TABLE quality_status_mix ( + id SERIAL PRIMARY KEY, + label VARCHAR(80) NOT NULL, + status VARCHAR(24) NOT NULL +); + +INSERT INTO quality_status_mix (label, status) +SELECT 'row ' || i, (ARRAY['active', 'Active', 'ACTIVE', 'inactive'])[1 + i % 4] +FROM generate_series(1, 120) i; + +-- Q8: constant + all-null combo on one table +CREATE TABLE quality_sensor_dump ( + id SERIAL PRIMARY KEY, + reading NUMERIC(12,4) NOT NULL, + firmware VARCHAR(16) NOT NULL, -- constant '1.0.0' + calibration_note TEXT -- all NULL +); + +INSERT INTO quality_sensor_dump (reading, firmware, calibration_note) +SELECT round((random() * 100)::numeric, 4), '1.0.0', NULL +FROM generate_series(1, 150) i; From d2b838c4c81a7914301c4355bde2b00477c49a22 Mon Sep 17 00:00:00 2001 From: Uday Bhan <158012869+udaybhan05@users.noreply.github.com> Date: Thu, 9 Jul 2026 12:54:09 +0530 Subject: [PATCH 2/2] test(dla): add large-fixture e2e findings report Full results of running the complete offline pipeline against the 125-table fixture: per-command results, performance numbers, idempotency verdict, and 18 ranked defects (manifest overcount on multi-schema sources, jsonb/array profiling failures with no readiness issue, SIGINT swallowed by dla run, recommender junction-vs-text scoring tie behavior, broken-fk type coercion, exit-code deviations, inference and pattern-detector false positives). --- .../tests/fixtures/postgres_large/FINDINGS.md | 158 ++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100644 apps/dla/tests/fixtures/postgres_large/FINDINGS.md diff --git a/apps/dla/tests/fixtures/postgres_large/FINDINGS.md b/apps/dla/tests/fixtures/postgres_large/FINDINGS.md new file mode 100644 index 0000000..cd6fddf --- /dev/null +++ b/apps/dla/tests/fixtures/postgres_large/FINDINGS.md @@ -0,0 +1,158 @@ +# L1 (dla) Large-Fixture End-to-End Validation Report + +Date: 2026-07-09 · Branch: `chore/makefile-cross-platform` (isolated worktree) +Environment: macOS, Python 3.11, uv workspace, Docker postgres:16-alpine + +## 1. Fixture summary + +New fixture at `apps/dla/tests/fixtures/postgres_large/` (compose + 8 seed files + +README). Container `dla_fixture_postgres_large`, host port **55433** (the existing +15-table fixture on 55432 is untouched). Configs: +`apps/dla/config/examples/postgres_large.yaml` (all 5 schemas → `./bundle_large`) +and `postgres_large_staging_only.yaml` (no-FK schema only → `./bundle_staging`). + +- **125 tables / 673 columns / 5 schemas**, ~130k rows total. +- `sales` (26): star #1 — 4 facts on conformed dims, two snowflake chains + (product→subcategory→category→department; store→region→country→continent), + 3 bridge tables, 2 text-heavy tables, composite-PK facts. +- `finance` (21): star #2 — self-referencing `dim_accounts`, 4 composite PKs, + **multi-column FK** (ledger→fiscal_periods), cross-schema FK (→hr.employees), + 2 junctions, text-heavy audit journal. +- `hr` (16): self-referencing `employees.manager_id`, 3 junctions, enum column, + **110-column wide table**, composite-PK job history. +- `staging` (16): **no-FK zone** — 0 declared FKs; engineered joins for + inference (overlap, type-mismatch, orphaned, unmatchable naming), 2 PK-less tables. +- `analytics` (46): reserved-word `"order"` (cols `"select"`, `"group"`), + mixed-case `"CamelCaseEvents"`, 60-char identifiers, `typed_showcase` + (uuid/jsonb/numeric[]/text[]/enum/interval/bytea/inet/daterange), 100k-row + `events_tall`, zero-row table, **25 generated distractors** all shaped + `(id, name, status, created_at)`, 6 quality-issue tables (incl. a **declared + broken FK via `NOT VALID`** and a mixed-case status column). + +## 2. Results per command + +| # | Command | Wall time | Exit | Verdict | +|---|---------|-----------|------|---------| +| 1 | `discover --dry-run` | 37.9s cold / ~1.5s warm | 0 | PASS (see A1 count anomaly) | +| 2 | `discover` | **1.49s** | 0 | PASS w/ anomalies A1 | +| 3 | `profile` (sampling) | **13.4s** (667 ok, 6 errors) | 0 | FAIL-partial → D2 | +| 4 | `profile --table analytics.order` / `"CamelCaseEvents"` | 0.9s | 0 | PASS (quoted/reserved OK) | +| 5 | `profile --table sales.no_such_table` | 0.6s | **0** | FAIL → D7 (silent no-op, expected exit 4) | +| 6 | `readiness` | 3.9s | 0 | PASS — all seeded issues found (see §4) | +| 7 | `patterns detect` | 0.8s | 0 | PASS w/ misclassifications → D12 | +| 8 | `glossary build --mode dry-run` | ~1s | 0 | PASS w/ term noise → D14 | +| 9 | `describe --table hr.employee_survey_wide --mode dry-run` | ~1s | 0 | PASS w/ formatting bug → D15 | +| 10 | `describe --column column:analytics.order:select --mode dry-run` | ~1s | 0 | PASS (sane prompt) | +| 11 | `kpi add` (valid ×2) | <1s | 0 | PASS | +| 12 | `kpi add` (ghost tables) | <1s | 4 | PASS — rejected, not written | +| 13 | `kpi add` (valid table, fake dims + fake formula col) | <1s | 0 | accepted — dims/formula are free text (see §6) | +| 14 | `recommend --explain` (full) | 0.5s | 0 | PASS — result analyzed in §5 | +| 15 | `recommend --override knowledge_graph --reason …` | 0.5s | 0 | PASS (attributed to default `developer`) | +| 16 | `bundle validate` / `--strict` | 0.5s | 0 / 5 | PASS (125 warnings; strict exits 5) but → D1b | +| 17 | `bundle export-schema` | <1s | 0 | PASS (63KB schema) | +| 18 | `run` (clean dir, offline) | **17.4s** | 0 | PASS (discover 0.8 / profile 12.4 / readiness 3.7 / patterns+recommend+validate ~0.12) | +| 19 | `run` + SIGINT mid-profile | — | 0 | FAIL → D3 (SIGINT swallowed; run completes) | +| 20 | `run` + SIGTERM, then `run --resume` | 16.5s | 0 | PASS — resumed exactly `profile…validate` | +| 21 | `run --resume` with nothing left | 0.4s | 6 | PASS (documented exit 6) | +| 22 | Idempotency: re-run 7 commands, stat+diff 3,350 files | ~25s | diff=0 | **PASS — zero diffs, not even mtimes** | +| 23 | `import --client-docs` (9-row fabricated CSV dictionary) | <1s | 0 | PASS | +| 24 | `reconcile` / `--bucket match` | 0.5s | 0 | PASS w/ conflict gap → D8 | +| 25 | `coverage` / `--format json` | <1s | 0 | PASS (imported 0/9, kpi 2/2) | +| 26 | staging-only `run` + `recommend --explain` | 1.25s | 0 | PASS — result in §5 | +| 27 | discover with password env var unset | — | **2** | DEVIATION → D6 (docs promise fail-fast exit 3) | +| 28 | `describe --column ` | <1s | 4 | PASS (clean artifact-not-found) | + +Term mappings: no CLI surface exists (bundle dir + reconciliation precedence only) — nothing to exercise offline beyond reconcile. + +## 3. Performance + +- **connect + discover + profile on 125 tables / 673 cols: ~15s** (org target < 2h — beaten by ~480×). Full offline pipeline: 17.4s. +- Tall table (100k rows): profiled within the 10,000-row budget; sample is the **head of the table** (values 1…10000), i.e. LIMIT-style, not random — sampling-bias caveat for skewed data. +- Bundle size: **13MB, 3,350 files** (full); staging-only 1.3MB. +- Cold-start outlier: the very first CLI invocation took 37.9s (uv first-run + cold container); all subsequent invocations 0.4–13s. + +## 4. Readiness vs seeded ground truth (10 critical / 3 warning / 58 info) + +| Seeded issue | Detected? | +|---|---| +| Q1 empty tables ×2 | YES — both Critical | +| Q2 all-null columns ×2 | YES — both Critical | +| Q3 constant columns ×2 | YES — Info (plus 56 incidental true constants, mostly same-instant `created_at` defaults — correct but noisy at scale) | +| Q4 high-null 70% | YES — Warning (plus 2 genuine incidental: `fact_sales.promotion_id` 0.8, `job_history.ended_on` 0.67) | +| Q5 broken FK on **inferred** rel | YES — Critical, 40 orphans sampled | +| Q6 broken FK on **declared `NOT VALID`** FK | YES — Critical (declared rels are value-checked too) | +| Q8 staging orphan join | YES — Critical | +| Q7 mixed-case status ('active'/'Active'/'ACTIVE') | NO — as expected; no case-consistency check exists. `type_mismatch` confirmed absent (deferred, matches docs/v1-deferred-scope) | + +False positive: the deliberate varchar↔int join (`stg_shipments.stg_order_id`) is reported as a Critical broken_fk with **350/350 orphans** — values are equal but compared as `'2' ≠ 2` (no type coercion) → D5. +Gap: the 6 jsonb/array columns whose profiles errored produce **no readiness issue at all** (only `unprofiled` status is surfaced; `error` status is invisible) → part of D2. + +## 5. Recommendation outputs + +**Full 5-schema bundle** (11 junctions, 88 rels, 6 prose columns): +- Strategy: **vector**, confidence medium. vector 6 pts (6 free-text cols ≥3; avg 298 chars ≥200) vs knowledge_graph 4 pts (11 junctions ≥2; ≥1 bridge) vs plain 1. +- KG lost its +2 density bonus because rel_density 0.704 < threshold — the 125-table denominator (distractors + no-FK schema) dilutes density. **Structural insight: vector and KG both max at 6 points and ties break toward vector, so a junction-rich schema can never out-rank a text-rich schema** — the "junctions ⇒ knowledge_graph" behavior seen on the small fixture does not generalize → D4. +- `coverage_pct: 1.0` reported with zero review work done (empty coverage = full coverage) → D17. + +**Staging-only (no-FK) bundle** (16 tables, 9 inferred-only rels): +- Strategy: **plain_schema**, confidence **low**; plain 1 vs KG 1 vs vector 0 — a bare tie broken by precedence. Inferred rels fully feed rel_density (0.562) and one inferred "junction" (`stg_inventory`, actually a fact) feeds the KG score. A cloud-warehouse dump with obvious entity joins lands on the least-capable strategy at low confidence. + +**Override**: recorded cleanly (`recommender chose: vector / SME override: knowledge_graph`), attributed to default `developer` when `DLA_SME_NAME` is unset. + +## 6. Defects and anomalies (ranked) + +**P1 — correctness of the contract deliverable** +- **D1. Manifest counts are wrong on multi-schema sources.** `bundle.json` says 130 tables / 703 cols / 92 rels / 5 idx; disk has 125 / 673 / 88 / 4. Cause: `PostgresConnector.introspect_schema()` (`apps/dla/src/dla/connectors/postgres.py`) calls `MetaData.reflect(schema=s)` per schema with default `resolve_fks=True`, which pulls cross-schema FK **targets** (and their FK closure) into the wrong schema pass — hr.employees(+departments/positions/locations) duplicated via finance, sales.dim_customers via analytics (= exactly +5/+30/+4/+1). The writer dedupes by path so disk is right, but downstream L2 consumers read the manifest. Repro: discover with the large config; compare `bundle.json.artifact_counts` to `ls schema/tables | wc -l`. Fix: `resolve_fks=False` or filter `metadata.tables` by `sa_table.schema == schema`. +- **D1b. `bundle validate` does not check manifest↔disk consistency**, so D1 ships silently even under `--strict`. +- **D2. jsonb / array columns cannot be profiled**: 6/673 columns fail with `TypeError: unhashable type: 'dict'|'list'` (distinct/top-value counting hashes raw values) — and profile_status `error` generates **no readiness issue**, so the failure is invisible in the report. Repro: profile the large config; see `bundle_large/profiles/analytics.typed_showcase.payload.json`. +- **D3. `dla run` cannot be aborted with Ctrl-C**: SIGINT mid-profile is swallowed and the pipeline runs to completion (verified twice; SIGTERM works). Operator-facing hazard on long engagements. + +**P2 — behavior contradicts docs or produces wrong signals** +- **D4. Recommender scoring makes knowledge_graph structurally unable to beat vector** (both cap at 6; ties break to vector), and rel_density (rels ÷ **all** tables) is diluted by distractor/no-FK tables. 11-junction schema → vector. Consider density over connected tables, or weighting junction count above text saturation, or a higher KG cap. +- **D5. broken_fk check compares sampled values without type coercion** → varchar/int joins report 100% orphans (false Critical). `apps/dla/src/dla/readiness/checks.py::check_broken_fk`. +- **D6. Unset password env var exits 2 with a raw SQLAlchemy stack tail**, but README §Secrets promises the loader "fails fast with exit code 3 if a required variable is unset". The loader accepts the missing var and the connector attempts an empty password. +- **D7. `profile --table ` exits 0, "profiles: 0"** — silent no-op; exit-code table says 4 (resource not found). `describe` gets this right. +- **D8. Reconciliation never produces `conflict` for type mismatches from a CSV dictionary**: documented `money` vs discovered `numeric(10,2)` classified `match (exact, 100.0)`. README/M5 promises "conflict (they disagree, e.g. a type mismatch)". + +**P3 — quality / robustness** +- **D9. Relationship inference singularization only strips a trailing 's'**: `stg_category_id → stg_categories` missed (`categorie_id` ≠ `category_id`); `customer_id → stg_customers` missed (prefix). -ies plurals are common; cheap fix. +- **D10. Value-overlap on serial ids is weak evidence**: distractor `analytics.stores (id 1..20)` attracted two **Strong (name+type+value_overlap)** cross-schema false-positive joins from `staging.*.store_id (1..25)`. Overlap of dense small-int surrogate ranges shouldn't upgrade confidence. +- **D11. Zero value overlap does not demote**: `stg_returns.stg_order_id` (100% orphans) still tagged **Strong** from name+type. Failed overlap check should be negative evidence, not neutral. +- **D12. Pattern-detector shape heuristics misfire at the margins**: compact facts (`fact_inventory_snapshots`, `stg_inventory`) classified junction (≤2 non-FK columns); master-data `hr.employees` classified a star fact; inferred false-positive rels propagate into "stars" (`stg_web_events` with the distractor store "dimension"). Self-referencing FKs are dropped from the graph, so `dim_accounts` snowflakes are undetectable (by design, worth documenting). +- **D13. Multi-column FK is flattened into two independent single-column relationships** (ledger fiscal_year→…, fiscal_month→…) — compositeness is lost in the contract. +- **D14. Glossary term extraction has no stop-list**: top proposals on this source are `name`, `status`, `created`, `stg`, `dim`, `fact` — technical prefixes and generic column names would be drafted as business terms in live mode. +- **D15. Table-describe prompt renders all column bullets as one 12KB line** (Jinja whitespace handling in `table_v1.j2` rendering) and has **no cap on column count** — 110 cols ≈ 3.3k tokens is fine, but a 1,000-column warehouse table would blow the prompt up ~10×. +- **D16. `bundle.json` is only maintained by discover**: `last_run_at` stays at discover time; profiles/readiness/patterns/kpi/recommendation counts never enter `artifact_counts`. +- **D17. Empty coverage reads as 100%**: `coverage_pct=1.0` when no reviewable artifacts exist, so FR-023's low-coverage confidence reduction can never trigger on a fresh bundle. +- **D18. Head-biased sampling**: profile samples are the first N rows (`events_tall` top values 1,2,3…), so stats on time-ordered tables reflect the oldest data. + +## 7. What worked well (verified positives) + +- **Idempotency is real**: 7 commands re-run over a 3,350-file bundle → zero diffs, zero mtime changes. +- Readiness caught **every** seeded issue class, including orphans behind a declared `NOT VALID` FK, with correct severities and useful `details`/suggestions. +- Reserved-word, quoted mixed-case, and 60-char identifiers survive the entire pipeline (discover→profile→describe→import→reconcile) with correct quoting. +- Exotic scalar types (uuid, enum, interval, bytea, inet, daterange) profile cleanly; the 110-column wide table and 100k-row tall table pose no functional problem; sample budget respected exactly. +- All 9 true junctions, both snowflake chains, and every real fact found by the detectors; staging contributed 0 declared / 9 inferred rels exactly as designed. +- `run`/`--resume`/exit-6, `--strict` exit-5, KPI ghost-table rejection exit-4, describe not-found exit-4 all match the documented contract. +- Recommender is deterministic and self-explaining; override flow works. +- Cross-schema FK, composite PKs, multi-col FK (modulo D13), self-referencing FKs all discovered. + +## 8. Ranked fix list + +1. D1 + D1b — manifest overcount + validate blind spot (contract-breaking for L2). +2. D2 — jsonb/array profiling failure + invisible `error` profiles (common column types). +3. D4 — recommender tie-break/density scaling (wrong hand-off signal at scale). +4. D5 — broken_fk type coercion (false Criticals erode trust in the report). +5. D3 — SIGINT handling in `dla run`. +6. D6/D7 — exit-code contract deviations (missing env var; silent no-op table filter). +7. D8 — unreachable conflict bucket for dictionary type mismatches. +8. D9/D10/D11 — inference naming + overlap-evidence improvements. +9. D14/D15 — glossary stop-list; prompt column-list newline + cap. +10. D16/D17/D18 — manifest freshness, empty-coverage semantics, sampling bias note. + +## 9. Artifacts + +- Fixture: `apps/dla/tests/fixtures/postgres_large/` (committed) +- Configs: `apps/dla/config/examples/postgres_large.yaml`, `postgres_large_staging_only.yaml` (committed) +- Bundles produced during the run (worktree-local, not committed): `bundle_large/`, `bundle_run/`, `bundle_resume*/`, `bundle_staging/` +- Raw logs: this scratchpad directory (`t1…t25`, `i1…i7`, `idem_*`, `e1…e3`)