Data Factory OS: Deterministic Econometrics & Calibrated AI Synthesis

Complete mathematical isolation between deterministic Python econometric compute and downstream probabilistic LLM synthesis via typed Pydantic v2 contracts. Multi-format tabular ingestion, sample selection drift monitoring, causal inference estimators, and automated multi-profile audience calibration.

---

1. System Overview

Large Language Models generate arithmetic inconsistencies and erratic parameter estimates when tasked with numerical analysis or statistical inference.

Data Factory OS addresses this limitation through strict architectural decoupling:

1. Deterministic Core: 100% of mathematical, statistical, econometric, and machine learning computations execute in native Python (Polars, SciPy, Statsmodels, Scikit-Learn, PyArrow, PyReadStat).
2. Contractual Decoupling: Compute results populate immutable Pydantic v2 schemas (EconometricDossier, ODCSContract, DeliverableSpec).
3. Calibrated Downstream Synthesis: The LLM operates purely as a strategic reasoning agent, translating structured econometric facts into executive decision memoranda, academic research notes, or technical audits in Italian or English.

---

2. System Architecture

                          DATA FACTORY OS PIPELINE
                                     │
                   Trigger (Desktop Dropzone / API / CLI)
                                     │
                                     ▼
                         ┌──────────────────────┐
                         │ Universal Reader     │  Zero-RAM metadata inspection
                         │ (Memory-Safe Guard)  │  Agnostic I/O: Parquet, DTA, XLSX, Feather, CSV
                         └──────────┬───────────┘  Equidistant downsampling ceiling
                                     │
                                     ▼
                         ┌──────────────────────┐
                         │ Deterministic Cleaner│  ODCS 4-gate verification
                         │ & Selection Monitor  │  Selection Drift Gate (Δμ <= 15%)
                         └──────────┬───────────┘  Cryptographic SHA-256 sample digest
                                     │
                                     ▼
                         ┌──────────────────────┐
                         │ Econometric Engine & │  Distributional metrics (Gini, HHI, VIF, Tukey)
                         │ Deterministic ML     │  Causal inference (OLS HC3/CR1, DiD 2x2, RDD)
                         └──────────┬───────────┘  Deterministic ML (K-Means, HistGB, Supervised Trees)
                                     │
                             EconometricDossier (Pydantic v2 Contract)
                                     │
                                     ▼
                         ┌──────────────────────┐
                         │ Audience Router &    │  Calibration matrix (Academic, Executive, Technical)
                         │ Epistemic Scoper     │  Epistemic gates (blocks causal overclaims)
                         └──────────┬───────────┘  Language localization (IT/EN catalogs)
                                     │
                                     ▼
                         ┌──────────────────────┐
                         │ Strategic Economist  │  Frame control, trade-off mapping
                         │ (Downstream LLM)     │  Provenance bindings: {{dossier.<path>}}
                         └──────────┬───────────┘  Externalized prompt templates
                                     │
                                     ▼
                         ┌──────────────────────┐
                         │ Executive Reporter   │  Vector PDF engine via Headless Chromium
                         │ & Table Transposer   │  Author typography (EB Garamond, Inter)
                         └──────────────────────┘  Zero RAM usage at rest (ephemeral runtime)

The runtime operates with zero memory footprint at rest (0 MB RAM), terminating immediately after PDF generation.

---

3. Core Engine Capabilities

3.1 Multi-Format Tabular Ingestion (worker/reader.py)

• Format Agnostic: Natively ingests .parquet, .pq, .dta (Stata), .xlsx, .xls (Excel), .feather, .arrow, .csv, and .tsv.
• Zero-RAM Header Introspection: inspect_schema_fast() reads metadata headers (PyArrow schemas, Stata binary headers via PyReadStat) in milliseconds without loading data arrays.
• Deterministic Memory Ceiling: load_dataset_memory_safe() samples large datasets exceeding 100 MB to 100,000 rows via equidistant indexing, maintaining representative temporal and cross-sectional coverage under 450 MB RAM.

3.2 Neurosymbolic Cleaning & Selection Drift Invariant (worker/cleaner.py)

• ODCS Contract Enforcement: Validates schema integrity, data types, primary key uniqueness, and feasibility boundaries across four sequential gates.
• Selection Drift Monitoring: Tracks percentage shift in target mean before and after cleaning: $$\Delta \mu = \frac{|\mu_{\text{post}} - \mu_{\text{pre}}|}{\mu_{\text{pre}}} \times 100$$ If $\Delta \mu > 15%$, execution halts via SelectionDriftError, preventing parameter distortion from sample truncation.
• Cryptographic Provenance: Generates an immutable SHA-256 hash over the cleaned data matrix, recorded alongside sample attrition counts in table1_sample_attrition.csv.

3.3 Advanced Econometrics & Causal Inference (worker/econometric_engine.py, worker/econometric_advanced.py)

• Distributional Metrics: Vectorized Gini inequality index ($O(n \log n)$), Herfindahl-Hirschman market concentration ($HHI \in [0, 10,000]$), Tukey interquartile extreme fences.
• Diagnostics: Correlation matrix inversion for Variance Inflation Factors ($\text{VIF} > 5.0$), Mann-Whitney U test, Cliff's delta non-parametric effect sizes.
• Robust Regression: OLS with MacKinnon-White HC3 leverage correction and Cameron-Miller cluster-robust standard errors (CR1).
• Causal Estimators: Callaway-Sant'Anna 2x2 Difference-in-Differences on clean cohorts with pre-trend testing, local linear Regression Discontinuity Design (RDD) with triangular kernel and Imbens-Kalyanaraman bandwidth, and Simpson's Paradox slope concordance tests.

3.4 Structural Sensitivity Engine (worker/sensitivity.py)

• Oster (2019) Delta Parameter ($\delta$): Quantifies selection on unobservables relative to observables required to nullify treatment effects ($|\delta| \ge 1.0$ indicates robustness).
• Specification Curve Analysis: Combinatorial estimation over valid subsets of non-colliding controls.

3.5 Deterministic Machine Learning & Segmentation (worker/ml_engine.py)

• Unsupervised Clustering: Automated K-Means with optimal cluster count selection via Silhouette analysis ($k \in [2, 6]$) on RobustScaler features.
• Supervised Classification: HistGradientBoosting with stratified 5-fold cross-validation, Out-Of-Fold (OOF) ROC-AUC, Brier score calibration, and Permutation Importance.
• Target-Guided Supervised Tree Clustering: Shallow decision trees partitioning feature space into interpretable segments with lift ratios and explained variance.

3.6 Audience & Objective Calibration Layer (worker/contracts/reporting.py, worker/quality_gate.py)

• Three Client Profiles: Dedicated formatting, masthead styling, metrics, and vocabulary standards for Academic, Executive, and Technical-Operative readers.
• Epistemic Scoping Gate: Strictly restricts conclusions to declared analytical objectives (DESCRIPTIVE, DIAGNOSTIC_ASSOCIATIVE, CAUSAL_INFERENCE, PREDICTIVE_SEGMENTATION), rejecting causal claims on descriptive data.

---

4. Computational Limits & Invariants

Dataset Scale	Memory Profile	Processing Mode	Execution Guarantees
$< 100 \text{ MB}$ ($< 100\text{k}$ rows)	$< 250 \text{ MB}$ RAM	Eager in-memory Polars / Pandas	Exact asymptotic estimation across all econometric batteries.
$100 \text{ MB} - 2 \text{ GB}$ ($100\text{k} - 5\text{M}$ rows)	$< 450 \text{ MB}$ RAM	Deterministic Equidistant Sampling	Downsampled to 100,000 observations; temporal and cross-sectional representations preserved.
$> 2 \text{ GB}$ ($> 5\text{M}$ rows / Multi-file archives)	Out-of-Core Disk Spilling	Streaming Polars LazyFrame / Parquet Partitioning	High-throughput aggregation; requires columnar partitioning prior to econometric estimation.

---

5. Quickstart & CLI Reference

5.1 Automated Dropzone Processing

Drop any tabular file into Desktop/inbox (or custom directory) and execute:

python worker/process_inbox.py

The script profiles the schema, runs the econometric engine, compiles the vector PDF report into reports/, archives the original dataset into processed/, and terminates with zero memory at rest.

5.2 Deterministic Harness CLI (worker/factory_cli.py)

# 1. Inspect schema and calculate exact null rates without loading data into RAM
python worker/factory_cli.py inspect data/sample_sales.csv --out results/schema.json

# 2. Clean dataset against ODCS contract and export sample attrition table
python worker/factory_cli.py clean data/sample_sales.csv `
  --contract domain_contracts/default_economic_contract.json `
  --out results/clean.parquet `
  --attrition results/table1_attrition.csv

# 3. Compute econometric dossier with deterministic ML modules
python worker/factory_cli.py compute results/clean.parquet `
  --lang en `
  --module ml `
  --ml-cluster `
  --out results/dossier.json

# 4. Run Oster sensitivity and specification curve
python worker/factory_cli.py sensitivity results/clean.parquet `
  --dependent churn_risk `
  --treatment discount_rate `
  --controls revenue `
  --dossier results/dossier.json `
  --out results/sensitivity.json

# 5. Verify deliverable compliance against audience and objective gates
python worker/factory_cli.py verify results/spec.json `
  --dossier results/dossier.json `
  --audience academic `
  --objective descriptive

# 6. Render vector PDF report
python worker/factory_cli.py render results/clean.parquet `
  --spec results/spec.json `
  --dossier results/dossier.json `
  --audience executive `
  --out-dir results

5.3 Headless REST API (worker/api.py)

# Start FastAPI service
uvicorn worker.api:app --host 127.0.0.1 --port 8000

# Health check
curl http://127.0.0.1:8000/health

# Synchronous execution
curl -X POST http://127.0.0.1:8000/api/v1/analyze-sync `
  -H "Content-Type: application/json" `
  -d '{
    "csv_path": "data/sample_sales.csv",
    "objective": "Identify discount leakage and margin churn drivers",
    "author": "Francesco Colombini",
    "lang": "en"
  }'

5.4 Docker Containerized Deployment

# Build and run containerized service
docker compose up -d

# Verify container health
docker compose ps
curl http://localhost:8000/health

---

6. Technical Documentation Index

Detailed specifications and mathematical proofs are maintained in dedicated reference documents:

• System Architecture & Invariants (docs/ARCHITECTURE.md)
• Econometric Specifications & Mathematical Reference (docs/ECONOMETRIC_SPECIFICATIONS.md)
• Audience Calibration & Objective Scoping (docs/AUDIENCE_AND_OBJECTIVES.md)
• Guida Calcoli Econometrici in Italiano (docs/it/CALCOLI_ECONOMETRICI_E_BACKEND_GUIDE.md)

---

7. License & Intellectual Property

Copyright © 2026 Francesco Colombini. All rights reserved.
Source code and architectural specifications are released under a Proprietary Source-Available License solely for technical portfolio inspection and engineering review. Any commercial usage, redistribution, mirroring, or extraction is strictly prohibited. Detailed terms are defined in LICENSE.

Author: Francesco Colombini
GitHub Profile · LinkedIn