Skip to content

SPECIFICATION.md

Latest commit

 

History

History
1211 lines (937 loc) · 61.6 KB

SPECIFICATION.md

File metadata and controls

1211 lines (937 loc) · 61.6 KB
Raw

SPECIFICATION.md — Synthdata Plugin

Maintenance: When the implementation changes, re-run /thinkkit:create-spec to incrementally update this document.

1. Purpose and Architecture Overview

What it does

Synthdata is a Claude Code plugin that generates realistic synthetic tabular datasets from YAML schemas. It ships as both a plugin (installable via claude --plugin-dir) and a marketplace (installable via /plugin marketplace add). The plugin contains eight skills covering the full lifecycle: generation, extraction, extension, anonymization, computation, serving, workflow planning, and interactive tutorial.

Who it's for

Developers, data engineers, QA teams, and analysts who need realistic test/demo data without touching production databases. Common use cases: populating dev environments, building dashboards, training ML models on realistic distributions, security tabletop exercises, and stakeholder demos.

High-level architecture

                      ┌─────────────────────────────────────────────┐
                      │            Claude Code Plugin System         │
                      │                                             │
                      │  .claude-plugin/                            │
                      │    plugin.json    (plugin manifest)         │
                      │    marketplace.json (marketplace manifest)  │
                      └────────────────┬────────────────────────────┘
                                       │
                           ┌───────────┴───────────┐
                           │     skills/            │
          ┌────────────────┼────────────────────────┼────────────────────┐
          │                │                        │                    │
   ┌──────┴──────┐  ┌─────┴─────┐  ┌──────────┐ ┌─┴──────────┐ ┌──────┴──────┐
   │  generate   │  │  extract  │  │  extend  │ │ anonymize  │ │  compute    │
   │             │  └───────────┘  └──────────┘ └────────────┘ └─────────────┘
   │ ┌─────────┐│                                               ┌─────────────┐
   │ │ engine/ ││                                               │   serve     │
   │ │ schema  ││                                               │  (MCP svr)  │
   │ │ distrib ││                                               └─────────────┘
   │ │ faker   ││   ┌───────────────┐  ┌───────────┐
   │ │ profile ││   │prompt-builder │  │ tutorial   │
   │ │ relship ││   │  (advisory)   │  │(walkthrough│
   │ │temporal ││   └───────────────┘  └───────────┘
   │ │writers/ ││
   │ └─────────┘│
   │ templates/ │
   │ references/│
   └────────────┘

Key architectural patterns

  • Skill-per-directory: Each skill is self-contained with its own SKILL.md, scripts/, and optional templates//references/. Skills do not import from each other.
  • Schema-driven generation: YAML schema in, dict[str, pd.DataFrame] out. The schema fully describes the output structure; the engine is a pure function of (schema, effort, seed).
  • Pipeline architecture (generate engine): parent tables first (no FK), then child tables (FK resolved from parents), profiles assigned, temporal columns generated, then writers serialize.
  • Plugin/marketplace duality: The repo root has both plugin.json (for direct plugin use) and marketplace.json (for marketplace discovery). The marketplace's plugin source is "./" (self-referencing).

How components communicate

Skills communicate only through files on disk. The generate skill writes xlsx/csv/json/sql/parquet files. Other skills (extract, extend, anonymize, compute, serve) read those files. There is no shared in-process state, no IPC, no database. Each skill's Python script is a standalone CLI tool.

2. Module Organization and Responsibilities

Logical decomposition

The plugin is organized as eight independent skill modules plus shared infrastructure at the repo root:

Module Responsibility Has scripts? Key dependencies
synthdata-generate Schema-driven data generation — the core engine Yes (generate.py + engine/) pandas, numpy, faker, openpyxl, pyyaml
synthdata-extract Excel → JSON conversion with title-row detection Yes (extract.py) pandas, openpyxl
synthdata-extend Add rows/columns to existing datasets Yes (extend.py) pandas, numpy, faker (imports from generate engine)
synthdata-anonymize PII detection and Faker-based replacement Yes (anonymize.py) pandas, faker (imports from generate engine)
synthdata-compute Derive aggregated/scored/transformed tables Yes (compute.py) pandas, numpy, openpyxl
synthdata-serve Read-only MCP server from datasets Yes (serve.py, export.py) pandas, numpy, mcp, openpyxl
synthdata-prompt-builder Plan multi-step generation workflows No (advisory SKILL.md only) None
synthdata-tutorial Guided interactive walkthrough No (SKILL.md only) None

Root-level infrastructure:

  • .claude-plugin/ — plugin + marketplace manifests
  • install.sh — skill installation to ~/.claude/skills/
  • package.sh — distributable archive builder
  • registry.json — plugin registry with skill catalog

Dependency direction between modules

synthdata-extend ──imports──▶ synthdata-generate/engine (faker_fields, distributions)
synthdata-anonymize ──imports──▶ synthdata-generate/engine (faker_fields, writers/xlsx)

All other skills are fully independent — no cross-skill imports. The two import arrows above are the only code-level dependencies between skills. They use relative path manipulation (sys.path.insert) to reach the generate engine.

Skills communicate exclusively through files on disk: generate writes xlsx/csv/json/sql/parquet; other skills read those files. No shared in-process state, no IPC, no database.

Internal vs. exposed boundaries

Exposed (public interfaces):

  • Each skill's CLI entry point (scripts/*.py) — invoked by Claude via python3 <path> [flags]
  • Each skill's SKILL.md — defines trigger phrases, workflow, and allowed tools
  • YAML schema format — user-authored inputs to the generator
  • MCP protocol tools (serve.py) — exposed to Claude via MCP stdio transport
  • Plugin/marketplace manifests — consumed by Claude Code plugin system

Internal (implementation details):

  • engine/ submodules (schema.py, distributions.py, faker_fields.py, profiles.py, relationships.py, temporal.py) — only consumed by Generator.__init__.py
  • engine/writers/ — pluggable but only dispatched internally by the writer registry
  • _row_id columns — internal FK resolution bookkeeping, stripped by writers
  • Profile assignment mechanics — jitter, weight normalization (users see profiles: in YAML)

Organizational constraints

  1. Skills must be self-contained directories under skills/. The Claude Code plugin system discovers skills by scanning this directory. Each skill must have a SKILL.md at its root.
  2. No shared code library: By design, skills duplicate rather than share code (~50 lines of load_tables() duplicated across extend, anonymize, compute, serve). This avoids import-path issues with the plugin system.
  3. Naming convention: synthdata-<verb> for skill directories; SKILL.md uppercase per Claude Code convention (Cowork bundle renames to Skill.md).
  4. Templates are YAML files in synthdata-generate/templates/ — the --template CLI flag resolves to templates/<name>.yaml.
  5. References are per-skill markdown docs in references/ subdirectories — not shared across skills.

Generated vs. hand-authored

  • dist/: generated by package.sh — 3 archive formats
  • examples/: exists but empty (reserved)
  • Everything else: hand-authored

3. Public Interfaces

3.1 CLI: synthdata-generate (generate.py)

python3 scripts/generate.py [flags]
Flag Type Default Description
--template <name> string — Built-in template name (resolves to templates/<name>.yaml)
--schema <path> string — Custom YAML schema file path
--list-templates flag — Print available templates and exit
--effort choice medium quick, medium, or thorough
--output <path> string ./synthdata_output Output file or directory
--format choice schema default xlsx, csv, json, sql, or parquet
--seed int 42 Random seed for reproducibility
--locale string en_US Faker locale code

Behavior:

  • Exactly one of --template or --schema must be provided (else exit with error)
  • --list-templates causes early return after printing template names
  • Schema is loaded via load_schema(), validated via validate_schema()
  • Format defaults to schema.writers[0] if not specified, else "xlsx"
  • Output path semantics depend on format (see Writers section)
  • Prints per-table row/column counts and output path on success

Exit codes: 0 on success, 1 on validation error or missing arguments

3.2 CLI: synthdata-extract (extract.py)

python3 scripts/extract.py [flags]
Flag Type Default Description
--input <path> string . Input xlsx file or directory of xlsx files
--output <path> string ./json Output directory for JSON files
--title-row choice auto auto, yes, or no
--flatten flag false Bundle all sheets per workbook into one JSON
--indent int 2 JSON indentation (0 for compact)

Behavior:

  • If input is a directory, globs *.xlsx
  • Title-row auto-detection: row 1 is a banner if it has exactly one non-empty cell and multiple total cells
  • Output filenames: <sheet>.json (single workbook) or <workbook>_<sheet>.json (multiple workbooks, not flattened)
  • With --flatten: one JSON per workbook as <workbook>.json (dict mapping sheet names to row arrays); no prefix applied even with multiple workbooks
  • Dates serialized as ISO strings
  • Columns with empty or None headers are silently skipped

3.3 CLI: synthdata-extend (extend.py)

python3 scripts/extend.py [flags]
Flag Type Default Description
--input <path> string required Existing dataset (xlsx/csv/json)
--output <path> string <input>_extended.<ext> Output path
--table <name> string required Target table name
--add-rows <N> int 0 Number of new rows to append
--add-column <name> string — New column name
--col-type choice — int, float, bool, choice, faker, constant
--distribution string normal Distribution for numeric columns
--mean float 0 Distribution mean
--std float 1 Distribution std deviation
--sigma float 0.5 Lognormal sigma
--values string — Comma-separated values for choice type
--method string word Faker method name (for faker type)
--value string — Constant value (for constant type)
--seed <N> int timestamp-derived Random seed (int(time.time()) & 0xFFFF if not set)
--overwrite flag — Allow overwriting existing column

Behavior:

  • Row extension: continues ID sequences, respects FK constraints (samples from existing parent IDs), infers distributions from existing data
  • Column addition: generates values of specified type; errors if column exists unless --overwrite
  • Never modifies existing rows; appends only
  • Preserves output format from input format
  • Imports get_faker, faker_value from generate engine's faker_fields, and sample_column from distributions

3.4 CLI: synthdata-anonymize (anonymize.py)

python3 scripts/anonymize.py [flags]
Flag Type Default Description
--input <path> string required Source dataset
--output <path> string <input>_anon.<ext> Output path
--scan flag — Detect-only mode (print report, exit)
--map <mappings> string — Comma-separated col=faker_method overrides (parsed by splitting on , then =)
--keep <cols> string — Comma-separated columns to pass through unchanged
--drop <cols> string — Comma-separated columns to remove entirely
--preserve-joins <cols> string — Comma-separated columns that must map consistently across tables
--locale string en_US Faker locale
--seed int 42 Random seed

Behavior:

  • PII detection via two signals: column name heuristics + value regex patterns
  • Confidence scoring: High (0.95) if both signals match, Medium (0.70-0.75) if one, Low (0.35) if high-cardinality string
  • Only columns with confidence >= 0.6 are auto-anonymized (unless overridden)
  • Deterministic mapping: same input value always produces same fake value (via hash-keyed cache)
  • --preserve-joins: shares cache across tables for specified columns
  • Imports get_faker, faker_value from generate engine's faker_fields; uses generate engine's xlsx writer for xlsx output

3.5 CLI: synthdata-compute (compute.py)

python3 scripts/compute.py [flags]
Flag Type Default Description
--input <path> string required Source dataset
--output <path> string <input>_computed.<ext> Output path
--inspect flag — Print table schemas and exit
--code <file.py> string — Python file with computation logic
--expr <string> string — Inline Python expression
--append flag — Add results to copy of input (xlsx only)

Computation namespace: user code receives tables (dict of DataFrames), pd, np, result (empty dict to populate)

3.6 CLI: synthdata-serve (serve.py)

python3 scripts/serve.py [flags]
Flag Type Default Description
--input <path> string required Dataset path (xlsx/csv/json/parquet/dir)
--name <string> string synthdata-<filename> MCP server name
--inspect flag — Print table schemas and exit

Behavior:

  • Without --inspect: starts MCP stdio server exposing 5 tools (see section 5.6)
  • Loads all tables into memory at startup; all operations are read-only
  • Server name derived from input filename if not specified

3.7 CLI: synthdata-serve export (export.py)

python3 scripts/export.py [flags]
Flag Type Default Description
--input <path> string required Dataset path
--output <path> string required Output directory for standalone project
--name <string> string synthdata-<filename> Server name

Produces: self-contained directory with server.py, data/dataset.json, requirements.txt, README.md

3.8 Shell: install.sh

./install.sh [flags]
Flag Description
(none) Install all skills via symlink
--copy Copy instead of symlink
--skill <name> Install single skill
--uninstall Remove all synthdata skills
--list List available skills
--help Show help

Target directory: $CLAUDE_SKILLS_DIR or ~/.claude/skills/ Behavior: creates symlinks (or copies) from skills/<name> to target; skips if destination already exists

3.9 Shell: package.sh

./package.sh [--clean]

Produces 3 archives in dist/:

  1. synthdata-marketplace-v<VERSION>.<ext> — full repo (README, LICENSE, registry, .claude-plugin, skills)
  2. synthdata-plugin-v<VERSION>.<ext> — standalone plugin (plugin.json + skills)
  3. synthdata-v<VERSION>.plugin — Cowork bundle (SKILL.md renamed to Skill.md)

Format: zip if zip is available, else tar.gz --clean: removes dist/ and exits

3.10 Configuration Schemas

plugin.json

{
  "name": "synthdata",
  "description": "...",
  "version": "0.3.0",
  "author": { "name": "Dan Rapp" },
  "license": "MIT",
  "keywords": ["synthetic-data", "test-data", "faker", ...],
  "skills": "./skills/"
}

Required fields: name, skills. The skills field is a relative path to the skills directory.

marketplace.json

{
  "name": "synthdata-marketplace",
  "owner": { "name": "Dan Rapp" },
  "metadata": { "description": "...", "version": "0.3.0" },
  "plugins": [
    {
      "name": "synthdata",
      "source": "./",
      "description": "...",
      "author": { "name": "Dan Rapp" },
      "license": "MIT",
      "keywords": [...],
      "category": "data-tools",
      "tags": [...]
    }
  ]
}

The "source": "./" self-reference means the marketplace's plugin source is the repo root itself.

registry.json

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "name": "synthdata-plugin-marketplace",
  "description": "...",
  "version": "0.3.0",
  "plugins": [
    {
      "name": "synthdata",
      "description": "...",
      "version": "0.3.0",
      "path": ".",
      "author": { "name": "Dan Rapp" },
      "tags": [...],
      "skills": [
        { "name": "synthdata-generate", "description": "..." },
        { "name": "synthdata-extract", "description": "..." },
        ...
      ]
    }
  ]
}

SKILL.md Frontmatter (all skills)

---
name: <skill-name>
description: >
  <Multi-line description with trigger phrases for skill activation>
version: 0.1.0
allowed-tools: Read Bash Glob Write
---

Required fields: name, description, version, allowed-tools.

4. Data Models and State Management

4.1 Schema Dataclasses (schema.py)

@dataclass
class ColumnSpec:
    name: str                              # Column name in output
    type: str                              # One of: id, faker, choice, int, float, bool,
                                           #   date, timestamp, constant, formula, ref
    params: dict[str, Any] = {}            # Type-specific parameters (see column types)

@dataclass
class ProfileSpec:
    name: str                              # Profile label (e.g., "whale", "dormant")
    weight: float                          # Relative probability (normalized at assignment)
    overrides: dict[str, Any] = {}         # Field name -> value (numbers for multipliers)

@dataclass
class TableSpec:
    name: str
    columns: list[ColumnSpec] = []
    rows: dict[str, int] | int | None = None     # {quick, medium, thorough} or single int
    profiles: list[ProfileSpec] = []
    foreign_key: dict[str, Any] | None = None    # {column, references, distribution, alpha}
    rows_per_parent: dict[str, Any] | None = None # {distribution, lam/value/min/max, lam_expr}
    temporal: dict[str, Any] | None = None        # {column, start, end, pattern, weekday_only}

@dataclass
class Schema:
    name: str
    tables: list[TableSpec]
    writers: list[str] = ["xlsx"]          # Default output format
    effort_defaults: dict[str, Any] = {}   # Reserved for future use
    post_process: str | None = None         # Reserved for future use
    description: str | None = None

Row count resolution (TableSpec.resolve_rows(effort)):

  • None -> 100
  • int -> that value directly
  • dict -> rows[effort], falling back to rows.get("medium", 100)

4.2 Generator State

class Generator:
    schema: Schema
    effort: str                            # "quick", "medium", or "thorough"
    seed: int
    fake: Faker                            # Seeded Faker instance
    tables: dict[str, pd.DataFrame]        # Populated during generate()
    profiles: dict[str, pd.DataFrame]      # Profile assignments per table

The tables dict is the central data structure. Each key is a table name, each value is a pandas DataFrame where columns match the schema. Internal columns prefixed with _ (e.g., _row_id) are stripped by writers.

4.3 State Management

All state is ephemeral and in-process. There is no persistent state, no database, no cache files. Each CLI invocation:

  1. Loads schema (YAML) or dataset (xlsx/csv/json) from disk
  2. Processes entirely in memory
  3. Writes output to disk
  4. Exits

The MCP server (serve.py) holds tables in memory for the lifetime of the server process but makes no mutations.

5. Key Algorithms and Business Logic

5.1 Generation Pipeline

Input: Schema YAML + effort level + seed

1. Parse YAML -> Schema object (load_schema -> parse_schema)
2. Validate schema (validate_schema: check FK references exist)
3. Initialize Generator(schema, effort, seed, locale)
   - Seeds: random.seed(seed), np.random.seed(seed)
   - Creates Faker(locale), seeds it, clears unique cache
4. For each table in schema.tables (order matters: parents first):
   a. If no foreign_key -> _generate_parent_table(table, n_rows)
   b. If foreign_key    -> _generate_child_table(table)
   c. If table.profiles -> assign_profiles(df, profiles, effort, seed)
      - Merges profile_type column into DataFrame
5. Return dict[str, pd.DataFrame]

Output: Dictionary of DataFrames (one per table)

5.2 Column Generation (_generate_column)

def _generate_column(col, n_rows, context, parent_df=None,
                     parent_profiles=None, fk_values=None,
                     fk_column=None, parent_col=None) -> list:
Type Algorithm
id [f"{prefix}{i+start:0{width}d}" for i in range(n_rows)]
faker Call faker_value(fake, method, unique, **args) per row; unique retries up to 100 times
choice random.choices(values, weights=weights, k=n_rows); if weights_by_profile: per-row profile lookup to select weights
int/float sample_column(n_rows, params, as_int) — delegates to distributions module
bool [random.random() < p for _ in range(n_rows)]; if p_by_profile or weights_by_profile: per-row profile lookup
date Random selection from pd.date_range(start, end), returns date objects
timestamp generate_timestamps(n_rows, params, effort)
constant [value] * n_rows
formula Per-row eval(expr, {"__builtins__": {}}, row_context) with random available; returns None on error
ref Lookup parent_df[parent_col] -> parent_df[from_col] per FK value; None if not found

5.3 Statistical Distributions (distributions.py)

sample_column(n, params, as_int=False) -> list

Distribution NumPy call Key params
uniform np.random.uniform(min, max, n) min=0, max=100
normal np.random.normal(mean, std, n) mean=0, std=1
lognormal np.random.lognormal(mu, sigma, n) mean (linear space), sigma=0.5. Conversion: mu = log(mean) - 0.5*sigma^2
exponential np.random.exponential(scale, n) scale=1.0
poisson np.random.poisson(lam, n) lam=1.0
gamma np.random.gamma(shape, scale, n) shape=2.0, scale=1.0
pareto np.random.pareto(alpha, n) + 1 alpha=2.0 (shifted +1)
constant np.full(n, value) value (required)

Post-processing: clip to [min, max] if specified, then round(decimals) (default 2 for float), cast to int if as_int.

sample_count(params) -> int: samples single count value for rows_per_parent. Supports constant, poisson, uniform, normal. Returns max(0, ...).

zipfian_weights(n, alpha=1.5) -> ndarray: weights[i] = 1/(rank[i]^alpha), normalized to sum=1.

5.4 Foreign Key Resolution (relationships.py)

EFFORT_SCALES = {"quick": 0.1, "medium": 1.0, "thorough": 2.0}

def resolve_foreign_keys(parent_df, parent_col, rows_per_parent,
                         distribution="uniform", alpha=1.5,
                         parent_profiles=None, effort="medium") -> list:

Algorithm:

  1. Extract parent IDs list and effort scale factor
  2. If distribution == "zipfian": shuffle parent indices, compute zipfian weights
  3. If parent_profiles provided: build {parent_id: profile_row_dict} lookup
  4. For each parent:
    • If lam_expr defined + parent in profile map:
      • Evaluate expression in profile context (eval with profile fields + random, np)
      • Fallback to rows_per_parent.get("lam", 1.0) on exception
      • Result: lam = max(0.01, float(result) * scale)
    • Else: scale params via _scale_params(rows_per_parent, scale)
    • Sample n_children = sample_count(params)
    • Extend FK list: [parent_id] * n_children
  5. If zipfian weights computed: resample entire FK list with weighted random.choices
  6. Return FK value list (length = total child rows)

_scale_params(params, scale): multiplies distribution parameters by effort scale.

  • constant: max(0, round(value * scale))
  • poisson: max(0.01, lam * scale)
  • uniform: max(0, min * scale), max(min, max * scale)
  • normal: mean * scale

5.5 Profile Assignment (profiles.py)

def assign_profiles(df, profiles, effort="medium", seed=42) -> pd.DataFrame:

Quick mode (effort == "quick"):

  • All rows: profile_type = "baseline"
  • All numeric overrides: 1.0 (no variation)
  • Returns DataFrame with [id_col, profile_type, ...override_fields]

Normal mode:

  1. Normalize weights: count[name] = round(n * weight / total_weight)
  2. Adjust first profile to absorb rounding error
  3. Shuffle assignments
  4. For each row: look up profile, apply jitter to each numeric override
  5. Jitter: value * random.uniform(0.85, 1.15), rounded to 3 decimals

_pick_id_col(df): finds first column ending with _id, or _row_id, or id, or falls back to first column.

5.6 MCP Server Tools (serve.py)

The server exposes 5 read-only tools via MCP stdio protocol:

list_tables

  • Input: none
  • Output: {tables: [{name, row_count, column_count, columns: [{name, dtype}]}], count}

describe_table

  • Input: {table: string}
  • Output: {name, row_count, column_count, columns: [{name, dtype, null_count, unique_count, sample_values, possible_fk?}]}
  • FK hints: if column ends with _id and exists in another table, adds possible_fk field

query_table

  • Input: {table, filters?, columns?, sort_by?, sort_order?, limit?, offset?}
  • Output: {rows, total_matched, returned, truncated, note?}
  • Default limit: 50. Max: 500. Response cap: 100KB (auto-truncates by halving rows)
  • Filter operators: = (implicit), >, <, >=, <=, !=, in, contains (case-insensitive), between
  • Multiple filters combine with AND logic

sample_rows

  • Input: {table, n?} (default n=5)
  • Output: {rows, sampled, total_rows}

get_stats

  • Input: {table, columns?} (default: all non-internal columns)
  • Output: {table, row_count, columns: [{name, dtype, count, null_count, unique_count, mean?, std?, min?, max?, percentiles?, top_values?}]}
  • Numeric columns get descriptive stats; string/categorical get value counts (top 10)

5.7 Filter Engine (serve.py)

def apply_filters(df, filters) -> pd.DataFrame:
    mask = pd.Series(True, index=df.index)
    for col, condition in filters.items():
        if isinstance(condition, dict):
            for op, val in condition.items():
                # Apply operator to mask
        else:
            mask &= df[col] == condition  # equality
    return df[mask]

5.8 PII Detection (anonymize.py)

Two-signal approach:

Signal 1 — Column name heuristics (keyword matching). Structure is (keywords, faker_method, exclusions) — a column matches if any keyword is a substring of the normalized column name AND no exclusion is a substring. Heuristics are checked in order; first match wins:

# Keywords Faker method Exclusions
1 email, mail email (none)
2 ssn, socialsecurity ssn (none)
3 fullname name (none)
4 firstname, givenname first_name (none)
5 lastname, surname, familyname last_name (none)
6 phone, mobile, cellphone, telephone phone_number (none)
7 streetaddress, address1, addressline, mailingaddress street_address (none)
8 street street_address (none)
9 zipcode, postalcode, postcode postcode (none)
10 city city (none)
11 dob, dateofbirth, birthdate, birthday date_of_birth (none)
12 creditcard, ccnum, cardnumber credit_card_number (none)
13 ipaddress, ipv4 ipv4 (none)
14 username, login, handle user_name (none)
15 name name username, hostname, filename, pathname, company, product, event, module, campaign, policy, department, role, city, state, country, brand
16 company, employer, organization company (none)
17 url, website, homepage url (none)

Note: Entry 15 (name) is last in order so that more specific entries (firstname, lastname, username) match first. The exclusion list prevents false positives on columns like "department_name" or "campaign_name".

Signal 2 — Value regex patterns (>= 80% sample match threshold):

Pattern Regex
Email ^[^@\s]+@[^@\s]+\.[^@\s]+$
Phone ^(?!\d{4}-\d{2}-\d{2})[\+\(]?[\d][\d\s\-\(\)\.]{8,}\d$
SSN ^\d{3}-?\d{2}-?\d{4}$
IPv4 ^(\d{1,3}\.){3}\d{1,3}$
URL ^https?://
UUID4 ^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$

Signal 3 — High cardinality: nunique/count > 0.9 for string columns

Confidence scoring:

  • 0.95: heuristic + regex both match
  • 0.75: heuristic only (non-numeric)
  • 0.70: regex only
  • 0.35: high cardinality + string only
  • 0.0: numeric columns (excluded unless heuristic is date_of_birth)

5.9 Dataset Extension (extend.py)

ID continuation (continue_ids):

  • Parses existing IDs with regex ^([A-Za-z\-_]*)(\d+)$
  • Finds max numeric value across all existing IDs
  • Generates new IDs continuing the sequence with same prefix and zero-pad width
  • Fallback: if no values match the pattern, generates X-N style IDs starting from X-{len(existing)}

Column type inference (infer_column_type):

  • Checks dtype first: bool -> "bool", integer -> "int", float -> "float", datetime -> "timestamp"
  • For string-like columns: samples 20 values, checks all against regex ^[A-Za-z\-_]*\d+$; if all match -> "id"
  • Default: "str"

Row synthesis (synthesize_new_rows):

  • Per inferred column type:
    • ID columns: continue_ids()
    • Integer/float: sample from normal distribution fitted to existing data (mean, std, min, max)
    • Boolean: Bernoulli with p = existing_mean
    • String/categorical: random.choices() from existing unique values
    • FK columns (*_id): sample from matching parent table's ID values

5.10 Temporal Generation (temporal.py)

def generate_timestamps(n, params, effort="medium") -> list[datetime]:
Pattern Algorithm
uniform Random offset in [0, total_seconds] from start. Weekday-only: shift weekend to prior Friday
business-hours Random day + random hour in [8,17], minute [0,59], second [0,59]. Retry up to 20 times if weekend (when weekday_only)
diurnal Random day + hour from np.random.triangular(0, 14, 23) (peak at 2pm), random minute/second

Post-processing: sort chronologically if params.get("sorted", False).

_parse_dt(v): accepts datetime, date, or string. Tries formats: %Y-%m-%dT%H:%M:%S, %Y-%m-%d %H:%M:%S, %Y-%m-%d, then fromisoformat().

6. Capabilities and External Integrations

Required capabilities and chosen libraries

Capability needed Library Why this choice
Tabular data representation + I/O pandas (any version) DataFrames are the natural representation for tabular data. All downstream operations (filtering, grouping, statistics) use pandas APIs. No alternative offers comparable breadth.
Statistical distributions numpy (any version) Required for normal, lognormal, exponential, gamma, pareto, poisson, Zipfian — not available in Python stdlib. Also used for vectorized weight computation.
Excel (.xlsx) read/write with styling openpyxl (any version) Standard Python xlsx library. Used in generation (title banners, styled headers) and reading (extract, extend, anonymize, compute, serve). Alternative xlsxwriter is write-only.
Realistic fake data faker (any version) Best-in-class: 100+ provider methods, locale support, uniqueness tracking, deterministic seeding. No real alternative at this breadth.
YAML parsing pyyaml (any version) Standard YAML parser. Schemas are YAML for human readability and LLM-friendliness (Claude can read/write YAML in conversation).
MCP server protocol mcp (>= 1.0.0) Official MCP SDK from Anthropic. Provides Server class, stdio_server transport, Tool/TextContent types. No alternative — this is the standard.
Parquet file format pyarrow (any, optional) Only needed if user chooses parquet output format. Heavy (~150MB) so not required. Lazy-imported with clear error if missing.

No version pinning. No lock files. All packages installed with --break-system-packages (for system Python on modern distros). The project targets broad compatibility rather than reproducible builds.

Runtime requirements

  • Python 3.11+ (uses f-strings, match statements, dataclass features)
  • bash (for install.sh and package.sh)
  • zip (optional; package.sh falls back to tar.gz)
  • OS: Linux, macOS, or Windows (all Python, no native extensions except optional pyarrow)

External service integrations

The project integrates with no external services at runtime. All data generation, transformation, and serving is local. The MCP server communicates via stdio (not network sockets).

The only external integration is the Claude Code plugin system: the .claude-plugin/ manifests are consumed by Claude Code to discover and activate skills. This is a declarative integration (JSON manifests), not a runtime API call.

7. Build, Test, and Run Instructions

System prerequisites

  • Python 3.11+
  • pip3
  • bash (for install.sh and package.sh)
  • zip (optional; package.sh falls back to tar.gz)

Install dependencies

pip install openpyxl faker numpy pandas pyyaml mcp --break-system-packages
# Optional: pip install pyarrow --break-system-packages

Run the generator

# List templates
python3 skills/synthdata-generate/scripts/generate.py --list-templates

# Generate from template
python3 skills/synthdata-generate/scripts/generate.py \
  --template hr-directory --effort quick --output /tmp/hr.xlsx

# Generate from custom schema
python3 skills/synthdata-generate/scripts/generate.py \
  --schema path/to/schema.yaml --format json --output /tmp/out/

Smoke test all templates

for t in skills/synthdata-generate/templates/*.yaml; do
  python3 skills/synthdata-generate/scripts/generate.py \
    --template "$(basename "$t" .yaml)" --effort quick --output /tmp/
done

Run other skills

# Extract Excel to JSON
python3 skills/synthdata-extract/scripts/extract.py --input data.xlsx --output ./json/

# Extend dataset
python3 skills/synthdata-extend/scripts/extend.py --input data.xlsx --table employees --add-rows 100

# Anonymize
python3 skills/synthdata-anonymize/scripts/anonymize.py --input data.xlsx --scan
python3 skills/synthdata-anonymize/scripts/anonymize.py --input data.xlsx --output anon.xlsx

# Compute derived tables
python3 skills/synthdata-compute/scripts/compute.py --input data.xlsx --inspect
python3 skills/synthdata-compute/scripts/compute.py --input data.xlsx --code transform.py --output derived.xlsx

# Serve as MCP server
python3 skills/synthdata-serve/scripts/serve.py --input data.xlsx --inspect
python3 skills/synthdata-serve/scripts/serve.py --input data.xlsx --name my-data

# Export standalone MCP server
python3 skills/synthdata-serve/scripts/export.py --input data.xlsx --output ./server-project/

Install as Claude Code plugin

# Option 1: Marketplace
# /plugin marketplace add rappdw/synthdata
# /plugin install synthdata@synthdata-marketplace

# Option 2: Plugin directory
claude --plugin-dir /path/to/synthdata

# Option 3: Manual skill copy
./install.sh              # symlink all skills to ~/.claude/skills/
./install.sh --copy       # copy instead
./install.sh --uninstall  # remove

# Option 4: Package and distribute
./package.sh              # produces dist/ with 3 archive formats

Testing approach

No test framework. Validation is manual: run generator against templates, inspect output.

8. Design Decisions and Constraints

Skills are self-contained (no shared imports)

Decision: Each skill duplicates data-loading code rather than sharing a common library. Alternative rejected: Shared synthdata-common module. Reason: Claude Code plugins discover skills as independent directories. A shared module would require import path manipulation and break the plugin system's assumptions. The duplication (~50 lines of load_tables()) is preferable to coupling.

YAML schemas, not Python DSL

Decision: Schema format is YAML with a fixed vocabulary of column types and distributions. Alternative rejected: Python-based schema definition (like SQLAlchemy models or factory_boy). Reason: YAML is LLM-friendly — Claude can read, write, and modify schemas in conversation. A Python DSL would require Claude to generate executable code for schema definition, adding a code-gen step before data-gen.

Fixed column type vocabulary

Decision: 11 column types (id, faker, choice, int, float, bool, date, timestamp, constant, formula, ref). Alternative rejected: Arbitrary expression columns only. Reason: Fixed types make schemas declarative and predictable. The formula type is the escape hatch for computed values. The ref type handles parent-column copying without formulas.

Effort controls scale, not shape

Decision: quick/medium/thorough changes row counts and profile richness, never schema structure. Alternative rejected: Effort levels that add/remove columns or simplify relationships. Reason: Predictable schema shape across effort levels means dashboards and downstream tools work identically regardless of effort — only the statistical fidelity changes.

Profile jitter (+-15%)

Decision: Profile override values get random jitter of +-15% per row. Alternative rejected: Exact override values for all rows in a profile. Reason: Without jitter, all "whale" customers would have identical behavior metrics, which is unrealistic and visible in dashboards as flat lines.

Writers strip _row_id columns

Decision: Internal columns prefixed with _ are removed before serialization. Alternative rejected: Exposing internal columns or using a separate metadata channel. Reason: _row_id exists solely for FK resolution when no explicit ID column is defined. It's an implementation detail that shouldn't appear in output.

MCP serve: fixed tools, not per-table tools

Decision: 5 fixed tools (list_tables, describe_table, query_table, sample_rows, get_stats) with a table parameter. Alternative rejected: One tool per table (e.g., get_customers, get_orders). Reason: Fixed tool count means the MCP server works identically regardless of dataset size. Per-table tools would flood Claude's tool list and scale poorly for many-table datasets.

MCP response size cap (100KB)

Decision: query_table auto-truncates responses exceeding 100KB by halving the row count. Alternative rejected: No cap (risk of overloading Claude's context window) or hard error. Reason: Graceful degradation is better than failure. The note field in the response tells the caller that truncation occurred.

Export produces frozen snapshots

Decision: Exported MCP server projects embed data as JSON and are self-contained. Alternative rejected: Exported projects that reference original data files. Reason: Portability. The exported project should work on any machine without the synthdata plugin or original data files.

No test framework

Decision: Validation is running templates and inspecting output. Alternative rejected: pytest suite with assertions on output distributions. Reason: The output is stochastic (even with fixed seeds, distribution shapes are what matter, not exact values). Template smoke testing catches structural regressions; statistical validation would require tolerance thresholds that add complexity without proportional value at this project's scale.

Plugin + marketplace duality

Decision: Repo serves as both a plugin (plugin.json) and a marketplace (marketplace.json) simultaneously. Alternative rejected: Separate repos for plugin and marketplace. Reason: Single repo is simpler for a single-plugin marketplace. The "source": "./" self-reference in marketplace.json eliminates the need for a nested plugins/ directory.

Security: formula eval with restricted builtins

Decision: Formula columns use eval() with __builtins__ set to empty dict, but random module available. Reason: Schemas are author-controlled (not user-input in the web-security sense). Restricting builtins prevents accidental use of open(), exec(), etc. while allowing mathematical expressions.

Performance characteristics

  • Memory: All tables held in memory simultaneously. Datasets up to ~1M rows are practical; beyond that, memory becomes the bottleneck.
  • Generation time: Dominated by Faker calls (unique values are slowest due to retry loop). Distributions via NumPy are fast.
  • MCP serve: All data loaded at startup. Query latency is pandas DataFrame indexing time — sub-millisecond for most datasets.
  • No streaming: Writers buffer entire DataFrames before writing. No row-by-row streaming.

9. Edge Cases and Error Handling

Schema Validation

  • validate_schema() checks: FK references must contain "." separator; referenced parent table must exist in schema
  • Does NOT validate: column type strings, distribution parameter completeness, weight array lengths
  • Invalid column types pass through to generation time and raise unhandled exceptions

Generator Edge Cases

Scenario Behavior
Parent table with no ID column Auto-adds _row_id column (0-indexed) for FK resolution
Faker unique after 100 retries Accepts the 100th value (collision accepted, not error)
Formula eval exception Returns None for that cell, continues
FK parent not yet generated Raises KeyError (tables must be ordered: parents before children)
Empty template (0 rows at quick) Generates empty DataFrame (valid, 0 rows)
Negative sample_count result Clipped to 0 (no children for that parent)
lam_expr eval failure Falls back to rows_per_parent.get("lam", 1.0)
Missing profile in child weights_by_profile Falls back to default weights
_parse_dt invalid string Tries 4 format strings, then fromisoformat(), then raises ValueError
business-hours 20 retry exhaustion Returns start datetime as fallback

Writer Edge Cases

Scenario Behavior
Excel sheet name > 31 chars Truncated to 31 characters
Excel sheet name with :\/?*[] Replaced with _
Single-table CSV to .csv path Writes single file
Multi-table CSV to .csv path Creates directory, writes per-table files
JSON with numpy types Custom _default serializer handles int64, float64, datetime, Period
SQL NaN values Serialized as NULL
SQL single quotes in strings Escaped by doubling: ' -> ''
Parquet without pyarrow ImportError at write time

Data Loading Edge Cases (shared across extract/extend/anonymize/compute/serve)

Scenario Behavior
XLSX row 1 is title banner Auto-detected (single non-empty cell in row 1); headers read from row 2
XLSX all-empty rows Skipped
XLSX None-valued header cells Excluded from column list
JSON not a dict Raises ValueError
CSV directory with mixed formats Reads *.csv and *.json files
Parquet without pyarrow (serve.py) Prints error message, exits with code 1
Parquet without pyarrow (in directory scan) Silently skips parquet files
Empty dataset (0 tables loaded) Error message and exit

Anonymize Edge Cases

Scenario Behavior
Numeric column flagged by name heuristic Confidence set to 0.0 (excluded)
Column below 0.6 confidence threshold Not anonymized (passed through)
--preserve-joins column in multiple tables Shared cache ensures same real value -> same fake value
--drop + --keep conflict Implementation-specific (last flag wins)

MCP Server Edge Cases

Scenario Behavior
Unknown table name Returns {error: "Table 'x' not found. Available: [...]"}
Unknown column in filter Returns {error: "Unknown column: x. Available: [...]"}
Unknown filter operator Returns {error: "Unknown operator: x. Supported: {...}"}
between with non-2-element list Returns {error: "'between' requires [min, max]"}
Response > 100KB Auto-truncates by halving rows; adds note field
Limit > 500 Capped to 500
sample_rows with n > table size Capped to table size
Unknown tool name Returns {error: "Unknown tool: x"}
Tool handler exception Caught, returned as {error: str(e)}

Extend Edge Cases

Scenario Behavior
ID column with non-standard format Falls back to X-N style IDs
FK column without matching parent table Samples from own unique values
Add column that already exists Error unless --overwrite flag
Seed not provided Derived from current timestamp

10. Implementation Gaps and Opportunities

Test coverage gaps (CRITICAL)

No automated tests exist. There are no test files (test_*.py), no test framework (pytest, unittest), no CI pipeline, and no assertion-based validation anywhere in the codebase. The only testing approach is manual: run templates, inspect output visually.

Gap Location Impact
No unit tests for distributions engine/distributions.py Statistical correctness of sample_column is unverified — a regression in lognormal mu conversion (mu = log(mean) - 0.5*sigma²) would go undetected
No unit tests for FK resolution engine/relationships.py Zipfian weighting, lam_expr evaluation, and effort scaling are untested
No unit tests for profile assignment engine/profiles.py Weight normalization, jitter bounds (±15%), and quick-mode bypass are untested
No schema validation tests engine/schema.py Malformed YAML schemas (missing required fields, invalid types) may produce confusing errors rather than clear validation messages
No PII detection accuracy tests scripts/anonymize.py The 17-entry heuristic table and regex patterns have no precision/recall benchmarks
No MCP protocol integration tests scripts/serve.py Tool handlers are tested manually; no automated test that sends MCP protocol messages and validates responses
No writer round-trip tests engine/writers/ No verification that generate → write → read preserves data fidelity (especially for dates, numpy types, NaN handling)
No template regression tests templates/*.yaml Template changes could silently break downstream expectations (row counts, column names, FK integrity)

Recommendation: A pytest suite with seeded deterministic tests would catch regressions without tolerance-threshold complexity. Key tests: schema validation (valid + invalid), distribution shape (mean within ±10% for 10K samples), FK integrity (all child FKs resolve), writer round-trip (write + read == original).

Security risks

Risk Location Severity Detail
eval() with restricted builtins engine/__init__.py:178 Medium Formula columns use eval(expr, {"__builtins__": {}}, local). While __builtins__ is empty, Python's type hierarchy is still accessible (e.g., ().__class__.__bases__[0].__subclasses__()), allowing sandbox escape. Schemas are author-controlled (not untrusted user input in the web sense), so this is defense-in-depth, not a security boundary.
eval() in FK resolution relationships.py:57 Medium lam_expr uses same restricted-builtins pattern with np (numpy) in context. Same trust boundary as formula columns.
exec() without any restriction compute.py:174 High exec(code, namespace) runs user-provided Python files with full access to the Python runtime. The namespace includes pd, np, and tables, but __builtins__ is NOT restricted. Arbitrary code execution is by design (compute transforms are arbitrary Python), but there's no documentation warning about the trust boundary.
No input validation on MCP filter values serve.py:262-305 Low Filter values from MCP tool calls are passed directly to pandas comparison operators. Pandas handles type mismatches gracefully (raises TypeError caught by the outer handler), so this is not exploitable, but unexpected types could produce confusing errors.
contains filter doesn't escape regex serve.py:162 Low str.contains(val, case=False) treats val as a regex pattern by default. A value like "invalid[" triggers a regex error (caught by the outer handler). Not exploitable but produces confusing error messages.

Recommendation: Document the trust model explicitly: schemas and compute scripts are trusted inputs (same trust level as source code). The eval() sandbox in formula/lam_expr is defense-in-depth, not a security boundary. For compute.py, add a prominent warning in SKILL.md that --code files execute arbitrary Python.

Performance concerns

Concern Location Impact
Faker unique retry loop engine/faker_fields.py faker_value with unique=True retries up to 100 times per value. For large row counts approaching Faker's namespace (e.g., 50K unique emails), this degrades to O(n²) as collisions increase. No warning is emitted when approaching exhaustion.
Per-row eval() in formula columns engine/__init__.py:170-181 Formula evaluation builds a local dict per row and calls eval() per row. For thorough-effort child tables (10K+ rows), this is significantly slower than vectorized pandas operations.
Per-row profile lookup in choice/bool engine/__init__.py weights_by_profile and p_by_profile iterate per-row to find the profile type and select appropriate weights. Could be vectorized with groupby.
All tables in memory All skills Entire dataset held in memory simultaneously. Practical limit ~1M total rows across all tables. No streaming, no chunked processing.
Response truncation loop serve.py _truncate_response halves row count iteratively until under 100KB. For very wide tables, this could loop many times.

Reliability issues

Issue Location Impact
Silent exception swallowing engine/__init__.py:179-180 Formula eval() catches all Exception and returns None. A typo in an expression silently produces null values rather than surfacing the error.
Silent exception swallowing relationships.py:58-59 lam_expr eval failure silently falls back to default lam. The user never learns their expression failed.
No validation of column type strings engine/schema.py validate_schema() checks FK references but not column type values. A typo like type: floatt passes validation and raises ValueError at generation time with a less helpful message.
No pre-validation of distribution names engine/distributions.py Unknown distributions raise ValueError at generation time (line 46), but this isn't caught by validate_schema(). A typo in distribution: normall only surfaces mid-generation, not at schema load.
Unvalidated column type strings at load time engine/__init__.py Unknown column types (e.g., type: floatt) only raise ValueError during generation (line 189). Not caught by validate_schema(), which only checks FK references.
Global Faker seed is a class method engine/faker_fields.py Faker.seed(seed) affects all Faker instances globally. The _unique_cache dict persists across Generator invocations unless get_faker() is called again. Safe for the current single-invocation CLI pattern, but would break if multiple generators ran in the same process.
Distribution parameters unchecked engine/distributions.py No validation that std >= 0, scale > 0, alpha > 0, etc. Negative std passed to np.random.normal produces valid but semantically wrong output.
Missing input file validation Multiple skills Most skills check Path(input).exists() but don't verify the file is readable, non-empty, or the expected format. Bad input produces stack traces rather than user-friendly errors.

Dead code and tech debt

Item Location Impact
Empty examples/ directory synthdata-generate/examples/ Reserved but unused since initial release. Should either be populated or removed.
effort_defaults and post_process in Schema engine/schema.py Marked "reserved for future use" — dead fields that are never read.
apply_profile_overrides() never called engine/profiles.py:85 Defined and imported in engine/__init__.py:18 but never invoked anywhere. Dead code.
Duplicated load_tables() function extend.py, anonymize.py, compute.py, serve.py, export.py ~50 lines duplicated 5 times. Intentional (self-contained skills), but increases maintenance surface when data-loading logic changes (e.g., adding parquet support required updating each copy).
Duplicated JSON serialization helpers serve.py, export.py, compute.py, anonymize.py, extract.py Custom _json_default/_default/json_default functions with near-identical logic for numpy types, datetimes, and Periods.
Cross-skill imports via sys.path manipulation extend.py, anonymize.py Use sys.path.insert to reach generate engine's faker_fields and distributions. Fragile — breaks if directory structure changes.

Missing documentation

Gap Location Impact
No docstrings on engine functions engine/*.py Public functions like sample_column, assign_profiles, resolve_foreign_keys lack docstrings. Parameter semantics require reading implementation.
Compute --code file format undocumented compute.py SKILL.md The SKILL.md describes the namespace (tables, pd, np, result) but doesn't provide a template .py file or document error handling.
Trust model not documented compute.py, engine/__init__.py No explicit warning that --code/--expr execute arbitrary Python, or that formula columns use eval().
Parquet support inconsistencies Various Generate engine has parquet writer; serve.py has parquet reader. But extend.py, anonymize.py, and compute.py don't mention parquet support in their SKILL.md files.
--preserve-joins behavior anonymize.py SKILL.md The flag is listed but its cross-table cache-sharing semantics aren't explained in detail.

Appendix A: YAML Schema Format Reference

Top-level structure

name: <dataset-name>              # Required
description: <string>             # Optional
tables:                           # Required, non-empty array
  - <table-definition>
writers: [xlsx, json]             # Optional, default ["xlsx"]
effort_defaults: {}               # Reserved
post_process: null                # Reserved

Table definition

- name: <table-name>
  rows: { quick: 50, medium: 1000, thorough: 5000 }   # Or single int
  columns:
    - <column-definition>
  profiles:                       # Optional
    - { name: whale, weight: 0.05, overrides: { rate: 24.0 } }
  foreign_key:                    # Present = child table
    column: <fk-column-name>
    references: <parent-table>.<parent-column>
    distribution: uniform         # Or "zipfian"
    alpha: 1.5                    # Zipfian only
  rows_per_parent:                # Child tables only
    distribution: poisson         # Or constant, uniform, normal
    lam: 5                        # Or value, min/max, mean/std
    lam_expr: "order_rate"        # Expression using profile fields
  temporal:                       # Optional timestamp config
    column: ts
    start: "2025-01-01"
    end: "2025-12-31"
    pattern: business-hours       # Or uniform, diurnal
    weekday_only: true

Column types with all parameters

# ID: auto-incrementing string identifiers
- { name: id, type: id, prefix: "U", width: 4, start: 1 }

# Faker: any Faker provider method
- { name: email, type: faker, method: email, unique: true }
- { name: bio, type: faker, method: sentence, args: { nb_words: 12 } }

# Choice: weighted categorical
- { name: dept, type: choice, values: [A, B, C], weights: [0.5, 0.3, 0.2] }
- { name: tier, type: choice, values: [High, Low],
    weights_by_profile: { whale: [0.8, 0.2], dormant: [0.2, 0.8] } }

# Numeric: statistical distribution
- { name: salary, type: float, distribution: lognormal, mean: 75000,
    sigma: 0.4, min: 30000, max: 500000, decimals: 0 }
- { name: count, type: int, distribution: poisson, lam: 5 }

# Boolean: Bernoulli
- { name: active, type: bool, p: 0.92 }
- { name: clicked, type: bool, p: 0.05,
    weights_by_profile: { high_risk: [0.75, 0.25], champion: [0.985, 0.015] } }

# Date/timestamp
- { name: hire_date, type: date, start: "2015-01-01", end: "2025-12-31" }
- { name: ts, type: timestamp, start: "2025-01-01", end: "2025-12-31",
    pattern: diurnal, weekday_only: true }

# Constant: fixed value for all rows
- { name: region, type: constant, value: "NORAM" }

# Formula: Python expression evaluated per row
- { name: total, type: formula, expr: "quantity * unit_price" }

# Ref: copy column from parent table via FK
- { name: customer_name, type: ref, from: name }

Appendix B: Template Catalog

Template Tables Profiles FK distributions Temporal patterns
blank-slate users (1) none none none
hr-directory departments, employees (2) high_performer, at_risk none none
ecommerce-orders customers, products, orders (3) whale, regular, dormant zipfian (orders) diurnal (orders)
saas-metrics accounts, users, events, subscriptions (4) power_user, healthy, at_risk zipfian (events) business-hours (events)
healthcare-patients providers, patients, encounters, claims (4) high_utilizer, chronic, well, rare zipfian (encounters, claims) business-hours (encounters)
financial-transactions customers, accounts, transactions (3) fraud_ring, power_user, mainstream, dormant zipfian (transactions) diurnal (transactions)
security-events users, devices, alerts, incidents (4) high_risk, normal, low_risk zipfian (alerts, incidents) diurnal (alerts)
log-events services, requests, errors (3) none zipfian (both) diurnal (requests), uniform (errors)
iot-sensors devices, readings, events (3) faulty, healthy, retired zipfian (events) uniform (both)
crm-pipeline companies, contacts, deals, activities (4) none zipfian (deals, activities) business-hours (activities)
survey-responses respondents, questions, responses (3) detractor, neutral, promoter none diurnal (respondents)
healthcare-hrm-security users, threat_events, phishing_sims, training, dlp_events, abuse_mailbox (6) high_risk, champion, baseline zipfian (threats, dlp, abuse) business-hours (threats, dlp, abuse)

Appendix C: Writer Output Formats

XLSX

  • Row 1: merged title banner (bold, 14pt, left-aligned). Text: "{dataset_name} — {table_name}"
  • Row 2: column headers (bold, 11pt, blue fill #D9E1F2)
  • Row 3+: data rows
  • One sheet per table (sheet names truncated to 31 chars, invalid chars replaced with _)
  • Internal _row_id columns dropped

CSV

  • Single-table + .csv path: one file
  • Multi-table or directory path: one {table}.csv per table in directory
  • Standard CSV encoding, index=False

JSON

  • .json path: single file with {table_name: [row_objects]} structure
  • Directory path: one {table}.json per table
  • Indent: 2 spaces
  • Custom serializer for dates (ISO), numpy types (.item()), Period (str())

SQL

  • Single .sql file with all tables
  • Per table: CREATE TABLE IF NOT EXISTS "{name}" (col_defs); then INSERT INTO per row
  • Type mapping: int->INTEGER, float->REAL, bool->BOOLEAN, datetime->TIMESTAMP, else TEXT
  • Values: NULL for None/NaN, numeric as literals, strings single-quoted (escaped by doubling)
  • Column/table names double-quoted

Parquet

  • Directory output: one {table}.parquet per table
  • index=False
  • Requires pyarrow at runtime