Skip to content

DATA_CONTRACT.md

Latest commit

 

History

History
96 lines (71 loc) · 3.99 KB

DATA_CONTRACT.md

File metadata and controls

96 lines (71 loc) · 3.99 KB
Raw

BYO Data Contract

Penrose can read local, pre-collected data through PENROSE_DATA_DIR. Point the variable at a directory containing loader.py; Penrose imports that module and calls a small catalog surface:

load_series(name: str) -> tuple[pandas.Series, str] | None
available() -> list[str]
domain_of(name: str) -> str | None
domains() -> list[str]
describe(name: str) -> dict
describe_brief(name: str) -> str

The typed reference is penrose.data.loader_protocol.CatalogLoaderProtocol.

When PENROSE_DATA_DIR is unset, the catalog directory defaults to the repository-local data/. Run make init to create a valid empty data/catalog.yaml, then follow data/README.md to populate it. For one migration release, Penrose warns and uses the old sibling ../penrose-data only when the repo-local catalog is absent or invalid and the sibling has a valid catalog. Set PENROSE_DATA_DIR or move that catalog into data/; the sibling fallback will be removed in the next breaking release.

When PENROSE_DATA_DIR is unset, points at the wrong directory, or a catalog-only series cannot be delivered, Penrose marks that series Unavailable and dependent claims route to needs_data. Synthetic substitution is opt-in for calibration/dev runs only:

PENROSE_ALLOW_SYNTHETIC=1

Leave PENROSE_ALLOW_SYNTHETIC unset or 0 for real adjudication.

Series Shape

load_series() returns (series, provenance) or None if the series cannot be delivered. The series must be a scalar daily time series:

  • pandas.Series
  • DatetimeIndex, UTC daily timestamps
  • float values
  • sorted, deterministic, point-in-time

If a returned index is timezone-naive, Penrose localizes it to UTC before placing it in a penrose.data.contract.Series. Return None for missing files, missing columns, empty data, or unsupported names. Do not fabricate fallback values in a loader.

There is no OHLC or bar primitive in this catalog seam. OHLC is four scalar daily series by convention: asset_open, asset_high, asset_low, and asset_close.

Cleaned-data vintage & the sentry loader

Vendor bad prints, including zero or near-zero futures prices, must be repaired before serving as a versioned, content-addressed derived cleaned artifact. Preserve and hash the raw bytes, write and hash the cleaned bytes actually served, and record the deterministic repair in a manifest. The repair must be causal: it may use only information available at or before the repaired observation, never future data.

The serve-time loader is a non-mutating sentry. It validates the cleaned artifact and flags residual anomalies loudly; it never repairs, substitutes, clips, or fabricates values while loading. This reconciles the no-fallback rule above with data cleaning: a repair belongs in a new derived vintage, not in the loader. A loader that silently repairs is a loader that disagrees with its own hash, because the catalog's artifact_sha256 would no longer describe the bytes actually served.

Intraday Data

Collapse intraday data to daily before returning it from load_series(), and declare the aggregation in catalog metadata. Supported aggregation names for scalar series should be explicit:

  • last
  • sum
  • mean
  • count
  • first
  • max
  • min

For OHLC input, use the usual daily collapse: open = first, high = max, low = min, close = last. Penrose also exposes penrose.data.granularity.resample_ohlc(frame) for that generic conversion.

Minimal Example

The reference implementation is in examples/reference_loader/:

PENROSE_DATA_DIR=examples/reference_loader penrose run --claims claims.json

It contains:

  • loader.py implementing CatalogLoaderProtocol
  • catalog.yaml with names, domains, units, provenance, aggregation, and descriptions
  • tiny CSV files under data/

The sample catalog publishes equity_spy_close and btc_spot_close as daily scalar close series. Use it as a template for private catalogs; keep private data outside the repository and point PENROSE_DATA_DIR at that directory.