Add Cboe NSE and Deribit public research sources

docs/free-derivatives.md

@@ -0,0 +1,63 @@
+# Public derivative research sources
+
+`qdk-derivatives free-fetch` adds bounded official-source ingestion. No API key,
+broker connection or live order is involved. A successful request publishes an
+immutable contracts/quotes/manifest bundle with source URLs, response hashes,
+normalization assumptions, excluded records and allowed research modes.
+
+| Provider | Explicit selection | Supported data | Boundary |
+| --- | --- | --- | --- |
+| `cboe-vix` | Actual VX expiry dates, comma separated | Cboe per-contract daily OHLC, settlement, volume and OI | VX futures only; select actual expiry, including holiday adjustments |
+| `nse-fo` | `FinInstrmNm`, e.g. `NIFTY25JAN23500PE` | NSE UDiFF daily index futures/options | Historical `NewBrdLotQty` and actual expiry come from each row; stock/physical delivery unsupported |
+| `deribit` | BTC/ETH option names | `trades`: historical public trades aggregated daily; `snapshot`: current vendor marks | USD analytical projection of native coin premiums; **analysis only** |
+
+Example, run from an installed locked environment:
+
+```powershell
+qdk-derivatives free-fetch --provider cboe-vix --kind future --selection 2025-02-19,2025-03-18 --start 2025-01-08 --end 2025-01-10 --output D:/research/vx-v1 --rights-note "Personal local research; no raw redistribution"
+qdk-derivatives free-fetch --provider nse-fo --kind option --selection NIFTY25JAN23500PE --start 2025-01-08 --end 2025-01-10 --output D:/research/nse-v1 --rights-note "Personal local research; no raw redistribution"
+qdk-derivatives free-fetch --provider deribit --kind option --selection BTC-31JAN25-100000-C --source-mode trades --start 2025-01-08 --end 2025-01-10 --output D:/research/btc-v1 --rights-note "Personal local research; no raw redistribution"
+```
+
+Each request covers 1–32 inclusive dates and explicit contracts (20 maximum;
+Deribit 10). A snapshot requires today's UTC date and unexpired options. The
+source's daily session is retained in the normalized quote; date requests are
+calendar bounds, not a claim of complete exchange-calendar coverage.
+
+Daily zero-volume records are disclosed and excluded, not replaced by fabricated
+OHLC. Missing NSE HTTP 404 dates are disclosed without calling them holidays.
+Access denials stop the request without bypass or automatic retry. No incomplete
+download is published. Deribit pagination is capped at 20 pages per instrument;
+sequence gaps, overlaps or an ambiguous boundary fail closed. Same-millisecond
+trades are ordered by their official sequence number.
+
+For official files downloaded by the user, add `--files-directory PATH`. Use the
+original `VX_YYYY-MM-DD.csv` or
+`BhavCopy_NSE_FO_0_0_0_YYYYMMDD_F_0000.csv.zip` names. Files are read locally;
+untrusted ZIP paths and oversized decompression are rejected.
+
+Research defaults are analysis only. Cboe/NSE may enable modeled replay with both
+`--initial-margin` and `--maintenance-margin` (per contract, contract currency).
+These and `--tick` are **explicit research assumptions**, not historical exchange
+margin certification. Samples must end before expiry; no final settlement is
+inferred. Publication times and first-observed listing times in daily files are
+conservative retrospective assumptions, not point-in-time evidence.
+
+Deribit trade premiums use each trade's USD index. Snapshot marks use one shared
+index fixing within a bounded capture interval; the source timestamps are kept.
+These are calculation snapshots, not synchronized executable quotes. Native coin
+settlement, inverse payoff, collateral and variable tick rules are not modeled
+by the USD ledger. Bundle validation and the research/replay layers must enforce
+`allowed_modes`. Multi-leg historical analysis also requires synchronized last
+observations; using different trades as if simultaneous is rejected.
+
+Public access does not grant redistribution rights. Raw downloaded responses and
+customer data stay outside Git. Unit tests use synthetic official-shaped records.
+
+Official references:
+
+- [Cboe historical files](https://www.cboe.com/markets/us/futures/market-statistics/historical-data/futures)
+- [VX specifications](https://www.cboe.com/tradable-products/vix/vix-futures/specifications)
+- [NSE derivative reports](https://www.nseindia.com/all-reports-derivatives)
+- [NSE index contract specifications](https://www.nseindia.com/static/products-services/equity-derivatives-contract-specifications)
+- [Deribit public API](https://docs.deribit.com/)

src/quant_data_kit/derivatives/bundle.py

@@ -84,6 +84,21 @@ class DerivativeBundle:
     quotes: tuple[Quote, ...]
     identity: str
 
+    @property
+    def allowed_modes(self):
+        source = self.manifest.get("source", {})
+        return (
+            source.get("allowed_modes", ["analysis", "replay"])
+            if isinstance(source, dict)
+            else ["analysis", "replay"]
+        )
+
+    def require_mode(self, mode):
+        if mode not in self.allowed_modes:
+            raise ValueError(
+                f"input permits {self.allowed_modes}; {mode} is blocked by source usage restrictions"
+            )
+
     def contract(self, instrument_id):
         try:
             return next(c for c in self.contracts if c.instrument_id == instrument_id)
@@ -123,6 +138,7 @@ def summary(self):
             "venues": sorted({c.venue for c in self.contracts}),
             "input_sha256": self.identity,
             "limits": self.manifest["limits"],
+            "allowed_modes": self.allowed_modes,
         }
 
 
@@ -144,6 +160,25 @@ def load_bundle(root):
         if not isinstance(manifest.get(name), str) or not manifest[name].strip():
             raise ValueError(f"{name} must be declared")
     utc(manifest["captured_at"])
+    source = manifest.get("source", {})
+    if manifest["provider"] in {"cboe-vix", "nse-fo", "deribit"}:
+        if (
+            not isinstance(source, dict)
+            or source.get("schema") != "qdk.free-derivatives/v1"
+            or "allowed_modes" not in source
+        ):
+            raise ValueError("public derivative input must declare its usage restrictions")
+        if manifest["provider"] == "deribit" and source["allowed_modes"] != ["analysis"]:
+            raise ValueError("Deribit USD projections are analysis-only")
+    if isinstance(source, dict) and "allowed_modes" in source:
+        modes = source["allowed_modes"]
+        if (
+            not isinstance(modes, list)
+            or not modes
+            or any(m not in {"analysis", "replay"} for m in modes)
+            or len(set(modes)) != len(modes)
+        ):
+            raise ValueError("invalid input usage restrictions")
     files = manifest.get("files", {})
     if set(files) != {"contracts.json", "quotes.csv"}:
         raise ValueError("unexpected bundle file set")

src/quant_data_kit/derivatives/cli.py

@@ -27,6 +27,30 @@ def main(argv=None):
     fetch.add_argument("--end", required=True)
     fetch.add_argument("--dataset")
     fetch.add_argument("--max-cost-usd")
+    free = sub.add_parser("free-fetch", help="Cboe VX, NSE index F&O, Deribit coin-option analysis")
+    free.add_argument("--provider", choices=["cboe-vix", "nse-fo", "deribit"], required=True)
+    free.add_argument("--kind", choices=["future", "option"], required=True)
+    free.add_argument(
+        "--selection",
+        required=True,
+        help="comma-separated VX expiry dates, NSE FinInstrmNm or Deribit symbols",
+    )
+    free.add_argument("--start", required=True)
+    free.add_argument("--end", required=True)
+    free.add_argument("--source-mode", choices=["daily", "trades", "snapshot"], default="daily")
+    free.add_argument(
+        "--tick",
+        default="0.05",
+        help="explicit research tick assumption, not historical rule certification",
+    )
+    free.add_argument("--initial-margin")
+    free.add_argument("--maintenance-margin")
+    free.add_argument(
+        "--files-directory",
+        help="offline official files; expected exact filenames appear in documentation/receipts",
+    )
+    free.add_argument("--output", required=True)
+    free.add_argument("--rights-note", required=True)
     for p in (imp, fetch):
         p.add_argument("--contracts", required=True, help="explicit contract rules JSON")
         p.add_argument("--output", required=True)
@@ -37,6 +61,13 @@ def main(argv=None):
             result = load_bundle(args.bundle)
         elif args.command == "demo":
             result = write_demo(args.output, args.kind)
+        elif args.command == "free-fetch":
+            from .free_sources import collect_free
+
+            options = vars(args).copy()
+            options.pop("command")
+            output = options.pop("output")
+            result = collect_free(output, **options)
         else:
             contracts = [
                 record(Contract, c)
@@ -67,7 +98,7 @@ def main(argv=None):
                     rights_note=args.rights_note,
                     **extra,
                 )
-        print(json.dumps(result.summary(), ensure_ascii=False, indent=2))
+        print(json.dumps(result.summary(), ensure_ascii=True, indent=2))
         return 0
     except (ValueError, OSError, KeyError, TypeError) as exc:
         parser.exit(2, f"derivative input rejected: {exc}\n")

src/quant_data_kit/derivatives/free_common.py

@@ -0,0 +1,160 @@
+"""Bounded public-file/API ingestion; never bypass a source access rejection."""
+
+import csv
+import io
+import json
+import re
+from datetime import date, datetime, time, timedelta, timezone
+from pathlib import Path
+from urllib.error import HTTPError, URLError
+from urllib.request import Request, urlopen
+from zoneinfo import ZoneInfo
+
+from .bundle import digest
+from .models import decimal
+
+MAX_BYTES = 32 * 1024 * 1024
+PROVIDERS = {"cboe-vix", "nse-fo", "deribit"}
+
+
+class PublicSourceError(ValueError):
+    def __init__(self, status):
+        self.status = status
+        super().__init__(f"public source HTTP {status}; use official files if access is restricted")
+
+
+def public_request(url):
+    try:
+        req = Request(url, headers={"User-Agent": "QuantResearch/1.0 (read-only research)"})
+        with urlopen(req, timeout=30) as response:
+            raw = response.read(MAX_BYTES + 1)
+    except HTTPError as exc:
+        raise PublicSourceError(exc.code) from None
+    except (URLError, OSError, TimeoutError):
+        raise ValueError(
+            "public data source unavailable; no automatic retry or partial publication"
+        ) from None
+    if len(raw) > MAX_BYTES:
+        raise ValueError("public response exceeds 32 MiB; narrow the request")
+    return raw
+
+
+def selections(value):
+    values = value.split(",") if isinstance(value, str) else list(value)
+    values = [v.strip() for v in values]
+    if not 1 <= len(values) <= 20 or len(set(values)) != len(values):
+        raise ValueError("select 1-20 distinct contracts, separated by commas")
+    if any(not re.fullmatch(r"[A-Za-z0-9_.-]{1,80}", v) for v in values):
+        raise ValueError("invalid contract selection")
+    return values
+
+
+def validate_request(provider, kind, selection, start, end, source_mode="daily"):
+    if provider not in PROVIDERS or kind not in {"future", "option"}:
+        raise ValueError("unsupported free source or instrument kind")
+    values = selections(selection)
+    first, last = date.fromisoformat(start), date.fromisoformat(end)
+    if not 0 <= (last - first).days <= 31 or last > datetime.now(timezone.utc).date():
+        raise ValueError("select 1-32 inclusive completed/current UTC dates")
+    if provider == "cboe-vix":
+        if kind != "future" or source_mode != "daily":
+            raise ValueError("Cboe adapter supports VX futures daily contract files")
+        for value in values:
+            expiry = date.fromisoformat(value)
+            if last >= expiry:
+                raise ValueError(
+                    "Cboe sample must end before the selected expiry; final fixing is separate"
+                )
+    elif provider == "nse-fo" and source_mode != "daily":
+        raise ValueError("NSE adapter supports daily UDiFF files")
+    elif provider == "deribit":
+        if kind != "option" or source_mode not in {"trades", "snapshot"}:
+            raise ValueError("Deribit supports BTC/ETH option trades or analytical snapshots")
+        if len(values) > 10:
+            raise ValueError("Deribit collection is bounded to ten explicit option contracts")
+        if any(
+            not re.fullmatch(r"(?:BTC|ETH)-\d{1,2}[A-Z]{3}\d{2}-\d+(?:d\d+)?-[CP]", v)
+            for v in values
+        ):
+            raise ValueError("select explicit BTC/ETH coin-quoted vanilla option symbols")
+        if len({v.split("-")[0] for v in values}) != 1:
+            raise ValueError("one underlying currency per Deribit request")
+        if source_mode == "snapshot" and (
+            first != last or last != datetime.now(timezone.utc).date()
+        ):
+            raise ValueError("a current snapshot requires today's UTC date; it cannot be backdated")
+    return values, first, last
+
+
+def csv_rows(raw, required):
+    if len(raw) > MAX_BYTES:
+        raise ValueError("CSV exceeds 32 MiB")
+    reader = csv.DictReader(io.StringIO(raw.decode("utf-8-sig"), newline=""))
+    if (
+        not reader.fieldnames
+        or len(set(reader.fieldnames)) != len(reader.fieldnames)
+        or not set(required) <= set(reader.fieldnames)
+    ):
+        raise ValueError("public CSV schema changed or required fields are absent")
+    rows = list(reader)
+    if any(None in row for row in rows):
+        raise ValueError("malformed public CSV record")
+    return rows
+
+
+def daily_time(day, zone, hour, minute=0):
+    at = datetime.combine(day, time(hour, minute), ZoneInfo(zone)).astimezone(timezone.utc)
+    # Explicit conservative research assumption, not historical publication evidence.
+    available = datetime.combine(day + timedelta(days=1), time(9), ZoneInfo(zone)).astimezone(
+        timezone.utc
+    )
+    return at, available
+
+
+def margin_values(initial=None, maintenance=None):
+    if initial is None and maintenance is None:
+        return None, None
+    if initial is None or maintenance is None:
+        raise ValueError("declare both research margin amounts or neither")
+    initial, maintenance = decimal(initial), decimal(maintenance)
+    if not 0 <= maintenance <= initial or initial <= 0:
+        raise ValueError("margin requires 0 <= maintenance <= positive initial")
+    return initial, maintenance
+
+
+class Resources:
+    def __init__(self, *, fetch=public_request, directory=None):
+        self.fetch = fetch
+        self.directory = Path(directory).resolve() if directory else None
+        self.receipts = []
+
+    def read(self, filename, url):
+        if self.directory:
+            path = (self.directory / filename).resolve()
+            if (
+                path.parent != self.directory
+                or not path.is_file()
+                or path.stat().st_size > MAX_BYTES
+            ):
+                raise ValueError(f"missing, escaped or oversized official source file: {filename}")
+            raw = path.read_bytes()
+        else:
+            raw = self.fetch(url)
+        if len(raw) > MAX_BYTES:
+            raise ValueError("public source response exceeds 32 MiB")
+        self.receipts.append(
+            {
+                "file": filename,
+                "url": url,
+                "bytes": len(raw),
+                "sha256": digest(raw),
+                "transport": "local-file" if self.directory else "https",
+            }
+        )
+        return raw
+
+    def rpc(self, filename, url):
+        data = json.loads(self.read(filename, url))
+        if not isinstance(data, dict) or "error" in data or "result" not in data:
+            raise ValueError("public API returned a JSON-RPC error or changed schema")
+        return data["result"]