TASK_BOARD.md

TASK_BOARD.md — Volatility Relative Value Toolkit

Companion task board for PROJECT_SPEC.md
Purpose: break each stage into small, agent-executable tasks with dependencies, outputs, and checks.

---

How to use this board

• Work top-down by Stage.
• Complete one Stage at a time unless explicitly approved to parallelize.
• For each task:
  • Mark status ([ ] -> [x])
  • Record file changes
  • Run listed checks
  • Save outputs to outputs/
• If blocked, document blocker and downgrade to MVP per PROJECT_SPEC.md priority rules.

Status Legend

• [ ] Not started
• [-] In progress
• [x] Done
• [!] Blocked
• [~] Partial / MVP only

---

Stage 1 — Repo Scaffold & Reproducible Environment

Goal

Create runnable project scaffold, dependency management, Makefile, and test harness.

Exit Criteria (DoD)

• make test passes
• make reproduce runs successfully (stub allowed)
• README includes project goal + stage roadmap
• Modules import cleanly

S1.1 Create repository skeleton

• Create directories: data_pipeline/, signals/, backtest/, risk/, report/, config/, tests/, outputs/, notebooks/
• Add __init__.py files to Python packages
• Add .gitignore for Python/data outputs
• Add outputs/.gitkeep (optional)

Outputs

• Repo directory structure exists

Checks

• find . -maxdepth 2 -type d shows required folders
• python -c "import data_pipeline, signals, backtest, risk, report"

---

S1.2 Set up Python project metadata

• Add pyproject.toml (preferred) or requirements.txt
• Pin core dependencies (pandas, numpy, pyarrow, pytest, pyyaml, scikit-learn, matplotlib, etc.)
• Add dev dependencies (ruff/black optional but recommended)

Outputs

• pyproject.toml or requirements*.txt

Checks

• [~] Fresh env install succeeds
• python -c "import pandas, numpy, pyarrow, pytest"

---

S1.3 Create Makefile commands (MVP)

• make setup
• make test
• make lint (can be no-op initially)
• make clean
• make reproduce (stub pipeline)

Outputs

• Makefile

Checks

• make test returns 0
• make reproduce returns 0 and prints stage placeholders

---

S1.4 Add config placeholders

• Create config/data.yaml
• Create config/signals.yaml
• Create config/backtest.yaml
• Create config/risk.yaml
• Create config/report.yaml

Outputs

• YAML placeholder configs with comments

Checks

• YAML files parse: python -c "import yaml,glob; [yaml.safe_load(open(f)) for f in glob.glob('config/*.yaml')]"

---

S1.5 Add tests + smoke test

• Create tests/test_smoke.py
• Add at least 1 passing test
• Add test command in Makefile

Outputs

• tests/test_smoke.py

Checks

• pytest -q

---

S1.6 README initial version

• Project summary (RV toolkit, not prediction)
• Stage roadmap
• Local setup instructions (Mac M2)
• How to run make test / make reproduce

Outputs

• README.md initial draft

Checks

• README includes exact command examples
• [~] New user can follow setup steps end-to-end

---

Stage 2 — Data Pipeline v1 (Load + Standardize)

Goal

Build a minimal data pipeline that ingests raw data and produces standardized parquet outputs.

Exit Criteria (DoD)

• Standardized parquet snapshot generated
• Coverage/metadata report produced
• Schema consistent and basic sanity checks performed

S2.1 Define data schema contract

• Create data_pipeline/schema.py (or constants module)
• Define required standardized columns
• Document dtypes and nullable rules
• Define validation helper (validate_standardized_schema(df))

Outputs

• Schema contract module

Checks

• Validation helper passes on toy DataFrame
• Fails on missing required column

---

S2.2 Implement raw loaders (MVP)

• Create data_pipeline/loaders/ package
• Implement CSV loader utility
• Implement optional yfinance loader stub (or real if available)
• Add source config parsing from config/data.yaml

Outputs

• Raw loader modules
• Optional sample raw data fixtures

Checks

• Loader returns DataFrame with source-native columns
• Error handling for missing file/path

---

S2.3 Implement standardization transforms

• Create data_pipeline/standardize/ package
• Map source columns -> canonical columns
• Normalize dates / timezone handling
• Cast dtypes
• Add source and asof_timestamp

Outputs

• Standardization functions

Checks

• Standardized output matches schema contract
• Dates sortable and parse correctly

---

S2.4 Build pipeline entrypoint

• Implement data_pipeline/build_dataset.py
• Read config and run loaders + standardizers
• Write outputs/data/raw/*.parquet
• Write outputs/data/standardized/*.parquet

Outputs

• Raw and standardized parquet files

Checks

• Script runs from CLI
• Output folders created automatically
• Parquet files readable

---

S2.5 Metadata / coverage reporting

• Generate outputs/data/metadata/source_summary.json
• Include rows, symbols, date ranges, missing counts
• Add duplicate key check on (date, symbol) and report counts

Outputs

• Source summary JSON
• Duplicate report (JSON/CSV/parquet acceptable)

Checks

• Metadata file exists and valid JSON
• Duplicate detection runs even when no duplicates

---

S2.6 Data sanity checks (MVP)

• Negative price check
• Zero-heavy field check (warning only)
• Null rate summary by column

Outputs

• Sanity check logs / summary report

Checks

• Pipeline does not silently ignore anomalies
• Warnings are visible and persisted

---

Stage 3 — Data QA / Calendar Alignment / Roll Rules

Goal

Add auditable QA, trading calendar alignment, and VX roll rule logic.

Exit Criteria (DoD)

• QA reports generated
• Continuous/rolled series produced
• Roll log generated and inspectable
• No lookahead in roll decision logic

S3.1 Implement trading calendar alignment

• Create data_pipeline/calendars/ utilities
• Define target trading calendar (config-driven)
• Align per-symbol data to calendar
• Mark is_market_closed vs is_data_missing (if source allows; otherwise placeholder flags)

Outputs

• Aligned datasets
• Missing classification flags

Checks

• Aligned index is monotonic and calendar-consistent
• Missing flags populated (or explicitly unavailable)

---

S3.2 Missing data handling (configurable)

• Implement fill rules for price fields (limited, config-based)
• Prevent blind fill for volume by default
• Add fill audit columns/logs (is_filled_close, etc.)

Outputs

• Cleaned dataset with fill markers
• Fill audit log/report

Checks

• Fill actions counted and reported
• Volume untouched unless config explicitly enables

---

S3.3 Outlier detection

• Implement z-score or MAD-based outlier detection (MVP)
• Output outlier records without deleting by default
• Add optional config for handling policy (mark-only/drop/winsorize later)

Outputs

• outputs/data/qa/outlier_report.parquet

Checks

• Outlier report produced even if empty
• No silent deletion in MVP

---

S3.4 Implement VX roll rule engine

• Create data_pipeline/rolls/
• Implement rule: roll before expiry by configurable N trading days (MVP)
• Produce selected active contract series
• Generate roll_log with date/from_contract/to_contract/reason

Outputs

• Continuous/active-contract dataset
• outputs/data/qa/roll_log.parquet

Checks

• Roll log non-empty when multiple contracts exist
• No contract overlap inconsistency
• Roll decision uses only same-day available metadata

---

S3.5 Build QA reports

• Generate qa_report.json (missing/duplicates/outliers summary)
• Generate missing_report.parquet
• Write clean and continuous outputs

Outputs

• outputs/data/qa/qa_report.json
• outputs/data/qa/missing_report.parquet
• outputs/data/clean/*.parquet
• outputs/data/continuous/*.parquet

Checks

• All expected files exist
• QA report readable and references generated artifacts

---

S3.6 Add tests for calendar + roll logic

• Unit test: alignment preserves chronological order
• Unit test: roll rule triggers on expected dates
• Unit test: no lookahead in roll selection

Outputs

• tests/test_roll_rules.py
• tests/test_calendar_alignment.py

Checks

• pytest -q tests/test_roll_rules.py
• Edge-case fixtures included

---

Stage 4 — Signals (RV Logic, Not Prediction)

Goal

Implement interpretable volatility RV signals and unified signal outputs.

Exit Criteria (DoD)

• Signals parquet generated
• Diagnostics summary generated
• Signal interfaces standardized
• No lookahead in feature computation

S4.1 Define signal interface contract

• Create signals/base.py (or equivalent)
• Define expected input/output columns
• Establish naming conventions (signal_*, z_*)
• Add helper for lagging/anti-leakage (apply_signal_lag)

Outputs

• Signal interface utilities

Checks

• Two toy signals conform to same schema

---

S4.2 Implement term structure slope signal

• signals/term_structure.py::compute_slope
• Configurable contract pairs / maturities
• Optional rolling z-score normalization

Outputs

• Slope signal columns

Checks

• Values generated for valid dates
• Missing values only where expected (insufficient inputs)

---

S4.3 Implement curvature signal

• signals/term_structure.py::compute_curvature
• Define 3-point curve formula
• Optional normalization

Outputs

• Curvature signal columns

Checks

• Formula documented in code comments/docstring
• Curvature computed only when required tenors available

---

S4.4 Implement carry / roll-down proxy

• signals/carry_roll.py
• Define carry proxy and roll-down approximation
• Make assumptions configurable and documented

Outputs

• Carry / roll-down signal columns

Checks

• Signal names and units documented
• No forward-looking fields used

---

S4.5 Implement VRP proxy (IV - RV)

• signals/vrp_proxy.py
• IV proxy from VIX (or configured source)
• RV proxy from realized vol window (configurable)
• Annualization convention documented

Outputs

• VRP proxy column(s)

Checks

• Rolling window shift/lag is explicit
• Initial NaNs are expected and reported

---

S4.6 Implement PCA factors

• signals/pca_factors.py
• Fit PCA on term structure matrix
• Output factor scores + explained variance ratios
• Save loadings metadata/artifact

Outputs

• PCA factor columns
• PCA diagnostics metadata

Checks

• Explained variance ratio sums to <= 1
• Loadings dimensions correct

---

S4.7 Signal orchestration + diagnostics

• signals/signal_registry.py or pipeline entrypoint
• Produce outputs/data/signals.parquet
• Produce outputs/data/signal_diagnostics.json (stats summary, missing rates)
• Ensure downstream-compatible field names

Outputs

• Signals parquet
• Diagnostics JSON

Checks

• Each configured signal appears in output
• Diagnostics include mean/std/min/max/missing per signal
• Backtest module can read output without renaming

---

S4.8 Signal tests

• Unit tests for slope/curvature formulas
• Unit tests for VRP lagging/no-leakage
• Unit tests for PCA output dimensions

Outputs

• tests/test_signals.py

Checks

• pytest -q tests/test_signals.py

---

Stage 5 — Backtest Engine (Execution + Costs + Roll + Attribution)

Goal

Implement configurable backtest engine with explicit execution assumptions and PnL attribution.

Exit Criteria (DoD)

• Trades, positions, pnl, attribution files generated
• Cost assumptions configurable and visible
• Roll execution handled
• Basic accounting consistency checks pass

S5.1 Define backtest data contract

• Create backtest/contracts.py (optional)
• Define schemas for trades / positions / pnl / attribution
• Define required signal input fields

Outputs

• Backtest schema contract

Checks

• Validation helpers pass/fail correctly on toy data

---

S5.2 Implement execution timing model

• backtest/execution.py
• Make signal timestamp vs execution timestamp explicit
• MVP execution price rule (e.g., next-period close/open proxy)
• Anti-lookahead assertions

Outputs

• Execution model functions
• Timing assumptions documented in code + config

Checks

• Unit test fails if same-bar future data is used
• Execution lag configurable

---

S5.3 Implement position sizing & constraints

• backtest/positioning.py
• Signal -> target position mapping
• Position cap
• Optional leverage cap
• Risk targeting (target vol, MVP acceptable)

Outputs

• Target and realized position series

Checks

• Positions respect caps
• Risk targeting can be disabled/enabled by config

---

S5.4 Implement transaction costs & slippage (MVP)

• Fixed bps cost model
• Slippage proxy (optional simple model)
• Separate roll costs vs regular trade costs

Outputs

• Cost columns in trades/pnl

Checks

• Zero-cost config changes net PnL materially (vs non-zero config)
• Costs never silently omitted

---

S5.5 Implement roll-aware trade generation

• Integrate Stage 3 roll outputs
• Generate roll trades on roll dates
• Carry forward positions across contracts correctly

Outputs

• Roll events reflected in trades/positions

Checks

• Roll dates in trades match roll_log
• No “ghost positions” in expired contracts

---

S5.6 Implement PnL attribution (MVP + residual)

• backtest/attribution.py
• Attribute PnL into carry/roll, spot-curve move, costs, residual
• Add convexity_proxy_pnl placeholder column if full implementation deferred

Outputs

• outputs/backtests/attribution.parquet

Checks

• PnL_total approximately equals sum of attribution components
• Residual is reported (not hidden)

---

S5.7 Orchestrate backtest run

• backtest/engine.py
• Read config, signals, continuous data
• Write trades/positions/pnl/attribution/summary outputs
• Include parameter snapshot in summary

Outputs

• outputs/backtests/trades.parquet
• outputs/backtests/positions.parquet
• outputs/backtests/pnl.parquet
• outputs/backtests/attribution.parquet
• outputs/backtests/summary.json

Checks

• All files created
• Summary includes turnover/hit-rate/sharpe placeholders or computed values

---

S5.8 Backtest tests

• Unit test for position/trade accounting consistency
• Unit test for cost application
• Unit test for no-lookahead execution timing
• Unit test for attribution summation identity

Outputs

• tests/test_backtest.py

Checks

• pytest -q tests/test_backtest.py

---

Stage 6 — Risk Analytics (VaR/CVaR/Exposure/Stress)

Goal

Generate risk metrics and regime/stress diagnostics from backtest outputs.

Exit Criteria (DoD)

• Risk metrics JSON generated
• Stress report generated
• Exposure series generated and aligned by date

S6.1 Implement drawdown analytics

• risk/drawdown.py
• MaxDD, drawdown series, duration, recovery time (MVP)
• Utility to attach to report pipeline

Outputs

• Drawdown metrics and series

Checks

• MaxDD consistent with equity curve in toy test

---

S6.2 Implement VaR/CVaR (historical)

• risk/var_cvar.py
• 95% and/or 99% historical VaR/CVaR
• Configurable horizon (MVP = 1 day)

Outputs

• VaR/CVaR metrics for risk report

Checks

• CVaR magnitude >= VaR magnitude (loss convention documented)

---

S6.3 Implement exposures (proxy-based)

• risk/exposures.py
• Beta proxy (e.g., to SPX/returns)
• Vega proxy (e.g., sensitivity to VIX level moves)
• Gamma proxy (nonlinear proxy or placeholder with documented formula)

Outputs

• outputs/backtests/exposures.parquet

Checks

• Exposure series indexed by date
• Column names/units documented

---

S6.4 Implement stress/regime analysis

• risk/stress.py
• Predefined windows from config (e.g., crisis windows)
• Performance/risk stats by window
• Optional rolling regime summaries

Outputs

• outputs/backtests/stress_report.parquet

Checks

• Stress report generated even for partial overlap windows
• Window labels preserved

---

S6.5 Orchestrate risk pipeline

• Risk entrypoint (risk/run_risk.py or module function)
• Read pnl/positions inputs
• Write risk_metrics.json, stress_report.parquet, exposures.parquet

Outputs

• Risk artifacts

Checks

• Artifacts exist
• JSON is valid and includes core metrics

---

S6.6 Risk tests

• Unit test for drawdown
• Unit test for VaR/CVaR ordering
• Unit test for stress window slicing

Outputs

• tests/test_risk.py

Checks

• pytest -q tests/test_risk.py

---

Stage 7 — Report Generator (HTML/PDF Dashboard)

Goal

Auto-generate HTML/PDF dashboard after pipeline runs, including assumptions and key metrics.

Exit Criteria (DoD)

• HTML report generated
• PDF report generated if environment supports it (else documented fallback)
• Required content sections included

S7.1 Define report data model

• Enumerate required inputs from Stage 3/5/6
• Build report context assembler
• Add config version/sample window metadata

Outputs

• Report context object/dict builder

Checks

• Missing required artifact yields actionable error

---

S7.2 Implement metrics tables and summaries

• Compute/display Sharpe, MaxDD, turnover, hit-rate
• Include PnL attribution summary table
• Include QA summary and backtest assumptions section

Outputs

• Report tables data structures

Checks

• Required metrics present (placeholder allowed if documented)

---

S7.3 Generate plots

• Equity curve
• Drawdown curve
• Attribution chart
• Exposure chart(s)
• Optional signal diagnostics plots

Outputs

• Plot images or embedded charts in report build directory

Checks

• Titles, axes, units, sample dates visible
• Plots render without manual notebook steps

---

S7.4 Build HTML report

• report/dashboard.py + template(s)
• Render outputs/reports/latest_report.html
• Include links/paths to generated artifacts if helpful

Outputs

• HTML report

Checks

• HTML opens locally and sections are present
• No broken image references

---

S7.5 Build PDF export (best effort)

• Implement PDF export path (WeasyPrint / wkhtmltopdf / reportlab fallback)
• If unsupported, document fallback and still pass Stage with HTML-only + logged limitation

Outputs

• outputs/reports/latest_report.pdf (or documented fallback)

Checks

• PDF file exists OR limitation is clearly recorded in logs/readme

---

S7.6 Report tests / content validation

• Add content validation checks (required headings/sections in HTML)
• Smoke test for report generation with minimal artifacts

Outputs

• tests/test_report.py

Checks

• pytest -q tests/test_report.py

---

Stage 8 — Reproducibility Hardening (make reproduce)

Goal

Wire all stages into one reproducible command and add regression/smoke tests.

Exit Criteria (DoD)

• make reproduce runs full pipeline from scratch (or cached data path)
• run_manifest.json saved
• Reproducibility smoke test passes
• README matches actual commands

S8.1 Create end-to-end pipeline commands

• Add Makefile targets:
  • build-data
  • build-signals
  • run-backtest
  • run-risk
  • build-report
  • reproduce (depends on all above)
• Ensure commands call scripts/modules with config paths

Outputs

• Finalized Makefile workflow

Checks

• Running each target individually works
• make reproduce executes in correct order

---

S8.2 Add run manifest + config snapshotting

• Write outputs/run_manifest.json
• Include timestamp, git commit hash (if available), config file hashes/contents, Python version
• Save random seed and key parameters

Outputs

• outputs/run_manifest.json

Checks

• Manifest valid JSON
• Config snapshot present and readable

---

S8.3 Add caching strategy (MVP)

• Avoid re-downloading data when raw cache exists (configurable force_refresh)
• Log cache hits/misses
• Keep cache behavior deterministic

Outputs

• Cache behavior documented and implemented

Checks

• Repeat run uses cache when expected
• Force refresh path works

---

S8.4 Add reproducibility smoke test

• tests/test_reproducibility.py
• Run small-sample pipeline
• Assert output files exist
• Assert key metrics are finite and within broad ranges (not exact equality)

Outputs

• Reproducibility smoke test

Checks

• pytest -q tests/test_reproducibility.py

---

S8.5 README final hardening

• Add exact make reproduce command
• Document assumptions (data quality, costs/slippage/roll)
• Add results table example and artifact locations
• Add troubleshooting section for Mac M2 / PDF export issues

Outputs

• Final README

Checks

• README commands match Makefile exactly
• New user can reproduce report with documented steps

---

Cross-Stage QA Checklist (run before major merges)

• No core logic only in notebooks
• Config-driven parameters (not hardcoded magic numbers)
• No lookahead in signals/backtest/roll logic
• Data cleaning and fill actions are auditable
• Outputs written to outputs/ with stable paths
• Stage artifacts are reusable by later stages
• README and PROJECT_SPEC.md remain aligned

---

Optional Parallelization Map (only after Stage 1 is stable)

If using multiple agents, split by module boundaries and merge carefully.

Safe-ish parallel work after Stage 2

• Agent A: Stage 3 QA + calendar alignment
• Agent B: Stage 3 roll rule engine
• Agent C: Signal interface + term structure signals (Stage 4)

Safe-ish parallel work after Stage 4

• Agent A: Backtest execution/positioning
• Agent B: Backtest attribution
• Agent C: Risk analytics scaffolding
• Agent D: Report templates + chart rendering

Merge guardrails

• Shared schema contracts must be finalized first
• Use consistent column names from contracts
• Run full test suite after each merge

---

Per-Task Execution Note Template (copy into agent replies)

Task: <Task ID / Name>
Status: [x] Done / [~] Partial / [!] Blocked

Files changed:
- ...

Commands run:
- ...

Validation:
- [PASS] ...
- [FAIL] ...

Artifacts:
- ...

Blockers / limitations:
- ...

Next task:
- ...

---

Suggested First Execution Order (practical)

1. S1.1 → S1.6
2. S2.1 → S2.6
3. S3.1 → S3.6
4. S4.1 → S4.8
5. S5.1 → S5.8
6. S6.1 → S6.6
7. S7.1 → S7.6
8. S8.1 → S8.5

---

File name

Save this file as: TASK_BOARD.md