feat: joint convex portfolio optimization and Barra risk contracts

.github/workflows/ci.yml

@@ -11,9 +11,11 @@ permissions:
 
 jobs:
   test:
-    runs-on: ubuntu-24.04
+    name: test (${{ matrix.python-version }})${{ matrix.os == 'windows-2025' && ' Windows' || '' }}
+    runs-on: ${{ matrix.os }}
     strategy:
       matrix:
+        os: [ubuntu-24.04, windows-2025]
         python-version: ["3.10", "3.11", "3.12"]
 
     steps:
@@ -30,7 +32,7 @@ jobs:
         run: |
           python -m pip install --no-deps --requirement requirements.lock
           python -m pip check
-          python -m pip install --no-deps --no-build-isolation --editable .
+          python -m pip install --no-deps --no-build-isolation .
           python -m pip check
 
       - name: Ruff check
@@ -66,6 +68,7 @@ jobs:
           print(f"cross_asset.py pure branch coverage: {covered}/{total} ({covered / total:.2%})")
 
       - name: Run deterministic synthetic demo
+        shell: bash
         run: |
           python scripts/run_synthetic_demo.py --out "$RUNNER_TEMP/synthetic-demo"
           grep -q "SYNTHETIC DEMONSTRATION" "$RUNNER_TEMP/synthetic-demo/metrics.json"

README.md

@@ -173,9 +173,82 @@ it emits QExec `OrderIntent` suggestions and never alters a ledger, positions, o
 The module has no live-order, network, or credential capability. Missing, future, duplicate, or
 non-finite PIT inputs fail closed.
 
-## M6 governance and reproducibility
+## Optimizer protocol v2 and migration
+
+`allocate_research` returns an `AllocationResult` with continuous account weights,
+independent constraint residuals, forecast/risk units, exact solver versions and
+backend attempts. `research_allocation_weights` remains a Series convenience API.
+New configurations explicitly specify `protocol_version: 2`, `mode`,
+`risk_objective: absolute|active`, `forecast_horizon_sessions` and
+`expected_return_model: ic_vol|provided|none`. For example:
+
+```yaml
+allocation:
+  protocol_version: 2
+  mode: cost_aware
+  risk_objective: absolute
+  forecast_horizon_sessions: 5
+  expected_return_model: ic_vol
+  lookback: 120
+  min_observations: 20
+  ic_lookback: 120
+  ic_min_periods: 20
+  max_turnover: 2.0
+  timeout_seconds: 30
+```
 
-Version `0.4.2` consumes only published annotated internal tags:
+The expected-return and risk penalty use the declared decision horizon. Costs
+are charged once per rebalance. Covariance inputs and tracking-error limits are
+annualized. IC uses only labels available and matured by the decision, selects
+non-overlapping periods and fails on insufficient history. `provided` forecasts
+must cover every candidate. Black–Litterman remains an explicit lower-level
+views API; v2 does not infer missing views or market weights.
+
+Mean–variance maximizes forecast return minus half the risk-aversion coefficient
+times variance and trading costs. `active` centers risk on the full benchmark
+and is currently supported by `cost_aware`. Target methods minimize squared
+distance to equal, inverse-volatility, risk-parity or HRP targets plus declared
+cost penalties; risk-contribution changes are reported. Ratio methods maximize
+the positive excess-return/volatility or weighted-volatility/portfolio-volatility
+ratio; transaction costs are reported and checked during executable repair,
+but are not subtracted from these ratio definitions. CVaR/CDaR use exact empirical
+tail-risk auxiliary variables; EVaR uses exponential cones. Their objectives are
+forecast return minus risk aversion times scenario risk and trading costs.
+CDaR uses cumulative simple-return paths, not compounded wealth drawdowns.
+
+All constraints enter the same problem, including trades out of the candidate
+pool. Cash is retained in whole-account weights; benchmark weights are never
+renormalized to the candidate pool or invested sleeve. `RiskSnapshot` carries
+X/F/D or a dense covariance, availability time, horizon, annualization and model
+identity. The factor path never constructs an asset-by-asset covariance.
+HRP/risk-parity target construction currently requires explicit dense candidate
+covariance; research configuration rejects their combination with factor-only
+Barra inputs. The default exposure definitions are unchanged.
+
+CVXPY 1.7.5, OSQP 1.0.5 and Clarabel 0.11.1 are pinned. OSQP may retry the same
+problem in Clarabel; an inaccurate result is never directly published. Failures
+distinguish `infeasible`, `not_converged`, `numerical_failure`, `timeout`,
+`no_positive_excess_return`, `undefined_ratio` and `repair_failed`. The legacy
+mean–variance algorithm is available only through `solver_backend="legacy"`
+for comparison; it is never an automatic fallback. Integer targets use exact
+execution-layer cash quotes and deterministic local repair. They are verified
+feasible targets, with no claim of a global integer optimum.
+
+Configurations without a protocol version retain their score interpretation
+and are marked as compatibility mode. Solver correctness fixes can change their
+weights. Exact historical reproduction requires the original frozen environment;
+new runs receive new identities and must not overwrite previous evidence.
+
+Run `scripts/benchmark_optimizer.py --help` for the seeded 200/1,000/6,000 asset
+factor benchmark. The regression suite contains the bounded-simplex KKT failure,
+three six-asset projection cases, joint-turnover maximum Sharpe, pool-external
+benchmark/cash tests and 200 independent SLSQP objective comparisons.
+
+## M6 governance and reproducibility (historical baseline)
+
+The original `0.4.2` governance baseline consumed these annotated internal tags.
+The optimizer upgrade pins its reviewed protocol and execution revisions in
+`pyproject.toml` and `requirements.lock` instead:
 
 - `quant-data-kit v0.8.1` (`b5621379e1a15562371be31c03f354a6acf512e4`)
 - `quant-execution v0.5.1` (`99ab9b1445d72164fa4e8c6d1ebde859b80dba1f`)

pyproject.toml

@@ -10,8 +10,11 @@ readme = "README.md"
 license = { text = "MIT" }
 requires-python = ">=3.10"
 dependencies = [
-    "quant-data-kit @ git+https://github.com/PureSaber/quant-data-kit.git@f85128307dd0e3407d24350282fe7b679b591309",
-    "quant-execution @ git+https://github.com/PureSaber/quant-execution.git@49ef53d2c502ad6faad6818f853dadd1f8b8b182",
+    "cvxpy==1.7.5",
+    "osqp==1.0.5",
+    "clarabel==0.11.1",
+    "quant-data-kit @ git+https://github.com/PureSaber/quant-data-kit.git@ad3440700b226497c40093197f1df4321b49d1e8",
+    "quant-execution @ git+https://github.com/PureSaber/quant-execution.git@8423726a9d80b6509d2c016a7478fc0f40a29774",
     "pandas>=2.0",
     "pyarrow>=14.0",
     "pyyaml>=6.0",

requirements.lock

@@ -1,105 +1,44 @@
-#
-# This file is autogenerated by pip-compile with Python 3.10
-# by the following command:
-#
-#    pip-compile --allow-unsafe --build-deps-for=editable --constraint=requirements-constraints.txt --extra=dev --index-url=https://pypi.org/simple --no-index --output-file=requirements.lock --pip-args='--timeout=15 --retries=1' --strip-extras pyproject.toml
-#
+# Exact dependency closure; optimizer additions resolved universally from solver-requirements.in.
 attrs==26.1.0
-    # via
-    #   jsonschema
-    #   referencing
+cffi==2.1.1
+clarabel==0.11.1
+cloudpickle==3.1.2
 colorama==0.4.6
-    # via pytest
 coverage==7.16.0
-    # via pytest-cov
+cvxpy==1.7.5
 duckdb==1.5.5
-    # via quant-data-kit
 exceptiongroup==1.3.1
-    # via
-    #   pytest
-    #   quant-portfolio (pyproject.toml)
 iniconfig==2.3.0
-    # via pytest
+jinja2==3.1.6
+joblib==1.6.0
 jsonschema==4.26.0
-    # via
-    #   quant-data-kit
-    #   quant-execution
 jsonschema-rs==0.52.1
-    # via quant-data-kit
 jsonschema-specifications==2025.9.1
-    # via jsonschema
+markupsafe==3.0.4
 numpy==2.2.6
-    # via
-    #   pandas
-    #   quant-data-kit
-    #   quant-portfolio (pyproject.toml)
 orjson==3.12.0
-    # via quant-data-kit
+osqp==1.0.5
 packaging==26.3
-    # via pytest
 pandas==2.3.3
-    # via
-    #   quant-data-kit
-    #   quant-portfolio (pyproject.toml)
 pluggy==1.6.0
-    # via
-    #   pytest
-    #   pytest-cov
 pyarrow==25.0.1
-    # via
-    #   quant-data-kit
-    #   quant-execution
-    #   quant-portfolio (pyproject.toml)
+pycparser==3.11 ; implementation_name != 'PyPy'
 pygments==2.21.0
-    # via pytest
 pytest==9.1.1
-    # via
-    #   pytest-cov
-    #   quant-portfolio (pyproject.toml)
 pytest-cov==7.1.0
-    # via quant-portfolio (pyproject.toml)
 python-dateutil==2.9.0.post0
-    # via pandas
 pytz==2026.3.post1
-    # via pandas
 pyyaml==6.0.3
-    # via
-    #   quant-data-kit
-    #   quant-portfolio (pyproject.toml)
-quant-data-kit @ git+https://github.com/PureSaber/quant-data-kit.git@f85128307dd0e3407d24350282fe7b679b591309
-    # via
-    #   quant-execution
-    #   quant-portfolio (pyproject.toml)
-quant-execution @ git+https://github.com/PureSaber/quant-execution.git@49ef53d2c502ad6faad6818f853dadd1f8b8b182
-    # via quant-portfolio (pyproject.toml)
+quant-data-kit @ git+https://github.com/PureSaber/quant-data-kit.git@ad3440700b226497c40093197f1df4321b49d1e8
+quant-execution @ git+https://github.com/PureSaber/quant-execution.git@8423726a9d80b6509d2c016a7478fc0f40a29774
 referencing==0.37.0
-    # via
-    #   jsonschema
-    #   jsonschema-specifications
 rpds-py==0.30.0
-    # via
-    #   -c requirements-constraints.txt
-    #   jsonschema
-    #   referencing
 ruff==0.16.10
-    # via quant-portfolio (pyproject.toml)
+scipy==1.15.3
+scs==3.3.1
+setuptools==84.0.0
 six==1.17.0
-    # via python-dateutil
 tomli==2.4.1 ; python_version < "3.11"
-    # via
-    #   coverage
-    #   pytest
-    #   quant-portfolio (pyproject.toml)
 typing-extensions==4.16.0
-    # via
-    #   exceptiongroup
-    #   quant-data-kit
-    #   referencing
 tzdata==2026.3
-    # via pandas
 websockets==15.0.1
-    # via quant-data-kit
-
-# The following packages are considered to be unsafe in a requirements file:
-setuptools==84.0.0
-    # via quant-portfolio (pyproject.toml::build-system.requires)

scripts/benchmark_optimizer.py

@@ -0,0 +1,96 @@
+"""Reproducible factor-only scalability probe (not a live trading latency claim)."""
+
+import argparse
+import json
+import os
+from pathlib import Path
+from time import perf_counter
+
+import numpy as np
+
+from quant_portfolio.optimization import OptimizationConstraints
+from quant_portfolio.solver import RiskMatrix, solve_portfolio
+
+
+def peak_memory():
+    if os.name == "nt":
+        import ctypes
+        from ctypes import wintypes
+
+        class Counters(ctypes.Structure):
+            _fields_ = [("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD)] + [
+                (name, ctypes.c_size_t)
+                for name in (
+                    "PeakWorkingSetSize",
+                    "WorkingSetSize",
+                    "QuotaPeakPagedPoolUsage",
+                    "QuotaPagedPoolUsage",
+                    "QuotaPeakNonPagedPoolUsage",
+                    "QuotaNonPagedPoolUsage",
+                    "PagefileUsage",
+                    "PeakPagefileUsage",
+                )
+            ]
+
+        counters = Counters()
+        counters.cb = ctypes.sizeof(counters)
+        api = ctypes.windll.psapi.GetProcessMemoryInfo
+        api.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters), wintypes.DWORD]
+        api.restype = wintypes.BOOL
+        if not api(wintypes.HANDLE(-1), ctypes.byref(counters), counters.cb):
+            raise ctypes.WinError()
+        return counters.PeakWorkingSetSize
+    import resource
+
+    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--assets", type=int, default=200)
+    parser.add_argument("--repeats", type=int, default=5)
+    parser.add_argument("--timeout", type=float, default=30)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    n, k = args.assets, 25
+    rng = np.random.default_rng(20261014 + n)
+    x = rng.normal(0, 0.4, (n, k))
+    f = np.eye(k) * 0.003
+    d = rng.uniform(0.01, 0.04, n)
+    risk = RiskMatrix(exposures=x, factor_covariance=f, specific_variance=d)
+    current = np.ones(n) / n
+    mu = rng.normal(0.001, 0.0001, n)
+    times, records = [], []
+    for _ in range(args.repeats):
+        start = perf_counter()
+        got = solve_portfolio(
+            mu,
+            risk,
+            current,
+            OptimizationConstraints(max_weight=3 / n, max_turnover=0.4),
+            risk_scale=1 / 252,
+            risk_aversion=5,
+            linear_costs=np.ones(n) * 0.00005,
+            timeout_seconds=args.timeout,
+        )
+        times.append(perf_counter() - start)
+        records.append(got.diagnostics)
+    payload = {
+        "assets": n,
+        "factors": k,
+        "seed": 20261014 + n,
+        "repeats": args.repeats,
+        "timeout_seconds": args.timeout,
+        "elapsed_seconds": times,
+        "p95_seconds": float(np.quantile(times, 0.95)),
+        "process_peak_memory_bytes": peak_memory(),
+        "asset_covariance_materialized": False,
+        "diagnostics": records,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2), encoding="utf-8")
+    print(json.dumps({k: v for k, v in payload.items() if k != "diagnostics"}))
+
+
+if __name__ == "__main__":
+    main()