diff --git a/.env.example b/.env.example index 132638ab..7aa4804d 100644 --- a/.env.example +++ b/.env.example @@ -54,6 +54,10 @@ EVOLVER_JOB_TIMEOUT_S=300 EVOLVER_RUN_TIMEOUT_S=1200 EVOLVER_JOB_MEM_GB=2 EVOLVER_LLM_TIMEOUT_S=120 +# E2 事件 Campaign 分阶段开关;关闭时 E1 单代 run 完全不受影响。 +EVENT_EVOLUTION_ENABLED=false +CAMPAIGN_LEASE_TTL_S=90 +CAMPAIGN_MAX_CONCURRENT=1 # Evolver 仅用该地址按 owner/config_id 解析既有加密凭据;不会持久化明文 key。 DASHBOARD_SERVICE_URL=http://localhost:3001 # Ed25519 DER 的 base64:私钥仅给 orchestration,公钥给 Dashboard 验证逐操作 grant。 @@ -61,6 +65,14 @@ DASHBOARD_SERVICE_URL=http://localhost:3001 EVOLUTION_CREDENTIAL_PRIVATE_KEY_B64= EVOLUTION_CREDENTIAL_PUBLIC_KEY_B64= +# 事件唯一事实源:历史结构化导入 + 上线后的精选新闻 forward 归档。 +COINMARKETCAL_API_KEY= +COINMARKETCAL_BASE_URL=https://api.coinmarketcal.com +EVENT_PROVIDER_TIMEOUT_S=20 +EVENT_HISTORICAL_LATENCY_S=300 +EVENT_ARCHIVE_ENABLED=false +EVENT_ARCHIVE_INTERVAL_S=900 + # factor 服务可选项(ADR-0043):qlib Alpha158 风格因子纯 pandas 本地算,默认开; # 设 false 可整源关闭。snapshot top-N 去相关阈值默认 0.85(1.0 = 关闭去相关) # FACTOR_QLIB_ENABLED=true diff --git a/AGENTS.md b/AGENTS.md index 432528e4..d18e6372 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,7 +104,7 @@ pnpm scheduler:trigger daily_btc_deep_dive # 手动触发一次 ## 6. 当前 Phase 状态 -Phase **D-12 + E1 生产闭环**已落地:单 orchestrator + plan/exec 三件套 +Phase **D-12 + E2 事件演化闭环**已落地:单 orchestrator + plan/exec 三件套 (create_plan / approve_plan / execute_plan)+ hooks + permissions deny + approval_token 状态机(D-8/D-9)→ LLM 自创策略沙盒 + 风控引擎 + 多市场数据 (D-9/D-10)→ 跨币种 cash + **live runner**(promoted 候选按行情自动跑,机器审批 @@ -112,9 +112,10 @@ approval_token 状态机(D-8/D-9)→ LLM 自创策略沙盒 + 风控引擎 + (candidate 归属校验 / per-account run 上限 / 错误可重试分类);D-11.2 收口运维; D-12 完成 factor 血缘、衰减巡检与因子发现。research-hub 三方辩论已收口。 E1 已拆出 `services/evolver:8005`:真实 frozen bars、单代 unified-diff 变异、异步 -owner-scoped 状态、显式逐次审批与可复现实验元数据;演化链路不会自动 promote、启动策略或下单。 -当前收口项是冻结 LLM/定价快照、owner key 即时获取与 token/cost 审计;下一里程碑为 -E2 best-parent 多代选择与 early stopping(issue #7),MAP-Elites / Island Model 后置。 +owner-scoped 状态、显式逐次审批与可复现实验元数据。E2 在 feature flag 后增加双时态事件 +snapshot、HypothesisSpec 确定性编译、五代 8×3 共演化、单冠军 Forward、一次性 holdout 与 +`runner_eligible=false` 的人工实验性采用;不会自动 promote、启动策略或下单。 +下一步是在小流量运行中校准事件覆盖与信用分配;MAP-Elites / Island Model 继续后置。 详见 [`docs/04-current-state.md`](docs/04-current-state.md) / `CLAUDE.md` §3 / 仓库根 `README.md`。 diff --git a/CLAUDE.md b/CLAUDE.md index 3196a80f..d45ccf98 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,11 +9,11 @@ - **不是**开箱即用策略平台 / LangChain / AutoGen 包装 - **三层**:Next.js + CopilotKit → Mastra(TS)→ Python services。详 `docs/01-architecture-overview.md` -## 2. 文档入口 & 当前 Phase(D-12 + E1) +## 2. 文档入口 & 当前 Phase(D-12 + E2) - `README.md` / `README.zh-CN.md` 首页;`AGENTS.md` 多工具入口;`docs/00-context.md` 背景 / `01-architecture-overview.md` 架构 / `03-kernel-design.md` services / `04-current-state.md` 进度 - 内部 ADR 在 `docs/miro/`(gitignored,公开文档勿引用) -- D-8~D-12 已完成 Plan/Exec、策略创作、风控、多市场模拟盘与因子闭环;research-hub 已收口;E1 独立 Evolver 已落地真实 frozen bars、显式审批、owner 隔离与可复现实验元数据。当前收口冻结 LLM/定价快照与费用审计;下一 E2 best-parent 多代演化 #7 +- D-8~D-12、research-hub 与 E1 已收口;E2 在 feature flag 后落地双时态事件 snapshot、确定性 DSL、五代 8×3 共演化、Forward/Holdout 和非 Runner 实验性采用。E1 原语义不变 ## 3. 协作硬约束 @@ -61,8 +61,8 @@ bash scripts/dev.sh # data:8001…evolver:8005 + mastra:4 D-9/D-9.1a 收口 + D-10 多市场数据(web/基本面)+ D-11 多市场模拟盘 (跨币种 cash + live runner #1)+ D-12 因子库闭环(血缘 + 衰减巡检 + -monthly 宏观 + 因子发现 L1 + 三方研究辩论)和 E1 独立 Evolver 已落地。 -当前:冻结 LLM/定价快照、owner key 即时获取、token/cost 审计;下一:E2 best-parent 多代演化 #7 +monthly 宏观 + 因子发现 L1 + 三方研究辩论)、E1 与 feature-flag E2 事件共演化已落地。 +当前:小流量校准事件覆盖、FDR、费用与 Forward 证据;MAP-Elites / Island Model 后置 --- diff --git a/README.md b/README.md index 4c4ed36c..bfb7dc68 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ Several capability lines sit on top of that harness: The name combines **Ina**ri (the Japanese fox deity of prosperity) with **alpha** (the quant term for excess return) — a companion that reads your direction and keeps every step on the record. -> **Status:** Inalpha is in **alpha** — 79 factors with lineage & decay watch, restricted-DSL factor discovery, a three-party research debate, multi-market paper trading, and an E1 strategy-evolution service. Each evolution run requires explicit approval, freezes its dataset and non-secret LLM/pricing snapshot, and never auto-promotes or starts a candidate. Read the code, weigh in on design — **do not run this against real money** (real-money trading is out of scope). +> **Status:** Inalpha is in **alpha** — 79 factors with lineage & decay watch, restricted-DSL factor discovery, a three-party research debate, multi-market paper trading, and E1/E2 strategy evolution. Feature-flagged E2 adds point-in-time event snapshots, deterministic hypothesis DSL compilation, five-generation campaigns, Forward evidence, and one-shot sealed holdout. Evolution never auto-promotes, starts, or trades a candidate. Read the code, weigh in on design — **do not run this against real money** (real-money trading is out of scope). --- @@ -128,7 +128,7 @@ Three software layers over one data layer. A request flows down; results flow ba | `services/paper` | The event-driven kernel — backtest + paper on the **same code** — plus the LLM-authored-strategy sandbox and the live runner. | | `services/research` | Multi-agent deep dive: 6 analysts in parallel, then a bull / bear / risk debate (triggered only when they disagree, with a soft early-stop and the decision chain persisted for replay). | | `services/factor` | The factor library (pandas-ta / Alpha101 / qlib + FRED macro): IC screening, current-effective factor timing, lineage & decay watch, DSL factor discovery. **Signals only — never places an order.** | -| `services/evolver` | Owner-scoped E1 strategy evolution: unified-diff mutation, frozen-data evaluation, candidate/run lineage, cost accounting, and an explicit approval boundary. **Never auto-promotes, starts, or trades a candidate.** | +| `services/evolver` | Owner-scoped E1/E2 evolution: unified-diff mutation, event-hypothesis campaigns, frozen-data evaluation, lineage, cost accounting, Forward/holdout gates, and explicit final adoption. **Never auto-promotes, starts, or trades a candidate.** | **L4 · Persistence & external.** Postgres + TimescaleDB holds all time-series and business state. External venues span crypto, US / A-share / HK and other Asian & European single-name equities, global indices, and FRED macro — the orchestrator routes each venue automatically by market type. @@ -173,7 +173,7 @@ Human-written strategies hit a velocity ceiling, and parameter tuning can only a - **Reproducible end to end.** One run freezes `as_of`, the closed-bar dataset manifest/hash, seed source, baseline, candidate source/diff, evaluation snapshots, and non-secret LLM/provider/pricing metadata. The user's encrypted API key is resolved only for that owner and never stored in the run. - **Explicitly authorized and non-promoting.** Starting a run requires a trusted approval bound to the owner, operation ID, request, estimated cost, and frozen LLM snapshot. Completion never promotes, starts, or routes a candidate into the order path. -> E1 now runs as the separate `services/evolver` service on port 8005. E2 is intentionally narrower than the original research plan: best-parent multi-generation selection plus early stopping first; MAP-Elites and Island Model wait for real run data to justify the complexity. +> E1 and feature-flagged E2 run in the separate `services/evolver` service on port 8005. E2 uses fixed 8×3 generations, novelty-aware inheritance, one locked Forward champion, and a single-use sealed holdout; MAP-Elites and Island Model wait for real run data to justify the complexity. ### 4. Swarm — run dozens of backtests in parallel @@ -234,6 +234,7 @@ Where each capability stands today. Live module inventory and the end-to-end dec | ✅ Shipped | LLM-authored strategies — E1 MVP | D-9 | three sandbox gates (AST · subprocess · `Strategy` contract) + multi-objective fitness + baseline auto-run | | ✅ Shipped | Strategy evolution — E1 production loop | E1 | `services/evolver:8005` · explicit cost-bearing approval · unified-diff mutation · frozen dataset/hash · seed/baseline/candidates evaluated on the same bars · owner-scoped async run/slot state | | ✅ Shipped | Frozen LLM approval snapshot | E1 closure | Dashboard approve/deny · owner/operation/model/pricing binding · Ed25519 replay-safe credential grant · per-slot token/cost accounting, including rejected mutations | +| ✅ Feature flag | Event-driven automatic evolution | E2 | bitemporal event snapshots · HypothesisSpec DSL · five 8×3 generations · Forward + one-shot sealed holdout · human experimental adoption only, never Runner eligible | | ✅ Shipped | Risk engine at the HTTP boundary | D-9 | declarative `risk_rules.toml` · pre-trade `enforce` · `risk_locks` table with independent commit | | ✅ Shipped | Bull / bear researcher debate | D-9 | opposing-stance researchers under `services/research` | | ✅ Shipped | Scheduler / cron agent mode | D-9 | `scheduler_jobs` + advisory lock + `/api/scheduler/*` management plane | @@ -255,7 +256,6 @@ Where each capability stands today. Live module inventory and the end-to-end dec | ✅ Shipped | Cross-sectional factor scoring | D-12 | `factor.panel_score` · `POST /panel/score` · cross-sectional Rank IC (rank the pool each period vs forward cross-sectional return) · native Alpha101 a1/a3 · orthogonal to single-name timing | | ✅ Shipped | Time-series cross-validation — anti-overfitting | D-12 | WalkForward / PurgedKFold / Combinatorial Purged CV + Deflated Sharpe · `POST /backtest/cv` · test fold always includes the latest bar · auto-fallback to walk-forward when samples are short | | ✅ Shipped | Point-in-time fundamentals | D-12 | Baostock financials filtered by actual publication date · `GET /fundamentals?as_of=` · prevents look-ahead (yfinance v1 not yet PIT, explicitly flagged) | -| 🗓️ Planned | Strategy evolution — E2 | E2 | best-parent multi-generation loop + early stopping; MAP-Elites / Island Model deferred until real run data shows a diversity problem | | 🗓️ Planned | Factor discovery — L2 / L3 | L2 / L3 | multi-agent factor crew (L2) + weekly automated scans (L3), on top of the L1 DSL pipeline already shipped | | 🗓️ Planned | Automated decay handling | TBD | reflection-driven backtest + auto-trim of decaying factors — today the decay patrol only alerts, never moves the book | | 🔬 Exploring | Alpha Zoo cold start | E1+ | seed factor library with public alphas (Qlib / Kakushadze / GTJA) | diff --git a/README.zh-CN.md b/README.zh-CN.md index e4dc2c73..49bbea0c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -232,6 +232,7 @@ Inalpha 把*调度*和*算力*分开:agent runtime 负责扇出网格、聚合 | ✅ 已上线 | LLM 自创策略 — E1 MVP | D-9 | 三道沙盒(AST 审计 / 子进程 / `Strategy` 协议契约) + 多目标 fitness + baseline 自动并跑 | | ✅ 已上线 | 策略演化 — E1 生产闭环 | E1 | `services/evolver:8005` · 计费动作显式审批 · unified-diff 变异 · 冻结数据集/hash · seed/baseline/candidate 同 bars 评估 · owner 隔离异步 run/slot 状态机 | | ✅ 已上线 | 冻结 LLM 审批快照 | E1 收口 | Dashboard 批准/拒绝 · owner/operation/模型/计价绑定 · Ed25519 可重试凭据 grant · 被拒变异也记 token/费用 | +| ✅ Feature flag | 事件驱动自动演化 | E2 | 双时态事件 snapshot · HypothesisSpec DSL · 每代 8×3、连续五代 · Forward + 一次性 holdout · 仅人工实验性采用,禁止 Runner | | ✅ 已上线 | 风控引擎落到 HTTP 边界 | D-9 | 声明式 `risk_rules.toml` · 撮合前 `enforce` · `risk_locks` 表(独立 commit) | | ✅ 已上线 | Bull / Bear 研究员辩论 | D-9 | `services/research` 立场对抗研究员 | | ✅ 已上线 | Scheduler / cron agent 模式 | D-9 | `scheduler_jobs` + advisory lock + `/api/scheduler/*` 管理面 | diff --git a/TODOS.md b/TODOS.md new file mode 100644 index 00000000..3539fa24 --- /dev/null +++ b/TODOS.md @@ -0,0 +1,5 @@ +# Inalpha 后续路线 + +- [ ] 事件演化的多资产组合级回测与 cross-market lane。本期只支持单一 crypto 标的,不能把多个市场的事件效应当作组合收益。 +- [ ] 在基础信用分配与五代搜索有足够线上样本后,再评估 MAP-Elites / Island Model。 +- [ ] 评估 1m/5m 事件内执行、盘口、partial fill;当前事件研究硬限定 15m/1h/4h。 diff --git a/apps/dashboard/messages/en.json b/apps/dashboard/messages/en.json index 01a30424..035b47d8 100644 --- a/apps/dashboard/messages/en.json +++ b/apps/dashboard/messages/en.json @@ -13,6 +13,7 @@ "risk": "Risk", "divination": "Omikuji", "evolution": "Evolution", + "dataHealth": "Event Data Health", "soon": "soon", "menu": "Menu", "close": "Close", @@ -65,6 +66,12 @@ "overview": { "title": "Portfolio Overview", "subtitle": "A live snapshot of your account, positions, orders, and strategy pool.", + "evolution": { + "active": "Active Campaigns", + "waiting": "Waiting for Events", + "adoption": "Awaiting Adoption", + "sources": "Event Source Faults" + }, "kpi": { "totalEquity": "Total Equity", "totalReturn": "Total Return", @@ -271,6 +278,7 @@ "order": "Orders", "backtest": "Backtests", "runner": "Runners", + "evolution": "Evolution", "conversation": "Chats" }, "empty": "No activity yet.", @@ -305,6 +313,13 @@ "resize": "Drag to resize height" }, "lab": { + "experimental": { + "title": "Experimental Evolution Strategies", + "empty": "No adopted event-evolution winner yet.", + "badge": "Limited evidence · Experimental", + "grade": "Evidence grade: {grade}", + "lineage": "View evolution lineage" + }, "title": "Strategy Lab", "subtitle": "LLM-authored strategy candidates, ranked by multi-objective fitness — with their latest backtest metrics.", "candidates": "Candidates", @@ -565,7 +580,7 @@ }, "evolution": { "title": "Evolution", - "subtitle": "LLM-driven strategy mutation and evaluation — each run spawns candidates through code mutation, AST audit, sandboxing, and backtest evaluation.", + "subtitle": "Event facts and paper feedback drive two-level evolution: agents diversify hypotheses, deterministic DSL arms validate them, and one champion enters durable Forward.", "runs": "Total runs", "running": "Running", "llmCost": "LLM cost", @@ -632,10 +647,58 @@ }, "loadMore": "Load more", "loadingMore": "Loading…" + ,"campaign": { + "title": "Event Campaigns", + "empty": "No event campaign yet; E1 single-generation runs keep their existing semantics.", + "id": "Campaign", + "status": "Status", + "stage": "Generation", + "budget": "Candidates/gen", + "events": "Forward events", + "holdout": "Sealed holdout", + "updated": "Updated", + "sealed": "Unconsumed", + "consumed": "Consumed", + "back": "Back to evolution", + "detailTitle": "Event Evolution", + "detailSubtitle": "The audit trail for frozen facts, hypothesis lineage, selection, ablations, Forward, and one-shot holdout.", + "track": "Generation Track", + "frozenData": "Frozen Facts & Execution", + "snapshot": "Event snapshot", + "facts": "Facts", + "eventHash": "Event hash", + "executionModel": "Execution model", + "forwardHoldout": "Forward & Holdout", + "forwardEvents": "Independent events", + "deadline": "Deadline", + "winner": "Locked champion", + "hypotheses": "Hypotheses, Evidence & Lineage", + "selected": "Selected parent", + "ablations": "Implementation Ablations & Event Effects", + "failure": "Failure", + "adopt": "Adopt into Strategy Lab", + "adopting": "Adopting…", + "adopted": "Adopted as an experimental strategy", + "adoptHint": "Adoption does not promote, start Runner, or trade; runner_eligible is enforced false." + } }, "backtests": { "back": "Agent Activity", "notFound": "Backtest run not found.", "viewCandidate": "View strategy" } + ,"dataHealth": { + "title": "Event Data Health", + "subtitle": "Coverage, first acceptance, revisions, and snapshot integrity for the event source of truth.", + "rawEvents": "Raw events", + "facts": "Normalized facts", + "retractions": "Revisions / retractions", + "lastAccepted": "Last accepted", + "sources": "Source coverage", + "source": "Source", + "count": "Events", + "latest": "Last fetch", + "versions": "Max version", + "empty": "No event source has been archived yet." + } } diff --git a/apps/dashboard/messages/zh.json b/apps/dashboard/messages/zh.json index fb19157a..f32171e4 100644 --- a/apps/dashboard/messages/zh.json +++ b/apps/dashboard/messages/zh.json @@ -13,6 +13,7 @@ "risk": "风控", "divination": "狐神签", "evolution": "策略演化", + "dataHealth": "事件数据健康", "soon": "即将", "menu": "菜单", "close": "关闭", @@ -65,6 +66,12 @@ "overview": { "title": "组合总览", "subtitle": "账户、持仓、订单与策略池的实时快照。", + "evolution": { + "active": "活跃 Campaign", + "waiting": "等待事件", + "adoption": "待人工采用", + "sources": "事件源故障" + }, "kpi": { "totalEquity": "总权益", "totalReturn": "总收益率", @@ -271,6 +278,7 @@ "order": "订单", "backtest": "回测", "runner": "模拟盘", + "evolution": "演化", "conversation": "会话" }, "empty": "暂无活动。", @@ -305,6 +313,13 @@ "resize": "拖动调整高度" }, "lab": { + "experimental": { + "title": "实验性演化策略", + "empty": "暂无已采用的事件演化赢家。", + "badge": "有限证据 · 实验性", + "grade": "证据等级:{grade}", + "lineage": "查看演化血缘" + }, "title": "策略实验室", "subtitle": "LLM 自创的策略候选,按多目标 fitness 排序 —— 附最近一次回测指标。", "candidates": "候选", @@ -565,7 +580,7 @@ }, "evolution": { "title": "策略演化", - "subtitle": "LLM 驱动的策略自动变异与评估 —— 每轮运行通过代码变异、AST 审计、沙盒执行、回测评估产出候选策略。", + "subtitle": "事件事实与模拟盘反馈驱动的双层自动演化 —— Agent 发散假设,确定性 DSL 消融验证,单冠军进入耐久 Forward。", "runs": "总运行轮次", "running": "运行中", "llmCost": "LLM 费用", @@ -632,10 +647,58 @@ }, "loadMore": "加载更多", "loadingMore": "正在加载…" + ,"campaign": { + "title": "事件 Campaign", + "empty": "暂无事件演化 Campaign;E1 单代演化仍保留原语义。", + "id": "Campaign", + "status": "状态", + "stage": "代际", + "budget": "每代候选", + "events": "Forward 事件", + "holdout": "封存测试", + "updated": "更新", + "sealed": "未消费", + "consumed": "已消费", + "back": "返回演化列表", + "detailTitle": "事件演化", + "detailSubtitle": "冻结事实、假设血缘、代际选择、消融验证与一次性 Holdout 的完整审计轨迹。", + "track": "代际轨道", + "frozenData": "冻结事实与执行契约", + "snapshot": "事件快照", + "facts": "事实数", + "eventHash": "事件哈希", + "executionModel": "撮合模型", + "forwardHoldout": "Forward 与 Holdout", + "forwardEvents": "独立事件", + "deadline": "最长等待", + "winner": "锁定冠军", + "hypotheses": "假设、证据与血缘", + "selected": "入选父代", + "ablations": "实现消融与事件效应", + "failure": "失败原因", + "adopt": "采用到策略实验室", + "adopting": "正在采用…", + "adopted": "已作为实验性策略采用", + "adoptHint": "采用不会 promote、启动 Runner 或下单;后端强制 runner_eligible=false。" + } }, "backtests": { "back": "Agent 活动", "notFound": "未找到该回测记录。", "viewCandidate": "查看策略详情" } + ,"dataHealth": { + "title": "事件数据健康", + "subtitle": "事件唯一事实源的覆盖、首次接收、修订与冻结快照完整性。", + "rawEvents": "原始事件", + "facts": "规范事实", + "retractions": "修订 / 撤回", + "lastAccepted": "最后接收", + "sources": "来源覆盖", + "source": "来源", + "count": "事件数", + "latest": "最后抓取", + "versions": "最高版本", + "empty": "尚未归档事件来源。" + } } diff --git a/apps/dashboard/src/app/[locale]/data-health/page.tsx b/apps/dashboard/src/app/[locale]/data-health/page.tsx new file mode 100644 index 00000000..a8dc8591 --- /dev/null +++ b/apps/dashboard/src/app/[locale]/data-health/page.tsx @@ -0,0 +1,10 @@ +import { setRequestLocale } from "next-intl/server"; + +import { DataHealthClient } from "@/components/data-health/DataHealthClient"; + +/** Event source coverage and point-in-time ledger health. */ +export default async function DataHealthPage({ params }: { params: Promise<{ locale: string }> }) { + const { locale } = await params; + setRequestLocale(locale); + return ; +} diff --git a/apps/dashboard/src/app/[locale]/evolution/campaigns/[campaignId]/page.tsx b/apps/dashboard/src/app/[locale]/evolution/campaigns/[campaignId]/page.tsx new file mode 100644 index 00000000..07a12ec6 --- /dev/null +++ b/apps/dashboard/src/app/[locale]/evolution/campaigns/[campaignId]/page.tsx @@ -0,0 +1,10 @@ +import { setRequestLocale } from "next-intl/server"; + +import { EvolutionCampaignDetailClient } from "@/components/evolution/EvolutionCampaignDetailClient"; + +/** Event-driven multi-generation campaign workbench. */ +export default async function EvolutionCampaignPage({ params }: { params: Promise<{ locale: string; campaignId: string }> }) { + const { locale, campaignId } = await params; + setRequestLocale(locale); + return ; +} diff --git a/apps/dashboard/src/app/api/activity/route.ts b/apps/dashboard/src/app/api/activity/route.ts index 84e63166..4f72b228 100644 --- a/apps/dashboard/src/app/api/activity/route.ts +++ b/apps/dashboard/src/app/api/activity/route.ts @@ -8,6 +8,7 @@ import type { ActivityPayload, ActivityStat, ActivityTone, + EvolutionCampaign, OrderRecord, StrategyRunDecisionRecord, StrategyRunRecord, @@ -109,6 +110,7 @@ export async function GET() { strategyRunsR, ordersR, backtestsR, + campaignsR, threadsR, ] = await Promise.allSettled([ backendFetch("mastra", "/scheduler/jobs", { @@ -141,6 +143,10 @@ export async function GET() { backendFetch("paper", "/backtest_runs", { query: { limit: 30 }, }), + backendFetch<{ items: EvolutionCampaign[] }>("evolver", "/api/v1/campaigns", { + query: { limit: 20 }, + timeoutMs: 5_000, + }), // 用户对话(mastra memory threads)—— 让"发起了什么会话"进入活动流。 // 8s 轮询热路径:跳过标题回填(下方用 `#id` 兜底),回填只在历史下拉里做。 listChatThreads(200, { backfillTitles: false }), @@ -391,6 +397,38 @@ export async function GET() { sources.backtests = false; } + // ── 事件演化:只展示规范化提案、选择和终态,永不展示原始新闻正文。── + if (campaignsR.status === "fulfilled") { + for (const campaign of campaignsR.value.items) { + events.push({ + id: `campaign:${campaign.campaign_id}:${campaign.state_version}`, + kind: "evolution", + ts: campaign.updated_at, + title: `campaign ${campaign.campaign_id.slice(0, 8)}`, + detail: `generation ${campaign.active_generation}/${campaign.max_generations} · snapshot ${campaign.event_snapshot_id.slice(0, 8)}`, + outcome: campaign.status, + tone: campaign.status === "graduated" ? "bull" : campaign.failure_code ? "fox" : "cyan", + href: `/evolution/campaigns/${campaign.campaign_id}`, + stats: [{ text: `${campaign.hypothesis_budget * campaign.implementations_per_hypothesis} candidates/gen`, tone: "muted" }], + }); + for (const hypothesis of campaign.hypotheses.filter((item) => item.selected).slice(-8)) { + events.push({ + id: `hypothesis:${hypothesis.hypothesis_id}:selected`, + kind: "evolution", + ts: hypothesis.created_at, + title: `G${hypothesis.generation} ${hypothesis.lane}`, + detail: `selected · ${hypothesis.lineage_kind} · ${hypothesis.spec_hash.slice(0, 10)}`, + outcome: "selected", + tone: "gold", + href: `/evolution/campaigns/${campaign.campaign_id}`, + stats: hypothesis.upper_credit === null ? undefined : [{ text: `credit ${hypothesis.upper_credit.toFixed(3)}`, tone: "cyan" }], + }); + } + } + } else { + sources.evolution = false; + } + // ── 用户对话(会话发起 / 最近活跃)── if (threadsR.status === "fulfilled") { for (const th of threadsR.value) { @@ -496,6 +534,7 @@ function countByKind(events: ActivityEvent[]): Record { order: 0, backtest: 0, runner: 0, + evolution: 0, conversation: 0, }; for (const e of events) c[e.kind] += 1; diff --git a/apps/dashboard/src/app/api/data-health/route.ts b/apps/dashboard/src/app/api/data-health/route.ts new file mode 100644 index 00000000..b804a33e --- /dev/null +++ b/apps/dashboard/src/app/api/data-health/route.ts @@ -0,0 +1,21 @@ +import { NextResponse } from "next/server"; + +import { backendFetch, BackendError } from "@/lib/backend"; +import type { EventDataCoverage } from "@/lib/types"; + +export const dynamic = "force-dynamic"; + +/** Proxy event-ledger health while keeping service JWTs on the BFF. */ +export async function GET() { + try { + const coverage = await backendFetch("data", "/events/coverage", { + timeoutMs: 5_000, + }); + return NextResponse.json(coverage, { headers: { "Cache-Control": "no-store" } }); + } catch (error) { + return NextResponse.json( + { error: error instanceof Error ? error.message : "unknown error" }, + { status: error instanceof BackendError ? error.status : 500 }, + ); + } +} diff --git a/apps/dashboard/src/app/api/evolution/campaigns/[campaignId]/route.ts b/apps/dashboard/src/app/api/evolution/campaigns/[campaignId]/route.ts new file mode 100644 index 00000000..b79408ff --- /dev/null +++ b/apps/dashboard/src/app/api/evolution/campaigns/[campaignId]/route.ts @@ -0,0 +1,60 @@ +import { NextResponse } from "next/server"; + +import { backendFetch, BackendError } from "@/lib/backend"; +import { isEvolutionEnabled } from "@/lib/evolution-capability"; +import type { EvolutionCampaign, EvolutionCampaignDetailPayload } from "@/lib/types"; + +export const dynamic = "force-dynamic"; + +const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; + +/** Load a generation-level campaign read model; evidence curves stay lazy on the backend. */ +export async function GET( + _request: Request, + { params }: { params: Promise<{ campaignId: string }> }, +) { + if (!isEvolutionEnabled()) return NextResponse.json({ error: "disabled" }, { status: 503 }); + const { campaignId } = await params; + if (!UUID_RE.test(campaignId)) { + return NextResponse.json({ error: "invalid campaign id" }, { status: 400 }); + } + try { + const campaign = await backendFetch( + "evolver", + `/api/v1/campaigns/${campaignId}`, + { timeoutMs: 5_000 }, + ); + const payload: EvolutionCampaignDetailPayload = { campaign, asOf: new Date().toISOString() }; + return NextResponse.json(payload, { headers: { "Cache-Control": "no-store" } }); + } catch (error) { + return NextResponse.json( + { error: error instanceof Error ? error.message : "unknown error" }, + { status: error instanceof BackendError ? error.status : 500 }, + ); + } +} + +/** The only operator action is experimental adoption; it never promotes or starts Runner. */ +export async function POST( + _request: Request, + { params }: { params: Promise<{ campaignId: string }> }, +) { + if (!isEvolutionEnabled()) return NextResponse.json({ error: "disabled" }, { status: 503 }); + const { campaignId } = await params; + if (!UUID_RE.test(campaignId)) { + return NextResponse.json({ error: "invalid campaign id" }, { status: 400 }); + } + try { + const adoption = await backendFetch>( + "evolver", + `/api/v1/campaigns/${campaignId}/adopt`, + { method: "POST", body: {}, timeoutMs: 5_000 }, + ); + return NextResponse.json({ adoption }); + } catch (error) { + return NextResponse.json( + { error: error instanceof Error ? error.message : "unknown error" }, + { status: error instanceof BackendError ? error.status : 500 }, + ); + } +} diff --git a/apps/dashboard/src/app/api/evolution/campaigns/route.ts b/apps/dashboard/src/app/api/evolution/campaigns/route.ts new file mode 100644 index 00000000..53896828 --- /dev/null +++ b/apps/dashboard/src/app/api/evolution/campaigns/route.ts @@ -0,0 +1,31 @@ +import { NextResponse } from "next/server"; + +import { backendFetch, BackendError } from "@/lib/backend"; +import { isEvolutionEnabled } from "@/lib/evolution-capability"; +import type { EvolutionCampaign, EvolutionCampaignPayload } from "@/lib/types"; + +export const dynamic = "force-dynamic"; + +/** Return the owner-scoped event campaign projection without candidate source blobs. */ +export async function GET() { + if (!isEvolutionEnabled()) { + return NextResponse.json({ error: "evolution service is not enabled" }, { status: 503 }); + } + try { + const response = await backendFetch<{ items: EvolutionCampaign[] }>( + "evolver", + "/api/v1/campaigns", + { query: { limit: 50 }, timeoutMs: 5_000 }, + ); + const payload: EvolutionCampaignPayload = { + campaigns: response.items, + asOf: new Date().toISOString(), + }; + return NextResponse.json(payload, { headers: { "Cache-Control": "no-store" } }); + } catch (error) { + return NextResponse.json( + { error: error instanceof Error ? error.message : "unknown error" }, + { status: error instanceof BackendError ? error.status : 500 }, + ); + } +} diff --git a/apps/dashboard/src/app/api/internal/llm-config/[id]/route.test.ts b/apps/dashboard/src/app/api/internal/llm-config/[id]/route.test.ts index 70547db0..c73c9342 100644 --- a/apps/dashboard/src/app/api/internal/llm-config/[id]/route.test.ts +++ b/apps/dashboard/src/app/api/internal/llm-config/[id]/route.test.ts @@ -117,6 +117,31 @@ describe("internal owner LLM credential route", () => { expect(mockedDecryptUserApiKey).toHaveBeenCalledTimes(3); }); + it("allows bounded campaign recovery redemptions for the grant lifetime", async () => { + const query = vi.fn().mockResolvedValue({ rowCount: 1 }); + mockedGetPool.mockReturnValue({ query } as never); + mockedDecryptUserApiKey.mockResolvedValue({ + id: "config-1", + provider: "deepseek", + api_key: "owner-key", + } as never); + + expect( + (await callRoute(`Bearer ${await token({ grant_purpose: "event_campaign" })}`)).status, + ).toBe(200); + expect(query.mock.calls[0]?.[1]).toEqual([ + "11111111-1111-4111-8111-111111111111", + "user:alice", + "config-1", + "operation-1", + "a".repeat(64), + "b".repeat(64), + "event_campaign", + 8, + "30 hours", + ]); + }); + it("rejects a reused jti whose recorded scope differs", async () => { mockedGetPool.mockReturnValue({ query: vi.fn().mockResolvedValue({ rowCount: 0 }), diff --git a/apps/dashboard/src/app/api/internal/llm-config/[id]/route.ts b/apps/dashboard/src/app/api/internal/llm-config/[id]/route.ts index e351310f..b713ed53 100644 --- a/apps/dashboard/src/app/api/internal/llm-config/[id]/route.ts +++ b/apps/dashboard/src/app/api/internal/llm-config/[id]/route.ts @@ -35,11 +35,13 @@ export async function GET( const issuedAt = payload.iat; const expiresAt = payload.exp; const now = Math.floor(Date.now() / 1_000); + const grantPurpose = payload.grant_purpose ?? "e1_run"; if ( payload.token_use !== "evolution_credential" || payload.config_id !== configId || typeof payload.operation_id !== "string" || payload.operation_id.length < 8 || + (grantPurpose !== "e1_run" && grantPurpose !== "event_campaign") || typeof payload.llm_config_digest !== "string" || !/^[0-9a-f]{64}$/.test(payload.llm_config_digest) || typeof payload.request_digest !== "string" || @@ -67,11 +69,13 @@ export async function GET( if (!config) return NextResponse.json({ error: "not_found" }, { status: 404 }); let recorded; + const maxRedemptions = grantPurpose === "event_campaign" ? 8 : 2; + const redemptionWindow = grantPurpose === "event_campaign" ? "30 hours" : "2 minutes"; try { recorded = await getPool().query( `INSERT INTO evolution_credential_grant_uses - (jti,owner_sub,config_id,operation_id,config_digest,request_digest,consumed_at) - VALUES ($1,$2,$3,$4,$5,$6,NOW()) + (jti,owner_sub,config_id,operation_id,config_digest,request_digest,grant_purpose,consumed_at) + VALUES ($1,$2,$3,$4,$5,$6,$7,NOW()) ON CONFLICT (jti) DO UPDATE SET redemption_count=evolution_credential_grant_uses.redemption_count+1, last_redeemed_at=NOW() @@ -80,8 +84,9 @@ export async function GET( AND evolution_credential_grant_uses.operation_id=EXCLUDED.operation_id AND evolution_credential_grant_uses.config_digest=EXCLUDED.config_digest AND evolution_credential_grant_uses.request_digest=EXCLUDED.request_digest - AND evolution_credential_grant_uses.redemption_count<2 - AND evolution_credential_grant_uses.consumed_at>=NOW()-INTERVAL '2 minutes' + AND evolution_credential_grant_uses.grant_purpose=EXCLUDED.grant_purpose + AND evolution_credential_grant_uses.redemption_count<$8 + AND evolution_credential_grant_uses.consumed_at>=NOW()-$9::INTERVAL RETURNING jti`, [ payload.jti, @@ -90,6 +95,9 @@ export async function GET( payload.operation_id, payload.llm_config_digest, payload.request_digest, + grantPurpose, + maxRedemptions, + redemptionWindow, ], ); } catch { diff --git a/apps/dashboard/src/app/api/lab/route.ts b/apps/dashboard/src/app/api/lab/route.ts index 3efc327d..76e025fd 100644 --- a/apps/dashboard/src/app/api/lab/route.ts +++ b/apps/dashboard/src/app/api/lab/route.ts @@ -1,7 +1,7 @@ import { NextResponse } from "next/server"; import { backendFetch, BackendError } from "@/lib/backend"; -import type { LabPayload, StrategyCandidateSummary } from "@/lib/types"; +import type { ExperimentalStrategyAdoption, LabPayload, StrategyCandidateSummary } from "@/lib/types"; export const dynamic = "force-dynamic"; @@ -13,11 +13,15 @@ export async function GET() { try { // 多取 1 条探测「是否还有更多候选」(命中上限 → 截断提示,不静默)。 const CANDIDATES_SHOWN = 100; - const raw = await backendFetch( - "paper", - "/strategy_candidates", - { query: { limit: CANDIDATES_SHOWN + 1 } }, - ); + const [raw, adoptions] = await Promise.all([ + backendFetch("paper", "/strategy_candidates", { + query: { limit: CANDIDATES_SHOWN + 1 }, + }), + backendFetch<{ items: ExperimentalStrategyAdoption[] }>("evolver", "/api/v1/adoptions", { + query: { limit: 50 }, + timeoutMs: 5_000, + }).catch(() => ({ items: [] })), + ]); const truncated = raw.length > CANDIDATES_SHOWN; const candidates = raw.slice(0, CANDIDATES_SHOWN); const counts = { @@ -30,6 +34,7 @@ export async function GET() { candidates, counts, truncated, + experimentalAdoptions: adoptions.items, asOf: new Date().toISOString(), }; return NextResponse.json(payload, { diff --git a/apps/dashboard/src/app/api/overview/route.ts b/apps/dashboard/src/app/api/overview/route.ts index e198b780..7ab64a84 100644 --- a/apps/dashboard/src/app/api/overview/route.ts +++ b/apps/dashboard/src/app/api/overview/route.ts @@ -3,6 +3,8 @@ import { NextResponse } from "next/server"; import { backendFetch, BackendError } from "@/lib/backend"; import type { AccountSnapshot, + EventDataCoverage, + EvolutionCampaign, OrderRecord, OverviewPayload, PositionRecord, @@ -40,7 +42,7 @@ export async function GET() { // positions / orders / runs —— 任一失败降级为空,不整页挂。 // orders 多取 1 条探测「是否还有更早的」(命中上限 → 截断提示,不静默)。 const ORDERS_SHOWN = 20; - const [positionsRes, ordersRes, runsRes, candidatesRes] = + const [positionsRes, ordersRes, runsRes, candidatesRes, campaignsRes, coverageRes] = await Promise.allSettled([ backendFetch("paper", "/positions"), backendFetch("paper", "/orders", { @@ -54,6 +56,11 @@ export async function GET() { backendFetch("paper", "/strategy_candidates", { query: { limit: 200 }, }), + backendFetch<{ items: EvolutionCampaign[] }>("evolver", "/api/v1/campaigns", { + query: { limit: 50 }, + timeoutMs: 3_000, + }), + backendFetch("data", "/events/coverage", { timeoutMs: 3_000 }), ]); const positions = settledOr(positionsRes, []); @@ -70,6 +77,15 @@ export async function GET() { candidate: candidatesAll.filter((c) => c.status === "candidate").length, }; const candidates = candidatesAll.slice(0, CANDIDATES_SHOWN); + const campaigns = settledOr(campaignsRes, { items: [] }).items; + const coverage = settledOr(coverageRes, null); + const activeCampaignStates = new Set(["draft", "replaying", "candidate_locked", "waiting_forward", "holdout_ready"]); + const evolutionSummary = { + activeCampaigns: campaigns.filter((campaign) => activeCampaignStates.has(campaign.status)).length, + waitingEvents: campaigns.filter((campaign) => campaign.status === "waiting_forward").length, + readyForAdoption: campaigns.filter((campaign) => campaign.status === "graduated").length, + eventSourceFailures: coverage === null || coverage.sources.length === 0 ? 1 : 0, + }; // 每个持仓 best-effort 补最新价(fresh=false:只读 DB 缓存,不触发慢 backfill)。 const marked: PositionWithMark[] = await Promise.all( @@ -102,6 +118,7 @@ export async function GET() { candidates, candidateCounts, ordersTruncated, + evolutionSummary, asOf: new Date().toISOString(), }; diff --git a/apps/dashboard/src/components/activity/ActivityClient.tsx b/apps/dashboard/src/components/activity/ActivityClient.tsx index 374168c4..e7bcca51 100644 --- a/apps/dashboard/src/components/activity/ActivityClient.tsx +++ b/apps/dashboard/src/components/activity/ActivityClient.tsx @@ -24,6 +24,7 @@ const KINDS: ActivityKind[] = [ "order", "backtest", "runner", + "evolution", "conversation", ]; type Filter = ActivityKind | "all"; diff --git a/apps/dashboard/src/components/activity/KindTag.tsx b/apps/dashboard/src/components/activity/KindTag.tsx index b7c28dd3..dd053d7f 100644 --- a/apps/dashboard/src/components/activity/KindTag.tsx +++ b/apps/dashboard/src/components/activity/KindTag.tsx @@ -9,6 +9,7 @@ import { Radio, ShieldAlert, ShieldQuestion, + Workflow, type LucideIcon, } from "lucide-react"; @@ -22,6 +23,7 @@ const META: Record = { order: { icon: ArrowLeftRight, cls: "border-border-subtle bg-bg-elev/40 text-fg-muted" }, backtest: { icon: FlaskConical, cls: "border-cyan/30 bg-cyan/10 text-cyan" }, runner: { icon: Radio, cls: "border-bull/30 bg-bull/10 text-bull" }, + evolution: { icon: Workflow, cls: "border-cyan/30 bg-cyan/10 text-cyan" }, conversation: { icon: MessageSquare, cls: "border-seal/30 bg-seal/10 text-seal" }, }; diff --git a/apps/dashboard/src/components/data-health/DataHealthClient.tsx b/apps/dashboard/src/components/data-health/DataHealthClient.tsx new file mode 100644 index 00000000..5fa605f4 --- /dev/null +++ b/apps/dashboard/src/components/data-health/DataHealthClient.tsx @@ -0,0 +1,34 @@ +"use client"; + +import { useLocale, useTranslations } from "next-intl"; +import useSWR from "swr"; + +import { jsonFetcher } from "@/lib/fetcher"; +import { fmtRelative } from "@/lib/format"; +import type { EventDataCoverage } from "@/lib/types"; +import { ErrorState, SkeletonBlock } from "@/components/ui/Feedback"; +import { LiveStrip } from "@/components/ui/LiveStrip"; +import { PageHeader } from "@/components/ui/PageHeader"; +import { Panel } from "@/components/ui/Panel"; +import { Td, TableEmpty, TableHeadRow, Th } from "@/components/ui/Table"; + +/** Operational view for the append-only event ledger and source freshness. */ +export function DataHealthClient() { + const t = useTranslations("dataHealth"); + const locale = useLocale(); + const { data, error, isLoading, isValidating, mutate } = useSWR( + "/api/data-health", + jsonFetcher, + { refreshInterval: 30_000, keepPreviousData: true }, + ); + if (isLoading && !data) return ; + if (error && !data) return mutate()} />; + if (!data) return null; + const cards = [ + [t("rawEvents"), data.raw_event_count], + [t("facts"), data.fact_count], + [t("retractions"), data.retraction_count], + [t("lastAccepted"), data.latest_accepted_at ? fmtRelative(data.latest_accepted_at, Date.now(), locale) : "—"], + ] as const; + return
} />
{cards.map(([label, value]) =>
{label}
{value}
)}
{data.sources.length === 0 ? {t("empty")} :
{data.sources.map((source) => )}
{t("source")}{t("count")}{t("versions")}{t("latest")}
{source.source}{source.raw_event_count}{source.max_version}{source.latest_accepted_at ? fmtRelative(source.latest_accepted_at, Date.now(), locale) : "—"}
}
; +} diff --git a/apps/dashboard/src/components/evolution/EvolutionCampaignDetailClient.tsx b/apps/dashboard/src/components/evolution/EvolutionCampaignDetailClient.tsx new file mode 100644 index 00000000..d0da3b9f --- /dev/null +++ b/apps/dashboard/src/components/evolution/EvolutionCampaignDetailClient.tsx @@ -0,0 +1,98 @@ +"use client"; + +import { useMemo, useState } from "react"; +import { useLocale, useTranslations } from "next-intl"; +import { toast } from "sonner"; +import useSWR from "swr"; + +import { Link } from "@/i18n/navigation"; +import { jsonFetcher } from "@/lib/fetcher"; +import { fmtRelative } from "@/lib/format"; +import type { EvolutionCampaignDetailPayload, EvolutionImplementation } from "@/lib/types"; +import { ErrorState, SkeletonBlock } from "@/components/ui/Feedback"; +import { LiveStrip } from "@/components/ui/LiveStrip"; +import { PageHeader } from "@/components/ui/PageHeader"; +import { Panel } from "@/components/ui/Panel"; +import { StatusBadge } from "@/components/ui/StatusBadge"; + +const ACTIVE = new Set(["draft", "replaying", "candidate_locked", "waiting_forward", "holdout_ready"]); + +/** Campaign workbench: lineage, generations, ablations, Forward and one-shot holdout. */ +export function EvolutionCampaignDetailClient({ campaignId }: { campaignId: string }) { + const t = useTranslations("evolution.campaign"); + const locale = useLocale(); + const [adopting, setAdopting] = useState(false); + const { data, error, isLoading, isValidating, mutate } = useSWR( + `/api/evolution/campaigns/${campaignId}`, + jsonFetcher, + { refreshInterval: (value) => value && ACTIVE.has(value.campaign.status) ? 4_000 : 0, keepPreviousData: true }, + ); + const campaign = data?.campaign; + const implementations = useMemo( + () => [...(campaign?.implementations ?? [])].sort((a, b) => b.generation - a.generation || (b.fitness ?? -Infinity) - (a.fitness ?? -Infinity)), + [campaign?.implementations], + ); + if (isLoading && !campaign) return ; + if (error && !campaign) return mutate()} />; + if (!campaign) return null; + + async function adopt() { + setAdopting(true); + try { + const response = await fetch(`/api/evolution/campaigns/${campaignId}`, { method: "POST" }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + toast.success(t("adopted")); + await mutate(); + } catch (cause) { + toast.error(cause instanceof Error ? cause.message : String(cause)); + } finally { + setAdopting(false); + } + } + + const snapshot = objectValue(campaign.frozen_config.event_snapshot); + return ( +
+
← {t("back")}
+ } /> + + }> +
+ {Array.from({ length: campaign.max_generations }, (_, index) => index + 1).map((generation) => { + const item = campaign.generations.find((value) => value.generation === generation); + return
G{generation}
{item?.hypothesis_count ?? 0} × 3
novelty {formatMetric(item?.best_novelty)}
; + })} +
+
+ +
+
+
{campaign.status === "graduated" &&

{t("adoptHint")}

}
+
+ + +
+ {campaign.hypotheses.map((hypothesis) =>
G{hypothesis.generation} · {hypothesis.lane} · {hypothesis.lineage_kind}
{hypothesis.selected && }

{String(hypothesis.spec.thesis ?? "—")}

{arrayValue(hypothesis.spec.event_types).map((value) => {value})}
credit {formatMetric(hypothesis.upper_credit)} · novelty {formatMetric(hypothesis.novelty_score)} · parents {hypothesis.parent_ids.map((value) => value.slice(0, 6)).join(", ") || "root"}
)} +
+
+ + +
{implementations.map((item) => )}
Gprofileoutcomefitnessevent advantageevidenceFDR
+
+ + {(campaign.failure_code || campaign.failure_message) &&
{campaign.failure_code}: {campaign.failure_message}
} +
+ ); +} + +/** Render one lower-level implementation without exposing source code in the aggregate view. */ +function ImplementationRow({ item }: { item: EvolutionImplementation }) { + const eventMetrics = objectValue(item.event_metrics); + return {item.generation}{item.profile}{formatMetric(item.fitness)}{formatMetric(numberValue(eventMetrics?.event_advantage_pct))}{formatMetric(item.evidence_quality)}{item.fdr_pass === null ? "—" : item.fdr_pass ? "pass" : "fail"}; +} + +function Metric({ label, value }: { label: string; value: string }) { return
{label}
{value}
; } +function objectValue(value: unknown): Record | null { return typeof value === "object" && value !== null && !Array.isArray(value) ? value as Record : null; } +function arrayValue(value: unknown): string[] { return Array.isArray(value) ? value.map(String) : []; } +function numberValue(value: unknown): number | null { return typeof value === "number" ? value : null; } +function formatMetric(value: number | null | undefined): string { return value === null || value === undefined ? "—" : value.toFixed(3); } diff --git a/apps/dashboard/src/components/evolution/EvolutionCampaignTable.tsx b/apps/dashboard/src/components/evolution/EvolutionCampaignTable.tsx new file mode 100644 index 00000000..0283d9ba --- /dev/null +++ b/apps/dashboard/src/components/evolution/EvolutionCampaignTable.tsx @@ -0,0 +1,51 @@ +"use client"; + +import { useLocale, useTranslations } from "next-intl"; + +import { Link } from "@/i18n/navigation"; +import { fmtRelative } from "@/lib/format"; +import type { EvolutionCampaign } from "@/lib/types"; +import { Panel } from "@/components/ui/Panel"; +import { StatusBadge } from "@/components/ui/StatusBadge"; +import { Td, TableEmpty, TableHeadRow, Th } from "@/components/ui/Table"; + +/** Compact campaign read model for the primary evolution workspace. */ +export function EvolutionCampaignTable({ campaigns }: { campaigns: EvolutionCampaign[] }) { + const t = useTranslations("evolution.campaign"); + const locale = useLocale(); + return ( + {campaigns.length}}> + {campaigns.length === 0 ? ( + {t("empty")} + ) : ( +
+ + + + {campaigns.map((campaign) => ( + + + + + + + + + + ))} + +
{t("id")}{t("status")}{t("stage")}{t("budget")}{t("events")}{t("holdout")}{t("updated")}
{campaign.campaign_id.slice(0, 8)}{campaign.active_generation}/{campaign.max_generations}{campaign.hypothesis_budget * campaign.implementations_per_hypothesis}{campaign.forward_event_count}{campaign.holdout_consumed_at ? t("consumed") : t("sealed")}{fmtRelative(campaign.updated_at, Date.now(), locale)}
+
+ )} +
+ ); +} + +/** Map campaign state to one consistent semantic dashboard tone. */ +function campaignTone(status: EvolutionCampaign["status"]) { + if (status === "graduated") return "bull" as const; + if (status === "replaying" || status === "holdout_ready") return "cyan" as const; + if (status === "waiting_forward" || status === "candidate_locked") return "gold" as const; + if (status === "failed" || status === "rejected") return "fox" as const; + return "muted" as const; +} diff --git a/apps/dashboard/src/components/evolution/EvolutionClient.tsx b/apps/dashboard/src/components/evolution/EvolutionClient.tsx index a88a1f29..6ed9f215 100644 --- a/apps/dashboard/src/components/evolution/EvolutionClient.tsx +++ b/apps/dashboard/src/components/evolution/EvolutionClient.tsx @@ -1,7 +1,10 @@ "use client"; import { useTranslations } from "next-intl"; +import useSWR from "swr"; +import { jsonFetcher } from "@/lib/fetcher"; +import type { EvolutionCampaignPayload } from "@/lib/types"; import { isEvolutionActive } from "@/lib/evolution"; import { useEvolutionRuns } from "@/lib/use-evolution-runs"; import { ErrorState, SkeletonBlock } from "@/components/ui/Feedback"; @@ -9,10 +12,16 @@ import { LiveStrip } from "@/components/ui/LiveStrip"; import { PageHeader } from "@/components/ui/PageHeader"; import { EvolutionRunTable } from "./EvolutionRunTable"; import { EvolutionStats } from "./EvolutionStats"; +import { EvolutionCampaignTable } from "./EvolutionCampaignTable"; /** E1 策略演化运行列表。 */ export function EvolutionClient() { const t = useTranslations("evolution"); + const campaigns = useSWR( + "/api/evolution/campaigns", + jsonFetcher, + { refreshInterval: 10_000, keepPreviousData: true }, + ); const { runs, asOf, @@ -35,6 +44,7 @@ export function EvolutionClient() {
} /> sum + run.llm_cost_usd, 0)} rejected={runs.reduce((sum, run) => sum + run.rejected, 0)} /> + void loadMore()} />
); diff --git a/apps/dashboard/src/components/lab/LabClient.tsx b/apps/dashboard/src/components/lab/LabClient.tsx index 8266dd79..c9d71176 100644 --- a/apps/dashboard/src/components/lab/LabClient.tsx +++ b/apps/dashboard/src/components/lab/LabClient.tsx @@ -88,6 +88,10 @@ export function LabClient() { } /> + + {data.experimentalAdoptions.length === 0 ? {t("experimental.empty")} :
{data.experimentalAdoptions.map((adoption) =>
{t("experimental.badge")}runner=false
{adoption.source_hash.slice(0, 16)}
{t("experimental.grade", { grade: adoption.evidence_grade })}
{adoption.campaign_id && {t("experimental.lineage")} →}
)}
} +
+ + + {/* 执行态一排:live runner + 持仓(在跑什么 ↔ 仓位什么样)。 @@ -102,6 +104,18 @@ export function OverviewClient() { ); } +/** Surface event-evolution operations without mixing sandbox state into Runner panels. */ +function EvolutionSummary({ summary }: { summary: OverviewPayload["evolutionSummary"] }) { + const t = useTranslations("overview.evolution"); + const items = [ + [t("active"), summary.activeCampaigns, "text-cyan"], + [t("waiting"), summary.waitingEvents, "text-gold"], + [t("adoption"), summary.readyForAdoption, "text-bull"], + [t("sources"), summary.eventSourceFailures, summary.eventSourceFailures ? "text-fox-red" : "text-bull"], + ] as const; + return
{items.map(([label, value, tone]) =>
{label}
{value}
)}
; +} + function OverviewSkeleton() { return (
diff --git a/apps/dashboard/src/components/shell/ConsoleSidebar.tsx b/apps/dashboard/src/components/shell/ConsoleSidebar.tsx index 13a476f0..03c47954 100644 --- a/apps/dashboard/src/components/shell/ConsoleSidebar.tsx +++ b/apps/dashboard/src/components/shell/ConsoleSidebar.tsx @@ -5,6 +5,7 @@ import { createPortal } from "react-dom"; import { useTranslations } from "next-intl"; import { Activity, + Database, FlaskConical, Key, LayoutDashboard, @@ -59,6 +60,7 @@ const NAV: NavItem[] = [ { key: "risk", href: "/risk", icon: ShieldAlert }, // E2 策略演化引擎(LLM 自动变异 + 评估) { key: "evolution", href: "/evolution", icon: Workflow }, + { key: "dataHealth", href: "/data-health", icon: Database }, // 玄学彩蛋占卜台(纯娱乐) { key: "divination", href: "/divination", icon: Sparkles }, { key: "activity", href: "/activity", icon: Activity }, diff --git a/apps/dashboard/src/lib/types.ts b/apps/dashboard/src/lib/types.ts index 2e8f5f54..6975bd86 100644 --- a/apps/dashboard/src/lib/types.ts +++ b/apps/dashboard/src/lib/types.ts @@ -173,6 +173,12 @@ export interface OverviewPayload { candidateCounts: { all: number; promoted: number; candidate: number }; /** orders 命中上限被截断(还有更早的订单未显示) —— UI 给「仅显示最近 N」提示,不静默。 */ ordersTruncated: boolean; + evolutionSummary: { + activeCampaigns: number; + waitingEvents: number; + readyForAdoption: number; + eventSourceFailures: number; + }; /** server 侧采集这一帧的时刻(ISO);UI 显示 "数据时间"。 */ asOf: string; } @@ -253,9 +259,24 @@ export interface LabPayload { counts: { all: number; promoted: number; candidate: number; rejected: number }; /** candidates 命中上限被截断(还有更多候选未显示) —— UI 给截断提示,不静默。 */ truncated: boolean; + experimentalAdoptions: ExperimentalStrategyAdoption[]; asOf: string; } +export interface ExperimentalStrategyAdoption { + adoption_id: string; + artifact_id: string; + campaign_id: string | null; + evidence_grade: "standard" | "limited"; + status: "experimental" | "accepted" | "rejected"; + runner_eligible: false; + evidence: Record; + adopted_at: string; + source_hash: string; + compiler_version: string | null; + campaign_status: string | null; +} + /** 该候选最近一次回测的概要(回测时间 / 区间)。 */ export interface BacktestRunSummary { runId: string; @@ -473,6 +494,7 @@ export type ActivityKind = | "order" | "backtest" | "runner" + | "evolution" | "conversation"; export type ActivityTone = "bull" | "fox" | "gold" | "cyan" | "muted"; @@ -622,3 +644,113 @@ export interface EvolutionCandidateDetailPayload { candidate: EvolutionCandidateSummary; asOf: string; } + +// ── E2 事件驱动 Campaign ── + +export type EvolutionCampaignStatus = + | "draft" + | "replaying" + | "candidate_locked" + | "waiting_forward" + | "holdout_ready" + | "graduated" + | "rejected" + | "insufficient_evidence" + | "failed" + | "aborted"; + +export interface EvolutionGenerationProjection { + generation: number; + hypothesis_count: number; + selected_count: number; + best_credit: number | null; + best_novelty: number | null; +} + +export interface EvolutionHypothesis { + hypothesis_id: string; + campaign_id: string; + generation: number; + slot: number; + lineage_kind: string; + lane: string; + parent_ids: string[]; + spec: Record; + spec_hash: string; + upper_credit: number | null; + novelty_score: number | null; + pareto_rank: number | null; + selected: boolean; + created_at: string; +} + +export interface EvolutionImplementation { + implementation_id: string; + campaign_id: string; + hypothesis_id: string; + generation: number; + profile: string; + source_hash: string; + outcome: string; + fitness: number | null; + validation_metrics: Record | null; + event_metrics: Record | null; + evidence_quality: number | null; + novelty_score: number | null; + fdr_pass: boolean | null; + error_code: string | null; + error_message: string | null; + created_at: string; + updated_at: string; +} + +export interface EvolutionCampaign { + campaign_id: string; + status: EvolutionCampaignStatus; + active_generation: number; + hypothesis_budget: number; + implementations_per_hypothesis: number; + max_generations: number; + event_snapshot_id: string; + frozen_config: Record; + locked_candidate_id: string | null; + holdout_consumed_at: string | null; + forward_started_at: string | null; + forward_deadline_at: string | null; + forward_event_count: number; + forward_metrics: Record | null; + failure_code: string | null; + failure_message: string | null; + state_version: number; + created_at: string; + updated_at: string; + finished_at: string | null; + generations: EvolutionGenerationProjection[]; + hypotheses: EvolutionHypothesis[]; + implementations: EvolutionImplementation[]; +} + +export interface EvolutionCampaignPayload { + campaigns: EvolutionCampaign[]; + asOf: string; +} + +export interface EvolutionCampaignDetailPayload { + campaign: EvolutionCampaign; + asOf: string; +} + +export interface EventDataCoverage { + as_of: string; + sources: Array<{ + source: string; + raw_event_count: number; + retractions: number; + latest_accepted_at: string | null; + max_version: number; + }>; + raw_event_count: number; + fact_count: number; + retraction_count: number; + latest_accepted_at: string | null; +} diff --git a/docs/00-context.md b/docs/00-context.md index 0639c11f..c750eac1 100644 --- a/docs/00-context.md +++ b/docs/00-context.md @@ -51,8 +51,9 @@ | D-11.1 / .2 | live runner 信任边界加固 + 运维收口(PnL 净口径 / TTL / build 退避) | ✅ | | D-12 | 因子血缘 + 衰减巡检 + monthly 宏观 + 因子发现 L1;research-hub 三方辩论收口 | ✅ | | E1 生产闭环 | 独立 Evolver:真实 frozen bars、单代 unified-diff 变异、显式审批、owner 隔离、异步持久化与可复现实验元数据 | ✅ | -| E1 收口 | 冻结 LLM/定价快照、owner key 即时获取、token/cost 审计 | 🚧 当前分支 | -| 下一 | E2 best-parent 多代选择 + early stopping(#7);MAP-Elites / Island Model 后置 | 🔲 | +| E1 收口 | 冻结 LLM/定价快照、owner key 即时获取、token/cost 审计 | ✅ | +| E2 事件共演化 | 双时态事件 snapshot、HypothesisSpec、五代 8×3 搜索、Forward/一次性 holdout、实验性采用 | ✅ feature flag | +| 下一 | 小流量校准事件覆盖与信用分配;MAP-Elites / Island Model 后置 | 🔲 | ## 不做的事(边界) diff --git a/docs/04-current-state.md b/docs/04-current-state.md index cec89f60..cacb1922 100644 --- a/docs/04-current-state.md +++ b/docs/04-current-state.md @@ -1,10 +1,11 @@ -# 04 · 当前状态:D-12 + E1 策略演化生产闭环 +# 04 · 当前状态:D-12 + E2 事件驱动自动演化 -> 状态:**D-12 因子库闭环 + E1 独立 Evolver 已落地(更新至 2026-08-27)**——因子血缘 + 衰减巡检 + monthly +> 状态:**D-12 因子库闭环 + E2 事件驱动 Evolver 已落地(更新至 2026-08-28)**——因子血缘 + 衰减巡检 + monthly > 宏观 + 因子发现 L1,在 D-11(多市场模拟盘)/ D-10(web 搜索 + 财报基本面 + > 多市场数据)/ D-9(Plan/Exec 闭环 + LLM 自创策略 + 风控引擎)/ D-9.1a 基础上落地。 -> research-hub(issue #6)已于 2026-06-12 收口;E1 生产代码由 PR #159 合入 main, -> 当前分支继续冻结 LLM/定价审批快照与费用审计。下一里程碑:E2 best-parent 多代演化(issue #7)。 +> research-hub(issue #6)已于 2026-06-12 收口;E1 生产代码由 PR #159 合入 main; +> E2 在 `EVENT_EVOLUTION_ENABLED` 后提供事件账本、确定性 DSL、五代共演化和实验性采用, +> 原 E1 run API 与逐次审批语义保持不变。 > > 本文回答的问题:**clone 仓库后,"现在到底做到哪里、决策链路长什么样"。** > 详细架构与设计取舍见 [`docs/03-kernel-design.md`](./03-kernel-design.md); @@ -81,8 +82,8 @@ sequenceDiagram | paper 内核 | `services/paper/src/inalpha_paper/kernel/` · `execution/` | `clock.py` · `msgbus.py` · `risk_engine.py` · `execution_engine.py` · `order_executor.py` · `gateway.py` | | data 多市场服务 | `services/data/` | CCXT / akshare / yfinance / FRED / web → Postgres + TimescaleDB | | research + factor | `services/research/` · `services/factor/` | 三方研究辩论;因子血缘、IC、衰减与发现工作流 | -| E1 Evolver | `services/evolver/` | frozen bars + unified diff + owner-scoped 异步 run/slot + PostgreSQL 候选审计 | -| 认证控制台 | `apps/dashboard/` | 登录/session、注册 waitlist、admin 审核、逐用户 LLM 配置、演化/回测/runner/因子/风控看板 | +| E1/E2 Evolver | `services/evolver/` | E1 frozen bars + unified diff;E2 事件 DSL + 五代 campaign + Forward/Holdout 门禁 | +| 认证控制台 | `apps/dashboard/` | 登录/session、注册 waitlist、admin 审核、agent 对话、逐用户 LLM 配置、演化/回测/runner/因子/风控看板 | ### 注册与试用准入(2026-08-28) @@ -236,6 +237,22 @@ seed / buy-and-hold / 全候选同数据哈希评估、owner 隔离、数据库 `EVOLVER_LLM_TIMEOUT_S`、凭据签名公私钥已进入环境模板和生产 Compose;Dashboard 暂时 不可用或返回 5xx 时 run 回到队列重试,不会在依赖启动窗口直接进入失败终态。 +### E2 事件驱动自动演化(feature flag) + +- Data 新增 append-only 双时态事件账本、事实版本与 cutoff snapshot;CoinMarketCal + Professional 可导入历史首次添加时间,精选 crypto 新闻仅从首次抓取开始 forward 归档。 +- Paper 明确 `bar_open_at` / `bar_known_at`,在同一决策点按稳定顺序先发布规范事件、再发布 + 已关闭 bar;事件订单最早下一根开盘成交,并使用上一根成交量容量、此前 ATR 与严重度的 + `event-fill-v1` adverse slippage。空事件输入保持 E1 路径不变。 +- Evolver 新增 `HypothesisSpec` DSL、direct/confirmed/hybrid 消融、匹配无事件窗口、block + bootstrap + BH-FDR、2 elite + 4 mutation + 1 crossover + 1 restart 的五代状态机。每代固定 + 两个 owner-scoped proposer 调用;只接收 DSL,凭据仍通过短时 owner grant 即时解析。 +- Champion 锁定后进入独立 Forward 状态,30 天/3 个独立事件/90 天上限;Forward 通过才可 + 一次性消费 holdout。赢家只能以 `runner_eligible=false` 的实验性 adoption 进入策略实验室, + 不会自动 promote、启动 Runner 或下单。 +- Dashboard 已增加 Campaign 工作台、代际/血缘/消融/Forward/Holdout 视图、首页摘要、实验性 + 策略标签、过滤后的 Agent Activity 和事件数据健康页。通过 `EVENT_EVOLUTION_ENABLED` 分阶段开放。 + --- ## D-10(2026-06-01)web 搜索 + 财报基本面 + 多市场数据源扩展 @@ -547,8 +564,8 @@ spot 仍严格 long-only(裸空 / 超卖翻空被守门拒),做空 / 杠 > 重心:模拟盘(paper)先于实盘(live)。 -- **E2 多代演化**(issue #7):在已完成的 E1 单代生产闭环上增加 best-parent 多代选择与 - early stopping;MAP-Elites / Island Model 后置到真实运行数据证明需要时再做 +- **E2 运行观测**:在 feature flag 小流量 campaign 上校准事件覆盖、费用、FDR 与 Forward + 样本积累;MAP-Elites / Island Model 继续后置到真实运行数据证明需要时再做 - **delegation hop**(issue #5 · ADR-0012 补丁):sub-strategy 派生计划的转授权链 > 已收口:paper live runner(#1,D-11)、PnL 净口径 / 运行时长 TTL / build 退避(#45 / diff --git a/infra/docker-compose.prod.yml b/infra/docker-compose.prod.yml index 5afb6b59..5d8abe61 100644 --- a/infra/docker-compose.prod.yml +++ b/infra/docker-compose.prod.yml @@ -170,6 +170,9 @@ services: DATA_SERVICE_URL: http://data:8001 DASHBOARD_SERVICE_URL: http://dashboard:3001 EVOLVER_LLM_TIMEOUT_S: ${EVOLVER_LLM_TIMEOUT_S:-120} + EVENT_EVOLUTION_ENABLED: ${EVENT_EVOLUTION_ENABLED:-false} + CAMPAIGN_LEASE_TTL_S: ${CAMPAIGN_LEASE_TTL_S:-90} + CAMPAIGN_MAX_CONCURRENT: ${CAMPAIGN_MAX_CONCURRENT:-1} # 签名私钥只属于 orchestration;覆盖 svc-common env_file,防 Evolver 自签 owner grant。 EVOLUTION_CREDENTIAL_PRIVATE_KEY_B64: "" EVOLUTION_CREDENTIAL_PUBLIC_KEY_B64: ${EVOLUTION_CREDENTIAL_PUBLIC_KEY_B64:?required} diff --git a/infra/migrations/tests/test_migration_0043.py b/infra/migrations/tests/test_migration_0043.py new file mode 100644 index 00000000..f302b8d6 --- /dev/null +++ b/infra/migrations/tests/test_migration_0043.py @@ -0,0 +1,116 @@ +"""0043 point-in-time event ledger and owner campaign integration tests.""" + +from __future__ import annotations + +import psycopg +import pytest +from migration_0038_support import alembic, db_url + +pytestmark = pytest.mark.integration + + +def test_0043_builds_event_campaign_schema_and_reversible_grant_scope( + migration_db_url: str, +) -> None: + """Exercise the critical foreign keys, one-shot records, and rolling downgrade.""" + alembic(migration_db_url, "upgrade", "0042") + alembic(migration_db_url, "upgrade", "0043") + + with psycopg.connect(db_url(migration_db_url), autocommit=True) as conn: + conn.execute( + """INSERT INTO evolution_credential_grant_uses + (jti,owner_sub,config_id,operation_id,config_digest,request_digest,grant_purpose) + VALUES ('44444444-4444-4444-8444-444444444443','user:alice','config-1', + 'campaign-operation',%s,%s,'event_campaign')""", + ("a" * 64, "b" * 64), + ) + raw_id = conn.execute( + """INSERT INTO raw_market_events( + source,source_event_id,version,title,content_hash,first_seen_at,fetched_at, + accepted_at,collector_version,policy_version,source_tier) + VALUES('coinmarketcal','event-1',1,'listing',%s,NOW(),NOW(),NOW(), + 'collector-v1','event-time-v1','structured') RETURNING event_id""", + ("c" * 64,), + ).fetchone()[0] + fact_id = conn.execute( + """INSERT INTO market_event_facts( + raw_event_id,fact_key,version,fact_hash,event_type,assets,action,severity, + confidence,effective_at,available_at,extractor_version,policy_version) + VALUES(%s,'listing:btc',1,%s,'listing',ARRAY['BTC'],'listed',0.8,0.9, + NOW(),NOW(),'extractor-v1','event-time-v1') RETURNING fact_id""", + (raw_id, "d" * 64), + ).fetchone()[0] + snapshot_id = conn.execute( + """INSERT INTO market_event_snapshots( + cutoff,policy_version,query_hash,events_sha256,fact_count) + VALUES(NOW(),'event-time-v1',%s,%s,1) RETURNING snapshot_id""", + ("e" * 64, "f" * 64), + ).fetchone()[0] + conn.execute( + """INSERT INTO market_event_snapshot_facts(snapshot_id,fact_id,ordinal) + VALUES(%s,%s,0)""", + (snapshot_id, fact_id), + ) + campaign_id = conn.execute( + """INSERT INTO evolution_campaigns( + owner_account_id,requested_by_sub,idempotency_key,request_hash,event_snapshot_id, + frozen_config,llm_snapshot,llm_config_digest,llm_credential_grant) + VALUES('00000000-0000-0000-0000-000000000099','user:alice','campaign-1',%s,%s, + '{}','{}',%s,%s) RETURNING campaign_id""", + ("1" * 64, snapshot_id, "2" * 64, "grant-" + "x" * 120), + ).fetchone()[0] + hypothesis_id = conn.execute( + """INSERT INTO evolution_hypotheses( + campaign_id,generation,slot,lineage_kind,lane,spec,spec_hash) + VALUES(%s,1,0,'seed','event','{}',%s) RETURNING hypothesis_id""", + (campaign_id, "3" * 64), + ).fetchone()[0] + implementation_id = conn.execute( + """INSERT INTO evolution_implementations( + campaign_id,hypothesis_id,generation,profile,source_code,source_hash) + VALUES(%s,%s,1,'direct','pass',%s) RETURNING implementation_id""", + (campaign_id, hypothesis_id, "4" * 64), + ).fetchone()[0] + assert conn.execute( + """SELECT i.implementation_id + FROM evolution_implementations i + JOIN evolution_hypotheses h ON h.hypothesis_id=i.hypothesis_id + JOIN evolution_campaigns c ON c.campaign_id=i.campaign_id + WHERE i.implementation_id=%s AND c.event_snapshot_id=%s""", + (implementation_id, snapshot_id), + ).fetchone() == (implementation_id,) + with pytest.raises(psycopg.errors.CheckViolation): + conn.execute( + """UPDATE evolution_credential_grant_uses + SET redemption_count=9 WHERE jti='44444444-4444-4444-8444-444444444443'""" + ) + + blocked = alembic(migration_db_url, "downgrade", "0042", check=False) + assert blocked.returncode != 0 + assert "cannot downgrade 0043 while E2 event or campaign records exist" in ( + blocked.stderr + ) + + with psycopg.connect(db_url(migration_db_url), autocommit=True) as conn: + conn.execute("DELETE FROM evolution_campaigns") + conn.execute("DELETE FROM market_event_snapshot_facts") + conn.execute("DELETE FROM market_event_snapshots") + conn.execute("DELETE FROM market_event_facts") + conn.execute("DELETE FROM raw_market_events") + conn.execute( + "DELETE FROM evolution_credential_grant_uses WHERE grant_purpose='event_campaign'" + ) + + alembic(migration_db_url, "downgrade", "0042") + with psycopg.connect(db_url(migration_db_url), autocommit=True) as conn: + assert conn.execute("SELECT to_regclass('evolution_campaigns')").fetchone() == ( + None, + ) + assert ( + conn.execute( + """SELECT 1 FROM information_schema.columns + WHERE table_name='evolution_credential_grant_uses' + AND column_name='grant_purpose'""" + ).fetchone() + is None + ) diff --git a/infra/migrations/tests/test_migration_0044.py b/infra/migrations/tests/test_migration_0044.py new file mode 100644 index 00000000..a8b1a57f --- /dev/null +++ b/infra/migrations/tests/test_migration_0044.py @@ -0,0 +1,28 @@ +"""0044 migration-head merge integration test.""" + +from __future__ import annotations + +import psycopg +from migration_0038_support import alembic, db_url + + +def test_0044_restores_a_single_upgrade_head(migration_db_url: str) -> None: + """Upgrade both 0043 branches through one deterministic head.""" + alembic(migration_db_url, "upgrade", "head") + + current = alembic(migration_db_url, "current") + assert "0044 (head)" in current.stdout + assert "0043_waitlist (head)" not in current.stdout + + alembic(migration_db_url, "downgrade", "0042") + with psycopg.connect(db_url(migration_db_url), autocommit=True) as conn: + assert conn.execute("SELECT to_regclass('evolution_campaigns')").fetchone() == ( + None, + ) + assert ( + conn.execute( + """SELECT 1 FROM information_schema.columns + WHERE table_name='users' AND column_name='access_status'""" + ).fetchone() + is None + ) diff --git a/infra/migrations/versions/0043_event_evolution_foundation.py b/infra/migrations/versions/0043_event_evolution_foundation.py new file mode 100644 index 00000000..50823464 --- /dev/null +++ b/infra/migrations/versions/0043_event_evolution_foundation.py @@ -0,0 +1,271 @@ +"""Add the point-in-time market event ledger and E2 campaign records.""" + +from __future__ import annotations + +from alembic import op + +revision: str = "0043" +down_revision: str | None = "0042" +branch_labels: str | tuple[str, ...] | None = None +depends_on: str | tuple[str, ...] | None = None + + +def upgrade() -> None: + """Create immutable event versions, snapshots, and owner-scoped campaigns.""" + op.execute( + """SET LOCAL lock_timeout = '10s'; +ALTER TABLE evolution_credential_grant_uses +ADD COLUMN grant_purpose TEXT NOT NULL DEFAULT 'e1_run' +CHECK (grant_purpose IN ('e1_run','event_campaign')), +DROP CONSTRAINT evolution_credential_grant_uses_redemption_count_check, +ADD CONSTRAINT evolution_credential_grant_uses_redemption_count_check +CHECK (redemption_count BETWEEN 1 AND 8); +CREATE TABLE raw_market_events ( + event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + source TEXT NOT NULL, + source_event_id TEXT NOT NULL, + version INTEGER NOT NULL CHECK (version > 0), + title TEXT NOT NULL DEFAULT '', + content TEXT NOT NULL DEFAULT '', + url TEXT, + content_hash TEXT NOT NULL CHECK (content_hash ~ '^[0-9a-f]{64}$'), + raw_payload JSONB NOT NULL DEFAULT '{}'::jsonb, + source_valid_at TIMESTAMPTZ, + claimed_published_at TIMESTAMPTZ, + first_seen_at TIMESTAMPTZ NOT NULL, + fetched_at TIMESTAMPTZ NOT NULL, + accepted_at TIMESTAMPTZ NOT NULL, + collector_version TEXT NOT NULL, + policy_version TEXT NOT NULL, + source_tier TEXT NOT NULL CHECK (source_tier IN ('official','professional_media','aggregator','structured')), + supersedes_event_id UUID REFERENCES raw_market_events(event_id), + retracted BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(source, source_event_id, version), + UNIQUE(source, source_event_id, content_hash) +); +CREATE INDEX ix_raw_market_events_lookup +ON raw_market_events(source, source_event_id, version DESC); +CREATE INDEX ix_raw_market_events_first_seen +ON raw_market_events(first_seen_at, event_id); + +CREATE TABLE market_event_facts ( + fact_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + raw_event_id UUID NOT NULL REFERENCES raw_market_events(event_id), + fact_key TEXT NOT NULL, + version INTEGER NOT NULL CHECK (version > 0), + fact_hash TEXT NOT NULL CHECK (fact_hash ~ '^[0-9a-f]{64}$'), + event_type TEXT NOT NULL CHECK (event_type IN ('listing','delisting','exploit','chain_halt','regulatory','upgrade','unlock','burn','partnership','macro','other')), + assets TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + actor TEXT, + action TEXT NOT NULL, + severity DOUBLE PRECISION NOT NULL CHECK (severity BETWEEN 0 AND 1), + confidence DOUBLE PRECISION NOT NULL CHECK (confidence BETWEEN 0 AND 1), + effective_at TIMESTAMPTZ NOT NULL, + available_at TIMESTAMPTZ NOT NULL, + evidence_spans JSONB NOT NULL DEFAULT '[]'::jsonb, + extractor_version TEXT NOT NULL, + policy_version TEXT NOT NULL, + supersedes_fact_id UUID REFERENCES market_event_facts(fact_id), + retracted BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(raw_event_id, fact_key, version), + UNIQUE(raw_event_id, fact_key, fact_hash) +); +CREATE INDEX ix_market_event_facts_available +ON market_event_facts(available_at, fact_id); +CREATE INDEX ix_market_event_facts_assets +ON market_event_facts USING GIN(assets); + +CREATE TABLE market_event_snapshots ( + snapshot_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + cutoff TIMESTAMPTZ NOT NULL, + policy_version TEXT NOT NULL, + query_hash TEXT NOT NULL CHECK (query_hash ~ '^[0-9a-f]{64}$'), + events_sha256 TEXT NOT NULL CHECK (events_sha256 ~ '^[0-9a-f]{64}$'), + coverage JSONB NOT NULL DEFAULT '{}'::jsonb, + event_types TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + assets TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + fact_count INTEGER NOT NULL CHECK (fact_count >= 0), + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(query_hash, events_sha256) +); +CREATE TABLE market_event_snapshot_facts ( + snapshot_id UUID NOT NULL REFERENCES market_event_snapshots(snapshot_id) ON DELETE CASCADE, + fact_id UUID NOT NULL REFERENCES market_event_facts(fact_id), + ordinal INTEGER NOT NULL CHECK (ordinal >= 0), + PRIMARY KEY(snapshot_id, fact_id), + UNIQUE(snapshot_id, ordinal) +); + +CREATE TABLE evolution_campaigns ( + campaign_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + owner_account_id UUID NOT NULL, + requested_by_sub TEXT NOT NULL, + idempotency_key TEXT NOT NULL, + request_hash TEXT NOT NULL CHECK (request_hash ~ '^[0-9a-f]{64}$'), + source_run_id UUID REFERENCES strategy_evo_runs(run_id), + status TEXT NOT NULL DEFAULT 'draft' CHECK (status IN ( + 'draft','replaying','candidate_locked','waiting_forward','holdout_ready', + 'graduated','rejected','insufficient_evidence','failed','aborted' + )), + active_generation INTEGER NOT NULL DEFAULT 0 CHECK (active_generation BETWEEN 0 AND 5), + hypothesis_budget INTEGER NOT NULL DEFAULT 8 CHECK (hypothesis_budget BETWEEN 1 AND 8), + implementations_per_hypothesis INTEGER NOT NULL DEFAULT 3 CHECK (implementations_per_hypothesis = 3), + max_generations INTEGER NOT NULL DEFAULT 5 CHECK (max_generations BETWEEN 1 AND 5), + event_snapshot_id UUID NOT NULL REFERENCES market_event_snapshots(snapshot_id), + frozen_config JSONB NOT NULL, + llm_snapshot JSONB NOT NULL, + llm_config_digest TEXT NOT NULL CHECK (llm_config_digest ~ '^[0-9a-f]{64}$'), + llm_credential_grant TEXT CHECK (llm_credential_grant IS NULL OR length(llm_credential_grant) >= 100), + llm_cost_usd DOUBLE PRECISION NOT NULL DEFAULT 0 CHECK (llm_cost_usd >= 0), + locked_candidate_id UUID, + holdout_consumed_at TIMESTAMPTZ, + forward_started_at TIMESTAMPTZ, + forward_deadline_at TIMESTAMPTZ, + forward_event_count INTEGER NOT NULL DEFAULT 0 CHECK (forward_event_count >= 0), + forward_metrics JSONB, + failure_code TEXT, + failure_message TEXT, + lease_owner TEXT, + lease_token UUID, + lease_expires_at TIMESTAMPTZ, + state_version BIGINT NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + finished_at TIMESTAMPTZ, + CHECK ((status <> 'waiting_forward') OR (locked_candidate_id IS NOT NULL AND forward_started_at IS NOT NULL AND forward_deadline_at IS NOT NULL)), + CHECK ((holdout_consumed_at IS NULL) OR locked_candidate_id IS NOT NULL) + ,UNIQUE(owner_account_id,idempotency_key) +); +CREATE INDEX ix_evolution_campaigns_owner_created +ON evolution_campaigns(owner_account_id, created_at DESC, campaign_id DESC); +CREATE INDEX ix_evolution_campaigns_dispatch +ON evolution_campaigns(status, lease_expires_at, created_at) +WHERE status IN ('draft','replaying','holdout_ready'); + +CREATE TABLE evolution_hypotheses ( + hypothesis_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + campaign_id UUID NOT NULL REFERENCES evolution_campaigns(campaign_id) ON DELETE CASCADE, + generation INTEGER NOT NULL CHECK (generation BETWEEN 1 AND 5), + slot INTEGER NOT NULL CHECK (slot BETWEEN 0 AND 7), + lineage_kind TEXT NOT NULL CHECK (lineage_kind IN ('seed','elite','mutation','crossover','restart')), + lane TEXT NOT NULL CHECK (lane IN ('event','event_regime','factor','execution_risk','regime','restart')), + parent_ids UUID[] NOT NULL DEFAULT ARRAY[]::UUID[], + spec JSONB NOT NULL, + spec_hash TEXT NOT NULL CHECK (spec_hash ~ '^[0-9a-f]{64}$'), + upper_credit DOUBLE PRECISION, + novelty_score DOUBLE PRECISION, + pareto_rank INTEGER, + selected BOOLEAN NOT NULL DEFAULT FALSE, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(campaign_id, generation, slot), + UNIQUE(campaign_id, generation, spec_hash) +); +CREATE INDEX ix_evolution_hypotheses_generation +ON evolution_hypotheses(campaign_id, generation, slot); + +CREATE TABLE evolution_implementations ( + implementation_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + campaign_id UUID NOT NULL REFERENCES evolution_campaigns(campaign_id) ON DELETE CASCADE, + hypothesis_id UUID NOT NULL REFERENCES evolution_hypotheses(hypothesis_id) ON DELETE CASCADE, + generation INTEGER NOT NULL CHECK (generation BETWEEN 1 AND 5), + profile TEXT NOT NULL CHECK (profile IN ('direct','confirmed','hybrid','conservative','canonical','aggressive')), + source_code TEXT NOT NULL, + source_hash TEXT NOT NULL CHECK (source_hash ~ '^[0-9a-f]{64}$'), + outcome TEXT NOT NULL DEFAULT 'pending' CHECK (outcome IN ('pending','succeeded','rejected','failed')), + fitness DOUBLE PRECISION, + validation_metrics JSONB, + event_metrics JSONB, + evidence_quality DOUBLE PRECISION, + novelty_score DOUBLE PRECISION, + fdr_pass BOOLEAN, + error_code TEXT, + error_message TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(hypothesis_id,profile) +); +CREATE INDEX ix_evolution_implementations_source_cache +ON evolution_implementations(campaign_id,source_hash) +WHERE outcome='succeeded'; +CREATE INDEX ix_evolution_implementations_generation +ON evolution_implementations(campaign_id,generation,hypothesis_id); +ALTER TABLE evolution_campaigns ADD CONSTRAINT evolution_campaign_locked_candidate_fk +FOREIGN KEY(locked_candidate_id) REFERENCES evolution_implementations(implementation_id); + +ALTER TABLE strategy_evo_candidates +ADD COLUMN campaign_id UUID REFERENCES evolution_campaigns(campaign_id), +ADD COLUMN hypothesis_id UUID REFERENCES evolution_hypotheses(hypothesis_id), +ADD COLUMN implementation_profile TEXT CHECK (implementation_profile IN ('direct','confirmed','hybrid','conservative','canonical','aggressive')); +CREATE INDEX ix_strategy_evo_candidates_campaign +ON strategy_evo_candidates(campaign_id,generation,slot) +WHERE campaign_id IS NOT NULL; + +CREATE TABLE strategy_artifacts ( + artifact_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + source_hash TEXT NOT NULL UNIQUE, + source_code TEXT NOT NULL, + dsl_spec JSONB, + compiler_version TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); +CREATE TABLE strategy_adoptions ( + adoption_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + artifact_id UUID NOT NULL REFERENCES strategy_artifacts(artifact_id), + owner_account_id UUID NOT NULL, + campaign_id UUID REFERENCES evolution_campaigns(campaign_id), + evidence_grade TEXT NOT NULL CHECK (evidence_grade IN ('standard','limited')), + status TEXT NOT NULL DEFAULT 'experimental' CHECK (status IN ('experimental','accepted','rejected')), + runner_eligible BOOLEAN NOT NULL DEFAULT FALSE CHECK (runner_eligible = FALSE), + evidence JSONB NOT NULL DEFAULT '{}'::jsonb, + adopted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(owner_account_id, artifact_id) +); +CREATE INDEX ix_strategy_adoptions_owner +ON strategy_adoptions(owner_account_id, adopted_at DESC); +""" + ) + + +def downgrade() -> None: + """Remove E2 records without changing the existing E1 schema.""" + op.execute( + """SET LOCAL lock_timeout = '10s'; +DO $$ +BEGIN + IF EXISTS (SELECT 1 FROM raw_market_events LIMIT 1) + OR EXISTS (SELECT 1 FROM market_event_facts LIMIT 1) + OR EXISTS (SELECT 1 FROM market_event_snapshots LIMIT 1) + OR EXISTS (SELECT 1 FROM evolution_campaigns LIMIT 1) + OR EXISTS (SELECT 1 FROM strategy_artifacts LIMIT 1) + OR EXISTS (SELECT 1 FROM strategy_adoptions LIMIT 1) + OR EXISTS ( + SELECT 1 FROM evolution_credential_grant_uses + WHERE grant_purpose = 'event_campaign' LIMIT 1 + ) THEN + RAISE EXCEPTION + 'cannot downgrade 0043 while E2 event or campaign records exist; archive and remove them first'; + END IF; +END $$; +DROP TABLE strategy_adoptions; +DROP TABLE strategy_artifacts; +ALTER TABLE evolution_campaigns DROP CONSTRAINT evolution_campaign_locked_candidate_fk; +ALTER TABLE strategy_evo_candidates +DROP COLUMN implementation_profile, +DROP COLUMN hypothesis_id, +DROP COLUMN campaign_id; +DROP TABLE evolution_implementations; +DROP TABLE evolution_hypotheses; +DROP TABLE evolution_campaigns; +DROP TABLE market_event_snapshot_facts; +DROP TABLE market_event_snapshots; +DROP TABLE market_event_facts; +DROP TABLE raw_market_events; +ALTER TABLE evolution_credential_grant_uses +DROP CONSTRAINT evolution_credential_grant_uses_redemption_count_check, +ADD CONSTRAINT evolution_credential_grant_uses_redemption_count_check +CHECK (redemption_count BETWEEN 1 AND 2), +DROP COLUMN grant_purpose; +""" + ) diff --git a/infra/migrations/versions/0044_merge_event_waitlist_heads.py b/infra/migrations/versions/0044_merge_event_waitlist_heads.py new file mode 100644 index 00000000..cc31fece --- /dev/null +++ b/infra/migrations/versions/0044_merge_event_waitlist_heads.py @@ -0,0 +1,21 @@ +"""Merge event-evolution and waitlist migration heads. + +Revision ID: 0044 +Revises: 0043, 0043_waitlist +Create Date: 2026-08-28 +""" + +from __future__ import annotations + +revision: str = "0044" +down_revision: tuple[str, str] = ("0043", "0043_waitlist") +branch_labels: str | tuple[str, ...] | None = None +depends_on: str | tuple[str, ...] | None = None + + +def upgrade() -> None: + pass + + +def downgrade() -> None: + pass diff --git a/infra/migrations/versions/0045_e2_campaign_approval_operations.py b/infra/migrations/versions/0045_e2_campaign_approval_operations.py new file mode 100644 index 00000000..da5540d3 --- /dev/null +++ b/infra/migrations/versions/0045_e2_campaign_approval_operations.py @@ -0,0 +1,36 @@ +"""Allow bounded E2 campaign approvals in the durable operation ledger.""" + +from __future__ import annotations + +from alembic import op + +revision: str = "0045" +down_revision: str | None = "0044" +branch_labels: str | tuple[str, ...] | None = None +depends_on: str | tuple[str, ...] | None = None + + +def upgrade() -> None: + """Permit both E1 runs and E2 campaigns in the shared approval-operation ledger.""" + op.execute( + """SET LOCAL lock_timeout = '10s'; +ALTER TABLE evolution_approval_operations +DROP CONSTRAINT evolution_approval_operations_tool_name_check; +ALTER TABLE evolution_approval_operations +ADD CONSTRAINT evolution_approval_operations_tool_name_check +CHECK (tool_name IN ('evolver.run_evolution','evolver.run_event_campaign'));""" + ) + + +def downgrade() -> None: + """Remove E2 recovery rows before restoring the E1-only ledger constraint.""" + op.execute( + """SET LOCAL lock_timeout = '10s'; +DELETE FROM evolution_approval_operations +WHERE tool_name='evolver.run_event_campaign'; +ALTER TABLE evolution_approval_operations +DROP CONSTRAINT evolution_approval_operations_tool_name_check; +ALTER TABLE evolution_approval_operations +ADD CONSTRAINT evolution_approval_operations_tool_name_check +CHECK (tool_name='evolver.run_evolution');""" + ) diff --git a/packages/orchestration/config/permissions.default.yaml b/packages/orchestration/config/permissions.default.yaml index 7ae908ae..7dbc42a6 100644 --- a/packages/orchestration/config/permissions.default.yaml +++ b/packages/orchestration/config/permissions.default.yaml @@ -43,6 +43,8 @@ allow: # 演化状态与候选详情只读 - "evolver.get_evolution" - "evolver.get_candidate" + # 事件 campaign 查询只读;启动 campaign 会产生 LLM 费用,必须先显式审批 + - "evolver.get_event_campaign" # Swarm 批量回测(ADR-0025):只读,无下单路径 - "swarm.*" @@ -90,8 +92,9 @@ ask: - "paper.deposit_cash" - "paper.reset_account" - # LLM 变异会产生费用;取消会改变运行状态,均需明确确认 + # LLM 演化会产生费用;E2 一次审批覆盖整个五代 campaign,不逐代重复审批 - "evolver.run_evolution" + - "evolver.run_event_campaign" - "evolver.abort_evolution" deny: diff --git a/packages/orchestration/src/clients/evolver.ts b/packages/orchestration/src/clients/evolver.ts index 9a8d68a3..3647ca90 100644 --- a/packages/orchestration/src/clients/evolver.ts +++ b/packages/orchestration/src/clients/evolver.ts @@ -22,6 +22,106 @@ export type EvolutionStartRequest = { llm: EvolutionLLMSnapshot; }; +export type EventCampaignRequest = { + event_snapshot_id: string; + source_run_id: string | null; + config: { + venue: string; + symbol: string; + timeframe: "15m" | "1h" | "4h"; + from_ts: string; + as_of: string; + initial_cash: number; + fee_rate: number; + trading_mode: "spot" | "perp"; + leverage: number; + discovery_ratio: 0.6; + generation_validation_ratio: 0.2; + sealed_holdout_ratio: 0.2; + execution_model_version: "event-fill-v1"; + control_matcher_version: "event-control-v1"; + random_seed: number; + }; + llm: EvolutionLLMSnapshot; + hypotheses: []; +}; + +export type EventCampaignResult = { + campaign_id: string; + status: string; + active_generation: number; + max_generations: number; + event_snapshot_id: string; + llm_cost_usd: number; +}; + +type EventCampaignConfigBase = Omit< + EventCampaignRequest["config"], + "discovery_ratio" | "generation_validation_ratio" | "sealed_holdout_ratio" | "execution_model_version" | "control_matcher_version" +>; +export type EventCampaignConfigInput = Omit< + EventCampaignConfigBase, + "initial_cash" | "fee_rate" | "trading_mode" | "leverage" | "random_seed" +> & Partial>; + +/** Build the fixed 60/20/20 automatic event campaign request. */ +export function buildEventCampaignRequest(options: { + eventSnapshotId: string; + sourceRunId?: string; + config: EventCampaignConfigInput; + llmSnapshot: EvolutionLLMSnapshot; +}): EventCampaignRequest { + return { + event_snapshot_id: options.eventSnapshotId, + source_run_id: options.sourceRunId ?? null, + config: { + ...options.config, + from_ts: new Date(options.config.from_ts).toISOString(), + as_of: new Date(options.config.as_of).toISOString(), + initial_cash: options.config.initial_cash ?? 10_000, + fee_rate: options.config.fee_rate ?? 0.001, + trading_mode: options.config.trading_mode ?? "perp", + leverage: options.config.leverage ?? 1, + random_seed: options.config.random_seed ?? 0, + discovery_ratio: 0.6, + generation_validation_ratio: 0.2, + sealed_holdout_ratio: 0.2, + execution_model_version: "event-fill-v1", + control_matcher_version: "event-control-v1", + }, + llm: options.llmSnapshot, + hypotheses: [], + }; +} + +/** Match Python's sorted compact JSON digest for the auto-campaign request. */ +export function eventCampaignRequestDigest(request: EventCampaignRequest): string { + const config = request.config; + const hypothesesHash = createHash("sha256").update("[]").digest("hex"); + const canonical = [ + request.event_snapshot_id, + request.source_run_id ?? "", + config.venue, + config.symbol, + config.timeframe, + numberText(Date.parse(config.from_ts)), + numberText(Date.parse(config.as_of)), + float64Hex(config.initial_cash), + float64Hex(config.fee_rate), + config.trading_mode, + numberText(config.leverage), + float64Hex(config.discovery_ratio), + float64Hex(config.generation_validation_ratio), + float64Hex(config.sealed_holdout_ratio), + config.execution_model_version, + config.control_matcher_version, + numberText(config.random_seed), + request.llm.config_digest, + hypothesesHash, + ]; + return createHash("sha256").update(JSON.stringify(canonical)).digest("hex"); +} + /** 将 tool 输入收口为签名与发送共用的唯一请求体。 */ export function buildEvolutionStartRequest(options: { budget?: number; @@ -148,6 +248,42 @@ export class EvolverClient { } } + /** Create and start one automatic five-generation event campaign. */ + async startEventCampaign(options: { + request: EventCampaignRequest; + idempotencyKey: string; + credentialGrant: string; + }): Promise { + const headers = { + "Idempotency-Key": options.idempotencyKey, + "X-Evolution-Credential": options.credentialGrant, + }; + const created = await this.http.post( + "/api/v1/campaigns", + options.request, + headers, + ); + if (created.status !== "draft") return created; + + try { + return await this.http.post( + `/api/v1/campaigns/${created.campaign_id}/start`, + {}, + ); + } catch (error) { + if (!(error instanceof HttpClientError) || error.code !== "CAMPAIGN_STATE_CONFLICT") { + throw error; + } + const current = await this.getEventCampaign(created.campaign_id); + if (current.status === "draft") throw error; + return current; + } + } + + async getEventCampaign(campaignId: string): Promise { + return await this.http.get(`/api/v1/campaigns/${campaignId}`); + } + async listRuns(limit = 20): Promise { return await this.http.get("/api/v1/runs", { limit }); } diff --git a/packages/orchestration/src/hooks/with-hooks.ts b/packages/orchestration/src/hooks/with-hooks.ts index e26d2e99..ca71d470 100644 --- a/packages/orchestration/src/hooks/with-hooks.ts +++ b/packages/orchestration/src/hooks/with-hooks.ts @@ -54,6 +54,12 @@ type GenericTool = { [key: string]: unknown; }; +const DURABLE_EVOLUTION_APPROVAL_TOOLS = new Set([ + "evolver.run_evolution", + "evolver.run_event_campaign", +]); +const E2_CAMPAIGN_RETRY_WINDOW_MS = 2 * 60 * 1_000; + /** * mastra ``server.middleware`` 从 Bearer JWT 解出的已认证主体(sub)写进 RequestContext * 的 key(#91)。getSessionId 最高优先读它 → askCache 按已认证主体 scope(替代 __global__)。 @@ -201,14 +207,14 @@ export function withHooks(tool: T, opts: WithHooksOptions if (permDecision === "ask") { const store = opts.pendingApprovals ?? defaultPendingApprovals; const projectedInput = projectApprovalInput(toolName, effectiveInput); - const llmSnapshot = - toolName === "evolver.run_evolution" - ? getRequestContextValue(ctx, USER_LLM_SNAPSHOT_KEY) - : undefined; + const durableEvolutionApproval = DURABLE_EVOLUTION_APPROVAL_TOOLS.has(toolName); + const llmSnapshot = durableEvolutionApproval + ? getRequestContextValue(ctx, USER_LLM_SNAPSHOT_KEY) + : undefined; const approvalInput = llmSnapshot ? { request: projectedInput, llm_snapshot: llmSnapshot } : projectedInput; - if (!authSub || !sessionId || (toolName === "evolver.run_evolution" && !llmSnapshot)) { + if (!authSub || !sessionId || (durableEvolutionApproval && !llmSnapshot)) { return { isError: true, deniedBy: "permission-ask", @@ -228,6 +234,10 @@ export function withHooks(tool: T, opts: WithHooksOptions toolName, approvalInput, reuseAfterConsume: toolName === "evolver.run_evolution", + reuseOnceAfterConsumeMs: + toolName === "evolver.run_event_campaign" + ? E2_CAMPAIGN_RETRY_WINDOW_MS + : undefined, }); if (!operationId) { const approvalViewInput = llmSnapshot @@ -242,7 +252,7 @@ export function withHooks(tool: T, opts: WithHooksOptions timeoutMs: opts.askTimeoutMs && opts.askTimeoutMs > 0 ? opts.askTimeoutMs - : toolName === "evolver.run_evolution" + : durableEvolutionApproval ? 300_000 : undefined, }); diff --git a/packages/orchestration/src/mastra/llm/evolution-credential-grant.ts b/packages/orchestration/src/mastra/llm/evolution-credential-grant.ts index 97055905..b494afea 100644 --- a/packages/orchestration/src/mastra/llm/evolution-credential-grant.ts +++ b/packages/orchestration/src/mastra/llm/evolution-credential-grant.ts @@ -18,6 +18,7 @@ export async function mintEvolutionCredentialGrant(args: { operationId: string; requestDigest: string; snapshot: EvolutionLLMSnapshot; + purpose?: "e1_run" | "event_campaign"; }): Promise { const encoded = process.env.EVOLUTION_CREDENTIAL_PRIVATE_KEY_B64?.trim(); if (!encoded) { @@ -39,6 +40,7 @@ export async function mintEvolutionCredentialGrant(args: { config_id: args.snapshot.config_id, provider: args.snapshot.provider, operation_id: args.operationId, + grant_purpose: args.purpose ?? "e1_run", request_digest: args.requestDigest, llm_config_digest: args.snapshot.config_digest, }) diff --git a/packages/orchestration/src/permissions/approval-identity.ts b/packages/orchestration/src/permissions/approval-identity.ts index 2d187455..ae006759 100644 --- a/packages/orchestration/src/permissions/approval-identity.ts +++ b/packages/orchestration/src/permissions/approval-identity.ts @@ -58,6 +58,8 @@ export const APPROVAL_IDENTITY_FIELDS: Readonly; + oneShot?: boolean; } const DEFAULT_TIMEOUT_MS = 30_000; @@ -64,13 +67,19 @@ export interface ApprovalPersistence { decision: PendingDecision, via: "user" | "timeout", ): Promise; - rememberEvolutionOperation(args: EvolutionOperationScope & { operationId: string }): Promise<{ + rememberEvolutionOperation( + args: EvolutionOperationScope & { operationId: string; retentionMs?: number }, + ): Promise<{ expiresAt: string; } | undefined>; findEvolutionOperation(args: EvolutionOperationScope): Promise<{ operationId: string; expiresAt: string; } | undefined>; + claimEvolutionOperation?(args: EvolutionOperationScope): Promise<{ + operationId: string; + expiresAt: string; + } | undefined>; } export interface EvolutionOperationScope { @@ -90,6 +99,7 @@ export class PendingApprovalsStore { private readonly records = new Map(); private readonly identityIndex = new Map(); private readonly consumedByIdentity = new Map(); + private readonly consumingByIdentity = new Map>(); private readonly telemetry: PendingTelemetrySink; private readonly persistence?: ApprovalPersistence; @@ -170,36 +180,97 @@ export class PendingApprovalsStore { /** Atomically consumes one approved decision and returns its restart-stable operation ID. */ async consumeApproved(args: PendingConsumeArgs): Promise { const identity = this.identityFor(args); - const consumed = args.reuseAfterConsume ? this.consumedByIdentity.get(identity) : undefined; - if (consumed) { - if (Date.now() >= Date.parse(consumed.expiresAt)) { - this.removeConsumed(identity); - } else { - this.telemetry({ - event: "ask_approval_operation_reused", - requestId: consumed.operationId, - toolName: args.toolName, - sessionId: args.sessionId, - authSub: args.authSub, - ts: new Date().toISOString(), - }); - return consumed.operationId; + const inFlight = this.consumingByIdentity.get(identity); + if (inFlight) { + await inFlight; + return await this.consumeApproved(args); + } + + const run = this.consumeApprovedUnlocked(args, identity); + this.consumingByIdentity.set(identity, run); + try { + return await run; + } finally { + if (this.consumingByIdentity.get(identity) === run) { + this.consumingByIdentity.delete(identity); } } + } + + private async consumeApprovedUnlocked( + args: PendingConsumeArgs, + identity: string, + ): Promise { const scope = this.operationScope(args); - if (args.reuseAfterConsume && this.persistence) { - const persisted = await this.persistence.findEvolutionOperation(scope); - if (persisted && Date.now() < Date.parse(persisted.expiresAt)) { - this.cacheConsumed(identity, persisted); - this.telemetry({ - event: "ask_approval_operation_recovered", - requestId: persisted.operationId, - toolName: args.toolName, - sessionId: args.sessionId, - authSub: args.authSub, - ts: new Date().toISOString(), - }); - return persisted.operationId; + const boundedRetryMs = + args.reuseOnceAfterConsumeMs && args.reuseOnceAfterConsumeMs > 0 + ? args.reuseOnceAfterConsumeMs + : undefined; + + if (boundedRetryMs !== undefined) { + const consumed = this.consumedByIdentity.get(identity); + if (consumed) { + if (Date.now() >= Date.parse(consumed.expiresAt)) { + this.removeConsumed(identity); + } else if (consumed.oneShot) { + const operationId = consumed.operationId; + this.removeConsumed(identity); + this.telemetry({ + event: "ask_approval_operation_reused", + requestId: operationId, + toolName: args.toolName, + sessionId: args.sessionId, + authSub: args.authSub, + ts: new Date().toISOString(), + }); + return operationId; + } + } + if (this.persistence?.claimEvolutionOperation) { + const claimed = await this.persistence.claimEvolutionOperation(scope); + if (claimed && Date.now() < Date.parse(claimed.expiresAt)) { + this.telemetry({ + event: "ask_approval_operation_recovered", + requestId: claimed.operationId, + toolName: args.toolName, + sessionId: args.sessionId, + authSub: args.authSub, + ts: new Date().toISOString(), + }); + return claimed.operationId; + } + } + } else if (args.reuseAfterConsume) { + const consumed = this.consumedByIdentity.get(identity); + if (consumed) { + if (Date.now() >= Date.parse(consumed.expiresAt)) { + this.removeConsumed(identity); + } else { + this.telemetry({ + event: "ask_approval_operation_reused", + requestId: consumed.operationId, + toolName: args.toolName, + sessionId: args.sessionId, + authSub: args.authSub, + ts: new Date().toISOString(), + }); + return consumed.operationId; + } + } + if (this.persistence) { + const persisted = await this.persistence.findEvolutionOperation(scope); + if (persisted && Date.now() < Date.parse(persisted.expiresAt)) { + this.cacheConsumed(identity, persisted); + this.telemetry({ + event: "ask_approval_operation_recovered", + requestId: persisted.operationId, + toolName: args.toolName, + sessionId: args.sessionId, + authSub: args.authSub, + ts: new Date().toISOString(), + }); + return persisted.operationId; + } } } const requestId = this.identityIndex.get(identity); @@ -209,19 +280,29 @@ export class PendingApprovalsStore { this.expire(record.requestId); return undefined; } - let reusable: { operationId: string; expiresAt: string } | undefined; - if (args.reuseAfterConsume) { + let reusable: { operationId: string; expiresAt: string; oneShot?: boolean } | undefined; + const shouldReuse = args.reuseAfterConsume || boundedRetryMs !== undefined; + let persistedReusable = false; + if (shouldReuse) { + const retentionMs = boundedRetryMs ?? EVOLUTION_OPERATION_RETENTION_MS; const fallback = { operationId: record.requestId, - expiresAt: new Date(Date.now() + EVOLUTION_OPERATION_RETENTION_MS).toISOString(), + expiresAt: new Date(Date.now() + retentionMs).toISOString(), + oneShot: boundedRetryMs !== undefined, }; try { const persisted = await this.persistence?.rememberEvolutionOperation({ ...scope, operationId: record.requestId, + retentionMs, }); + persistedReusable = Boolean(persisted); reusable = persisted - ? { operationId: record.requestId, expiresAt: persisted.expiresAt } + ? { + operationId: record.requestId, + expiresAt: persisted.expiresAt, + oneShot: boundedRetryMs !== undefined, + } : fallback; } catch (error) { this.telemetry({ @@ -237,7 +318,9 @@ export class PendingApprovalsStore { } } this.remove(record); - if (reusable) this.cacheConsumed(identity, reusable); + if (reusable && !(boundedRetryMs !== undefined && persistedReusable)) { + this.cacheConsumed(identity, reusable); + } this.telemetry({ event: "ask_approval_consumed", requestId: record.requestId, @@ -303,7 +386,7 @@ export class PendingApprovalsStore { private cacheConsumed( identity: string, - record: { operationId: string; expiresAt: string }, + record: { operationId: string; expiresAt: string; oneShot?: boolean }, ): void { const timer = setTimeout( () => this.removeConsumed(identity), @@ -373,4 +456,5 @@ export const pendingApprovals = new PendingApprovalsStore(undefined, { markResolved, rememberEvolutionOperation, findEvolutionOperation, + claimEvolutionOperation, }); diff --git a/packages/orchestration/src/permissions/repo.ts b/packages/orchestration/src/permissions/repo.ts index 53de7565..82c70305 100644 --- a/packages/orchestration/src/permissions/repo.ts +++ b/packages/orchestration/src/permissions/repo.ts @@ -141,11 +141,14 @@ export async function markResolved( /** Persist one approved evolution identity before the costful tool is allowed to execute. */ export async function rememberEvolutionOperation( - args: EvolutionOperationScope & { operationId: string }, + args: EvolutionOperationScope & { operationId: string; retentionMs?: number }, ): Promise<{ expiresAt: string } | undefined> { const pool = getPoolOrNull(); if (!pool) return undefined; - const expiresAt = new Date(Date.now() + 24 * 60 * 60 * 1_000); + const retentionMs = args.retentionMs && args.retentionMs > 0 + ? args.retentionMs + : 24 * 60 * 60 * 1_000; + const expiresAt = new Date(Date.now() + retentionMs); const result = await pool.query( `INSERT INTO evolution_approval_operations (operation_id,auth_sub,session_id,tool_name,input_digest,approved_at,expires_at) @@ -186,6 +189,25 @@ export async function findEvolutionOperation( : undefined; } +/** Atomically consume one unexpired recovery entitlement. */ +export async function claimEvolutionOperation( + args: EvolutionOperationScope, +): Promise<{ operationId: string; expiresAt: string } | undefined> { + const pool = getPoolOrNull(); + if (!pool) return undefined; + const result = await pool.query( + `DELETE FROM evolution_approval_operations + WHERE auth_sub=$1 AND session_id=$2 AND tool_name=$3 AND input_digest=$4 + AND expires_at>NOW() + RETURNING operation_id,expires_at`, + [args.authSub, args.sessionId, args.toolName, args.inputDigest], + ); + const row = result.rows[0]; + return row + ? { operationId: String(row.operation_id), expiresAt: toIso(row.expires_at) } + : undefined; +} + /** * 启动扫尾:上一进程遗留的 pending 行批量置 expired_restart。 * 返回扫掉的行数(log / 测试用);DB 不可用返回 0。 diff --git a/packages/orchestration/src/tools/evolver-shared.ts b/packages/orchestration/src/tools/evolver-shared.ts index 14362a7f..2d83141c 100644 --- a/packages/orchestration/src/tools/evolver-shared.ts +++ b/packages/orchestration/src/tools/evolver-shared.ts @@ -1,13 +1,15 @@ /** Evolver Mastra tools 的共享 schema 与客户端解析。 */ import { z } from "zod"; - import { resolveRequestToken } from "../auth.js"; import { buildEvolutionStartRequest, + buildEventCampaignRequest, + eventCampaignRequestDigest, evolutionRequestDigest, EvolverClient, type EvolutionConfig, type EvolutionStartRequest, + type EventCampaignConfigInput, } from "../clients/evolver.js"; import { getSettings } from "../config.js"; import { AUTH_SUB_KEY } from "../hooks/with-hooks.js"; @@ -67,6 +69,48 @@ export async function getApprovedEvolutionRunContext( }; } +/** Build one approved campaign context; its internal five generations remain automatic. */ +export async function getApprovedEventCampaignContext( + input: { + eventSnapshotId: string; + sourceRunId?: string; + config: EventCampaignConfigInput; + }, + ctx?: ToolRequestContext, +) { + const operationId = getRequestContextValue( + { requestContext: ctx }, + APPROVAL_OPERATION_ID_KEY, + ); + const llmSnapshot = getRequestContextValue( + { requestContext: ctx }, + USER_LLM_SNAPSHOT_KEY, + ); + const authSub = ctx?.get?.(AUTH_SUB_KEY); + if (!operationId || !llmSnapshot || typeof authSub !== "string" || !authSub) { + throw new Error("explicit event campaign approval context is missing"); + } + const request = buildEventCampaignRequest({ + eventSnapshotId: input.eventSnapshotId, + sourceRunId: input.sourceRunId, + config: input.config, + llmSnapshot, + }); + const credentialGrant = await mintEvolutionCredentialGrant({ + authSub, + operationId, + purpose: "event_campaign", + requestDigest: eventCampaignRequestDigest(request), + snapshot: llmSnapshot, + }); + return { + client: await getEvolverClient(ctx), + operationId, + credentialGrant, + request, + }; +} + export const evolutionConfigSchema = z.object({ venue: z.string().min(1).describe("数据 venue;按标的市场选择,不预设市场"), symbol: z.string().min(1).describe("该 venue 使用的标的代码"), @@ -80,3 +124,16 @@ export const evolutionConfigSchema = z.object({ fee_rate: z.number().min(0).max(0.1).default(0.001), validation_split: z.number().min(0).max(0.5).default(0.3), }); + +export const eventCampaignConfigSchema = z.object({ + venue: z.string().min(1), + symbol: z.string().min(1), + timeframe: z.enum(["15m", "1h", "4h"]), + from_ts: z.string().datetime(), + as_of: z.string().datetime(), + initial_cash: z.number().min(100).default(10_000), + fee_rate: z.number().min(0).max(0.1).default(0.001), + trading_mode: z.enum(["spot", "perp"]).default("perp"), + leverage: z.number().int().min(1).max(20).default(1), + random_seed: z.number().int().min(0).max(2 ** 31 - 1).default(0), +}); diff --git a/packages/orchestration/src/tools/evolver.ts b/packages/orchestration/src/tools/evolver.ts index 04744ded..0c5355ce 100644 --- a/packages/orchestration/src/tools/evolver.ts +++ b/packages/orchestration/src/tools/evolver.ts @@ -4,11 +4,47 @@ import { z } from "zod"; import { evolutionConfigSchema, + eventCampaignConfigSchema, + getApprovedEventCampaignContext, getApprovedEvolutionRunContext, getEvolverClient, type ToolRequestContext, } from "./evolver-shared.js"; +export const evolverRunEventCampaignTool = createTool({ + id: "evolver.run_event_campaign", + description: ` +基于冻结事件事实与模拟盘反馈启动五代双层自动演化;每代两个 Agent proposer 产生八个假设,每个确定性展开三条实现,并锁定唯一冠军等待独立 Forward。 +何时用:用户要求从事件机制发散新策略方向,且已有 point-in-time event snapshot 与 15m/1h/4h 冻结行情。 +何时不用:只优化既有代码用 evolver.run_evolution;没有事件快照、需要实盘或希望自动下单时不要用。 +坑:启动整个五代 campaign 需要一次显式审批;审批后内部自动迭代不逐代审批,只产出 sandbox 候选;不会 promote、启动 Runner 或下单,Forward 与一次性 holdout 仍是硬门禁。 + `.trim(), + inputSchema: z.object({ + eventSnapshotId: z.string().uuid(), + sourceRunId: z.string().uuid().optional(), + config: eventCampaignConfigSchema, + }), + execute: async (inputData, ctx) => { + const approved = await getApprovedEventCampaignContext( + inputData, + ctx?.requestContext as ToolRequestContext | undefined, + ); + return await approved.client.startEventCampaign({ + request: approved.request, + idempotencyKey: approved.operationId, + credentialGrant: approved.credentialGrant, + }); + }, +}); + +export const evolverGetEventCampaignTool = createTool({ + id: "evolver.get_event_campaign", + description: "查询本人事件演化 campaign 的代际、Forward、holdout 与费用状态;不用它读取原始新闻或普通 E1 run。", + inputSchema: z.object({ campaignId: z.string().uuid() }), + execute: async (inputData, ctx) => + await (await getEvolverClient(ctx?.requestContext as ToolRequestContext | undefined)).getEventCampaign(inputData.campaignId), +}); + export const evolverRunEvolutionTool = createTool({ id: "evolver.run_evolution", description: ` @@ -65,6 +101,8 @@ export const evolverAbortEvolutionTool = createTool({ }); export const evolverTools = [ + evolverRunEventCampaignTool, + evolverGetEventCampaignTool, evolverRunEvolutionTool, evolverGetEvolutionTool, evolverGetCandidateTool, diff --git a/packages/orchestration/src/tools/index.ts b/packages/orchestration/src/tools/index.ts index aaa7efbb..6a953f0f 100644 --- a/packages/orchestration/src/tools/index.ts +++ b/packages/orchestration/src/tools/index.ts @@ -103,7 +103,9 @@ import { import { evolverAbortEvolutionTool, evolverGetCandidateTool, + evolverGetEventCampaignTool, evolverGetEvolutionTool, + evolverRunEventCampaignTool, evolverRunEvolutionTool, evolverTools, } from "./evolver.js"; @@ -126,7 +128,9 @@ export { executeTradePlanTool, evolverAbortEvolutionTool, evolverGetCandidateTool, + evolverGetEventCampaignTool, evolverGetEvolutionTool, + evolverRunEventCampaignTool, evolverRunEvolutionTool, factorCatalogTool, factorEvaluateCandidateTool, @@ -341,6 +345,8 @@ export const orchestratorToolList = [ divinationCastHexagramTool, divinationDrawTarotTool, // E1 演化引擎(显式单代变异 + 真实冻结数据评估) + evolverRunEventCampaignTool, + evolverGetEventCampaignTool, evolverRunEvolutionTool, evolverGetEvolutionTool, evolverGetCandidateTool, diff --git a/packages/orchestration/tests/e2-campaign-authorization.test.ts b/packages/orchestration/tests/e2-campaign-authorization.test.ts new file mode 100644 index 00000000..22258933 --- /dev/null +++ b/packages/orchestration/tests/e2-campaign-authorization.test.ts @@ -0,0 +1,220 @@ +import { describe, expect, it, vi } from "vitest"; + +import { AUTH_SUB_KEY, HookRunner, withHooks } from "../src/hooks/index.js"; +import { + APPROVAL_OPERATION_ID_KEY, + buildEvolutionLLMSnapshot, + USER_LLM_SNAPSHOT_KEY, +} from "../src/mastra/llm/evolution-snapshot.js"; +import { DEFAULT_PERMISSIONS, PermissionEngine } from "../src/permissions/index.js"; +import { PendingApprovalsStore } from "../src/permissions/pending.js"; +import { getApprovedEventCampaignContext } from "../src/tools/evolver-shared.js"; + +const snapshotA = buildEvolutionLLMSnapshot({ + id: "config-a", + provider: "deepseek", + model: "deepseek-v4-pro", + api_key: "not-forwarded-a", +}); + +const snapshotB = buildEvolutionLLMSnapshot({ + id: "config-b", + provider: "deepseek", + model: "deepseek-v4-pro", + api_key: "not-forwarded-b", +}); + +const campaignInput = { + eventSnapshotId: "11111111-1111-4111-8111-111111111111", + sourceRunId: "22222222-2222-4222-8222-222222222222", + config: { + venue: "binance", + symbol: "BTCUSDT", + timeframe: "1h" as const, + from_ts: "2026-08-01T00:00:00Z", + as_of: "2026-08-02T00:00:00Z", + initial_cash: 10_000, + fee_rate: 0.001, + trading_mode: "perp" as const, + leverage: 1, + random_seed: 7, + }, +}; + +type ToolCtx = { requestContext: Map }; + +function context(snapshot = snapshotA): ToolCtx { + return { + requestContext: new Map([[USER_LLM_SNAPSHOT_KEY, snapshot]]), + }; +} + +function makeApprovedTool(options?: { + store?: PendingApprovalsStore; + owner?: () => string | undefined; +}) { + const store = options?.store ?? new PendingApprovalsStore(); + const execute = vi.fn(async (_input: unknown, ctx?: unknown) => { + const requestContext = (ctx as ToolCtx).requestContext; + return { operationId: requestContext.get(APPROVAL_OPERATION_ID_KEY) }; + }); + const wrapped = withHooks( + { id: "evolver.run_event_campaign", execute }, + { + runner: new HookRunner(), + permissionResolver: () => "ask", + pendingApprovals: store, + getAuthSub: options?.owner ?? (() => "user:alice"), + getSessionId: () => "thread-e2", + }, + ); + return { store, execute, wrapped }; +} + +describe("E2 campaign authorization", () => { + it("requires ask permission instead of the automatic allow path", () => { + const engine = new PermissionEngine(DEFAULT_PERMISSIONS); + expect(engine.authorize("evolver.run_event_campaign", campaignInput).decision).toBe("ask"); + expect(engine.authorize("evolver.get_event_campaign", {}).decision).toBe("allow"); + }); + + it("rejects direct campaign-context construction without a trusted approval operation", async () => { + const requestContext = new Map([ + [AUTH_SUB_KEY, "user:alice"], + [USER_LLM_SNAPSHOT_KEY, snapshotA], + ]); + + await expect( + getApprovedEventCampaignContext(campaignInput, requestContext), + ).rejects.toThrow("explicit event campaign approval context is missing"); + }); + + it("fails closed when the frozen LLM approval context is missing", async () => { + const { store, execute, wrapped } = makeApprovedTool(); + const result = (await wrapped.execute!(campaignInput, { + requestContext: new Map(), + })) as { requiresApproval: boolean; message: string }; + + expect(result.requiresApproval).toBe(true); + expect(result.message).toContain("APPROVAL_UNAVAILABLE"); + expect(execute).not.toHaveBeenCalled(); + expect(store.list("user:alice")).toHaveLength(0); + store.clearAll(); + }); + + it("allows exactly one matching compensation retry for the approved E2 operation", async () => { + const { store, execute, wrapped } = makeApprovedTool(); + const ctx = context(); + + const pending = (await wrapped.execute!(campaignInput, ctx)) as { + requiresApproval: boolean; + requestId: string; + }; + expect(pending.requiresApproval).toBe(true); + expect(execute).not.toHaveBeenCalled(); + expect(store.respond(pending.requestId, "allow", "user:alice")).toBe(true); + + const first = (await wrapped.execute!(campaignInput, ctx)) as { operationId: string }; + const retry = (await wrapped.execute!(campaignInput, ctx)) as { operationId: string }; + const exhausted = (await wrapped.execute!(campaignInput, ctx)) as { + requiresApproval: boolean; + requestId: string; + }; + + expect(first.operationId).toBe(pending.requestId); + expect(retry.operationId).toBe(pending.requestId); + expect(exhausted.requiresApproval).toBe(true); + expect(exhausted.requestId).not.toBe(pending.requestId); + expect(execute).toHaveBeenCalledTimes(2); + store.clearAll(); + }); + + it("expires the E2 compensation retry after two minutes", async () => { + vi.useFakeTimers(); + const { store, execute, wrapped } = makeApprovedTool(); + const ctx = context(); + + const pending = (await wrapped.execute!(campaignInput, ctx)) as { requestId: string }; + expect(store.respond(pending.requestId, "allow", "user:alice")).toBe(true); + const first = (await wrapped.execute!(campaignInput, ctx)) as { operationId: string }; + expect(first.operationId).toBe(pending.requestId); + + vi.advanceTimersByTime(2 * 60 * 1_000 + 1); + const expired = (await wrapped.execute!(campaignInput, ctx)) as { + requiresApproval: boolean; + requestId: string; + }; + + expect(expired.requiresApproval).toBe(true); + expect(expired.requestId).not.toBe(pending.requestId); + expect(execute).toHaveBeenCalledTimes(1); + store.clearAll(); + vi.useRealTimers(); + }); + + it("does not consume an approval after material campaign input is changed", async () => { + const { store, execute, wrapped } = makeApprovedTool(); + const ctx = context(); + + const pending = (await wrapped.execute!(campaignInput, ctx)) as { requestId: string }; + expect(store.respond(pending.requestId, "allow", "user:alice")).toBe(true); + + const changed = { + ...campaignInput, + config: { ...campaignInput.config, symbol: "ETHUSDT" }, + }; + const blocked = (await wrapped.execute!(changed, ctx)) as { requiresApproval: boolean }; + + expect(blocked.requiresApproval).toBe(true); + expect(execute).not.toHaveBeenCalled(); + + const original = (await wrapped.execute!(campaignInput, ctx)) as { operationId: string }; + expect(original.operationId).toBe(pending.requestId); + store.clearAll(); + }); + + it("does not consume an approval after the frozen LLM snapshot is substituted", async () => { + const { store, execute, wrapped } = makeApprovedTool(); + const ctx = context(snapshotA); + + const pending = (await wrapped.execute!(campaignInput, ctx)) as { requestId: string }; + expect(store.respond(pending.requestId, "allow", "user:alice")).toBe(true); + + ctx.requestContext.set(USER_LLM_SNAPSHOT_KEY, snapshotB); + const blocked = (await wrapped.execute!(campaignInput, ctx)) as { + requiresApproval: boolean; + }; + expect(blocked.requiresApproval).toBe(true); + expect(execute).not.toHaveBeenCalled(); + + ctx.requestContext.set(USER_LLM_SNAPSHOT_KEY, snapshotA); + const original = (await wrapped.execute!(campaignInput, ctx)) as { operationId: string }; + expect(original.operationId).toBe(pending.requestId); + store.clearAll(); + }); + + it("does not let another owner consume the approved operation", async () => { + let owner = "user:alice"; + const store = new PendingApprovalsStore(); + const { execute, wrapped } = makeApprovedTool({ + store, + owner: () => owner, + }); + const ctx = context(); + + const pending = (await wrapped.execute!(campaignInput, ctx)) as { requestId: string }; + expect(store.respond(pending.requestId, "allow", "user:alice")).toBe(true); + + owner = "user:bob"; + const blocked = (await wrapped.execute!(campaignInput, ctx)) as { + requiresApproval: boolean; + }; + expect(blocked.requiresApproval).toBe(true); + expect(execute).not.toHaveBeenCalled(); + + owner = "user:alice"; + const original = (await wrapped.execute!(campaignInput, ctx)) as { operationId: string }; + expect(original.operationId).toBe(pending.requestId); + store.clearAll(); + }); +}); diff --git a/packages/orchestration/tests/evolver-client.test.ts b/packages/orchestration/tests/evolver-client.test.ts index 4f1d04a5..4eff3642 100644 --- a/packages/orchestration/tests/evolver-client.test.ts +++ b/packages/orchestration/tests/evolver-client.test.ts @@ -5,6 +5,8 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import { buildEvolutionStartRequest, + buildEventCampaignRequest, + eventCampaignRequestDigest, evolutionRequestDigest, EvolverClient, } from "../src/clients/evolver.js"; @@ -14,7 +16,10 @@ import { buildEvolutionLLMSnapshot, USER_LLM_SNAPSHOT_KEY, } from "../src/mastra/llm/evolution-snapshot.js"; -import { getApprovedEvolutionRunContext } from "../src/tools/evolver-shared.js"; +import { + getApprovedEvolutionRunContext, + getApprovedEventCampaignContext, +} from "../src/tools/evolver-shared.js"; const snapshot = buildEvolutionLLMSnapshot({ id: "config-1", @@ -34,6 +39,33 @@ function response(status: number): Response { ); } +function campaignResponse( + status = 200, + campaignStatus = "replaying", +): Response { + return new Response( + JSON.stringify({ + campaign_id: "33333333-3333-4333-8333-333333333333", + status: campaignStatus, + active_generation: campaignStatus === "draft" ? 0 : 1, + max_generations: 5, + event_snapshot_id: "11111111-1111-4111-8111-111111111111", + llm_cost_usd: 0, + }), + { status, headers: { "Content-Type": "application/json" } }, + ); +} + +function campaignConflict(): Response { + return new Response( + JSON.stringify({ + code: "CAMPAIGN_STATE_CONFLICT", + message: "campaign cannot start", + }), + { status: 409, headers: { "Content-Type": "application/json" } }, + ); +} + function options() { const request = buildEvolutionStartRequest({ budget: 1, @@ -96,6 +128,173 @@ describe("EvolverClient", () => { expect(Number(credential.exp) - Number(credential.iat)).toBe(108_000); }); + it("binds an approved E2 campaign grant to the shared durable operation identity", async () => { + const keys = generateKeyPairSync("ed25519"); + vi.stubEnv( + "EVOLUTION_CREDENTIAL_PRIVATE_KEY_B64", + keys.privateKey.export({ format: "der", type: "pkcs8" }).toString("base64"), + ); + const requestContext = new Map([ + [AUTH_SUB_KEY, "user:alice"], + [APPROVAL_OPERATION_ID_KEY, "approval-operation-e2"], + [USER_LLM_SNAPSHOT_KEY, snapshot], + ]); + const config = { + venue: "binance", + symbol: "BTCUSDT", + timeframe: "1h" as const, + from_ts: "2026-08-01T00:00:00Z", + as_of: "2026-08-02T00:00:00Z", + }; + + const campaign = await getApprovedEventCampaignContext( + { eventSnapshotId: "11111111-1111-4111-8111-111111111111", config }, + requestContext, + ); + const { payload } = await jwtVerify(campaign.credentialGrant, keys.publicKey, { + algorithms: ["EdDSA"], + audience: "inalpha-dashboard-credential", + }); + + expect(payload).toMatchObject({ + sub: "user:alice", + grant_purpose: "event_campaign", + operation_id: "approval-operation-e2", + llm_config_digest: snapshot.config_digest, + request_digest: eventCampaignRequestDigest(campaign.request), + }); + expect(campaign.operationId).toBe("approval-operation-e2"); + expect(campaign.request).toEqual( + buildEventCampaignRequest({ + eventSnapshotId: "11111111-1111-4111-8111-111111111111", + config, + llmSnapshot: snapshot, + }), + ); + }); + + it("recovers a lost start response by returning the already-started campaign on whole-operation retry", async () => { + const request = buildEventCampaignRequest({ + eventSnapshotId: "11111111-1111-4111-8111-111111111111", + config: { + venue: "binance", + symbol: "BTCUSDT", + timeframe: "1h", + from_ts: "2026-08-01T00:00:00Z", + as_of: "2026-08-02T00:00:00Z", + }, + llmSnapshot: snapshot, + }); + const fetchMock = vi + .fn() + .mockResolvedValueOnce(campaignResponse(201, "draft")) + .mockRejectedValueOnce(new TypeError("start response lost")) + .mockResolvedValueOnce(campaignResponse(201, "replaying")); + vi.stubGlobal("fetch", fetchMock); + + const client = new EvolverClient({ + baseUrl: "http://evolver.test", + token: "owner-token", + }); + const options = { + request, + idempotencyKey: "approval-operation-e2", + credentialGrant: "event-campaign-grant", + }; + + await expect(client.startEventCampaign(options)).rejects.toMatchObject({ + code: "UPSTREAM_UNREACHABLE", + }); + await expect(client.startEventCampaign(options)).resolves.toMatchObject({ + campaign_id: "33333333-3333-4333-8333-333333333333", + status: "replaying", + }); + + expect(fetchMock).toHaveBeenCalledTimes(3); + for (const index of [0, 2]) { + const [url, init] = fetchMock.mock.calls[index] as [string, RequestInit]; + expect(url).toContain("/api/v1/campaigns"); + expect(url).not.toContain("/start"); + expect((init.headers as Record)["Idempotency-Key"]).toBe( + "approval-operation-e2", + ); + expect((init.headers as Record)["X-Evolution-Credential"]).toBe( + "event-campaign-grant", + ); + expect(init.body).not.toContain("not-forwarded"); + } + }); + + it("reconciles a concurrent E2 start conflict when the campaign already advanced", async () => { + const request = buildEventCampaignRequest({ + eventSnapshotId: "11111111-1111-4111-8111-111111111111", + config: { + venue: "binance", + symbol: "BTCUSDT", + timeframe: "1h", + from_ts: "2026-08-01T00:00:00Z", + as_of: "2026-08-02T00:00:00Z", + }, + llmSnapshot: snapshot, + }); + const fetchMock = vi + .fn() + .mockResolvedValueOnce(campaignResponse(201, "draft")) + .mockResolvedValueOnce(campaignConflict()) + .mockResolvedValueOnce(campaignResponse(200, "replaying")); + vi.stubGlobal("fetch", fetchMock); + + await expect( + new EvolverClient({ + baseUrl: "http://evolver.test", + token: "owner-token", + }).startEventCampaign({ + request, + idempotencyKey: "approval-operation-e2", + credentialGrant: "event-campaign-grant", + }), + ).resolves.toMatchObject({ status: "replaying" }); + + expect(fetchMock).toHaveBeenCalledTimes(3); + expect(String(fetchMock.mock.calls[1]?.[0])).toContain("/start"); + expect(String(fetchMock.mock.calls[2]?.[0])).not.toContain("/start"); + }); + + it("preserves E2 start conflicts when reconciliation still finds a draft campaign", async () => { + const request = buildEventCampaignRequest({ + eventSnapshotId: "11111111-1111-4111-8111-111111111111", + config: { + venue: "binance", + symbol: "BTCUSDT", + timeframe: "1h", + from_ts: "2026-08-01T00:00:00Z", + as_of: "2026-08-02T00:00:00Z", + }, + llmSnapshot: snapshot, + }); + const fetchMock = vi + .fn() + .mockResolvedValueOnce(campaignResponse(201, "draft")) + .mockResolvedValueOnce(campaignConflict()) + .mockResolvedValueOnce(campaignResponse(200, "draft")); + vi.stubGlobal("fetch", fetchMock); + + await expect( + new EvolverClient({ + baseUrl: "http://evolver.test", + token: "owner-token", + }).startEventCampaign({ + request, + idempotencyKey: "approval-operation-e2", + credentialGrant: "event-campaign-grant", + }), + ).rejects.toMatchObject({ + code: "CAMPAIGN_STATE_CONFLICT", + status: 409, + }); + expect(fetchMock).toHaveBeenCalledTimes(3); + }); + it("retries 502/504 with the same approval-derived operation ID", async () => { const fetchMock = vi .fn() diff --git a/packages/orchestration/tests/permissions-pending.test.ts b/packages/orchestration/tests/permissions-pending.test.ts index 46613fa1..8390dca8 100644 --- a/packages/orchestration/tests/permissions-pending.test.ts +++ b/packages/orchestration/tests/permissions-pending.test.ts @@ -136,6 +136,196 @@ describe("PendingApprovalsStore", () => { store.clearAll(); }); + it("allows one bounded recovery retry and then exhausts it", async () => { + vi.useFakeTimers(); + const store = new PendingApprovalsStore(() => {}); + const args = { + authSub: "user:alice", + sessionId: "thread-E2", + toolName: "evolver.run_event_campaign", + toolInput: { eventSnapshotId: "event-1" }, + approvalInput: { request: { eventSnapshotId: "event-1" }, llm_snapshot: { config_digest: "digest" } }, + timeoutMs: 5_000, + }; + const view = store.request(args); + expect(store.respond(view.requestId, "allow", args.authSub)).toBe(true); + const consume = { + authSub: args.authSub, + sessionId: args.sessionId, + toolName: args.toolName, + approvalInput: args.approvalInput, + reuseOnceAfterConsumeMs: 120_000, + }; + + expect(await store.consumeApproved(consume)).toBe(view.requestId); + expect(await store.consumeApproved(consume)).toBe(view.requestId); + expect(await store.consumeApproved(consume)).toBeUndefined(); + store.clearAll(); + }); + + it("does not allow the bounded recovery retry after two minutes", async () => { + vi.useFakeTimers(); + const store = new PendingApprovalsStore(() => {}); + const args = { + authSub: "user:alice", + sessionId: "thread-E2", + toolName: "evolver.run_event_campaign", + toolInput: { eventSnapshotId: "event-1" }, + approvalInput: { request: { eventSnapshotId: "event-1" }, llm_snapshot: { config_digest: "digest" } }, + timeoutMs: 5_000, + }; + const view = store.request(args); + expect(store.respond(view.requestId, "allow", args.authSub)).toBe(true); + const consume = { + authSub: args.authSub, + sessionId: args.sessionId, + toolName: args.toolName, + approvalInput: args.approvalInput, + reuseOnceAfterConsumeMs: 120_000, + }; + expect(await store.consumeApproved(consume)).toBe(view.requestId); + vi.advanceTimersByTime(120_001); + expect(await store.consumeApproved(consume)).toBeUndefined(); + store.clearAll(); + }); + + it("serializes concurrent initial E2 consumption into initial plus one retry", async () => { + let releasePersist!: () => void; + const persistGate = new Promise((resolve) => { + releasePersist = resolve; + }); + const operations = new Map(); + const persistence = { + insertPending: vi.fn(async () => {}), + markResolved: vi.fn(async () => {}), + rememberEvolutionOperation: vi.fn(async (scope: { + inputDigest: string; + operationId: string; + retentionMs?: number; + }) => { + await persistGate; + const value = { + operationId: scope.operationId, + expiresAt: new Date(Date.now() + (scope.retentionMs ?? 86_400_000)).toISOString(), + }; + operations.set(scope.inputDigest, value); + return { expiresAt: value.expiresAt }; + }), + findEvolutionOperation: vi.fn(async (scope: { inputDigest: string }) => + operations.get(scope.inputDigest), + ), + claimEvolutionOperation: vi.fn(async (scope: { inputDigest: string }) => { + const value = operations.get(scope.inputDigest); + if (!value || Date.now() >= Date.parse(value.expiresAt)) return undefined; + operations.delete(scope.inputDigest); + return value; + }), + }; + const args = { + authSub: "user:alice", + sessionId: "thread-E2", + toolName: "evolver.run_event_campaign", + toolInput: { eventSnapshotId: "event-1" }, + approvalInput: { + request: { eventSnapshotId: "event-1" }, + llm_snapshot: { config_digest: "digest" }, + }, + timeoutMs: 5_000, + }; + const store = new PendingApprovalsStore(() => {}, persistence); + const view = store.request(args); + expect(store.respond(view.requestId, "allow", args.authSub)).toBe(true); + const consume = { + authSub: args.authSub, + sessionId: args.sessionId, + toolName: args.toolName, + approvalInput: args.approvalInput, + reuseOnceAfterConsumeMs: 120_000, + }; + + const initial = store.consumeApproved(consume); + const concurrent = store.consumeApproved(consume); + await Promise.resolve(); + expect(persistence.rememberEvolutionOperation).toHaveBeenCalledTimes(1); + + releasePersist(); + expect(await initial).toBe(view.requestId); + expect(await concurrent).toBe(view.requestId); + expect(persistence.rememberEvolutionOperation).toHaveBeenCalledTimes(1); + // Initial consumption probes for a prior durable retry before persisting this approval; + // the serialized concurrent call performs the second claim and consumes that retry. + expect(persistence.claimEvolutionOperation).toHaveBeenCalledTimes(2); + expect(await store.consumeApproved(consume)).toBeUndefined(); + store.clearAll(); + }); + + it("atomically allows only one bounded recovery across fresh stores", async () => { + const operations = new Map(); + const persistence = { + insertPending: vi.fn(async () => {}), + markResolved: vi.fn(async () => {}), + rememberEvolutionOperation: vi.fn(async (scope: { + inputDigest: string; + operationId: string; + retentionMs?: number; + }) => { + const value = { + operationId: scope.operationId, + expiresAt: new Date(Date.now() + (scope.retentionMs ?? 86_400_000)).toISOString(), + }; + operations.set(scope.inputDigest, value); + return { expiresAt: value.expiresAt }; + }), + findEvolutionOperation: vi.fn(async (scope: { inputDigest: string }) => + operations.get(scope.inputDigest), + ), + claimEvolutionOperation: vi.fn(async (scope: { inputDigest: string }) => { + const value = operations.get(scope.inputDigest); + if (!value || Date.now() >= Date.parse(value.expiresAt)) return undefined; + operations.delete(scope.inputDigest); + return value; + }), + }; + const args = { + authSub: "user:alice", + sessionId: "thread-E2", + toolName: "evolver.run_event_campaign", + toolInput: { eventSnapshotId: "event-1" }, + approvalInput: { + request: { eventSnapshotId: "event-1" }, + llm_snapshot: { config_digest: "digest" }, + }, + timeoutMs: 5_000, + }; + const first = new PendingApprovalsStore(() => {}, persistence); + const view = first.request(args); + expect(first.respond(view.requestId, "allow", args.authSub)).toBe(true); + const consume = { + authSub: args.authSub, + sessionId: args.sessionId, + toolName: args.toolName, + approvalInput: args.approvalInput, + reuseOnceAfterConsumeMs: 120_000, + }; + expect(await first.consumeApproved(consume)).toBe(view.requestId); + + const retryA = new PendingApprovalsStore(() => {}, persistence); + const retryB = new PendingApprovalsStore(() => {}, persistence); + const claims = await Promise.all([ + retryA.consumeApproved(consume), + retryB.consumeApproved(consume), + ]); + + expect(claims.filter((value) => value === view.requestId)).toHaveLength(1); + expect(claims.filter((value) => value === undefined)).toHaveLength(1); + expect(await new PendingApprovalsStore(() => {}, persistence).consumeApproved(consume)) + .toBeUndefined(); + + first.clearAll(); + retryA.clearAll(); + retryB.clearAll(); + }); + it("does not reuse a consumed evolution operation after its retention deadline", async () => { vi.useFakeTimers(); const store = new PendingApprovalsStore(() => {}); diff --git a/services/data/src/inalpha_data/api/events.py b/services/data/src/inalpha_data/api/events.py new file mode 100644 index 00000000..464257fb --- /dev/null +++ b/services/data/src/inalpha_data/api/events.py @@ -0,0 +1,176 @@ +"""Point-in-time market event ledger and frozen snapshot API.""" + +from __future__ import annotations + +from typing import Annotated +from uuid import UUID + +from fastapi import APIRouter, Depends +from inalpha_shared.auth import User, get_current_user +from inalpha_shared.db import DBConn +from inalpha_shared.errors import InalphaError, ValidationError + +from ..connectors.coinmarketcal import get_connector +from ..event_models import ( + CoinMarketCalImportRequest, + EventCoverageResponse, + EventFactRecord, + EventFactWriteRequest, + EventFactWriteResponse, + EventImportResponse, + EventSnapshotRecord, + EventSnapshotRequest, + RawEventIngestRequest, + RawEventIngestResponse, + RawEventRecord, +) +from ..storage import events as store + +router = APIRouter(prefix="/events", tags=["events"]) + + +class EventRecordNotFoundError(InalphaError): + """Requested raw event or immutable snapshot does not exist.""" + + code = "EVENT_RECORD_NOT_FOUND" + status_code = 404 + + +class EventProviderUnavailableError(InalphaError): + """Configured historical event provider cannot serve this request.""" + + code = "EVENT_PROVIDER_UNAVAILABLE" + status_code = 503 + + +@router.post("/raw", response_model=RawEventIngestResponse) +async def ingest_raw_event( + request: RawEventIngestRequest, + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> RawEventIngestResponse: + """Append one raw event version; identical retries return the existing version.""" + row, created = await store.ingest_raw_event(db, request) + return RawEventIngestResponse(event=RawEventRecord(**row), created=created) + + +@router.post("/facts", response_model=EventFactWriteResponse) +async def write_event_fact( + request: EventFactWriteRequest, + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> EventFactWriteResponse: + """Append one fact version without exposing its source content downstream.""" + try: + row, created = await store.write_fact(db, request) + except LookupError as exc: + raise EventRecordNotFoundError( + f"raw event {request.raw_event_id} not found", + details={"raw_event_id": str(request.raw_event_id)}, + ) from exc + except ValueError as exc: + raise ValidationError( + str(exc), + code="EVENT_AVAILABLE_AT_INVALID", + ) from exc + return EventFactWriteResponse(fact=EventFactRecord(**row), created=created) + + +@router.get("/raw/{event_id}", response_model=RawEventRecord) +async def get_raw_event( + event_id: UUID, + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> RawEventRecord: + """Expose raw evidence only to authenticated platform extraction services.""" + row = await store.get_raw_event(db, event_id) + if row is None: + raise EventRecordNotFoundError( + f"raw event {event_id} not found", + details={"raw_event_id": str(event_id)}, + ) + return RawEventRecord(**row) + + +@router.post("/snapshots", response_model=EventSnapshotRecord) +async def create_event_snapshot( + request: EventSnapshotRequest, + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> EventSnapshotRecord: + """Freeze latest visible event facts at ``cutoff`` with deterministic ordering.""" + snapshot, facts = await store.create_snapshot(db, request) + return EventSnapshotRecord( + **snapshot, + facts=[EventFactRecord(**row) for row in facts], + ) + + +@router.get("/snapshots/{snapshot_id}", response_model=EventSnapshotRecord) +async def get_event_snapshot( + snapshot_id: UUID, + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> EventSnapshotRecord: + """Load a frozen snapshot; facts preserve their original stable ordinal.""" + result = await store.get_snapshot(db, snapshot_id) + if result is None: + raise EventRecordNotFoundError( + f"event snapshot {snapshot_id} not found", + details={"snapshot_id": str(snapshot_id)}, + ) + snapshot, facts = result + return EventSnapshotRecord( + **snapshot, + facts=[EventFactRecord(**row) for row in facts], + ) + + +@router.get("/coverage", response_model=EventCoverageResponse) +async def get_event_coverage( + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> EventCoverageResponse: + """Return source freshness, versions, and retractions for operational monitoring.""" + return EventCoverageResponse(**await store.coverage(db)) + + +@router.post("/import/coinmarketcal", response_model=EventImportResponse) +async def import_coinmarketcal( + request: CoinMarketCalImportRequest, + db: DBConn, + _user: Annotated[User, Depends(get_current_user)], +) -> EventImportResponse: + """Import a bounded Professional catalog window into the immutable raw ledger.""" + connector = get_connector() + if not connector.configured: + raise EventProviderUnavailableError( + "CoinMarketCal Professional API is not configured", + details={"env": "COINMARKETCAL_API_KEY"}, + ) + try: + records = await connector.fetch(request) + except Exception as exc: + raise EventProviderUnavailableError( + f"CoinMarketCal import failed: {type(exc).__name__}", + ) from exc + created = unchanged = failed = 0 + for record in records: + try: + async with db.transaction(): + _, was_created = await store.ingest_raw_event(db, record) + except Exception: + failed += 1 + continue + created += int(was_created) + unchanged += int(not was_created) + return EventImportResponse( + source="coinmarketcal", + fetched=len(records), + created=created, + unchanged=unchanged, + failed=failed, + ) + + +__all__ = ["router"] diff --git a/services/data/src/inalpha_data/config.py b/services/data/src/inalpha_data/config.py index 489ffc61..3238a817 100644 --- a/services/data/src/inalpha_data/config.py +++ b/services/data/src/inalpha_data/config.py @@ -78,9 +78,7 @@ class DataSettings(BaseSettings): news_timeout_s: float = Field(default=15.0, alias="NEWS_TIMEOUT_S") """SEC、HKEX 与 RSS provider 的单请求超时。""" - sec_user_agent: str = Field( - default="Inalpha/0.2 contact@inalpha.dev", alias="SEC_USER_AGENT" - ) + sec_user_agent: str = Field(default="Inalpha/0.2 contact@inalpha.dev", alias="SEC_USER_AGENT") """SEC 要求可识别应用和联系方式;生产可覆盖为维护者邮箱。""" sec_min_interval_s: float = Field(default=0.11, alias="SEC_MIN_INTERVAL_S") @@ -97,6 +95,30 @@ class DataSettings(BaseSettings): """成分快照调度的检查间隔(小时)。幂等:每轮只补"今天还没快照"的指数, <24h 不会重复打源站(省 akshare + 防封),>1 轮/天纯为重启后尽快补当天。""" + coinmarketcal_api_key: str = Field(default="", alias="COINMARKETCAL_API_KEY") + """CoinMarketCal Professional v2 API key。为空时历史事件导入端点显式返回不可用。""" + + coinmarketcal_base_url: str = Field( + default="https://api.coinmarketcal.com", alias="COINMARKETCAL_BASE_URL" + ) + """CoinMarketCal 官方 API 根地址;保留覆盖能力用于测试 stub。""" + + event_provider_timeout_s: float = Field(default=20.0, alias="EVENT_PROVIDER_TIMEOUT_S") + """结构化事件 provider 单请求超时。""" + + event_historical_latency_s: int = Field( + default=300, ge=0, le=86_400, alias="EVENT_HISTORICAL_LATENCY_S" + ) + """历史结构化来源首次添加时间的保守可用延迟,冻结进 policy version。""" + + event_archive_enabled: bool = Field(default=False, alias="EVENT_ARCHIVE_ENABLED") + """是否启动精选 crypto RSS 的 forward 归档;默认关,避免未迁移 DB 时后台报错。""" + + event_archive_interval_s: int = Field( + default=900, ge=60, le=86_400, alias="EVENT_ARCHIVE_INTERVAL_S" + ) + """精选新闻 forward 归档轮询间隔。""" + @lru_cache(maxsize=1) def get_data_settings() -> DataSettings: diff --git a/services/data/src/inalpha_data/connectors/coinmarketcal.py b/services/data/src/inalpha_data/connectors/coinmarketcal.py new file mode 100644 index 00000000..384d3064 --- /dev/null +++ b/services/data/src/inalpha_data/connectors/coinmarketcal.py @@ -0,0 +1,157 @@ +"""CoinMarketCal v2 historical event connector. + +Only documented v2 fields are consumed. Provider payloads are retained in the raw +ledger so later extractor upgrades can reproduce normalized facts. +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from typing import Any + +import httpx + +from ..event_models import CoinMarketCalImportRequest, RawEventIngestRequest + + +class CoinMarketCalConnector: + """Fetch bounded pages from the configured Professional event catalog.""" + + def __init__( + self, + *, + api_key: str, + base_url: str, + timeout_s: float, + historical_latency_s: int, + ) -> None: + self._api_key = api_key + self._historical_latency = timedelta(seconds=historical_latency_s) + self._client = httpx.AsyncClient( + base_url=base_url.rstrip("/"), + timeout=timeout_s, + trust_env=False, + headers={"x-api-key": api_key, "User-Agent": "Inalpha/0.3 event-research"}, + ) + + @property + def configured(self) -> bool: + return bool(self._api_key) + + async def fetch(self, request: CoinMarketCalImportRequest) -> list[RawEventIngestRequest]: + """Fetch at most ``request.limit`` records across cursor pages.""" + if not self.configured: + raise RuntimeError("COINMARKETCAL_API_KEY is not configured") + records: list[RawEventIngestRequest] = [] + cursor: str | None = None + while len(records) < request.limit: + page_limit = min(100, request.limit - len(records)) + params: dict[str, Any] = { + "from": request.from_date.date().isoformat(), + "to": request.to_date.date().isoformat(), + "limit": page_limit, + } + if request.coins: + params["coins"] = ",".join(request.coins) + if request.categories: + params["categories"] = ",".join(request.categories) + if cursor: + params["cursor"] = cursor + response = await self._client.get("/v2/events", params=params) + response.raise_for_status() + payload = response.json() + data = payload.get("data") if isinstance(payload, dict) else None + if not isinstance(data, list): + raise ValueError("CoinMarketCal response data must be a list") + fetched_at = datetime.now(UTC) + records.extend( + self._convert(item, fetched_at) for item in data if isinstance(item, dict) + ) + meta = payload.get("meta") if isinstance(payload, dict) else None + cursor = ( + str(meta.get("cursor")) if isinstance(meta, dict) and meta.get("cursor") else None + ) + if not cursor or not data: + break + return records[: request.limit] + + def _convert(self, item: dict[str, Any], fetched_at: datetime) -> RawEventIngestRequest: + source_event_id = str(item.get("id") or item.get("slug") or "").strip() + if not source_event_id: + raise ValueError("CoinMarketCal event is missing id") + event_at = _parse_time(item.get("date")) + added_at = _parse_time( + item.get("dateAdded") or item.get("createdAt") or item.get("created_at") + ) + # Historical catalog rows use provider first-add time plus a frozen safety latency. + # If the provider omits it, the record is only known at this import's first_seen time. + first_seen_at = added_at + self._historical_latency if added_at else fetched_at + proof = item.get("proof") or item.get("source") or item.get("originalSource") + url = proof if isinstance(proof, str) else None + return RawEventIngestRequest( + source="coinmarketcal", + source_event_id=source_event_id, + title=str(item.get("title") or ""), + content=str(item.get("description") or ""), + url=url, + raw_payload=item, + source_valid_at=event_at, + claimed_published_at=added_at, + first_seen_at=first_seen_at, + fetched_at=fetched_at, + accepted_at=max(first_seen_at, fetched_at) if not added_at else first_seen_at, + collector_version="coinmarketcal-v2@1", + policy_version="structured-provider-latency-v1", + source_tier="structured", + retracted=bool(item.get("cancelled") or item.get("isCancelled")), + ) + + async def close(self) -> None: + """Close the underlying HTTP connection pool.""" + await self._client.aclose() + + +_connector: CoinMarketCalConnector | None = None + + +def init_connector( + *, api_key: str, base_url: str, timeout_s: float, historical_latency_s: int +) -> None: + """Initialize the process-wide connector during FastAPI lifespan startup.""" + global _connector + if _connector is not None: + raise RuntimeError("CoinMarketCal connector already initialized") + _connector = CoinMarketCalConnector( + api_key=api_key, + base_url=base_url, + timeout_s=timeout_s, + historical_latency_s=historical_latency_s, + ) + + +def get_connector() -> CoinMarketCalConnector: + """Return the initialized process-wide connector.""" + if _connector is None: + raise RuntimeError("CoinMarketCal connector not initialized") + return _connector + + +async def close_connector() -> None: + """Close and clear the process-wide connector.""" + global _connector + if _connector is not None: + await _connector.close() + _connector = None + + +def _parse_time(value: Any) -> datetime | None: + if not isinstance(value, str) or not value.strip(): + return None + try: + parsed = datetime.fromisoformat(value.strip().replace("Z", "+00:00")) + except ValueError: + return None + return parsed.replace(tzinfo=UTC) if parsed.tzinfo is None else parsed.astimezone(UTC) + + +__all__ = ["CoinMarketCalConnector", "close_connector", "get_connector", "init_connector"] diff --git a/services/data/src/inalpha_data/event_archive.py b/services/data/src/inalpha_data/event_archive.py new file mode 100644 index 00000000..55920c87 --- /dev/null +++ b/services/data/src/inalpha_data/event_archive.py @@ -0,0 +1,94 @@ +"""Forward-only archive for selected crypto news and official announcements.""" + +from __future__ import annotations + +import asyncio +import hashlib +from datetime import UTC, datetime + +from inalpha_shared import get_logger +from inalpha_shared.db import get_conn + +from .connectors.news import get_router +from .event_models import RawEventIngestRequest +from .news_models import NewsQuery +from .storage import events as store + +_logger = get_logger(__name__) + + +class EventArchiveScheduler: + """Poll selected snapshot-only feeds and preserve their first observed versions.""" + + def __init__(self, *, enabled: bool, interval_s: int) -> None: + self._enabled = enabled + self._interval_s = interval_s + self._task: asyncio.Task[None] | None = None + + def start(self) -> None: + """Start forward accumulation when explicitly enabled.""" + if not self._enabled: + _logger.info("event_archive_disabled") + return + if self._task is None: + self._task = asyncio.create_task(self._loop(), name="event-archive-scheduler") + _logger.info("event_archive_started", interval_s=self._interval_s) + + async def stop(self) -> None: + """Cancel the archive loop without delaying service shutdown.""" + if self._task is None: + return + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + self._task = None + + async def _loop(self) -> None: + while True: + try: + await self._tick() + except asyncio.CancelledError: + raise + except Exception as exc: + _logger.warning("event_archive_tick_failed", error=str(exc)) + await asyncio.sleep(self._interval_s) + + async def _tick(self) -> None: + response = await get_router().fetch(NewsQuery(market="crypto", limit=50)) + async with get_conn() as conn: + for item in response.items: + observed_at = item.accepted_at or item.fetched_at or response.fetched_at + source_event_id = ( + item.source_id + or item.link + or hashlib.sha256( + f"{item.source_name}\0{item.title}\0{item.published_at}".encode() + ).hexdigest() + ) + request = RawEventIngestRequest( + source=item.source_name or item.publisher or "crypto_news", + source_event_id=source_event_id, + title=item.title, + content=item.summary, + url=item.link or None, + raw_payload={ + "kind": item.kind, + "market": item.market, + "symbols": item.symbols, + "alternative_sources": item.alternative_sources, + }, + source_valid_at=item.published_at, + claimed_published_at=item.published_at, + first_seen_at=observed_at, + fetched_at=item.fetched_at or response.fetched_at, + accepted_at=max(observed_at, datetime.now(UTC)), + collector_version="selected-news-forward@1", + policy_version="first-seen-only-v1", + source_tier=item.source_tier, + ) + await store.ingest_raw_event(conn, request) + + +__all__ = ["EventArchiveScheduler"] diff --git a/services/data/src/inalpha_data/event_models.py b/services/data/src/inalpha_data/event_models.py new file mode 100644 index 00000000..3684a7fd --- /dev/null +++ b/services/data/src/inalpha_data/event_models.py @@ -0,0 +1,273 @@ +"""Point-in-time market event contracts owned by the data service.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any, Literal +from uuid import UUID + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +EventType = Literal[ + "listing", + "delisting", + "exploit", + "chain_halt", + "regulatory", + "upgrade", + "unlock", + "burn", + "partnership", + "macro", + "other", +] +EventSourceTier = Literal["official", "professional_media", "aggregator", "structured"] + + +def _utc(value: datetime | None) -> datetime | None: + """Normalize timestamps so hashing and database comparisons are stable.""" + if value is None: + return None + if value.tzinfo is None: + return value.replace(tzinfo=UTC) + return value.astimezone(UTC) + + +class RawEventIngestRequest(BaseModel): + """Append one immutable raw-event version, or return the matching version.""" + + model_config = ConfigDict(extra="forbid") + + source: str = Field(min_length=1, max_length=120) + source_event_id: str = Field(min_length=1, max_length=500) + title: str = Field(default="", max_length=2_000) + content: str = Field(default="", max_length=200_000) + url: str | None = Field(default=None, max_length=4_000) + raw_payload: dict[str, Any] = Field(default_factory=dict) + source_valid_at: datetime | None = None + claimed_published_at: datetime | None = None + first_seen_at: datetime + fetched_at: datetime + accepted_at: datetime + collector_version: str = Field(min_length=1, max_length=120) + policy_version: str = Field(min_length=1, max_length=120) + source_tier: EventSourceTier + retracted: bool = False + + @field_validator( + "source_valid_at", + "claimed_published_at", + "first_seen_at", + "fetched_at", + "accepted_at", + mode="after", + ) + @classmethod + def normalize_time(cls, value: datetime | None) -> datetime | None: + return _utc(value) + + @model_validator(mode="after") + def validate_observation_order(self) -> RawEventIngestRequest: + if self.fetched_at < self.first_seen_at: + raise ValueError("fetched_at cannot be earlier than first_seen_at") + if self.accepted_at < self.first_seen_at: + raise ValueError("accepted_at cannot be earlier than first_seen_at") + return self + + +class RawEventRecord(BaseModel): + """Persisted immutable raw-event version.""" + + event_id: UUID + source: str + source_event_id: str + version: int + title: str + content: str + url: str | None + content_hash: str + raw_payload: dict[str, Any] + source_valid_at: datetime | None + claimed_published_at: datetime | None + first_seen_at: datetime + fetched_at: datetime + accepted_at: datetime + collector_version: str + policy_version: str + source_tier: EventSourceTier + supersedes_event_id: UUID | None + retracted: bool + created_at: datetime + + +class RawEventIngestResponse(BaseModel): + """Idempotent ingest result.""" + + event: RawEventRecord + created: bool + + +class EvidenceSpan(BaseModel): + """Bounded evidence reference; downstream prompts never receive raw content.""" + + start: int = Field(ge=0) + end: int = Field(gt=0) + quote_hash: str = Field(pattern=r"^[0-9a-f]{64}$") + + @model_validator(mode="after") + def validate_bounds(self) -> EvidenceSpan: + if self.end <= self.start: + raise ValueError("evidence span end must be greater than start") + return self + + +class EventFactWriteRequest(BaseModel): + """Append a versioned normalized fact derived from one raw event.""" + + model_config = ConfigDict(extra="forbid") + + raw_event_id: UUID + fact_key: str = Field(min_length=1, max_length=240) + event_type: EventType + assets: list[str] = Field(default_factory=list, max_length=64) + actor: str | None = Field(default=None, max_length=500) + action: str = Field(min_length=1, max_length=2_000) + severity: float = Field(ge=0, le=1) + confidence: float = Field(ge=0, le=1) + effective_at: datetime + available_at: datetime + evidence_spans: list[EvidenceSpan] = Field(default_factory=list, max_length=32) + extractor_version: str = Field(min_length=1, max_length=120) + policy_version: str = Field(min_length=1, max_length=120) + retracted: bool = False + + @field_validator("effective_at", "available_at", mode="after") + @classmethod + def normalize_time(cls, value: datetime) -> datetime: + normalized = _utc(value) + assert normalized is not None + return normalized + + @field_validator("assets", mode="after") + @classmethod + def normalize_assets(cls, value: list[str]) -> list[str]: + return sorted({item.strip().upper() for item in value if item.strip()}) + + +class EventFactRecord(BaseModel): + """One point-in-time normalized event fact version.""" + + fact_id: UUID + raw_event_id: UUID + fact_key: str + version: int + fact_hash: str + event_type: EventType + assets: list[str] + actor: str | None + action: str + severity: float + confidence: float + effective_at: datetime + available_at: datetime + evidence_spans: list[EvidenceSpan] + extractor_version: str + policy_version: str + supersedes_fact_id: UUID | None + retracted: bool + created_at: datetime + + +class EventFactWriteResponse(BaseModel): + """Idempotent normalized fact write result.""" + + fact: EventFactRecord + created: bool + + +class EventSnapshotRequest(BaseModel): + """Freeze all latest visible fact versions at one point in time.""" + + model_config = ConfigDict(extra="forbid") + + cutoff: datetime + policy_version: str = Field(min_length=1, max_length=120) + event_types: list[EventType] = Field(default_factory=list) + assets: list[str] = Field(default_factory=list, max_length=64) + + @field_validator("cutoff", mode="after") + @classmethod + def normalize_cutoff(cls, value: datetime) -> datetime: + normalized = _utc(value) + assert normalized is not None + return normalized + + @field_validator("event_types", mode="after") + @classmethod + def normalize_types(cls, value: list[EventType]) -> list[EventType]: + return sorted(set(value)) + + @field_validator("assets", mode="after") + @classmethod + def normalize_assets(cls, value: list[str]) -> list[str]: + return sorted({item.strip().upper() for item in value if item.strip()}) + + +class EventSnapshotRecord(BaseModel): + """Immutable event set used by research and backtests.""" + + snapshot_id: UUID + cutoff: datetime + policy_version: str + query_hash: str + events_sha256: str + coverage: dict[str, Any] + event_types: list[EventType] + assets: list[str] + fact_count: int + created_at: datetime + facts: list[EventFactRecord] = Field(default_factory=list) + + +class EventCoverageResponse(BaseModel): + """Operational coverage summary for Dashboard health views.""" + + as_of: datetime + sources: list[dict[str, Any]] + raw_event_count: int + fact_count: int + retraction_count: int + latest_accepted_at: datetime | None + + +class CoinMarketCalImportRequest(BaseModel): + """Bounded historical import request for the configured Professional API.""" + + from_date: datetime + to_date: datetime + coins: list[str] = Field(default_factory=list, max_length=100) + categories: list[str] = Field(default_factory=list, max_length=100) + limit: int = Field(default=100, ge=1, le=500) + + @field_validator("from_date", "to_date", mode="after") + @classmethod + def normalize_time(cls, value: datetime) -> datetime: + normalized = _utc(value) + assert normalized is not None + return normalized + + @model_validator(mode="after") + def validate_window(self) -> CoinMarketCalImportRequest: + if self.from_date >= self.to_date: + raise ValueError("from_date must be earlier than to_date") + return self + + +class EventImportResponse(BaseModel): + """Historical import summary without returning raw provider payloads.""" + + source: str + fetched: int + created: int + unchanged: int + failed: int diff --git a/services/data/src/inalpha_data/main.py b/services/data/src/inalpha_data/main.py index 5369a7d3..03e6ae23 100644 --- a/services/data/src/inalpha_data/main.py +++ b/services/data/src/inalpha_data/main.py @@ -22,6 +22,7 @@ backfill, bars, constituents, + events, fundamentals, fx, health, @@ -38,6 +39,7 @@ from .connectors import baostock as baostock_conn from .connectors import binance as binance_conn from .connectors import cn_market as cn_market_conn +from .connectors import coinmarketcal as coinmarketcal_conn from .connectors import fred as fred_conn from .connectors import news as news_conn from .connectors import symbol_search as symbol_search_conn @@ -48,6 +50,7 @@ from .connectors.news.hkex import HkexNewsProvider from .connectors.news.rss import RssFeedProvider from .connectors.news.sec import SecNewsProvider +from .event_archive import EventArchiveScheduler from .scheduler import ConstituentSnapshotScheduler, parse_indices _settings = get_data_settings() @@ -75,6 +78,12 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]: fred_conn.init_connector(api_key=_settings.fred_api_key) web_search_conn.init_connector() cn_market_conn.init_connector() + coinmarketcal_conn.init_connector( + api_key=_settings.coinmarketcal_api_key, + base_url=_settings.coinmarketcal_base_url, + timeout_s=_settings.event_provider_timeout_s, + historical_latency_s=_settings.event_historical_latency_s, + ) news_conn.init_router( [ SecNewsProvider( @@ -98,9 +107,15 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]: interval_s=_settings.constituent_snapshot_interval_h * 3600, ) snapshot_scheduler.start() + event_archive = EventArchiveScheduler( + enabled=_settings.event_archive_enabled, + interval_s=_settings.event_archive_interval_s, + ) + event_archive.start() try: yield finally: + await event_archive.stop() await snapshot_scheduler.stop() await symbol_search_conn.close_connector() await news_conn.close_router() @@ -109,6 +124,7 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]: await yfinance_conn.close_connector() await cn_market_conn.close_connector() await web_search_conn.close_connector() + await coinmarketcal_conn.close_connector() await baostock_conn.close_connector() await alpaca_conn.close_connector() await binance_conn.close_connector() @@ -129,6 +145,7 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]: app.include_router(backfill.router) app.include_router(ticker.router) app.include_router(news.router) +app.include_router(events.router) app.include_router(market.router) app.include_router(constituents.router) app.include_router(fundamentals.router) diff --git a/services/data/src/inalpha_data/storage/events.py b/services/data/src/inalpha_data/storage/events.py new file mode 100644 index 00000000..b1e70a4b --- /dev/null +++ b/services/data/src/inalpha_data/storage/events.py @@ -0,0 +1,317 @@ +"""Immutable market-event ledger, fact versions, and frozen snapshots.""" + +from __future__ import annotations + +import hashlib +import json +from datetime import UTC, datetime +from typing import Any +from uuid import UUID, uuid4 + +from psycopg import AsyncConnection + +from ..event_models import EventFactWriteRequest, EventSnapshotRequest, RawEventIngestRequest + +_RAW_COLUMNS = """event_id,source,source_event_id,version,title,content,url,content_hash, +raw_payload,source_valid_at,claimed_published_at,first_seen_at,fetched_at,accepted_at, +collector_version,policy_version,source_tier,supersedes_event_id,retracted,created_at""" +_FACT_COLUMNS = """fact_id,raw_event_id,fact_key,version,fact_hash,event_type,assets,actor, +action,severity,confidence,effective_at,available_at,evidence_spans,extractor_version, +policy_version,supersedes_fact_id,retracted,created_at""" +_SNAPSHOT_COLUMNS = """snapshot_id,cutoff,policy_version,query_hash,events_sha256,coverage, +event_types,assets,fact_count,created_at""" + + +def _canonical_hash(value: Any) -> str: + """Hash JSON with stable ordering and compact separators.""" + encoded = json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + default=_json_default, + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def _json_default(value: Any) -> str: + if isinstance(value, datetime): + return value.astimezone(UTC).isoformat().replace("+00:00", "Z") + if isinstance(value, UUID): + return str(value) + raise TypeError(f"unsupported canonical JSON value: {type(value).__name__}") + + +def _advisory_lock_key(*parts: object) -> str: + """Return a PostgreSQL-safe, collision-resistant text key for advisory locks.""" + return _canonical_hash(list(parts)) + + +async def ingest_raw_event( + conn: AsyncConnection, + request: RawEventIngestRequest, +) -> tuple[dict[str, Any], bool]: + """Append an event revision while treating an identical payload as an idempotent retry.""" + payload = request.model_dump(mode="json") + content_hash = _canonical_hash( + { + "title": request.title, + "content": request.content, + "url": request.url, + "raw_payload": request.raw_payload, + "retracted": request.retracted, + } + ) + lock_key = _advisory_lock_key(request.source, request.source_event_id) + async with conn.cursor() as cur: + await cur.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (lock_key,)) + await cur.execute( + f"""SELECT {_RAW_COLUMNS} FROM raw_market_events +WHERE source=%s AND source_event_id=%s ORDER BY version DESC LIMIT 1""", + (request.source, request.source_event_id), + ) + latest = await cur.fetchone() + if latest is not None and latest["content_hash"] == content_hash: + return dict(latest), False + version = int(latest["version"]) + 1 if latest is not None else 1 + event_id = uuid4() + await cur.execute( + f"""INSERT INTO raw_market_events( +event_id,source,source_event_id,version,title,content,url,content_hash,raw_payload, +source_valid_at,claimed_published_at,first_seen_at,fetched_at,accepted_at, +collector_version,policy_version,source_tier,supersedes_event_id,retracted) +VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s::jsonb,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) +RETURNING {_RAW_COLUMNS}""", + ( + event_id, + request.source, + request.source_event_id, + version, + request.title, + request.content, + request.url, + content_hash, + json.dumps(payload["raw_payload"], ensure_ascii=False), + request.source_valid_at, + request.claimed_published_at, + request.first_seen_at, + request.fetched_at, + request.accepted_at, + request.collector_version, + request.policy_version, + request.source_tier, + latest["event_id"] if latest is not None else None, + request.retracted, + ), + ) + row = await cur.fetchone() + assert row is not None + return dict(row), True + + +async def write_fact( + conn: AsyncConnection, + request: EventFactWriteRequest, +) -> tuple[dict[str, Any], bool]: + """Append a normalized fact revision after validating its raw event boundary.""" + fact_payload = request.model_dump(mode="json", exclude={"raw_event_id"}) + fact_hash = _canonical_hash(fact_payload) + lock_key = _advisory_lock_key(request.raw_event_id, request.fact_key) + async with conn.cursor() as cur: + await cur.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s,0))", (lock_key,)) + await cur.execute( + """SELECT first_seen_at,accepted_at,source_tier,policy_version +FROM raw_market_events WHERE event_id=%s""", + (request.raw_event_id,), + ) + raw = await cur.fetchone() + if raw is None: + raise LookupError("raw event not found") + # Real-time sources may never be backdated. Structured historical providers + # may use their frozen first-added policy, but still cannot predate it. + earliest_available = ( + raw["first_seen_at"] if raw["source_tier"] == "structured" else raw["accepted_at"] + ) + if request.available_at < earliest_available: + raise ValueError( + "available_at predates the source's point-in-time availability boundary" + ) + await cur.execute( + f"""SELECT {_FACT_COLUMNS} FROM market_event_facts +WHERE raw_event_id=%s AND fact_key=%s ORDER BY version DESC LIMIT 1""", + (request.raw_event_id, request.fact_key), + ) + latest = await cur.fetchone() + if latest is not None and latest["fact_hash"] == fact_hash: + return dict(latest), False + version = int(latest["version"]) + 1 if latest is not None else 1 + fact_id = uuid4() + await cur.execute( + f"""INSERT INTO market_event_facts( +fact_id,raw_event_id,fact_key,version,fact_hash,event_type,assets,actor,action, +severity,confidence,effective_at,available_at,evidence_spans,extractor_version, +policy_version,supersedes_fact_id,retracted) +VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s::jsonb,%s,%s,%s,%s) +RETURNING {_FACT_COLUMNS}""", + ( + fact_id, + request.raw_event_id, + request.fact_key, + version, + fact_hash, + request.event_type, + request.assets, + request.actor, + request.action, + request.severity, + request.confidence, + request.effective_at, + request.available_at, + json.dumps([item.model_dump(mode="json") for item in request.evidence_spans]), + request.extractor_version, + request.policy_version, + latest["fact_id"] if latest is not None else None, + request.retracted, + ), + ) + row = await cur.fetchone() + assert row is not None + return dict(row), True + + +async def create_snapshot( + conn: AsyncConnection, + request: EventSnapshotRequest, +) -> tuple[dict[str, Any], list[dict[str, Any]]]: + """Freeze latest visible fact versions using only their point-in-time availability.""" + query_payload = request.model_dump(mode="json") + query_hash = _canonical_hash(query_payload) + conditions = [ + "f.available_at<=%s", + "r.accepted_at<=%s", + "f.policy_version=%s", + ] + params: list[Any] = [request.cutoff, request.cutoff, request.policy_version] + if request.event_types: + conditions.append("f.event_type=ANY(%s)") + params.append(request.event_types) + if request.assets: + conditions.append("f.assets&&%s::text[]") + params.append(request.assets) + where = " AND ".join(conditions) + async with conn.cursor() as cur: + await cur.execute( + f"""SELECT DISTINCT ON (r.source,r.source_event_id,f.fact_key) +{",".join(f"f.{name.strip()}" for name in _FACT_COLUMNS.split(","))} +FROM market_event_facts f JOIN raw_market_events r ON r.event_id=f.raw_event_id +WHERE {where} +ORDER BY r.source,r.source_event_id,f.fact_key,f.available_at DESC,f.version DESC,f.fact_id""", + params, + ) + visible = [dict(row) for row in await cur.fetchall()] + facts = [row for row in visible if not row["retracted"]] + facts.sort(key=lambda row: (row["available_at"], str(row["fact_id"]))) + events_sha256 = _canonical_hash( + [{"fact_id": str(row["fact_id"]), "fact_hash": row["fact_hash"]} for row in facts] + ) + await cur.execute( + """SELECT source,count(*) AS raw_count,max(accepted_at) AS latest_accepted_at +FROM raw_market_events WHERE accepted_at<=%s GROUP BY source ORDER BY source""", + (request.cutoff,), + ) + coverage_rows = [dict(row) for row in await cur.fetchall()] + coverage = {"sources": coverage_rows, "complete": bool(coverage_rows)} + await cur.execute( + f"""INSERT INTO market_event_snapshots( +snapshot_id,cutoff,policy_version,query_hash,events_sha256,coverage,event_types,assets,fact_count) +VALUES(%s,%s,%s,%s,%s,%s::jsonb,%s,%s,%s) +ON CONFLICT(query_hash,events_sha256) DO UPDATE SET query_hash=EXCLUDED.query_hash +RETURNING {_SNAPSHOT_COLUMNS}""", + ( + uuid4(), + request.cutoff, + request.policy_version, + query_hash, + events_sha256, + json.dumps(coverage, default=_json_default), + request.event_types, + request.assets, + len(facts), + ), + ) + snapshot = await cur.fetchone() + assert snapshot is not None + for ordinal, fact in enumerate(facts): + await cur.execute( + """INSERT INTO market_event_snapshot_facts(snapshot_id,fact_id,ordinal) +VALUES(%s,%s,%s) ON CONFLICT(snapshot_id,fact_id) DO NOTHING""", + (snapshot["snapshot_id"], fact["fact_id"], ordinal), + ) + return dict(snapshot), facts + + +async def get_snapshot( + conn: AsyncConnection, + snapshot_id: UUID, +) -> tuple[dict[str, Any], list[dict[str, Any]]] | None: + """Load an immutable snapshot and its stable fact ordering.""" + async with conn.cursor() as cur: + await cur.execute( + f"SELECT {_SNAPSHOT_COLUMNS} FROM market_event_snapshots WHERE snapshot_id=%s", + (snapshot_id,), + ) + snapshot = await cur.fetchone() + if snapshot is None: + return None + await cur.execute( + f"""SELECT {",".join(f"f.{name.strip()}" for name in _FACT_COLUMNS.split(","))} +FROM market_event_snapshot_facts sf JOIN market_event_facts f USING(fact_id) +WHERE sf.snapshot_id=%s ORDER BY sf.ordinal""", + (snapshot_id,), + ) + facts = [dict(row) for row in await cur.fetchall()] + return dict(snapshot), facts + + +async def get_raw_event( + conn: AsyncConnection, + event_id: UUID, +) -> dict[str, Any] | None: + """Load raw evidence only for the trusted extraction boundary.""" + async with conn.cursor() as cur: + await cur.execute( + f"SELECT {_RAW_COLUMNS} FROM raw_market_events WHERE event_id=%s", + (event_id,), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def coverage(conn: AsyncConnection) -> dict[str, Any]: + """Return source freshness and global ledger counts for operations.""" + async with conn.cursor() as cur: + await cur.execute( + """SELECT source,count(*) AS raw_event_count,count(*) FILTER(WHERE retracted) AS retractions, +max(accepted_at) AS latest_accepted_at,max(version) AS max_version +FROM raw_market_events GROUP BY source ORDER BY source""" + ) + sources = [dict(row) for row in await cur.fetchall()] + await cur.execute( + """SELECT (SELECT count(*) FROM raw_market_events) AS raw_event_count, +(SELECT count(*) FROM market_event_facts) AS fact_count, +(SELECT count(*) FROM raw_market_events WHERE retracted) + + (SELECT count(*) FROM market_event_facts WHERE retracted) AS retraction_count, +(SELECT max(accepted_at) FROM raw_market_events) AS latest_accepted_at""" + ) + totals = await cur.fetchone() + return {"as_of": datetime.now(UTC), "sources": sources, **dict(totals or {})} + + +__all__ = [ + "coverage", + "create_snapshot", + "get_raw_event", + "get_snapshot", + "ingest_raw_event", + "write_fact", +] diff --git a/services/data/tests/test_event_models.py b/services/data/tests/test_event_models.py new file mode 100644 index 00000000..e2a55dee --- /dev/null +++ b/services/data/tests/test_event_models.py @@ -0,0 +1,53 @@ +"""Pure point-in-time event contract tests without a database dependency.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import pytest +from pydantic import ValidationError + +from inalpha_data.event_models import EventFactWriteRequest, RawEventIngestRequest +from inalpha_data.storage.events import _advisory_lock_key + + +def test_event_advisory_lock_key_is_postgresql_safe_and_unambiguous() -> None: + first = _advisory_lock_key("local-demo", "listing-01") + + assert "\0" not in first + assert len(first) == 64 + assert first == _advisory_lock_key("local-demo", "listing-01") + assert _advisory_lock_key("a", "bc") != _advisory_lock_key("ab", "c") + + +def test_realtime_raw_event_cannot_claim_acceptance_before_first_seen() -> None: + now = datetime.now(UTC) + with pytest.raises(ValidationError, match="accepted_at"): + RawEventIngestRequest( + source="official-exchange", + source_event_id="1", + first_seen_at=now, + fetched_at=now, + accepted_at=now - timedelta(seconds=1), + collector_version="test@1", + policy_version="first-seen-only-v1", + source_tier="official", + ) + + +def test_fact_assets_are_normalized_and_deduplicated() -> None: + now = datetime.now(UTC) + fact = EventFactWriteRequest( + raw_event_id="3b67b111-1dac-4bf4-b70b-ab683c50469d", + fact_key="listing:btc", + event_type="listing", + assets=["btc", " BTC ", "ETH"], + action="exchange lists BTC", + severity=0.8, + confidence=0.9, + effective_at=now, + available_at=now, + extractor_version="test@1", + policy_version="event-time-policy-v1", + ) + assert fact.assets == ["BTC", "ETH"] diff --git a/services/evolver/src/inalpha_evolver/api/approval.py b/services/evolver/src/inalpha_evolver/api/approval.py index c00cb394..42b041d3 100644 --- a/services/evolver/src/inalpha_evolver/api/approval.py +++ b/services/evolver/src/inalpha_evolver/api/approval.py @@ -24,6 +24,7 @@ def verify_evolution_approval( provider: str, llm_config_digest: str, request_digest: str, + grant_purpose: str, settings: EvolverSettings, ) -> None: """Verify one owner/request-bound grant without giving Evolver signing authority.""" @@ -54,6 +55,7 @@ def verify_evolution_approval( "operation_id": operation_id, "config_id": config_id, "provider": provider, + "grant_purpose": grant_purpose, "llm_config_digest": llm_config_digest, "request_digest": request_digest, } diff --git a/services/evolver/src/inalpha_evolver/api/campaign_routes.py b/services/evolver/src/inalpha_evolver/api/campaign_routes.py new file mode 100644 index 00000000..3141722a --- /dev/null +++ b/services/evolver/src/inalpha_evolver/api/campaign_routes.py @@ -0,0 +1,320 @@ +"""E2 campaign, forward evidence, sealed holdout, and adoption API.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Annotated +from uuid import UUID + +from fastapi import APIRouter, BackgroundTasks, Depends, Header, Request, status +from inalpha_paper.account_id import account_id_from_user +from inalpha_shared.auth import User, get_current_user +from inalpha_shared.db import DBConn +from inalpha_shared.errors import ConflictError, NotFoundError, ValidationError + +from ..config import get_evolver_settings +from ..event_client import fetch_event_snapshot +from ..hypothesis.compiler import compile_hypothesis, expand_implementations +from ..hypothesis.models import HypothesisSpec +from ..hypothesis.seeding import seed_generation_one +from ..runtime.campaign import evaluate_sealed_holdout +from ..sandbox.ast_audit import assert_safe +from ..storage import campaigns as store +from .approval import verify_evolution_approval +from .schemas import ( + AdoptionListResponse, + AdoptionResponse, + CampaignListResponse, + CampaignResponse, + CreateCampaignRequest, + ForwardEvidenceRequest, + LockChampionRequest, + campaign_request_digest, +) + +router = APIRouter() + + +def _response(row: dict[str, object]) -> CampaignResponse: + return CampaignResponse(**row) + + +@router.post( + "/campaigns", + response_model=CampaignResponse, + status_code=status.HTTP_201_CREATED, +) +async def create_campaign( + body: CreateCampaignRequest, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], + idempotency_key: Annotated[str, Header(alias="Idempotency-Key", min_length=8, max_length=128)], + evolution_credential: Annotated[ + str, + Header(alias="X-Evolution-Credential", min_length=100, max_length=4096), + ], +) -> CampaignResponse: + """Create a frozen campaign after compiling all deterministic implementation arms.""" + owner = account_id_from_user(user) + settings = get_evolver_settings() + if not settings.event_evolution_enabled: + raise ValidationError( + "event evolution is disabled", + code="EVENT_EVOLUTION_DISABLED", + ) + snapshot = await fetch_event_snapshot( + body.event_snapshot_id, + owner_account_id=owner, + settings=settings, + ) + hypotheses = body.hypotheses or seed_generation_one( + snapshot, + body.config.symbol.split("/")[0].upper(), + ) + if len(hypotheses) != 8: + raise ValidationError( + "event campaign requires exactly eight generation-one hypothesis slots", + code="CAMPAIGN_DIRECTION_COVERAGE_REQUIRED", + ) + digest = campaign_request_digest(body) + verify_evolution_approval( + evolution_credential, + owner_sub=user.user_id, + operation_id=idempotency_key, + config_id=body.llm.config_id, + provider=body.llm.provider, + llm_config_digest=body.llm.config_digest, + request_digest=digest, + grant_purpose="event_campaign", + settings=settings, + ) + for hypothesis in hypotheses: + for implementation in expand_implementations(hypothesis): + compiled = compile_hypothesis(implementation) + assert_safe(compiled.source_code) + frozen_config = { + **body.config.model_dump(mode="json"), + "event_snapshot": { + "snapshot_id": snapshot["snapshot_id"], + "events_sha256": snapshot["events_sha256"], + "policy_version": snapshot["policy_version"], + "fact_count": snapshot["fact_count"], + "cutoff": snapshot["cutoff"], + }, + "compiler_version": "event-strategy-compiler-v1", + "selection_version": "pareto-novelty-v1", + "fdr_q": 0.10, + "llm_call_topology": {"calls_per_generation": 2, "hypotheses_per_call": 4}, + "estimated_reserved_llm_cost_usd": (10 * body.llm.pricing.estimated_max_usd_per_candidate), + "sealed_holdout_thresholds": { + "sharpe_gt": 0, + "net_return_pct_gt": 0, + "max_drawdown_pct_lte": 25, + "num_trades_gt": 0, + }, + "created_at": datetime.now(UTC).isoformat(), + } + async with db.transaction(): + row = await store.insert_campaign( + db, + owner_account_id=owner, + requested_by_sub=user.user_id, + idempotency_key=idempotency_key, + request_hash=digest, + source_run_id=body.source_run_id, + event_snapshot_id=body.event_snapshot_id, + frozen_config=frozen_config, + llm_snapshot=body.llm.model_dump(mode="json"), + llm_credential_grant=evolution_credential, + hypotheses=hypotheses, + ) + if row.get("request_hash") != digest: + raise ConflictError("idempotency key reused", code="IDEMPOTENCY_KEY_REUSED") + loaded = await store.get_campaign(db, row["campaign_id"], owner) + assert loaded is not None + return _response(loaded) + + +@router.get("/campaigns", response_model=CampaignListResponse) +async def list_campaigns( + db: DBConn, + user: Annotated[User, Depends(get_current_user)], + limit: int = 20, +) -> CampaignListResponse: + rows = await store.list_campaigns(db, account_id_from_user(user), limit=min(max(limit, 1), 50)) + return CampaignListResponse(items=[_response(row) for row in rows]) + + +@router.get("/campaigns/{campaign_id}", response_model=CampaignResponse) +async def get_campaign( + campaign_id: UUID, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], +) -> CampaignResponse: + row = await store.get_campaign(db, campaign_id, account_id_from_user(user)) + if row is None: + raise NotFoundError("campaign not found", code="CAMPAIGN_NOT_FOUND") + return _response(row) + + +@router.post("/campaigns/{campaign_id}/start", response_model=CampaignResponse) +async def start_campaign( + campaign_id: UUID, + request: Request, + background: BackgroundTasks, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], +) -> CampaignResponse: + owner = account_id_from_user(user) + manager = getattr(request.app.state, "campaign_manager", None) + if manager is None: + raise ValidationError("event evolution is disabled", code="EVENT_EVOLUTION_DISABLED") + row = await store.transition( + db, + campaign_id, + owner, + from_statuses=("draft",), + to_status="replaying", + values={ + "active_generation": 1, + "failure_code": None, + "failure_message": None, + "finished_at": None, + }, + ) + if row is None: + raise ConflictError("campaign cannot start", code="CAMPAIGN_STATE_CONFLICT") + loaded = await store.get_campaign(db, campaign_id, owner) + assert loaded is not None + background.add_task(manager.notify_async) + return _response(loaded) + + +@router.post("/campaigns/{campaign_id}/lock", response_model=CampaignResponse) +async def lock_campaign_champion( + campaign_id: UUID, + body: LockChampionRequest, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], +) -> CampaignResponse: + owner = account_id_from_user(user) + row = await store.lock_champion(db, campaign_id, owner, body.candidate_id) + if row is None: + raise ConflictError( + "campaign must finish generation five before locking one champion", + code="CAMPAIGN_LOCK_CONFLICT", + ) + loaded = await store.get_campaign(db, campaign_id, owner) + assert loaded is not None + return _response(loaded) + + +@router.post("/campaigns/{campaign_id}/forward", response_model=CampaignResponse) +async def record_forward_evidence( + campaign_id: UUID, + body: ForwardEvidenceRequest, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], +) -> CampaignResponse: + owner = account_id_from_user(user) + metrics = body.model_dump(mode="json") | {"passed": body.passed()} + row = await store.record_forward( + db, + campaign_id, + owner, + event_count=body.event_count, + metrics=metrics, + ) + if row is None: + raise ConflictError( + "campaign is not waiting for forward evidence", code="FORWARD_STATE_CONFLICT" + ) + loaded = await store.get_campaign(db, campaign_id, owner) + assert loaded is not None + return _response(loaded) + + +@router.post("/campaigns/{campaign_id}/holdout", response_model=CampaignResponse) +async def consume_campaign_holdout( + campaign_id: UUID, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], +) -> CampaignResponse: + """Run the pre-locked champion against sealed bars; request bodies cannot inject results.""" + owner = account_id_from_user(user) + campaign = await store.get_campaign(db, campaign_id, owner) + implementation = await store.locked_implementation(db, campaign_id, owner) + if campaign is None or implementation is None: + raise ConflictError( + "sealed holdout is unavailable or already consumed", + code="HOLDOUT_ALREADY_CONSUMED", + ) + reserved = await store.reserve_holdout(db, campaign_id, owner) + if reserved is None: + raise ConflictError( + "sealed holdout is unavailable or already consumed", + code="HOLDOUT_ALREADY_CONSUMED", + ) + try: + passed, evidence = await evaluate_sealed_holdout( + campaign, + source_code=implementation["source_code"], + hypothesis=HypothesisSpec.model_validate(implementation["spec"]), + settings=get_evolver_settings(), + ) + except Exception as exc: + passed = False + evidence = { + "error_code": str(getattr(exc, "code", "SEALED_HOLDOUT_FAILED")), + "error_message": str(exc)[:1000], + } + row = await store.finalize_holdout( + db, + campaign_id, + owner, + passed=passed, + evidence=evidence, + ) + if row is None: + raise ConflictError( + "sealed holdout finalization lost compare-and-swap", + code="HOLDOUT_FINALIZE_CONFLICT", + ) + loaded = await store.get_campaign(db, campaign_id, owner) + assert loaded is not None + return _response(loaded) + + +@router.post("/campaigns/{campaign_id}/adopt", response_model=AdoptionResponse) +async def adopt_campaign_winner( + campaign_id: UUID, + db: DBConn, + user: Annotated[User, Depends(get_current_user)], +) -> AdoptionResponse: + owner = account_id_from_user(user) + async with db.transaction(): + adoption = await store.adopt_graduated(db, campaign_id, owner) + if adoption is None: + raise ValidationError( + "graduated campaign has no adoptable locked source", + code="CAMPAIGN_NOT_ADOPTABLE", + ) + return AdoptionResponse(**adoption) + + +@router.get("/adoptions", response_model=AdoptionListResponse) +async def list_strategy_adoptions( + db: DBConn, + user: Annotated[User, Depends(get_current_user)], + limit: int = 50, +) -> AdoptionListResponse: + """List experimental campaign winners separately from promoted Paper candidates.""" + rows = await store.list_adoptions( + db, + account_id_from_user(user), + limit=min(max(limit, 1), 100), + ) + return AdoptionListResponse(items=rows) + + +__all__ = ["router"] diff --git a/services/evolver/src/inalpha_evolver/api/routes.py b/services/evolver/src/inalpha_evolver/api/routes.py index af19e36d..6a43f2c7 100644 --- a/services/evolver/src/inalpha_evolver/api/routes.py +++ b/services/evolver/src/inalpha_evolver/api/routes.py @@ -1,9 +1,12 @@ """Evolver API 路由集合。""" + from fastapi import APIRouter +from .campaign_routes import router as campaign_router from .detail_routes import router as detail_router from .run_routes import router as run_router router = APIRouter(prefix="/api/v1", tags=["evolution"]) router.include_router(run_router) router.include_router(detail_router) +router.include_router(campaign_router) diff --git a/services/evolver/src/inalpha_evolver/api/run_routes.py b/services/evolver/src/inalpha_evolver/api/run_routes.py index 76cda8b7..ee6e98c5 100644 --- a/services/evolver/src/inalpha_evolver/api/run_routes.py +++ b/services/evolver/src/inalpha_evolver/api/run_routes.py @@ -47,6 +47,7 @@ async def start_run( provider=body.llm.provider, llm_config_digest=body.llm.config_digest, request_digest=approval_request_digest(body), + grant_purpose="e1_run", settings=settings, ) async with db.transaction(): diff --git a/services/evolver/src/inalpha_evolver/api/schemas.py b/services/evolver/src/inalpha_evolver/api/schemas.py index 9a175074..9613d7a4 100644 --- a/services/evolver/src/inalpha_evolver/api/schemas.py +++ b/services/evolver/src/inalpha_evolver/api/schemas.py @@ -5,6 +5,7 @@ import hashlib import hmac import json +import struct from datetime import UTC, datetime, timedelta from decimal import Decimal from typing import Any, Literal @@ -16,6 +17,7 @@ from ..data.datetime_policy import MAX_AS_OF_CLOCK_SKEW from ..data.manifest import DatasetManifest +from ..hypothesis.models import HypothesisSpec class EvolutionConfig(BaseModel): @@ -155,6 +157,42 @@ def compute_llm_config_digest(snapshot: EvolutionLLMSnapshot) -> str: return hashlib.sha256(encoded).hexdigest() +def campaign_request_digest(request: CreateCampaignRequest) -> str: + """Bind the credential grant to every campaign input affecting cost or results.""" + config = request.config + hypotheses_hash = hashlib.sha256( + json.dumps( + [item.model_dump(mode="json") for item in request.hypotheses], + sort_keys=True, + ensure_ascii=False, + separators=(",", ":"), + ).encode() + ).hexdigest() + canonical = [ + str(request.event_snapshot_id), + str(request.source_run_id or ""), + config.venue, + config.symbol, + config.timeframe, + str(int(config.from_ts.timestamp() * 1_000)), + str(int(config.as_of.timestamp() * 1_000)), + struct.pack(">d", config.initial_cash).hex(), + struct.pack(">d", config.fee_rate).hex(), + config.trading_mode, + str(config.leverage), + struct.pack(">d", config.discovery_ratio).hex(), + struct.pack(">d", config.generation_validation_ratio).hex(), + struct.pack(">d", config.sealed_holdout_ratio).hex(), + config.execution_model_version, + config.control_matcher_version, + str(config.random_seed), + request.llm.config_digest, + hypotheses_hash, + ] + encoded = json.dumps(canonical, ensure_ascii=False, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + def _number_text(value: int | float) -> str: text = format(Decimal(str(value)), "f") if "." in text: @@ -164,7 +202,7 @@ def _number_text(value: int | float) -> str: class StartRunRequest(BaseModel): seed_strategy_id: str = Field(default="sma_cross_v1", max_length=128) - budget: int = Field(default=4, ge=1, le=20) + budget: int = Field(default=4, ge=1, le=24) config: EvolutionConfig llm: EvolutionLLMSnapshot @@ -220,3 +258,198 @@ class RunStatusResponse(BaseModel): class RunListResponse(BaseModel): items: list[RunStatusResponse] next_cursor: str | None = None + + +class CampaignConfig(BaseModel): + """Frozen statistical and execution contract for one E2 campaign.""" + + model_config = ConfigDict(extra="forbid") + + venue: str = Field(default="binance", min_length=1, max_length=40) + symbol: str = Field(default="BTC/USDT", min_length=1, max_length=80) + timeframe: Literal["15m", "1h", "4h"] = "1h" + from_ts: datetime + as_of: datetime + initial_cash: float = Field(default=10_000.0, ge=100) + fee_rate: float = Field(default=0.001, ge=0, le=0.1) + trading_mode: Literal["spot", "perp"] = "perp" + leverage: int = Field(default=1, ge=1, le=20) + discovery_ratio: float = 0.6 + generation_validation_ratio: float = 0.2 + sealed_holdout_ratio: float = 0.2 + execution_model_version: Literal["event-fill-v1"] = "event-fill-v1" + control_matcher_version: Literal["event-control-v1"] = "event-control-v1" + random_seed: int = Field(default=0, ge=0, le=2**31 - 1) + + @field_validator("from_ts", "as_of", mode="after") + @classmethod + def normalize_datetimes(cls, value: datetime) -> datetime: + return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC) + + @model_validator(mode="after") + def validate_window(self) -> CampaignConfig: + if self.from_ts >= self.as_of: + raise ValueError("from_ts must be earlier than as_of") + if ( + self.discovery_ratio, + self.generation_validation_ratio, + self.sealed_holdout_ratio, + ) != (0.6, 0.2, 0.2): + raise ValueError("campaign split is frozen at 60/20/20") + if self.trading_mode == "spot" and self.leverage != 1: + raise ValueError("spot campaigns must use leverage=1") + return self + + +class CreateCampaignRequest(BaseModel): + """Create a manual-hypothesis vertical or generation-one campaign.""" + + model_config = ConfigDict(extra="forbid") + + event_snapshot_id: UUID + source_run_id: UUID | None = None + config: CampaignConfig + llm: EvolutionLLMSnapshot + hypotheses: list[HypothesisSpec] = Field(default_factory=list, max_length=8) + + @model_validator(mode="after") + def validate_unique_hypotheses(self) -> CreateCampaignRequest: + ids = [item.hypothesis_id for item in self.hypotheses] + if len(ids) != len(set(ids)): + raise ValueError("hypothesis_id values must be unique") + return self + + +class GenerationProjection(BaseModel): + generation: int + hypothesis_count: int + selected_count: int + best_credit: float | None = None + best_novelty: float | None = None + + +class HypothesisResponse(BaseModel): + hypothesis_id: UUID + campaign_id: UUID + generation: int + slot: int + lineage_kind: str + lane: str + parent_ids: list[UUID] + spec: dict[str, Any] + spec_hash: str + upper_credit: float | None = None + novelty_score: float | None = None + pareto_rank: int | None = None + selected: bool = False + created_at: datetime + + +class ImplementationResponse(BaseModel): + implementation_id: UUID + campaign_id: UUID + hypothesis_id: UUID + generation: int + profile: str + source_hash: str + outcome: str + fitness: float | None = None + validation_metrics: dict[str, Any] | None = None + event_metrics: dict[str, Any] | None = None + evidence_quality: float | None = None + novelty_score: float | None = None + fdr_pass: bool | None = None + error_code: str | None = None + error_message: str | None = None + created_at: datetime + updated_at: datetime + + +class CampaignResponse(BaseModel): + campaign_id: UUID + owner_account_id: UUID + source_run_id: UUID | None = None + status: str + active_generation: int + hypothesis_budget: int + implementations_per_hypothesis: int + max_generations: int + event_snapshot_id: UUID + frozen_config: dict[str, Any] + llm_snapshot: EvolutionLLMSnapshot + llm_config_digest: str + llm_cost_usd: float + locked_candidate_id: UUID | None = None + holdout_consumed_at: datetime | None = None + forward_started_at: datetime | None = None + forward_deadline_at: datetime | None = None + forward_event_count: int = 0 + forward_metrics: dict[str, Any] | None = None + failure_code: str | None = None + failure_message: str | None = None + state_version: int + created_at: datetime + updated_at: datetime + finished_at: datetime | None = None + generations: list[GenerationProjection] = Field(default_factory=list) + hypotheses: list[HypothesisResponse] = Field(default_factory=list) + implementations: list[ImplementationResponse] = Field(default_factory=list) + + +class CampaignListResponse(BaseModel): + items: list[CampaignResponse] + + +class LockChampionRequest(BaseModel): + candidate_id: UUID + + +class ForwardEvidenceRequest(BaseModel): + """Aggregate pre-registered evidence; raw event text is not accepted.""" + + event_count: int = Field(ge=0) + net_return_pct: float + parent_excess_return_pct: float + control_excess_return_pct: float + positive_event_count: int = Field(ge=0) + risk_breaches: int = Field(default=0, ge=0) + data_quality_warnings: list[str] = Field(default_factory=list, max_length=100) + + @model_validator(mode="after") + def validate_counts(self) -> ForwardEvidenceRequest: + if self.positive_event_count > self.event_count: + raise ValueError("positive_event_count cannot exceed event_count") + return self + + def passed(self) -> bool: + return bool( + self.net_return_pct > 0 + and self.parent_excess_return_pct > 0 + and self.control_excess_return_pct > 0 + and self.event_count >= 3 + and self.positive_event_count * 3 >= self.event_count * 2 + and self.risk_breaches == 0 + and not self.data_quality_warnings + ) + + +class AdoptionResponse(BaseModel): + adoption_id: UUID + artifact_id: UUID + owner_account_id: UUID + campaign_id: UUID | None + evidence_grade: Literal["standard", "limited"] + status: Literal["experimental", "accepted", "rejected"] + runner_eligible: Literal[False] + evidence: dict[str, Any] + adopted_at: datetime + + +class AdoptionSummaryResponse(AdoptionResponse): + source_hash: str + compiler_version: str | None = None + campaign_status: str | None = None + + +class AdoptionListResponse(BaseModel): + items: list[AdoptionSummaryResponse] diff --git a/services/evolver/src/inalpha_evolver/config.py b/services/evolver/src/inalpha_evolver/config.py index 5d052461..47f52702 100644 --- a/services/evolver/src/inalpha_evolver/config.py +++ b/services/evolver/src/inalpha_evolver/config.py @@ -74,6 +74,20 @@ class EvolverSettings(BaseSettings): default="http://127.0.0.1:8001", alias="DATA_SERVICE_URL", ) + evolver_data_timeout_s: int = Field( + default=60, + alias="EVOLVER_DATA_TIMEOUT_S", + ge=5, + le=300, + description="E2 冻结行情与事件快照预检超时。", + ) + evolver_credential_timeout_s: int = Field( + default=60, + alias="EVOLVER_CREDENTIAL_TIMEOUT_S", + ge=5, + le=300, + description="Dashboard owner LLM 凭据兑换超时。", + ) dashboard_service_url: str = Field( default="http://127.0.0.1:3001", alias="DASHBOARD_SERVICE_URL", @@ -92,6 +106,23 @@ class EvolverSettings(BaseSettings): alias="EVOLUTION_CREDENTIAL_PUBLIC_KEY_B64", description="Orchestration Ed25519 签名公钥(SPKI DER base64)。", ) + event_evolution_enabled: bool = Field(default=False, alias="EVENT_EVOLUTION_ENABLED") + """E2 campaign API/dispatcher feature flag;E1 run 不受影响。""" + + campaign_lease_ttl_s: int = Field( + default=90, + ge=30, + le=600, + alias="CAMPAIGN_LEASE_TTL_S", + description="E2 campaign worker lease 与 fencing token 续租周期基准。", + ) + campaign_max_concurrent: int = Field( + default=1, + ge=1, + le=8, + alias="CAMPAIGN_MAX_CONCURRENT", + description="单个 Evolver 进程同时执行的 campaign 上限。", + ) # ---- LLM ---- llm_api_key: str = Field( diff --git a/services/evolver/src/inalpha_evolver/data/bar_hash.py b/services/evolver/src/inalpha_evolver/data/bar_hash.py index 960fd25d..8c020baf 100644 --- a/services/evolver/src/inalpha_evolver/data/bar_hash.py +++ b/services/evolver/src/inalpha_evolver/data/bar_hash.py @@ -16,12 +16,13 @@ def bars_content_hash( ) -> str: """按市场身份、时间戳和 OHLCV 的规范二进制编码计算 SHA-256。""" digest = hashlib.sha256() - identity = f"e1-bars-v1\0{instrument.venue}\0{instrument.symbol}\0{context.canonical_timeframe}\0" + identity = f"e2-bars-v2\0{instrument.venue}\0{instrument.symbol}\0{context.canonical_timeframe}\0" digest.update(identity.encode()) for bar in bars: digest.update( struct.pack( - "!q5d", + "!qq5d", + bar.bar_open_at, bar.ts_event, bar.open, bar.high, diff --git a/services/evolver/src/inalpha_evolver/data/bar_quality.py b/services/evolver/src/inalpha_evolver/data/bar_quality.py index 1949156c..e69225d7 100644 --- a/services/evolver/src/inalpha_evolver/data/bar_quality.py +++ b/services/evolver/src/inalpha_evolver/data/bar_quality.py @@ -83,7 +83,7 @@ def _validate_grid( def _bar_datetime(bar: Bar) -> datetime: - return datetime.fromtimestamp(bar.ts_event / 1_000_000_000, tz=UTC) + return datetime.fromtimestamp(bar.bar_open_at / 1_000_000_000, tz=UTC) def _utc(value: datetime) -> datetime: diff --git a/services/evolver/src/inalpha_evolver/data/frozen_bars.py b/services/evolver/src/inalpha_evolver/data/frozen_bars.py index e6cdc884..640c2760 100644 --- a/services/evolver/src/inalpha_evolver/data/frozen_bars.py +++ b/services/evolver/src/inalpha_evolver/data/frozen_bars.py @@ -55,7 +55,7 @@ async def load( context=context, as_of=cutoff, ) - first, latest = _bar_time(bars[0].ts_event), _bar_time(bars[-1].ts_event) + first, latest = _bar_time(bars[0].bar_open_at), _bar_time(bars[-1].bar_open_at) manifest = DatasetManifest( venue=venue, symbol=symbol, diff --git a/services/evolver/src/inalpha_evolver/evaluator/event_study.py b/services/evolver/src/inalpha_evolver/evaluator/event_study.py new file mode 100644 index 00000000..0b5a87b3 --- /dev/null +++ b/services/evolver/src/inalpha_evolver/evaluator/event_study.py @@ -0,0 +1,192 @@ +"""Deterministic event reaction and matched no-event control metrics.""" + +from __future__ import annotations + +import bisect +import statistics +from dataclasses import dataclass +from itertools import pairwise +from typing import Any + +from inalpha_paper.model.data import Bar +from inalpha_paper.model.market_events import MarketEvent + + +@dataclass(frozen=True, slots=True) +class EventStudyResult: + """Aggregate event-window evidence safe to expose to upper-level selection.""" + + event_count: int + matched_control_count: int + mean_event_return_pct: float + mean_control_return_pct: float + event_advantage_pct: float + positive_event_ratio: float + mean_mfe_pct: float + mean_mae_pct: float + unmatched_events: int + event_effects: tuple[float, ...] + + def as_dict(self) -> dict[str, Any]: + return { + "event_count": self.event_count, + "matched_control_count": self.matched_control_count, + "mean_event_return_pct": self.mean_event_return_pct, + "mean_control_return_pct": self.mean_control_return_pct, + "event_advantage_pct": self.event_advantage_pct, + "positive_event_ratio": self.positive_event_ratio, + "mean_mfe_pct": self.mean_mfe_pct, + "mean_mae_pct": self.mean_mae_pct, + "unmatched_events": self.unmatched_events, + "event_effects": list(self.event_effects), + } + + +def evaluate_event_reactions( + *, + bars: list[Bar], + events: list[MarketEvent], + asset: str, + direction: str, + holding_bars: int, + exclusion_bars: int, + volatility_tolerance: float, + volume_tolerance: float, +) -> EventStudyResult: + """Compare event windows with nearest pre-event regime/volume controls.""" + if len(bars) < holding_bars + 3: + return _empty(len(events)) + times = [bar.bar_known_at for bar in bars] + event_indices: list[int] = [] + for event in _independent_events(events, asset): + if event.assets and asset.upper() not in event.assets: + continue + index = bisect.bisect_left(times, event.available_at) + if 1 <= index < len(bars) - holding_bars: + event_indices.append(index) + excluded = { + index + for event_index in event_indices + for index in range( + max(1, event_index - exclusion_bars), + min(len(bars) - holding_bars, event_index + exclusion_bars + 1), + ) + } + event_returns: list[float] = [] + control_returns: list[float] = [] + mfes: list[float] = [] + maes: list[float] = [] + paired_effects: list[float] = [] + unmatched = 0 + for event_index in event_indices: + event_return, mfe, mae = _window_metrics(bars, event_index, holding_bars, direction) + event_returns.append(event_return) + mfes.append(mfe) + maes.append(mae) + control = _match_control( + bars, + event_index, + holding_bars=holding_bars, + excluded=excluded, + volatility_tolerance=volatility_tolerance, + volume_tolerance=volume_tolerance, + ) + if control is None: + unmatched += 1 + else: + control_return = _window_metrics(bars, control, holding_bars, direction)[0] + control_returns.append(control_return) + paired_effects.append(event_return - control_return) + event_mean = statistics.mean(event_returns) if event_returns else 0.0 + control_mean = statistics.mean(control_returns) if control_returns else 0.0 + return EventStudyResult( + event_count=len(event_returns), + matched_control_count=len(control_returns), + mean_event_return_pct=event_mean, + mean_control_return_pct=control_mean, + event_advantage_pct=event_mean - control_mean, + positive_event_ratio=( + sum(value > 0 for value in event_returns) / len(event_returns) if event_returns else 0.0 + ), + mean_mfe_pct=statistics.mean(mfes) if mfes else 0.0, + mean_mae_pct=statistics.mean(maes) if maes else 0.0, + unmatched_events=unmatched, + event_effects=tuple(paired_effects), + ) + + +def _independent_events(events: list[MarketEvent], asset: str) -> list[MarketEvent]: + """Cluster same-asset/type messages within 24 hours into one independent event.""" + cluster_ns = 24 * 60 * 60 * 1_000_000_000 + last_seen: dict[tuple[str, str], int] = {} + independent: list[MarketEvent] = [] + for event in sorted(events, key=lambda item: (item.available_at, item.event_id)): + assets = event.assets or (asset.upper(),) + relevant = tuple(item for item in assets if item == asset.upper()) + if not relevant: + continue + key = (relevant[0], event.event_type) + previous = last_seen.get(key) + if previous is not None and event.available_at - previous < cluster_ns: + continue + last_seen[key] = event.available_at + independent.append(event) + return independent + + +def _match_control( + bars: list[Bar], + event_index: int, + *, + holding_bars: int, + excluded: set[int], + volatility_tolerance: float, + volume_tolerance: float, +) -> int | None: + target_volatility, target_volume = _context(bars, event_index) + best: tuple[float, int] | None = None + for index in range(20, event_index - holding_bars): + if index in excluded: + continue + volatility, volume = _context(bars, index) + vol_distance = _relative_distance(volatility, target_volatility) + volume_distance = _relative_distance(volume, target_volume) + if vol_distance > volatility_tolerance or volume_distance > volume_tolerance: + continue + distance = vol_distance + volume_distance + if best is None or (distance, index) < best: + best = (distance, index) + return best[1] if best else None + + +def _context(bars: list[Bar], index: int) -> tuple[float, float]: + window = bars[max(0, index - 20) : index] + returns = [ + (right.close / left.close) - 1.0 for left, right in pairwise(window) if left.close > 0 + ] + volatility = statistics.pstdev(returns) if len(returns) > 1 else 0.0 + volume = statistics.mean(bar.volume for bar in window) if window else 0.0 + return volatility, volume + + +def _window_metrics( + bars: list[Bar], index: int, holding_bars: int, direction: str +) -> tuple[float, float, float]: + entry = bars[index].close + window = bars[index + 1 : index + holding_bars + 1] + sign = -1.0 if direction == "short" else 1.0 + returns = [sign * ((bar.close / entry) - 1.0) * 100 for bar in window if entry > 0] + final_return = returns[-1] if returns else 0.0 + return final_return, max(returns, default=0.0), min(returns, default=0.0) + + +def _relative_distance(left: float, right: float) -> float: + denominator = max(abs(right), 1e-12) + return abs(left - right) / denominator + + +def _empty(unmatched_events: int) -> EventStudyResult: + return EventStudyResult(0, 0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, unmatched_events, ()) + + +__all__ = ["EventStudyResult", "evaluate_event_reactions"] diff --git a/services/evolver/src/inalpha_evolver/evaluator/frozen.py b/services/evolver/src/inalpha_evolver/evaluator/frozen.py index 8c3544c7..0684ec4e 100644 --- a/services/evolver/src/inalpha_evolver/evaluator/frozen.py +++ b/services/evolver/src/inalpha_evolver/evaluator/frozen.py @@ -1,10 +1,13 @@ """冻结数据集上的真实策略评估。""" + from __future__ import annotations from dataclasses import dataclass from inalpha_paper.evaluation_executor import KillableEngineRunner +from inalpha_paper.execution.exchange import EventExecutionPolicy from inalpha_paper.kernel.identifiers import InstrumentId +from inalpha_paper.model.market_events import MarketEvent from inalpha_paper.strategy_evaluation import ( evaluate_buy_and_hold, evaluate_strategy_source, @@ -23,6 +26,10 @@ class FrozenDatasetEvaluator: initial_cash: float = 10_000.0 fee_rate: float = 0.001 validation_split: float = 0.3 + trading_mode: str = "spot" + leverage: int = 1 + events: tuple[MarketEvent, ...] = () + event_execution_policy: EventExecutionPolicy | None = None async def evaluate_baseline(self) -> dict: """在同一 frozen bars 上计算一次市场买入持有基准。""" @@ -52,6 +59,10 @@ async def evaluate(self, source_code: str) -> EvaluationResult: fee_rate=self.fee_rate, validation_split=self.validation_split, annualization_periods=float(manifest.annualization_periods), + events=list(self.events), + event_execution_policy=self.event_execution_policy, + trading_mode=self.trading_mode, + leverage=self.leverage, ) return EvaluationResult( report=result.snapshot.model_dump(mode="json"), diff --git a/services/evolver/src/inalpha_evolver/event_client.py b/services/evolver/src/inalpha_evolver/event_client.py new file mode 100644 index 00000000..f245acd8 --- /dev/null +++ b/services/evolver/src/inalpha_evolver/event_client.py @@ -0,0 +1,76 @@ +"""Data event snapshot client with short-lived owner-bound service JWT.""" + +from __future__ import annotations + +import time +from typing import Any +from uuid import UUID + +import httpx +import jwt +from inalpha_shared.errors import InalphaError, NotFoundError + +from .config import EvolverSettings + + +async def fetch_event_snapshot( + snapshot_id: UUID, + *, + owner_account_id: UUID, + settings: EvolverSettings, +) -> dict[str, Any]: + """Resolve and freeze Data-owned snapshot metadata without direct table access.""" + token = jwt.encode( + { + "sub": str(owner_account_id), + "token_use": "service", + "service_audience": "data", + "exp": int(time.time()) + min(settings.service_token_ttl_s, 300), + }, + settings.jwt_secret, + algorithm=settings.jwt_algorithm, + ) + url = f"{settings.data_service_url.rstrip('/')}/events/snapshots/{snapshot_id}" + try: + async with httpx.AsyncClient(timeout=15.0, trust_env=False) as client: + response = await client.get(url, headers={"Authorization": f"Bearer {token}"}) + except httpx.RequestError as exc: + raise InalphaError( + "event snapshot data service is unreachable", + code="EVENT_DATA_UNREACHABLE", + status_code=502, + ) from exc + if response.status_code != 200: + try: + detail = response.json() + except ValueError: + detail = {} + payload_detail = detail.get("detail", detail) if isinstance(detail, dict) else {} + code = ( + str(payload_detail.get("code")) + if isinstance(payload_detail, dict) and payload_detail.get("code") + else "EVENT_SNAPSHOT_UNAVAILABLE" + ) + message = ( + str(payload_detail.get("message")) + if isinstance(payload_detail, dict) and payload_detail.get("message") + else f"event snapshot unavailable: HTTP {response.status_code}" + ) + if response.status_code == 404: + raise NotFoundError(message, code=code) + raise InalphaError( + message, + code=code, + status_code=response.status_code if response.status_code in {401, 403, 503} else 502, + ) + payload = response.json() + if not isinstance(payload, dict) or payload.get("snapshot_id") != str(snapshot_id): + raise InalphaError( + "event snapshot response identity mismatch", + code="EVENT_SNAPSHOT_IDENTITY_MISMATCH", + status_code=502, + ) + return payload + + +__all__ = ["fetch_event_snapshot"] diff --git a/services/evolver/src/inalpha_evolver/hypothesis/__init__.py b/services/evolver/src/inalpha_evolver/hypothesis/__init__.py new file mode 100644 index 00000000..e1f0f7cf --- /dev/null +++ b/services/evolver/src/inalpha_evolver/hypothesis/__init__.py @@ -0,0 +1,11 @@ +"""Deterministic hypothesis DSL, compiler, and evolutionary selection.""" + +from .compiler import CompiledHypothesis, compile_hypothesis, expand_implementations +from .models import HypothesisSpec + +__all__ = [ + "CompiledHypothesis", + "HypothesisSpec", + "compile_hypothesis", + "expand_implementations", +] diff --git a/services/evolver/src/inalpha_evolver/hypothesis/compiler.py b/services/evolver/src/inalpha_evolver/hypothesis/compiler.py new file mode 100644 index 00000000..f9ba79cb --- /dev/null +++ b/services/evolver/src/inalpha_evolver/hypothesis/compiler.py @@ -0,0 +1,188 @@ +"""Deterministic HypothesisSpec to sandboxed Strategy source compiler.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass + +from .models import HypothesisSpec + + +@dataclass(frozen=True, slots=True) +class CompiledHypothesis: + """Auditable compiler output and its stable identity.""" + + spec: HypothesisSpec + source_code: str + source_hash: str + compiler_version: str + + +def expand_implementations(spec: HypothesisSpec) -> list[HypothesisSpec]: + """Expand one strong-event hypothesis into direct/confirmed/hybrid ablation arms.""" + direct_allowed = set(spec.event_types) <= {"listing", "delisting", "exploit", "chain_halt"} + modes = ( + ["direct", "confirmed", "hybrid"] + if direct_allowed + else [ + "confirmed", + "hybrid", + "confirmed", + ] + ) + profiles: list[HypothesisSpec] = [] + for index, mode in enumerate(modes): + update: dict[str, object] = {"trigger_mode": mode} + if not direct_allowed and index == 2: + update["confirmation"] = spec.confirmation.model_copy( + update={ + "min_price_change_pct": spec.confirmation.min_price_change_pct * 1.5, + "min_volume_ratio": spec.confirmation.min_volume_ratio * 1.25, + } + ) + profiles.append(spec.model_copy(deep=True, update=update)) + return profiles + + +def compile_hypothesis(spec: HypothesisSpec) -> CompiledHypothesis: + """Compile a validated DSL object; no LLM output is interpolated as executable text.""" + class_suffix = hashlib.sha256( + spec.model_dump_json(exclude={"hypothesis_id"}).encode("utf-8") + ).hexdigest()[:12] + class_name = f"EventHypothesis_{class_suffix}" + event_types = repr(tuple(spec.event_types)) + assets = repr(tuple(spec.assets)) + direction = repr(spec.direction) + trigger_mode = repr(spec.trigger_mode) + source = f"""class {class_name}(Strategy): + def __init__(self, name, clock, msgbus, instrument_id, timeframe="1h", initial_cash=10000.0, position_pct={spec.risk.position_pct!r}): + super().__init__(name, clock, msgbus) + self._instrument_id = instrument_id + self._timeframe = timeframe + self._initial_cash = float(initial_cash) + self._position_pct = min(float(position_pct), {spec.risk.position_pct!r}) + self._asset = str(instrument_id.symbol).split("/")[0].upper() + self._event_types = {event_types} + self._assets = {assets} + self._direction = {direction} + self._trigger_mode = {trigger_mode} + self._closes = deque(maxlen={spec.confirmation.lookback_bars}) + self._volumes = deque(maxlen={spec.confirmation.lookback_bars}) + self._last_bar = None + self._pending_event = None + self._pending_age = 0 + self._holding_age = 0 + self._position_qty = 0.0 + self._entry_price = 0.0 + self._initial_sent = False + + def on_start(self): + self.subscribe_bars(self._instrument_id, self._timeframe) + + def on_market_event(self, event): + if event.event_type not in self._event_types: + return + if self._assets and self._asset not in self._assets: + return + if event.assets and self._asset not in event.assets: + return + if event.severity < {spec.risk.min_severity!r} or event.confidence < {spec.risk.min_confidence!r}: + return + self._pending_event = event + self._pending_age = 0 + self._initial_sent = False + if self._trigger_mode == "direct" and self._last_bar is not None: + self._enter(self._last_bar, 1.0) + self._pending_event = None + elif self._trigger_mode == "hybrid" and self._last_bar is not None: + self._enter(self._last_bar, {spec.risk.hybrid_initial_fraction!r}) + self._initial_sent = True + + def on_bar(self, bar): + if bar.instrument_id != self._instrument_id or bar.timeframe != self._timeframe: + return + previous_close = self._closes[-1] if self._closes else None + average_volume = sum(self._volumes) / len(self._volumes) if self._volumes else None + self._closes.append(bar.close) + self._volumes.append(bar.volume) + self._last_bar = bar + if self._position_qty != 0.0: + self._holding_age += 1 + adverse = ((bar.close / self._entry_price) - 1.0) * 100.0 + if self._position_qty < 0: + adverse = -adverse + if adverse <= -{spec.invalidation.max_adverse_pct!r} or self._holding_age >= {spec.invalidation.holding_bars}: + self._exit() + if self._pending_event is None: + return + self._pending_age += 1 + if self._pending_age > {spec.invalidation.ttl_bars}: + self._pending_event = None + return + if self._trigger_mode == "direct": + if self._position_qty == 0.0: + self._enter(bar, 1.0) + self._pending_event = None + return + if previous_close is None or average_volume is None or previous_close <= 0 or average_volume <= 0: + return + change = ((bar.close / previous_close) - 1.0) * 100.0 + if self._direction == "short": + change = -change + confirmed = change >= {spec.confirmation.min_price_change_pct!r} and bar.volume / average_volume >= {spec.confirmation.min_volume_ratio!r} + if not confirmed: + return + fraction = 1.0 - {spec.risk.hybrid_initial_fraction!r} if self._trigger_mode == "hybrid" and self._initial_sent else 1.0 + self._enter(bar, fraction) + self._pending_event = None + + def on_position_opened(self, event): + self._position_qty = float(event.quantity) + self._entry_price = float(event.avg_open_price) + self._holding_age = 0 + + def on_position_changed(self, event): + self._position_qty = float(event.quantity) + self._entry_price = float(event.avg_open_price) + + def on_position_closed(self, event): + self._position_qty = 0.0 + self._entry_price = 0.0 + self._holding_age = 0 + + def _enter(self, bar, fraction): + if bar.close <= 0 or fraction <= 0: + return + quantity = self._initial_cash * self._position_pct * fraction / bar.close / 1.10 + side = OrderSide.BUY if self._direction == "long" else OrderSide.SELL + self.submit_order(Order(client_order_id=ClientOrderId("event-entry-" + uuid4().hex[:12]), instrument_id=self._instrument_id, side=side, type=OrderType.MARKET, quantity=quantity)) + + def _exit(self): + if self._position_qty == 0.0: + return + side = OrderSide.SELL if self._position_qty > 0 else OrderSide.BUY + self.submit_order(Order(client_order_id=ClientOrderId("event-exit-" + uuid4().hex[:12]), instrument_id=self._instrument_id, side=side, type=OrderType.MARKET, quantity=abs(self._position_qty))) +""" + source_hash = hashlib.sha256(source.encode("utf-8")).hexdigest() + return CompiledHypothesis( + spec=spec, + source_code=source, + source_hash=source_hash, + compiler_version=spec.compiler_version, + ) + + +def canonical_spec_hash(spec: HypothesisSpec) -> str: + """Return a stable genotype hash excluding its storage identity.""" + payload = spec.model_dump(mode="json", exclude={"hypothesis_id"}) + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + +__all__ = [ + "CompiledHypothesis", + "canonical_spec_hash", + "compile_hypothesis", + "expand_implementations", +] diff --git a/services/evolver/src/inalpha_evolver/hypothesis/models.py b/services/evolver/src/inalpha_evolver/hypothesis/models.py new file mode 100644 index 00000000..8673ab88 --- /dev/null +++ b/services/evolver/src/inalpha_evolver/hypothesis/models.py @@ -0,0 +1,130 @@ +"""Typed, falsifiable upper-level strategy hypothesis contract.""" + +from __future__ import annotations + +from typing import Literal +from uuid import UUID, uuid4 + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +HypothesisLane = Literal["event", "event_regime", "factor", "execution_risk", "regime", "restart"] +TriggerMode = Literal["direct", "confirmed", "hybrid"] +LineageKind = Literal["seed", "elite", "mutation", "crossover", "restart"] +Direction = Literal["long", "short"] + +_EVENT_TYPES = { + "listing", + "delisting", + "exploit", + "chain_halt", + "regulatory", + "upgrade", + "unlock", + "burn", + "partnership", + "macro", + "other", +} +_DIRECT_ALLOWED = {"listing", "delisting", "exploit", "chain_halt"} + + +class ConfirmationSpec(BaseModel): + """Price/volume confirmation evaluated only on closed bars.""" + + model_config = ConfigDict(extra="forbid") + + lookback_bars: int = Field(default=12, ge=2, le=100) + min_price_change_pct: float = Field(default=0.5, ge=0, le=30) + min_volume_ratio: float = Field(default=1.2, ge=0.1, le=20) + + +class InvalidationSpec(BaseModel): + """Pre-registered exit and expiry conditions.""" + + model_config = ConfigDict(extra="forbid") + + ttl_bars: int = Field(default=6, ge=1, le=100) + holding_bars: int = Field(default=12, ge=1, le=500) + max_adverse_pct: float = Field(default=4.0, gt=0, le=50) + + +class RiskSpec(BaseModel): + """Bounded strategy-level exposure; framework risk remains authoritative.""" + + model_config = ConfigDict(extra="forbid") + + position_pct: float = Field(default=0.10, gt=0, le=0.25) + hybrid_initial_fraction: float = Field(default=0.25, gt=0, lt=1) + min_severity: float = Field(default=0.5, ge=0, le=1) + min_confidence: float = Field(default=0.6, ge=0, le=1) + + +class CounterfactualSpec(BaseModel): + """Matching rules for event-free comparison windows.""" + + model_config = ConfigDict(extra="forbid") + + match_regime: bool = True + volatility_tolerance: float = Field(default=0.20, gt=0, le=1) + volume_tolerance: float = Field(default=0.25, gt=0, le=1) + exclusion_bars: int = Field(default=24, ge=1, le=500) + + +class HypothesisSpec(BaseModel): + """Versioned DSL genotype used by upper-level evolution.""" + + model_config = ConfigDict(extra="forbid") + + hypothesis_id: UUID = Field(default_factory=uuid4) + dsl_version: Literal["event-hypothesis-v1"] = "event-hypothesis-v1" + lane: HypothesisLane + lineage_kind: LineageKind = "seed" + parent_ids: list[UUID] = Field(default_factory=list, max_length=2) + thesis: str = Field(min_length=20, max_length=2_000) + evidence_ids: list[str] = Field(default_factory=list, max_length=64) + event_types: list[str] = Field(default_factory=list, min_length=1, max_length=8) + assets: list[str] = Field(default_factory=list, max_length=32) + applicable_regimes: list[str] = Field(default_factory=list, max_length=16) + direction: Direction + trigger_mode: TriggerMode + confirmation: ConfirmationSpec = Field(default_factory=ConfirmationSpec) + invalidation: InvalidationSpec = Field(default_factory=InvalidationSpec) + risk: RiskSpec = Field(default_factory=RiskSpec) + counterfactual: CounterfactualSpec = Field(default_factory=CounterfactualSpec) + compiler_version: Literal["event-strategy-compiler-v1"] = "event-strategy-compiler-v1" + + @field_validator("event_types", mode="after") + @classmethod + def validate_event_types(cls, value: list[str]) -> list[str]: + normalized = sorted({item.strip().lower() for item in value if item.strip()}) + unknown = set(normalized) - _EVENT_TYPES + if unknown: + raise ValueError(f"unsupported event types: {sorted(unknown)}") + return normalized + + @field_validator("assets", mode="after") + @classmethod + def normalize_assets(cls, value: list[str]) -> list[str]: + return sorted({item.strip().upper() for item in value if item.strip()}) + + @field_validator("evidence_ids", "applicable_regimes", mode="after") + @classmethod + def dedupe_strings(cls, value: list[str]) -> list[str]: + return list(dict.fromkeys(item.strip() for item in value if item.strip())) + + @model_validator(mode="after") + def validate_trigger_safety(self) -> HypothesisSpec: + if self.trigger_mode == "direct" and not set(self.event_types) <= _DIRECT_ALLOWED: + raise ValueError("direct trigger is restricted to listing/delisting/exploit/chain_halt") + if self.lane == "restart" and self.lineage_kind != "restart": + raise ValueError("restart lane requires lineage_kind='restart'") + return self + + +__all__ = [ + "ConfirmationSpec", + "CounterfactualSpec", + "HypothesisSpec", + "InvalidationSpec", + "RiskSpec", +] diff --git a/services/evolver/src/inalpha_evolver/hypothesis/proposer.py b/services/evolver/src/inalpha_evolver/hypothesis/proposer.py new file mode 100644 index 00000000..7f1a253a --- /dev/null +++ b/services/evolver/src/inalpha_evolver/hypothesis/proposer.py @@ -0,0 +1,165 @@ +"""Owner-scoped Agent proposer for structured hypothesis DSL only.""" + +from __future__ import annotations + +import asyncio +import json +from dataclasses import dataclass +from typing import Any +from uuid import uuid4 + +from inalpha_shared_llm.types import MutationRequest # type: ignore[import-untyped] + +from ..mutator import Mutator +from .models import HypothesisSpec + +_SYSTEM_PROMPT = """You are Inalpha's crypto strategy-hypothesis proposer. +Return only a JSON array of exactly four objects. Propose falsifiable event reaction mechanisms, +not prose strategies and never executable code. You may use only supplied frozen evidence IDs and +aggregate simulation feedback. Never infer publication time, unseen news, or future outcomes. +Each object may set: thesis,event_types,assets,applicable_regimes,direction,trigger_mode, +confirmation,invalidation,risk,counterfactual. Preserve diversity and avoid semantic duplicates.""" + +_ALLOWED_FIELDS = { + "thesis", + "event_types", + "assets", + "applicable_regimes", + "direction", + "trigger_mode", + "confirmation", + "invalidation", + "risk", + "counterfactual", +} + + +@dataclass(frozen=True, slots=True) +class ProposalResult: + """Two-call proposal output plus measured provider cost.""" + + hypotheses: tuple[HypothesisSpec, ...] + cost_usd: float + fallback_calls: int + + +async def propose_generation( + mutator: Mutator, + *, + generation: int, + scaffolds: list[HypothesisSpec], + feedback: list[dict[str, Any]], +) -> ProposalResult: + """Run exactly two proposer calls of four slots, falling back per invalid call.""" + if len(scaffolds) != 8: + raise ValueError("Agent proposer requires exactly eight scaffold slots") + calls = [ + _propose_four( + mutator, + generation=generation, + scaffolds=scaffolds[index : index + 4], + feedback=feedback, + ) + for index in (0, 4) + ] + results = await asyncio.gather(*calls, return_exceptions=True) + hypotheses: list[HypothesisSpec] = [] + cost = 0.0 + fallback_calls = 0 + for index, result in enumerate(results): + fallback = scaffolds[index * 4 : index * 4 + 4] + if isinstance(result, Exception): + hypotheses.extend(fallback) + fallback_calls += 1 + else: + batch, batch_cost, used_fallback = result + hypotheses.extend(batch) + cost += batch_cost + fallback_calls += int(used_fallback) + return ProposalResult(tuple(hypotheses), cost, fallback_calls) + + +async def _propose_four( + mutator: Mutator, + *, + generation: int, + scaffolds: list[HypothesisSpec], + feedback: list[dict[str, Any]], +) -> tuple[list[HypothesisSpec], float, bool]: + prompt = json.dumps( + { + "generation": generation, + "slots": [item.model_dump(mode="json") for item in scaffolds], + "aggregate_feedback": feedback, + }, + ensure_ascii=False, + separators=(",", ":"), + ) + if len(prompt.encode()) + len(_SYSTEM_PROMPT.encode()) > mutator.max_input_utf8_bytes: + raise ValueError("hypothesis proposer prompt exceeds frozen input budget") + response = await mutator.llm_client.mutate( + MutationRequest( + system_prompt=_SYSTEM_PROMPT, + user_prompt=prompt, + max_tokens=mutator.max_output_tokens, + ) + ) + cost = _cost(mutator, response.cache_metrics) + try: + payload = _parse_json_array(response.content) + if len(payload) != 4: + raise ValueError("hypothesis proposer must return exactly four objects") + proposals: list[HypothesisSpec] = [] + for scaffold, update in zip(scaffolds, payload, strict=True): + if not isinstance(update, dict): + raise ValueError("hypothesis proposal entries must be objects") + safe_update = {key: value for key, value in update.items() if key in _ALLOWED_FIELDS} + # Evidence, lineage, lane and compiler identity are platform-owned and cannot + # be fabricated or changed by the model. + safe_update.update( + { + "hypothesis_id": str(uuid4()), + "evidence_ids": scaffold.evidence_ids, + "lane": scaffold.lane, + "lineage_kind": scaffold.lineage_kind, + "parent_ids": scaffold.parent_ids, + "dsl_version": scaffold.dsl_version, + "compiler_version": scaffold.compiler_version, + } + ) + base = scaffold.model_dump(mode="json") + base.update(safe_update) + proposals.append(HypothesisSpec.model_validate(base)) + except (ValueError, TypeError, json.JSONDecodeError): + return scaffolds, cost, True + return proposals, cost, False + + +def _parse_json_array(content: str) -> list[Any]: + text = content.strip() + if text.startswith("```"): + text = text.split("\n", 1)[1] if "\n" in text else "" + text = text.rsplit("```", 1)[0] + start = text.find("[") + end = text.rfind("]") + if start < 0 or end < start: + raise ValueError("hypothesis proposer returned no JSON array") + parsed = json.loads(text[start : end + 1]) + if not isinstance(parsed, list): + raise ValueError("hypothesis proposer payload must be an array") + return parsed + + +def _cost(mutator: Mutator, metrics: Any) -> float: + if mutator.input_usd_per_million is None or mutator.output_usd_per_million is None: + return float(metrics.cost_usd) + return float( + ( + metrics.input_tokens * mutator.input_usd_per_million + + metrics.output_tokens * mutator.output_usd_per_million + ) + / 1_000_000 + ) + + +__all__ = ["ProposalResult", "propose_generation"] diff --git a/services/evolver/src/inalpha_evolver/hypothesis/seeding.py b/services/evolver/src/inalpha_evolver/hypothesis/seeding.py new file mode 100644 index 00000000..1612eb7c --- /dev/null +++ b/services/evolver/src/inalpha_evolver/hypothesis/seeding.py @@ -0,0 +1,106 @@ +"""Deterministic direction-coverage scaffold before owner Agent proposals.""" + +from __future__ import annotations + +from typing import Any + +from .models import HypothesisSpec + + +def seed_generation_one(snapshot: dict[str, Any], asset: str) -> list[HypothesisSpec]: + """Create eight diverse, falsifiable slots grounded only in frozen fact IDs.""" + facts = [item for item in snapshot.get("facts", []) if isinstance(item, dict)] + by_type: dict[str, list[dict[str, Any]]] = {} + for fact in facts: + by_type.setdefault(str(fact.get("event_type") or "other"), []).append(fact) + ranked_types = sorted( + by_type, + key=lambda event_type: ( + -max(float(item.get("severity") or 0) for item in by_type[event_type]), + event_type, + ), + ) + while len(ranked_types) < 3: + ranked_types.append(("listing", "exploit", "chain_halt")[len(ranked_types)]) + event_types = ranked_types[:3] + seeds: list[HypothesisSpec] = [] + for index, event_type in enumerate(event_types): + facts_for_type = by_type.get(event_type, []) + direction = ( + "short" if event_type in {"delisting", "exploit", "chain_halt", "unlock"} else "long" + ) + seeds.append( + HypothesisSpec( + lane="event", + thesis=f"冻结事实显示 {event_type} 可能产生可证伪的延迟反应,需以事后成本收益和匹配对照验证。", + evidence_ids=_evidence_ids(facts_for_type), + event_types=[event_type], + assets=[asset], + direction=direction, + trigger_mode=( + "direct" + if index == 0 + and event_type in {"listing", "delisting", "exploit", "chain_halt"} + else "confirmed" + ), + ) + ) + seeds.extend( + [ + HypothesisSpec( + lane="event_regime", + thesis="事件冲击只在高成交量或高波动状态持续,状态过滤应提高事件对照优势。", + evidence_ids=_evidence_ids(facts), + event_types=[event_types[0]], + assets=[asset], + applicable_regimes=["high_volume", "high_volatility"], + direction=seeds[0].direction, + trigger_mode="hybrid", + ), + HypothesisSpec( + lane="factor", + thesis="少量动量和成交量确认因子可作为事件后的证伪条件,而不是独立产生方向。", + event_types=["other"], + assets=[asset], + direction="long", + trigger_mode="confirmed", + ), + HypothesisSpec( + lane="execution_risk", + thesis="降低事件期仓位并缩短持有期可能在保留异常收益的同时减少不利滑点与回撤。", + event_types=["other"], + assets=[asset], + direction="long", + trigger_mode="confirmed", + risk={"position_pct": 0.05}, + invalidation={"ttl_bars": 4, "holding_bars": 6, "max_adverse_pct": 2.0}, + ), + HypothesisSpec( + lane="regime", + thesis="市场状态迁移本身可形成方向,但必须通过封闭验证集证明不依赖单一事件类别。", + event_types=["other"], + assets=[asset], + applicable_regimes=["trend_transition"], + direction="long", + trigger_mode="confirmed", + ), + HypothesisSpec( + lane="restart", + lineage_kind="restart", + thesis="自由重启探索安全事件后的反转机制,并与当前假设档案保持明显语义距离。", + evidence_ids=_evidence_ids(facts), + event_types=["exploit"], + assets=[asset], + direction="long", + trigger_mode="confirmed", + ), + ] + ) + return seeds + + +def _evidence_ids(facts: list[dict[str, Any]]) -> list[str]: + return [f"{item['fact_id']}:0" for item in facts[:16] if item.get("fact_id")] + + +__all__ = ["seed_generation_one"] diff --git a/services/evolver/src/inalpha_evolver/hypothesis/selection.py b/services/evolver/src/inalpha_evolver/hypothesis/selection.py new file mode 100644 index 00000000..518ac0f5 --- /dev/null +++ b/services/evolver/src/inalpha_evolver/hypothesis/selection.py @@ -0,0 +1,234 @@ +"""Deterministic upper-level credit, Pareto ranking, and next-generation plan.""" + +from __future__ import annotations + +import random +import statistics +from collections.abc import Iterable +from dataclasses import dataclass +from uuid import UUID + + +@dataclass(frozen=True, slots=True) +class ImplementationScore: + """Lower-level implementation evidence used to credit one hypothesis.""" + + fitness: float + max_drawdown_pct: float + event_advantage: float + stability: float + evidence_quality: float + novelty: float + complexity: float + + +@dataclass(frozen=True, slots=True) +class HypothesisScore: + """Upper-level multi-objective score and deterministic scalar credit.""" + + hypothesis_id: UUID + lane: str + event_family: str + trigger_mode: str + credit: float + objectives: tuple[float, ...] + + +@dataclass(frozen=True, slots=True) +class GenerationPlan: + """Exactly eight lineage operations for generations two through five.""" + + elites: tuple[UUID, UUID] + mutation_parents: tuple[UUID, UUID, UUID, UUID] + crossover_parents: tuple[UUID, UUID] + restart_slots: int = 1 + + +def credit_hypothesis( + hypothesis_id: UUID, + *, + lane: str, + event_family: str, + trigger_mode: str, + implementations: Iterable[ImplementationScore], +) -> HypothesisScore: + """Credit the genotype without hiding implementation fragility or drawdown.""" + scores = list(implementations) + if not scores: + raise ValueError("hypothesis credit requires at least one implementation") + fitnesses = [item.fitness for item in scores] + dispersion = statistics.pstdev(fitnesses) if len(fitnesses) > 1 else 0.0 + best = max(scores, key=lambda item: item.fitness) + drawdown_penalty = max(0.0, best.max_drawdown_pct - 20.0) / 20.0 + evidence_penalty = max(0.0, 0.6 - best.evidence_quality) + credit = ( + best.fitness + + 0.35 * best.event_advantage + + 0.20 * best.stability + + 0.15 * best.novelty + - 0.35 * dispersion + - 0.10 * best.complexity + - drawdown_penalty + - evidence_penalty + ) + objectives = ( + best.fitness, + -best.max_drawdown_pct, + best.stability, + best.event_advantage, + best.evidence_quality, + best.novelty, + ) + return HypothesisScore( + hypothesis_id=hypothesis_id, + lane=lane, + event_family=event_family, + trigger_mode=trigger_mode, + credit=credit, + objectives=objectives, + ) + + +def pareto_ranks(scores: Iterable[HypothesisScore]) -> dict[UUID, int]: + """Assign non-dominated sorting ranks; all objectives are maximized.""" + remaining = list(scores) + ranks: dict[UUID, int] = {} + rank = 0 + while remaining: + front = [ + candidate + for candidate in remaining + if not any( + _dominates(other, candidate) for other in remaining if other is not candidate + ) + ] + if not front: + raise RuntimeError("Pareto ranking failed to identify a front") + for candidate in front: + ranks[candidate.hypothesis_id] = rank + front_ids = {candidate.hypothesis_id for candidate in front} + remaining = [item for item in remaining if item.hypothesis_id not in front_ids] + rank += 1 + return ranks + + +def plan_next_generation( + scores: Iterable[HypothesisScore], + *, + seed: int, +) -> GenerationPlan: + """Select 2 elites, 4 novelty-weighted mutations, 1 crossover, and 1 restart.""" + candidates = _apply_niche_cap(list(scores), cap=2) + if len(candidates) < 2: + raise ValueError("next generation requires at least two scored hypothesis families") + ranks = pareto_ranks(candidates) + ordered = sorted( + candidates, + key=lambda item: (ranks[item.hypothesis_id], -item.credit, str(item.hypothesis_id)), + ) + elites = (ordered[0].hypothesis_id, ordered[1].hypothesis_id) + rng = random.Random(seed) + mutations = tuple(_novelty_tournament(candidates, ranks, rng).hypothesis_id for _ in range(4)) + left = _novelty_tournament(candidates, ranks, rng) + distinct = [ + item + for item in candidates + if item.hypothesis_id != left.hypothesis_id + and (item.lane != left.lane or item.event_family != left.event_family) + ] + right = _novelty_tournament(distinct or candidates, ranks, rng) + return GenerationPlan( + elites=elites, + mutation_parents=mutations, # type: ignore[arg-type] + crossover_parents=(left.hypothesis_id, right.hypothesis_id), + ) + + +def benjamini_hochberg(p_values: Iterable[float], *, q: float = 0.10) -> list[bool]: + """Return FDR rejections in original order using Benjamini-Hochberg.""" + values = list(p_values) + if not 0 < q < 1: + raise ValueError("q must be within (0,1)") + if any(not 0 <= value <= 1 for value in values): + raise ValueError("p-values must be within [0,1]") + ordered = sorted(enumerate(values), key=lambda pair: (pair[1], pair[0])) + cutoff_rank = 0 + for rank, (_, value) in enumerate(ordered, start=1): + if value <= rank * q / max(1, len(values)): + cutoff_rank = rank + accepted = {index for index, _ in ordered[:cutoff_rank]} + return [index in accepted for index in range(len(values))] + + +def block_bootstrap_p_value( + effects: Iterable[float], + *, + block_size: int = 3, + samples: int = 2_000, + seed: int = 0, +) -> float: + """Estimate one-sided P(mean<=0) while preserving short event clusters.""" + values = list(effects) + if not values: + return 1.0 + if block_size < 1 or samples < 100: + raise ValueError("block_size must be positive and samples >= 100") + blocks = [values[index : index + block_size] for index in range(0, len(values), block_size)] + rng = random.Random(seed) + non_positive = 0 + for _ in range(samples): + sample: list[float] = [] + while len(sample) < len(values): + sample.extend(rng.choice(blocks)) + if statistics.mean(sample[: len(values)]) <= 0: + non_positive += 1 + return (non_positive + 1) / (samples + 1) + + +def _dominates(left: HypothesisScore, right: HypothesisScore) -> bool: + return all(a >= b for a, b in zip(left.objectives, right.objectives, strict=True)) and any( + a > b for a, b in zip(left.objectives, right.objectives, strict=True) + ) + + +def _apply_niche_cap(scores: list[HypothesisScore], *, cap: int) -> list[HypothesisScore]: + buckets: dict[tuple[str, str, str], list[HypothesisScore]] = {} + for item in scores: + buckets.setdefault((item.lane, item.event_family, item.trigger_mode), []).append(item) + return [ + item + for bucket in buckets.values() + for item in sorted(bucket, key=lambda value: (-value.credit, str(value.hypothesis_id)))[ + :cap + ] + ] + + +def _novelty_tournament( + candidates: list[HypothesisScore], + ranks: dict[UUID, int], + rng: random.Random, +) -> HypothesisScore: + if not candidates: + raise ValueError("tournament candidate set is empty") + contenders = rng.sample(candidates, k=min(3, len(candidates))) + return max( + contenders, + key=lambda item: ( + -ranks[item.hypothesis_id], + item.objectives[-1], + item.credit, + ), + ) + + +__all__ = [ + "GenerationPlan", + "HypothesisScore", + "ImplementationScore", + "benjamini_hochberg", + "block_bootstrap_p_value", + "credit_hypothesis", + "pareto_ranks", + "plan_next_generation", +] diff --git a/services/evolver/src/inalpha_evolver/main.py b/services/evolver/src/inalpha_evolver/main.py index 0d29dbbd..af3f791c 100644 --- a/services/evolver/src/inalpha_evolver/main.py +++ b/services/evolver/src/inalpha_evolver/main.py @@ -9,6 +9,7 @@ import logging import os +from collections.abc import AsyncIterator from contextlib import asynccontextmanager from fastapi import FastAPI, Request @@ -19,12 +20,13 @@ from .api.routes import router from .config import get_evolver_settings from .runtime import EvolutionRunManager +from .runtime.campaign_manager import CampaignManager logger = logging.getLogger(__name__) @asynccontextmanager -async def lifespan(app: FastAPI): +async def lifespan(app: FastAPI) -> AsyncIterator[None]: """初始化 DB 队列与 manager;E1 强制单 API worker。""" settings = get_evolver_settings() workers = int(os.environ.get("WEB_CONCURRENCY", os.environ.get("WORKERS", "1"))) @@ -40,9 +42,15 @@ async def lifespan(app: FastAPI): manager = EvolutionRunManager(mutator=None, settings=settings) app.state.evolution_manager = manager await manager.start() + campaign_manager = CampaignManager(settings) if settings.event_evolution_enabled else None + app.state.campaign_manager = campaign_manager + if campaign_manager is not None: + await campaign_manager.start() try: yield finally: + if campaign_manager is not None: + await campaign_manager.close() await manager.close() await close_pool() @@ -72,4 +80,17 @@ async def health(request: Request) -> dict[str, str] | JSONResponse: "reason": reason, }, ) - return {"status": "ok", "service": "inalpha-evolver"} + campaign_manager = getattr(request.app.state, "campaign_manager", None) + if campaign_manager is not None and not campaign_manager.healthy: + return JSONResponse( + status_code=503, + content={ + "status": "unhealthy", + "service": "inalpha-evolver", + "reason": campaign_manager.unhealthy_reason or "campaign dispatcher unavailable", + }, + ) + response = {"status": "ok", "service": "inalpha-evolver"} + if campaign_manager is not None: + response["event_evolution"] = "enabled" + return response diff --git a/services/evolver/src/inalpha_evolver/owner_llm.py b/services/evolver/src/inalpha_evolver/owner_llm.py index 6606a75f..01f45faf 100644 --- a/services/evolver/src/inalpha_evolver/owner_llm.py +++ b/services/evolver/src/inalpha_evolver/owner_llm.py @@ -23,6 +23,8 @@ class CredentialTemporarilyUnavailable(RuntimeError): """Dashboard 暂时不可达;run 应回到队列而不是进入失败终态。""" + code = "EVOLUTION_CREDENTIAL_UNAVAILABLE" + async def build_owner_mutator( run: dict[str, Any], @@ -41,7 +43,10 @@ async def build_owner_mutator( f"{quote(config_id, safe='')}" ) try: - async with httpx.AsyncClient(timeout=10.0, trust_env=False) as client: + async with httpx.AsyncClient( + timeout=settings.evolver_credential_timeout_s, + trust_env=False, + ) as client: response = await client.get(url, headers={"Authorization": f"Bearer {token}"}) except httpx.HTTPError as exc: raise CredentialTemporarilyUnavailable( diff --git a/services/evolver/src/inalpha_evolver/runtime/campaign.py b/services/evolver/src/inalpha_evolver/runtime/campaign.py new file mode 100644 index 00000000..bc57710a --- /dev/null +++ b/services/evolver/src/inalpha_evolver/runtime/campaign.py @@ -0,0 +1,718 @@ +"""Durable five-generation event campaign executor.""" + +from __future__ import annotations + +import hashlib +import time +from datetime import UTC, datetime +from typing import Any +from uuid import UUID, uuid4 + +import jwt +from inalpha_paper.data_client import DataClient +from inalpha_paper.evaluation_executor import KillableEngineRunner +from inalpha_paper.event_conversion import market_event_from_fact +from inalpha_paper.execution.exchange import EventExecutionPolicy +from inalpha_shared.db import get_conn + +from ..api.schemas import CampaignConfig +from ..config import EvolverSettings +from ..data import FrozenBarsLoader, FrozenDataset +from ..evaluator.event_study import evaluate_event_reactions +from ..evaluator.frozen import FrozenDatasetEvaluator +from ..hypothesis.compiler import compile_hypothesis, expand_implementations +from ..hypothesis.models import HypothesisSpec +from ..hypothesis.proposer import propose_generation +from ..hypothesis.selection import ( + HypothesisScore, + ImplementationScore, + benjamini_hochberg, + block_bootstrap_p_value, + credit_hypothesis, + pareto_ranks, + plan_next_generation, +) +from ..mutator import Mutator +from ..owner_llm import build_owner_mutator +from ..storage import campaigns as store + + +async def execute_campaign(campaign: dict[str, Any], settings: EvolverSettings) -> None: + """Run remaining generations, then lock one champion for isolated forward evidence.""" + config = _campaign_config(campaign) + dataset, snapshot = await _load_frozen_inputs(campaign, config, settings) + mutator = await build_owner_mutator(campaign, settings) + try: + await _execute_campaign( + campaign, + settings, + mutator, + config=config, + dataset=dataset, + snapshot=snapshot, + ) + finally: + await mutator.close() + + +async def evaluate_sealed_holdout( + campaign: dict[str, Any], + *, + source_code: str, + hypothesis: HypothesisSpec, + settings: EvolverSettings, +) -> tuple[bool, dict[str, Any]]: + """Evaluate the locked champion once on the campaign's untouched final 20%.""" + config = CampaignConfig.model_validate( + { + key: campaign["frozen_config"][key] + for key in CampaignConfig.model_fields + if key in campaign["frozen_config"] + } + ) + token = _service_token(campaign["owner_account_id"], settings) + async with DataClient( + settings.data_service_url, + token, + timeout=settings.evolver_data_timeout_s, + ) as client: + dataset = await FrozenBarsLoader(client).load( + venue=config.venue, + symbol=config.symbol, + timeframe=config.timeframe, + from_ts=config.from_ts, + as_of=config.as_of, + ) + snapshot = await client.get_event_snapshot(str(campaign["event_snapshot_id"])) + if len(dataset.bars) < 5: + raise RuntimeError("sealed holdout requires at least five frozen bars") + events = tuple(market_event_from_fact(item) for item in snapshot["facts"]) + evaluator = FrozenDatasetEvaluator( + dataset=dataset, + runner=KillableEngineRunner( + timeout_s=settings.evolver_job_timeout_s, + mem_gb=settings.evolver_job_mem_gb, + ), + initial_cash=config.initial_cash, + fee_rate=config.fee_rate, + validation_split=0.8, + trading_mode=config.trading_mode, + leverage=config.leverage, + events=events, + event_execution_policy=EventExecutionPolicy(), + ) + result = await evaluator.evaluate(source_code) + validation = result.report.get("validation") or {} + holdout = validation.get("holdout") or {} + split_index = max(1, min(len(dataset.bars) - 1, int(len(dataset.bars) * 0.8))) + holdout_bars = list(dataset.bars[split_index:]) + start_known_at = holdout_bars[0].bar_known_at + holdout_events = [event for event in events if event.available_at >= start_known_at] + event_study = evaluate_event_reactions( + bars=holdout_bars, + events=holdout_events, + asset=_base_asset(config.symbol), + direction=hypothesis.direction, + holding_bars=hypothesis.invalidation.holding_bars, + exclusion_bars=hypothesis.counterfactual.exclusion_bars, + volatility_tolerance=hypothesis.counterfactual.volatility_tolerance, + volume_tolerance=hypothesis.counterfactual.volume_tolerance, + ) + sharpe = holdout.get("sharpe") + total_return = float(holdout.get("total_return_pct") or 0.0) + max_drawdown = float(holdout.get("max_drawdown_pct") or 100.0) + passed = bool( + isinstance(sharpe, (int, float)) + and sharpe > 0 + and total_return > 0 + and max_drawdown <= 25.0 + and int(holdout.get("num_trades") or 0) > 0 + ) + evidence = { + "execution_model_version": config.execution_model_version, + "snapshot_id": str(campaign["event_snapshot_id"]), + "source_hash": hashlib.sha256(source_code.encode()).hexdigest(), + "thresholds": { + "sharpe_gt": 0, + "net_return_pct_gt": 0, + "max_drawdown_pct_lte": 25, + "num_trades_gt": 0, + }, + "metrics": { + "sharpe": sharpe, + "total_return_pct": total_return, + "max_drawdown_pct": max_drawdown, + "num_trades": int(holdout.get("num_trades") or 0), + "num_bars": int(holdout.get("num_bars") or len(holdout_bars)), + }, + "event_study": event_study.as_dict(), + "limited_evidence": event_study.event_count < 3, + } + return passed, evidence + + +async def _execute_campaign( + campaign: dict[str, Any], + settings: EvolverSettings, + mutator: Mutator, + *, + config: CampaignConfig, + dataset: FrozenDataset, + snapshot: dict[str, Any], +) -> None: + """Execute one credential-bound campaign while keeping the key process-local.""" + all_events = tuple(market_event_from_fact(item) for item in snapshot["facts"]) + search_dataset, validation_bars = _search_dataset(dataset) + search_events = tuple( + event + for event in all_events + if search_dataset.bars[0].bar_known_at + <= event.available_at + <= search_dataset.bars[-1].bar_known_at + ) + evaluator = FrozenDatasetEvaluator( + dataset=search_dataset, + runner=KillableEngineRunner( + timeout_s=settings.evolver_job_timeout_s, + mem_gb=settings.evolver_job_mem_gb, + ), + initial_cash=config.initial_cash, + fee_rate=config.fee_rate, + validation_split=0.75, + trading_mode=config.trading_mode, + leverage=config.leverage, + events=search_events, + event_execution_policy=EventExecutionPolicy(), + ) + generation = max(1, int(campaign["active_generation"])) + while generation <= int(campaign["max_generations"]): + async with get_conn() as conn: + current = await store.get_campaign( + conn, campaign["campaign_id"], campaign["owner_account_id"] + ) + if current is None or current["status"] != "replaying": + return + hypotheses = [ + HypothesisSpec.model_validate(row["spec"]) + for row in current["hypotheses"] + if int(row["generation"]) == generation + ] + if not hypotheses: + raise RuntimeError(f"campaign generation {generation} has no hypotheses") + has_started_generation = any( + int(item["generation"]) == generation for item in current.get("implementations", []) + ) + if not has_started_generation: + proposed = await propose_generation( + mutator, + generation=generation, + scaffolds=hypotheses, + feedback=_proposal_feedback(current, generation - 1), + ) + hypotheses = list(proposed.hypotheses) + async with get_conn() as conn: + async with conn.transaction(): + await store.replace_generation_hypotheses( + conn, + campaign["campaign_id"], + generation, + hypotheses, + ) + await store.add_llm_cost(conn, campaign["campaign_id"], proposed.cost_usd) + scores = await _evaluate_generation( + campaign=current, + generation=generation, + hypotheses=hypotheses, + evaluator=evaluator, + validation_bars=list(validation_bars), + validation_events=list(search_events), + asset=config.symbol.split("/")[0], + seed=config.random_seed + generation, + ) + if generation == int(campaign["max_generations"]): + async with get_conn() as conn: + champion = await store.best_implementation( + conn, campaign["campaign_id"], generation + ) + if champion is None: + await store.transition( + conn, + campaign["campaign_id"], + campaign["owner_account_id"], + from_statuses=("replaying",), + to_status="rejected", + values={ + "failure_code": "NO_FDR_CHAMPION", + "failure_message": "generation five produced no FDR-passing implementation", + "finished_at": datetime.now(UTC), + }, + ) + return + await store.lock_champion( + conn, + campaign["campaign_id"], + campaign["owner_account_id"], + champion["implementation_id"], + ) + return + next_generation = generation + 1 + next_hypotheses = _next_hypotheses( + hypotheses, + scores, + seed=config.random_seed + next_generation, + ) + async with get_conn() as conn: + async with conn.transaction(): + await store.insert_hypotheses( + conn, campaign["campaign_id"], next_generation, next_hypotheses + ) + advanced = await store.advance_generation( + conn, + campaign["campaign_id"], + current_generation=generation, + next_generation=next_generation, + ) + if not advanced: + raise RuntimeError("campaign generation advance lost compare-and-swap") + generation = next_generation + + +def _proposal_feedback(campaign: dict[str, Any], generation: int) -> list[dict[str, Any]]: + """Expose only aggregate selection evidence, never bars, trades, or holdout rows.""" + if generation < 1: + return [] + return [ + { + "hypothesis_id": str(item["hypothesis_id"]), + "lane": item["lane"], + "upper_credit": item.get("upper_credit"), + "novelty_score": item.get("novelty_score"), + "pareto_rank": item.get("pareto_rank"), + "selected": item.get("selected", False), + } + for item in campaign.get("hypotheses", []) + if int(item["generation"]) == generation + ] + + +async def _evaluate_generation( + *, + campaign: dict[str, Any], + generation: int, + hypotheses: list[HypothesisSpec], + evaluator: FrozenDatasetEvaluator, + validation_bars: list[Any], + validation_events: list[Any], + asset: str, + seed: int, +) -> list[HypothesisScore]: + implementation_rows: list[tuple[dict[str, Any], HypothesisSpec, dict[str, Any]]] = [] + for hypothesis in hypotheses: + for implementation_index, implementation_spec in enumerate( + expand_implementations(hypothesis) + ): + compiled = compile_hypothesis(implementation_spec) + async with get_conn() as conn: + row = await store.insert_implementation( + conn, + campaign_id=campaign["campaign_id"], + hypothesis_id=hypothesis.hypothesis_id, + generation=generation, + profile=_implementation_profile( + hypothesis, implementation_spec, implementation_index + ), + source_code=compiled.source_code, + source_hash=compiled.source_hash, + ) + cached = await store.find_cached_implementation( + conn, campaign["campaign_id"], compiled.source_hash + ) + if row["outcome"] == "succeeded": + implementation_rows.append((row, implementation_spec, row)) + continue + if cached is not None and cached["implementation_id"] != row["implementation_id"]: + values = { + key: cached[key] + for key in ( + "fitness", + "validation_metrics", + "event_metrics", + "evidence_quality", + "novelty_score", + "fdr_pass", + ) + } | {"outcome": "succeeded"} + async with get_conn() as conn: + updated = await store.update_implementation( + conn, row["implementation_id"], values=values + ) + assert updated is not None + implementation_rows.append((updated, implementation_spec, updated)) + continue + try: + result = await evaluator.evaluate(compiled.source_code) + event_study = evaluate_event_reactions( + bars=validation_bars, + events=validation_events, + asset=asset, + direction=implementation_spec.direction, + holding_bars=implementation_spec.invalidation.holding_bars, + exclusion_bars=implementation_spec.counterfactual.exclusion_bars, + volatility_tolerance=implementation_spec.counterfactual.volatility_tolerance, + volume_tolerance=implementation_spec.counterfactual.volume_tolerance, + ) + validation = result.report.get("validation") or {} + holdout = validation.get("holdout") or {} + holdout_sharpe = float(holdout.get("sharpe") or 0.0) + holdout_return = float(holdout.get("total_return_pct") or 0.0) + holdout_drawdown = float(holdout.get("max_drawdown_pct") or 100.0) + selection_fitness = ( + holdout_sharpe + + 0.02 * holdout_return + - max(0.0, holdout_drawdown - 20.0) / 20.0 + ) + evidence_quality = min(1.0, event_study.event_count / 10.0) * ( + sum(event.confidence for event in validation_events) + / max(1, len(validation_events)) + ) + novelty = _spec_novelty(implementation_spec, hypotheses) + values = { + "outcome": "succeeded", + "fitness": selection_fitness, + "validation_metrics": validation, + "event_metrics": event_study.as_dict(), + "evidence_quality": evidence_quality, + "novelty_score": novelty, + } + except Exception as exc: + values = { + "outcome": "failed", + "error_code": str(getattr(exc, "code", "CAMPAIGN_EVALUATION_FAILED")), + "error_message": str(exc)[:1000], + } + async with get_conn() as conn: + updated = await store.update_implementation( + conn, row["implementation_id"], values=values + ) + assert updated is not None + implementation_rows.append((updated, implementation_spec, updated)) + + succeeded = [item for item in implementation_rows if item[0]["outcome"] == "succeeded"] + p_values = [ + block_bootstrap_p_value( + (item[0].get("event_metrics") or {}).get("event_effects") or (), + seed=seed + index, + ) + for index, item in enumerate(succeeded) + ] + fdr_passes = benjamini_hochberg(p_values, q=0.10) + for (row, _spec, _), fdr_pass in zip(succeeded, fdr_passes, strict=True): + async with get_conn() as conn: + await store.update_implementation( + conn, row["implementation_id"], values={"fdr_pass": fdr_pass} + ) + + by_hypothesis: dict[UUID, list[ImplementationScore]] = {} + spec_by_id = {item.hypothesis_id: item for item in hypotheses} + for row, implementation_spec, _ in succeeded: + validation = row.get("validation_metrics") or {} + event_metrics = row.get("event_metrics") or {} + decay = validation.get("decay_ratio") + by_hypothesis.setdefault(implementation_spec.hypothesis_id, []).append( + ImplementationScore( + fitness=float(row["fitness"]), + max_drawdown_pct=float( + (validation.get("holdout") or {}).get("max_drawdown_pct") or 100 + ), + event_advantage=float(event_metrics.get("event_advantage_pct") or 0), + stability=max(0.0, min(1.0, float(decay or 0))), + evidence_quality=float(row.get("evidence_quality") or 0), + novelty=float(row.get("novelty_score") or 0), + complexity=min(1.0, len(row["source_code"]) / 20_000), + ) + ) + hypothesis_scores: list[HypothesisScore] = [] + for hypothesis_id, implementations in by_hypothesis.items(): + spec = spec_by_id[hypothesis_id] + hypothesis_scores.append( + credit_hypothesis( + hypothesis_id, + lane=spec.lane, + event_family="+".join(spec.event_types), + trigger_mode=spec.trigger_mode, + implementations=implementations, + ) + ) + if not hypothesis_scores: + raise RuntimeError("all campaign implementations failed") + ranks = pareto_ranks(hypothesis_scores) + selected_ids: set[UUID] = set() + if len(hypothesis_scores) >= 2: + plan = plan_next_generation(hypothesis_scores, seed=seed) + selected_ids.update(plan.elites) + selected_ids.update(plan.mutation_parents) + selected_ids.update(plan.crossover_parents) + else: + selected_ids.add(hypothesis_scores[0].hypothesis_id) + score_rows = [ + { + "hypothesis_id": item.hypothesis_id, + "upper_credit": item.credit, + "novelty_score": item.objectives[-1], + "pareto_rank": ranks[item.hypothesis_id], + "selected": item.hypothesis_id in selected_ids, + } + for item in hypothesis_scores + ] + async with get_conn() as conn: + await store.update_hypothesis_scores(conn, campaign["campaign_id"], generation, score_rows) + return hypothesis_scores + + +def _next_hypotheses( + current: list[HypothesisSpec], + scores: list[HypothesisScore], + *, + seed: int, +) -> list[HypothesisSpec]: + by_id = {item.hypothesis_id: item for item in current} + if len(scores) == 1: + parent = by_id[scores[0].hypothesis_id] + return [ + _mutate(parent, index, lineage_kind="elite" if index < 2 else "mutation") + if index < 7 + else _restart(seed) + for index in range(8) + ] + plan = plan_next_generation(scores, seed=seed) + out = [ + _clone(by_id[parent_id], lineage_kind="elite", parent_ids=[parent_id]) + for parent_id in plan.elites + ] + out.extend( + _mutate(by_id[parent_id], index, lineage_kind="mutation") + for index, parent_id in enumerate(plan.mutation_parents) + ) + out.append( + _crossover( + by_id[plan.crossover_parents[0]], + by_id[plan.crossover_parents[1]], + ) + ) + out.append(_restart(seed)) + return out + + +def _clone( + parent: HypothesisSpec, + *, + lineage_kind: str, + parent_ids: list[UUID], + updates: dict[str, Any] | None = None, +) -> HypothesisSpec: + payload = parent.model_dump(mode="json") + payload.update(updates or {}) + if payload["lane"] == "restart" and lineage_kind != "restart": + payload["lane"] = ( + "event_regime" if payload.get("applicable_regimes") else "event" + ) + payload.update( + { + "hypothesis_id": str(uuid4()), + "lineage_kind": lineage_kind, + "parent_ids": [str(item) for item in parent_ids], + } + ) + return HypothesisSpec.model_validate(payload) + + +def _mutate( + parent: HypothesisSpec, + index: int, + *, + lineage_kind: str, +) -> HypothesisSpec: + confirmation = parent.confirmation.model_dump() + invalidation = parent.invalidation.model_dump() + risk = parent.risk.model_dump() + variant = index % 4 + if variant == 0: + confirmation["min_price_change_pct"] = max(0.0, confirmation["min_price_change_pct"] * 0.8) + confirmation["min_volume_ratio"] = min(20.0, confirmation["min_volume_ratio"] * 1.1) + elif variant == 1: + confirmation["min_price_change_pct"] = min(30.0, confirmation["min_price_change_pct"] * 1.2) + confirmation["min_volume_ratio"] = max(0.1, confirmation["min_volume_ratio"] * 0.9) + elif variant == 2: + invalidation["ttl_bars"] = min(100, invalidation["ttl_bars"] + 2) + invalidation["holding_bars"] = max(1, int(invalidation["holding_bars"] * 0.75)) + else: + risk["position_pct"] = max(0.01, risk["position_pct"] * 0.75) + invalidation["max_adverse_pct"] = max(0.5, invalidation["max_adverse_pct"] * 0.8) + return _clone( + parent, + lineage_kind=lineage_kind, + parent_ids=[parent.hypothesis_id], + updates={ + "thesis": f"{parent.thesis};基于上一代验证反馈执行第 {variant + 1} 类定向变异。", + "confirmation": confirmation, + "invalidation": invalidation, + "risk": risk, + }, + ) + + +def _crossover(left: HypothesisSpec, right: HypothesisSpec) -> HypothesisSpec: + event_types = sorted(set(left.event_types) | set(right.event_types)) + direct_allowed = set(event_types) <= {"listing", "delisting", "exploit", "chain_halt"} + payload = left.model_dump(mode="json") + payload.update( + { + "hypothesis_id": str(uuid4()), + "lineage_kind": "crossover", + "parent_ids": [str(left.hypothesis_id), str(right.hypothesis_id)], + "lane": "event_regime", + "thesis": f"交叉验证两个不同机制:{left.thesis[:500]};{right.thesis[:500]}", + "event_types": event_types, + "assets": sorted(set(left.assets) | set(right.assets)), + "evidence_ids": list(dict.fromkeys([*left.evidence_ids, *right.evidence_ids]))[:64], + "trigger_mode": left.trigger_mode if direct_allowed else "confirmed", + "risk": left.risk.model_copy( + update={"position_pct": min(left.risk.position_pct, right.risk.position_pct)} + ).model_dump(), + } + ) + return HypothesisSpec.model_validate(payload) + + +def _restart(seed: int) -> HypothesisSpec: + templates = [ + ("listing", "long", "confirmed", "新上市事件在成交量确认后可能出现延迟价格发现。"), + ("exploit", "short", "hybrid", "安全漏洞冲击可能先扩散后反转,分段确认能降低追空风险。"), + ("chain_halt", "short", "confirmed", "链暂停会造成流动性折价,恢复前维持风险规避方向。"), + ("upgrade", "long", "confirmed", "重大升级在价格与成交量共同确认后可能形成状态迁移。"), + ] + event_type, direction, mode, thesis = templates[seed % len(templates)] + return HypothesisSpec( + lane="restart", + lineage_kind="restart", + thesis=thesis, + event_types=[event_type], + direction=direction, # type: ignore[arg-type] + trigger_mode=mode, # type: ignore[arg-type] + ) + + +def _spec_novelty(spec: HypothesisSpec, population: list[HypothesisSpec]) -> float: + tokens = _spec_tokens(spec) + distances = [] + for other in population: + if other.hypothesis_id == spec.hypothesis_id: + continue + other_tokens = _spec_tokens(other) + union = tokens | other_tokens + similarity = len(tokens & other_tokens) / len(union) if union else 1.0 + distances.append(1.0 - similarity) + return sum(distances) / len(distances) if distances else 1.0 + + +def _spec_tokens(spec: HypothesisSpec) -> set[str]: + words = set(spec.thesis.lower().replace(",", " ").replace("。", " ").split()) + return words | set(spec.event_types) | {spec.lane, spec.trigger_mode, spec.direction} + + +def _search_dataset(dataset: FrozenDataset) -> tuple[FrozenDataset, tuple[Any, ...]]: + bars = dataset.bars + discovery_end = max(2, int(len(bars) * 0.6)) + validation_end = max(discovery_end + 2, int(len(bars) * 0.8)) + validation_end = min(validation_end, len(bars) - 1) + search_bars = bars[:validation_end] + validation_bars = bars[discovery_end:validation_end] + content_hash = hashlib.sha256( + b"".join( + f"{bar.bar_open_at}:{bar.bar_known_at}:{bar.open}:{bar.high}:{bar.low}:{bar.close}:{bar.volume}\n".encode() + for bar in search_bars + ) + ).hexdigest() + manifest = dataset.manifest.model_copy( + update={ + "effective_to": datetime.fromtimestamp(search_bars[-1].bar_open_at / 1e9, tz=UTC), + "latest_bar_ts": datetime.fromtimestamp(search_bars[-1].bar_open_at / 1e9, tz=UTC), + "bar_count": len(search_bars), + "content_sha256": content_hash, + "warnings": [*dataset.manifest.warnings, "sealed_holdout_excluded"], + } + ) + return FrozenDataset(tuple(search_bars), manifest), tuple(validation_bars) + + +def _campaign_config(campaign: dict[str, Any]) -> CampaignConfig: + return CampaignConfig.model_validate( + { + key: campaign["frozen_config"][key] + for key in CampaignConfig.model_fields + if key in campaign["frozen_config"] + } + ) + + +async def _load_frozen_inputs( + campaign: dict[str, Any], + config: CampaignConfig, + settings: EvolverSettings, +) -> tuple[FrozenDataset, dict[str, Any]]: + """Validate frozen market and event inputs before redeeming the owner LLM grant.""" + token = _service_token(campaign["owner_account_id"], settings) + async with DataClient( + settings.data_service_url, + token, + timeout=settings.evolver_data_timeout_s, + ) as client: + dataset = await FrozenBarsLoader(client).load( + venue=config.venue, + symbol=config.symbol, + timeframe=config.timeframe, + from_ts=config.from_ts, + as_of=config.as_of, + ) + snapshot = await client.get_event_snapshot(str(campaign["event_snapshot_id"])) + return dataset, snapshot + + +def _implementation_profile( + parent: HypothesisSpec, + implementation: HypothesisSpec, + index: int, +) -> str: + """Keep three unique ablation identities even when direct mode is prohibited.""" + if set(parent.event_types) <= {"listing", "delisting", "exploit", "chain_halt"}: + return implementation.trigger_mode + return ("canonical", "conservative", "aggressive")[index] + + +def _service_token(account_id: UUID, settings: EvolverSettings) -> str: + return jwt.encode( + { + "sub": str(account_id), + "token_use": "service", + "service_audience": "data", + "token_purpose": "event_campaign_snapshot", + "owner_account_id": str(account_id), + "exp": int(time.time()) + settings.service_token_ttl_s, + }, + settings.jwt_secret, + algorithm=settings.jwt_algorithm, + ) + + +def _base_asset(symbol: str) -> str: + """Normalize common slash and quote-suffixed crypto symbols for event matching.""" + normalized = symbol.upper().replace("-", "/") + if "/" in normalized: + return normalized.split("/", 1)[0] + for quote in ("USDT", "USDC", "USD", "BTC", "ETH"): + if normalized.endswith(quote) and len(normalized) > len(quote): + return normalized[: -len(quote)] + return normalized + + +__all__ = ["evaluate_sealed_holdout", "execute_campaign"] diff --git a/services/evolver/src/inalpha_evolver/runtime/campaign_manager.py b/services/evolver/src/inalpha_evolver/runtime/campaign_manager.py new file mode 100644 index 00000000..9508e31e --- /dev/null +++ b/services/evolver/src/inalpha_evolver/runtime/campaign_manager.py @@ -0,0 +1,181 @@ +"""Lease-backed campaign dispatcher that survives process restarts.""" + +from __future__ import annotations + +import asyncio +import socket +from datetime import UTC, datetime +from typing import Any +from uuid import UUID + +from inalpha_shared import get_logger +from inalpha_shared.db import get_conn + +from ..config import EvolverSettings +from ..owner_llm import CredentialTemporarilyUnavailable +from ..storage import campaigns as store +from .campaign import execute_campaign + +_logger = get_logger(__name__) + + +class CampaignManager: + """Dispatch E2 campaigns separately from the backward-compatible E1 run queue.""" + + def __init__(self, settings: EvolverSettings) -> None: + self.settings = settings + self.worker_id = f"{socket.gethostname()}:{id(self)}" + self.tasks: dict[UUID, asyncio.Task[None]] = {} + self.slots = asyncio.Semaphore(settings.campaign_max_concurrent) + self.dispatcher: asyncio.Task[None] | None = None + self.wake = asyncio.Event() + self.closing = False + self.unhealthy_reason: str | None = None + + @property + def healthy(self) -> bool: + return bool(self.dispatcher and not self.dispatcher.done() and not self.unhealthy_reason) + + async def start(self) -> None: + """Start polling; interrupted campaigns remain replaying until their lease expires.""" + self.dispatcher = asyncio.create_task(self._dispatch(), name="campaign-dispatch") + + async def notify_async(self) -> None: + self.wake.set() + + async def close(self) -> None: + self.closing = True + self.wake.set() + if self.dispatcher is not None: + self.dispatcher.cancel() + for task in self.tasks.values(): + task.cancel() + await asyncio.gather( + *self.tasks.values(), + *([self.dispatcher] if self.dispatcher else []), + return_exceptions=True, + ) + + async def _dispatch(self) -> None: + while not self.closing: + try: + if len(self.tasks) >= self.settings.campaign_max_concurrent: + await self._wait() + continue + async with get_conn() as conn: + campaign = await store.claim_next_campaign( + conn, + worker_id=self.worker_id, + ttl_s=self.settings.campaign_lease_ttl_s, + ) + self.unhealthy_reason = None + if campaign is None: + await self._wait() + continue + task = asyncio.create_task( + self._execute(campaign), + name=f"campaign-{campaign['campaign_id']}", + ) + self.tasks[campaign["campaign_id"]] = task + task.add_done_callback( + lambda done, cid=campaign["campaign_id"]: self._done(cid, done) + ) + except asyncio.CancelledError: + raise + except Exception as exc: + self.unhealthy_reason = f"campaign dispatcher failed: {type(exc).__name__}: {exc}" + await asyncio.sleep(1) + + async def _execute(self, campaign: dict[str, Any]) -> None: + async with self.slots: + try: + async with asyncio.TaskGroup() as group: + work = group.create_task(execute_campaign(campaign, self.settings)) + group.create_task(self._heartbeat(campaign, work)) + self.unhealthy_reason = None + except asyncio.CancelledError: + raise + except Exception as exc: + failure = _primary_exception(exc) + failure_message = str(getattr(failure, "message", failure)) + failure_code = str(getattr(failure, "code", "CAMPAIGN_FAILED")) + retryable = isinstance(failure, CredentialTemporarilyUnavailable) or ( + failure_code + in { + "EVOLUTION_DATA_FRESHNESS_FAILED", + "EVOLUTION_DATA_UNREACHABLE", + } + ) + _logger.exception( + "campaign_execution_failed", + campaign_id=str(campaign["campaign_id"]), + failure_type=type(failure).__name__, + failure_code=failure_code, + failure_message=failure_message, + retryable=retryable, + ) + async with get_conn() as conn: + await store.transition( + conn, + campaign["campaign_id"], + campaign["owner_account_id"], + from_statuses=("replaying",), + to_status="draft" if retryable else "failed", + values={ + "failure_code": failure_code, + "failure_message": failure_message[:1000], + "finished_at": None if retryable else datetime.now(UTC), + "lease_owner": None, + "lease_token": None, + "lease_expires_at": None, + }, + ) + + async def _heartbeat( + self, + campaign: dict[str, Any], + work: asyncio.Task[None], + ) -> None: + """Renew the fencing token until work completes; lease loss cancels the worker.""" + interval = max(1.0, self.settings.campaign_lease_ttl_s / 3) + while not work.done(): + await asyncio.sleep(interval) + if work.done(): + return + async with get_conn() as conn: + renewed = await store.renew_lease( + conn, + campaign["campaign_id"], + worker_id=self.worker_id, + lease_token=campaign["lease_token"], + ttl_s=self.settings.campaign_lease_ttl_s, + ) + if not renewed: + raise RuntimeError("campaign lease fencing token was lost") + + def _done(self, campaign_id: UUID, task: asyncio.Task[None]) -> None: + self.tasks.pop(campaign_id, None) + self.wake.set() + if not task.cancelled(): + task.exception() + + async def _wait(self) -> None: + self.wake.clear() + try: + await asyncio.wait_for(self.wake.wait(), timeout=1.0) + except TimeoutError: + pass + + +def _primary_exception(exc: Exception) -> Exception: + """Unwrap TaskGroup failures so persisted campaign errors remain actionable.""" + current = exc + while isinstance(current, BaseExceptionGroup): + nested = [item for item in current.exceptions if isinstance(item, Exception)] + if not nested: + return exc + current = nested[0] + return current + + +__all__ = ["CampaignManager"] diff --git a/services/evolver/src/inalpha_evolver/storage/campaigns.py b/services/evolver/src/inalpha_evolver/storage/campaigns.py new file mode 100644 index 00000000..895e1d2d --- /dev/null +++ b/services/evolver/src/inalpha_evolver/storage/campaigns.py @@ -0,0 +1,710 @@ +"""Owner-scoped E2 campaign state, lease, holdout, and adoption storage.""" + +from __future__ import annotations + +import hashlib +import json +from datetime import UTC, datetime, timedelta +from typing import Any +from uuid import UUID, uuid4 + +from psycopg import AsyncConnection + +from ..hypothesis.compiler import canonical_spec_hash +from ..hypothesis.models import HypothesisSpec + +_CAMPAIGN_COLUMNS = """campaign_id,owner_account_id,source_run_id,status,active_generation, +hypothesis_budget,implementations_per_hypothesis,max_generations,event_snapshot_id,frozen_config, +llm_snapshot,llm_config_digest,llm_credential_grant,llm_cost_usd, +locked_candidate_id,holdout_consumed_at,forward_started_at,forward_deadline_at, +forward_event_count,forward_metrics,failure_code,failure_message,lease_owner,lease_token, +lease_expires_at,state_version,created_at,updated_at,finished_at""" +_HYPOTHESIS_COLUMNS = """hypothesis_id,campaign_id,generation,slot,lineage_kind,lane,parent_ids, +spec,spec_hash,upper_credit,novelty_score,pareto_rank,selected,created_at""" +_IMPLEMENTATION_COLUMNS = """implementation_id,campaign_id,hypothesis_id,generation,profile, +source_code,source_hash,outcome,fitness,validation_metrics,event_metrics,evidence_quality, +novelty_score,fdr_pass,error_code,error_message,created_at,updated_at""" + + +async def insert_campaign( + conn: AsyncConnection, + *, + owner_account_id: UUID, + requested_by_sub: str, + idempotency_key: str, + request_hash: str, + source_run_id: UUID | None, + event_snapshot_id: UUID, + frozen_config: dict[str, Any], + llm_snapshot: dict[str, Any], + llm_credential_grant: str, + hypotheses: list[HypothesisSpec], +) -> dict[str, Any]: + """Create one campaign and its generation-one hypotheses atomically.""" + campaign_id = uuid4() + async with conn.cursor() as cur: + await cur.execute( + f"""INSERT INTO evolution_campaigns( +campaign_id,owner_account_id,requested_by_sub,idempotency_key,request_hash, +source_run_id,event_snapshot_id,frozen_config, +llm_snapshot,llm_config_digest,llm_credential_grant, +hypothesis_budget,implementations_per_hypothesis,max_generations) +VALUES(%s,%s,%s,%s,%s,%s,%s,%s::jsonb,%s::jsonb,%s,%s,%s,3,5) +ON CONFLICT(owner_account_id,idempotency_key) DO UPDATE +SET idempotency_key=EXCLUDED.idempotency_key RETURNING {_CAMPAIGN_COLUMNS},request_hash""", + ( + campaign_id, + owner_account_id, + requested_by_sub, + idempotency_key, + request_hash, + source_run_id, + event_snapshot_id, + json.dumps(frozen_config), + json.dumps(llm_snapshot), + llm_snapshot["config_digest"], + llm_credential_grant, + len(hypotheses), + ), + ) + campaign = await cur.fetchone() + assert campaign is not None + campaign_id = campaign["campaign_id"] + for slot, spec in enumerate(hypotheses): + await cur.execute( + f"""INSERT INTO evolution_hypotheses( +hypothesis_id,campaign_id,generation,slot,lineage_kind,lane,parent_ids,spec,spec_hash) +VALUES(%s,%s,1,%s,%s,%s,%s,%s::jsonb,%s) +ON CONFLICT(campaign_id,generation,slot) DO NOTHING +RETURNING {_HYPOTHESIS_COLUMNS}""", + ( + spec.hypothesis_id, + campaign_id, + slot, + spec.lineage_kind, + spec.lane, + spec.parent_ids, + spec.model_dump_json(), + canonical_spec_hash(spec), + ), + ) + return dict(campaign) + + +async def get_campaign( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, +) -> dict[str, Any] | None: + """Load one owner-scoped campaign with generation-level projection.""" + async with conn.cursor() as cur: + await cur.execute( + f"""SELECT {_CAMPAIGN_COLUMNS} FROM evolution_campaigns +WHERE campaign_id=%s AND owner_account_id=%s""", + (campaign_id, owner_account_id), + ) + row = await cur.fetchone() + if row is None: + return None + campaign = dict(row) + await cur.execute( + """SELECT generation,count(*) AS hypothesis_count, +count(*) FILTER(WHERE selected) AS selected_count, +max(upper_credit) AS best_credit,max(novelty_score) AS best_novelty +FROM evolution_hypotheses WHERE campaign_id=%s GROUP BY generation ORDER BY generation""", + (campaign_id,), + ) + campaign["generations"] = [dict(item) for item in await cur.fetchall()] + await cur.execute( + f"""SELECT {_HYPOTHESIS_COLUMNS} FROM evolution_hypotheses +WHERE campaign_id=%s ORDER BY generation,slot""", + (campaign_id,), + ) + campaign["hypotheses"] = [dict(item) for item in await cur.fetchall()] + await cur.execute( + f"""SELECT {_IMPLEMENTATION_COLUMNS} FROM evolution_implementations +WHERE campaign_id=%s ORDER BY generation,hypothesis_id,profile""", + (campaign_id,), + ) + campaign["implementations"] = [dict(item) for item in await cur.fetchall()] + return campaign + + +async def list_campaigns( + conn: AsyncConnection, + owner_account_id: UUID, + *, + limit: int, +) -> list[dict[str, Any]]: + """List recent campaigns without loading candidate curves or evidence.""" + async with conn.cursor() as cur: + await cur.execute( + f"""SELECT {_CAMPAIGN_COLUMNS} FROM evolution_campaigns +WHERE owner_account_id=%s ORDER BY created_at DESC,campaign_id DESC LIMIT %s""", + (owner_account_id, limit), + ) + return [dict(row) for row in await cur.fetchall()] + + +async def insert_implementation( + conn: AsyncConnection, + *, + campaign_id: UUID, + hypothesis_id: UUID, + generation: int, + profile: str, + source_code: str, + source_hash: str, +) -> dict[str, Any]: + """Insert one deterministic lower-level implementation idempotently.""" + async with conn.cursor() as cur: + await cur.execute( + f"""INSERT INTO evolution_implementations( +implementation_id,campaign_id,hypothesis_id,generation,profile,source_code,source_hash) +VALUES(%s,%s,%s,%s,%s,%s,%s) +ON CONFLICT(hypothesis_id,profile) DO UPDATE SET updated_at=NOW() +RETURNING {_IMPLEMENTATION_COLUMNS}""", + (uuid4(), campaign_id, hypothesis_id, generation, profile, source_code, source_hash), + ) + row = await cur.fetchone() + assert row is not None + return dict(row) + + +async def update_implementation( + conn: AsyncConnection, + implementation_id: UUID, + *, + values: dict[str, Any], +) -> dict[str, Any] | None: + """Persist evaluation evidence without changing source identity.""" + updates = {**values, "updated_at": datetime.now(UTC)} + assignments = ",".join(f"{key}=%s" for key in updates) + params = [json.dumps(value) if isinstance(value, dict) else value for value in updates.values()] + params.append(implementation_id) + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_implementations SET {assignments} +WHERE implementation_id=%s RETURNING {_IMPLEMENTATION_COLUMNS}""", + params, + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def find_cached_implementation( + conn: AsyncConnection, + campaign_id: UUID, + source_hash: str, +) -> dict[str, Any] | None: + """Reuse evaluation only within the same fully frozen campaign contract.""" + async with conn.cursor() as cur: + await cur.execute( + f"""SELECT {_IMPLEMENTATION_COLUMNS} FROM evolution_implementations +WHERE campaign_id=%s AND source_hash=%s AND outcome='succeeded' +ORDER BY updated_at DESC LIMIT 1""", + (campaign_id, source_hash), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def list_generation_implementations( + conn: AsyncConnection, + campaign_id: UUID, + generation: int, +) -> list[dict[str, Any]]: + """Return lightweight implementation evidence for selection.""" + async with conn.cursor() as cur: + await cur.execute( + f"""SELECT {_IMPLEMENTATION_COLUMNS} FROM evolution_implementations +WHERE campaign_id=%s AND generation=%s ORDER BY hypothesis_id,profile""", + (campaign_id, generation), + ) + return [dict(row) for row in await cur.fetchall()] + + +async def update_hypothesis_scores( + conn: AsyncConnection, + campaign_id: UUID, + generation: int, + scores: list[dict[str, Any]], +) -> None: + """Persist upper credit, novelty, rank, and selection flags.""" + async with conn.cursor() as cur: + for score in scores: + await cur.execute( + """UPDATE evolution_hypotheses SET upper_credit=%s,novelty_score=%s, +pareto_rank=%s,selected=%s WHERE campaign_id=%s AND generation=%s AND hypothesis_id=%s""", + ( + score["upper_credit"], + score["novelty_score"], + score["pareto_rank"], + score["selected"], + campaign_id, + generation, + score["hypothesis_id"], + ), + ) + + +async def add_llm_cost( + conn: AsyncConnection, + campaign_id: UUID, + amount_usd: float, +) -> None: + """Atomically append measured proposer cost without storing prompts or credentials.""" + if amount_usd < 0: + raise ValueError("LLM cost cannot be negative") + async with conn.cursor() as cur: + await cur.execute( + """UPDATE evolution_campaigns SET llm_cost_usd=llm_cost_usd+%s, +state_version=state_version+1,updated_at=NOW() WHERE campaign_id=%s""", + (amount_usd, campaign_id), + ) + + +async def replace_generation_hypotheses( + conn: AsyncConnection, + campaign_id: UUID, + generation: int, + hypotheses: list[HypothesisSpec], +) -> None: + """Replace an unevaluated scaffold with two-call Agent proposals atomically.""" + async with conn.cursor() as cur: + await cur.execute( + """DELETE FROM evolution_hypotheses h WHERE h.campaign_id=%s AND h.generation=%s +AND NOT EXISTS(SELECT 1 FROM evolution_implementations i + WHERE i.campaign_id=h.campaign_id AND i.generation=h.generation)""", + (campaign_id, generation), + ) + if cur.rowcount == 0: + return + await insert_hypotheses(conn, campaign_id, generation, hypotheses) + + +async def insert_hypotheses( + conn: AsyncConnection, + campaign_id: UUID, + generation: int, + hypotheses: list[HypothesisSpec], +) -> None: + """Append exactly one immutable next generation.""" + async with conn.cursor() as cur: + for slot, spec in enumerate(hypotheses): + await cur.execute( + f"""INSERT INTO evolution_hypotheses( +hypothesis_id,campaign_id,generation,slot,lineage_kind,lane,parent_ids,spec,spec_hash) +VALUES(%s,%s,%s,%s,%s,%s,%s,%s::jsonb,%s) +ON CONFLICT(campaign_id,generation,slot) DO NOTHING +RETURNING {_HYPOTHESIS_COLUMNS}""", + ( + spec.hypothesis_id, + campaign_id, + generation, + slot, + spec.lineage_kind, + spec.lane, + spec.parent_ids, + spec.model_dump_json(), + canonical_spec_hash(spec), + ), + ) + + +async def advance_generation( + conn: AsyncConnection, + campaign_id: UUID, + *, + current_generation: int, + next_generation: int, +) -> bool: + """Advance only once after every implementation in the current generation is terminal.""" + async with conn.cursor() as cur: + await cur.execute( + """UPDATE evolution_campaigns c SET active_generation=%s,state_version=state_version+1, +updated_at=NOW() WHERE campaign_id=%s AND status='replaying' AND active_generation=%s +AND NOT EXISTS(SELECT 1 FROM evolution_implementations i WHERE i.campaign_id=c.campaign_id +AND i.generation=%s AND i.outcome='pending')""", + (next_generation, campaign_id, current_generation, current_generation), + ) + return cur.rowcount == 1 + + +async def best_implementation( + conn: AsyncConnection, + campaign_id: UUID, + generation: int, +) -> dict[str, Any] | None: + """Choose the unique FDR-passing champion with stable tie-breakers.""" + async with conn.cursor() as cur: + await cur.execute( + f"""SELECT {_IMPLEMENTATION_COLUMNS} FROM evolution_implementations +WHERE campaign_id=%s AND generation=%s AND outcome='succeeded' AND fdr_pass IS TRUE +ORDER BY fitness DESC,novelty_score DESC,implementation_id LIMIT 1""", + (campaign_id, generation), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def transition( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, + *, + from_statuses: tuple[str, ...], + to_status: str, + values: dict[str, Any] | None = None, +) -> dict[str, Any] | None: + """Compare-and-swap campaign state and increment its projection version.""" + updates = dict(values or {}) + if to_status in {"graduated", "rejected", "insufficient_evidence", "failed", "aborted"}: + updates["llm_credential_grant"] = None + updates.update({"status": to_status, "updated_at": datetime.now(UTC)}) + assignments = ",".join(f"{key}=%s" for key in updates) + params = [json.dumps(value) if isinstance(value, dict) else value for value in updates.values()] + params.extend([campaign_id, owner_account_id, list(from_statuses)]) + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_campaigns SET {assignments},state_version=state_version+1 +WHERE campaign_id=%s AND owner_account_id=%s AND status=ANY(%s) +RETURNING {_CAMPAIGN_COLUMNS}""", + params, + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def acquire_lease( + conn: AsyncConnection, + campaign_id: UUID, + *, + worker_id: str, + ttl_s: int = 60, +) -> dict[str, Any] | None: + """Acquire or steal an expired campaign lease with a fresh fencing token.""" + now = datetime.now(UTC) + token = uuid4() + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_campaigns SET lease_owner=%s,lease_token=%s, +lease_expires_at=%s,state_version=state_version+1,updated_at=%s +WHERE campaign_id=%s AND (lease_expires_at IS NULL OR lease_expires_at<%s OR lease_owner=%s) +RETURNING {_CAMPAIGN_COLUMNS}""", + (worker_id, token, now + timedelta(seconds=ttl_s), now, campaign_id, now, worker_id), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def claim_next_campaign( + conn: AsyncConnection, + *, + worker_id: str, + ttl_s: int = 60, +) -> dict[str, Any] | None: + """Claim one replaying campaign with SKIP LOCKED and a fresh fencing token.""" + now = datetime.now(UTC) + token = uuid4() + async with conn.transaction(): + async with conn.cursor() as cur: + await cur.execute( + """SELECT campaign_id FROM evolution_campaigns +WHERE status='replaying' AND (lease_expires_at IS NULL OR lease_expires_at bool: + """Renew only the current fencing token; stale workers cannot extend ownership.""" + async with conn.cursor() as cur: + await cur.execute( + """UPDATE evolution_campaigns SET lease_expires_at=%s,updated_at=%s +WHERE campaign_id=%s AND lease_owner=%s AND lease_token=%s AND lease_expires_at>=NOW()""", + ( + datetime.now(UTC) + timedelta(seconds=ttl_s), + datetime.now(UTC), + campaign_id, + worker_id, + lease_token, + ), + ) + return cur.rowcount == 1 + + +async def lock_champion( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, + candidate_id: UUID, +) -> dict[str, Any] | None: + """Irreversibly lock one champion and start its 30-90 day forward window.""" + now = datetime.now(UTC) + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_campaigns SET status='waiting_forward',locked_candidate_id=%s, +forward_started_at=%s,forward_deadline_at=%s,active_generation=max_generations, +llm_credential_grant=NULL, +state_version=state_version+1,updated_at=%s +WHERE campaign_id=%s AND owner_account_id=%s AND status='replaying' +AND active_generation=max_generations AND locked_candidate_id IS NULL +AND EXISTS(SELECT 1 FROM evolution_implementations i + WHERE i.implementation_id=%s AND i.campaign_id=evolution_campaigns.campaign_id + AND i.generation=evolution_campaigns.max_generations AND i.outcome='succeeded' + AND i.fdr_pass IS TRUE) +RETURNING {_CAMPAIGN_COLUMNS}""", + ( + candidate_id, + now, + now + timedelta(days=90), + now, + campaign_id, + owner_account_id, + candidate_id, + ), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def record_forward( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, + *, + event_count: int, + metrics: dict[str, Any], +) -> dict[str, Any] | None: + """Persist aggregate forward evidence and derive holdout readiness server-side.""" + now = datetime.now(UTC) + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_campaigns SET +forward_event_count=GREATEST(forward_event_count,%s),forward_metrics=%s::jsonb, +status=CASE + WHEN %s>=forward_deadline_at AND GREATEST(forward_event_count,%s)<3 THEN 'insufficient_evidence' + WHEN %s>=forward_started_at+INTERVAL '30 days' AND GREATEST(forward_event_count,%s)>=3 + AND COALESCE((%s::jsonb->>'passed')::boolean,FALSE) THEN 'holdout_ready' + WHEN %s>=forward_started_at+INTERVAL '30 days' AND GREATEST(forward_event_count,%s)>=3 THEN 'rejected' + ELSE status END, +finished_at=CASE WHEN (%s>=forward_deadline_at AND GREATEST(forward_event_count,%s)<3) + OR (%s>=forward_started_at+INTERVAL '30 days' AND GREATEST(forward_event_count,%s)>=3 + AND NOT COALESCE((%s::jsonb->>'passed')::boolean,FALSE)) THEN %s ELSE finished_at END, +state_version=state_version+1,updated_at=%s +WHERE campaign_id=%s AND owner_account_id=%s AND status='waiting_forward' +RETURNING {_CAMPAIGN_COLUMNS}""", + ( + event_count, + json.dumps(metrics), + now, + event_count, + now, + event_count, + json.dumps(metrics), + now, + event_count, + now, + event_count, + now, + event_count, + json.dumps(metrics), + now, + now, + campaign_id, + owner_account_id, + ), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def reserve_holdout( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, +) -> dict[str, Any] | None: + """Irreversibly consume access before any sealed bars are evaluated.""" + now = datetime.now(UTC) + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_campaigns SET holdout_consumed_at=%s, +state_version=state_version+1,updated_at=%s +WHERE campaign_id=%s AND owner_account_id=%s AND status='holdout_ready' +AND locked_candidate_id IS NOT NULL AND holdout_consumed_at IS NULL +RETURNING {_CAMPAIGN_COLUMNS}""", + (now, now, campaign_id, owner_account_id), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def finalize_holdout( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, + *, + passed: bool, + evidence: dict[str, Any], +) -> dict[str, Any] | None: + """Finish the already-consumed sealed evaluation without allowing a second access.""" + now = datetime.now(UTC) + metrics = {"sealed_holdout": evidence, "holdout_passed": passed} + async with conn.cursor() as cur: + await cur.execute( + f"""UPDATE evolution_campaigns SET status=%s, +forward_metrics=COALESCE(forward_metrics,'{{}}'::jsonb)||%s::jsonb,finished_at=%s, +state_version=state_version+1,updated_at=%s +WHERE campaign_id=%s AND owner_account_id=%s AND status='holdout_ready' +AND locked_candidate_id IS NOT NULL AND holdout_consumed_at IS NOT NULL +AND finished_at IS NULL +RETURNING {_CAMPAIGN_COLUMNS}""", + ( + "graduated" if passed else "rejected", + json.dumps(metrics), + now, + now, + campaign_id, + owner_account_id, + ), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def locked_implementation( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, +) -> dict[str, Any] | None: + """Return the locked source and its DSL only inside the Evolver trust boundary.""" + async with conn.cursor() as cur: + await cur.execute( + """SELECT i.implementation_id,i.source_code,i.source_hash,h.spec +FROM evolution_campaigns c +JOIN evolution_implementations i ON i.implementation_id=c.locked_candidate_id + AND i.campaign_id=c.campaign_id +JOIN evolution_hypotheses h ON h.hypothesis_id=i.hypothesis_id +WHERE c.campaign_id=%s AND c.owner_account_id=%s AND c.status='holdout_ready' +AND c.holdout_consumed_at IS NULL""", + (campaign_id, owner_account_id), + ) + row = await cur.fetchone() + return dict(row) if row else None + + +async def adopt_graduated( + conn: AsyncConnection, + campaign_id: UUID, + owner_account_id: UUID, +) -> dict[str, Any] | None: + """Adopt the locked source as a non-runner-eligible owner asset.""" + async with conn.cursor() as cur: + await cur.execute( + """SELECT c.locked_candidate_id,c.forward_metrics,e.source_code +FROM evolution_campaigns c JOIN evolution_implementations e +ON e.implementation_id=c.locked_candidate_id AND e.campaign_id=c.campaign_id +WHERE c.campaign_id=%s AND c.owner_account_id=%s AND c.status='graduated'""", + (campaign_id, owner_account_id), + ) + row = await cur.fetchone() + if row is None or not row["source_code"]: + return None + source_hash = hashlib.sha256(row["source_code"].encode()).hexdigest() + await cur.execute( + """INSERT INTO strategy_artifacts(artifact_id,source_hash,source_code,compiler_version) +VALUES(%s,%s,%s,'event-strategy-compiler-v1') +ON CONFLICT(source_hash) DO UPDATE SET source_hash=EXCLUDED.source_hash +RETURNING artifact_id""", + (uuid4(), source_hash, row["source_code"]), + ) + artifact = await cur.fetchone() + assert artifact is not None + evidence = row["forward_metrics"] or {} + holdout_evidence = evidence.get("sealed_holdout") or {} + evidence_grade = "limited" if holdout_evidence.get("limited_evidence", True) else "standard" + await cur.execute( + """INSERT INTO strategy_adoptions( +adoption_id,artifact_id,owner_account_id,campaign_id,evidence_grade,status,runner_eligible,evidence) +VALUES(%s,%s,%s,%s,%s,'experimental',FALSE,%s::jsonb) +ON CONFLICT(owner_account_id,artifact_id) DO UPDATE SET evidence=EXCLUDED.evidence, +evidence_grade=EXCLUDED.evidence_grade +RETURNING adoption_id,artifact_id,owner_account_id,campaign_id,evidence_grade,status, +runner_eligible,evidence,adopted_at""", + ( + uuid4(), + artifact["artifact_id"], + owner_account_id, + campaign_id, + evidence_grade, + json.dumps(evidence), + ), + ) + adoption = await cur.fetchone() + return dict(adoption) if adoption else None + + +async def list_adoptions( + conn: AsyncConnection, + owner_account_id: UUID, + *, + limit: int, +) -> list[dict[str, Any]]: + """List owner-scoped experimental assets without returning executable source.""" + async with conn.cursor() as cur: + await cur.execute( + """SELECT a.adoption_id,a.artifact_id,a.owner_account_id,a.campaign_id, +a.evidence_grade,a.status,a.runner_eligible,a.evidence,a.adopted_at, +s.source_hash,s.compiler_version,c.status AS campaign_status +FROM strategy_adoptions a JOIN strategy_artifacts s USING(artifact_id) +LEFT JOIN evolution_campaigns c USING(campaign_id) +WHERE a.owner_account_id=%s ORDER BY a.adopted_at DESC,a.adoption_id DESC LIMIT %s""", + (owner_account_id, limit), + ) + return [dict(row) for row in await cur.fetchall()] + + +__all__ = [ + "acquire_lease", + "add_llm_cost", + "adopt_graduated", + "advance_generation", + "best_implementation", + "claim_next_campaign", + "finalize_holdout", + "find_cached_implementation", + "get_campaign", + "insert_campaign", + "insert_hypotheses", + "insert_implementation", + "list_adoptions", + "list_campaigns", + "list_generation_implementations", + "lock_champion", + "locked_implementation", + "record_forward", + "renew_lease", + "replace_generation_hypotheses", + "reserve_holdout", + "transition", + "update_hypothesis_scores", + "update_implementation", +] diff --git a/services/evolver/tests/llm_snapshot_fixtures.py b/services/evolver/tests/llm_snapshot_fixtures.py index b1bbb832..21e332df 100644 --- a/services/evolver/tests/llm_snapshot_fixtures.py +++ b/services/evolver/tests/llm_snapshot_fixtures.py @@ -53,6 +53,7 @@ def approval_token( { "sub": subject, "token_use": "evolution_credential", + "grant_purpose": "e1_run", "aud": ["inalpha-evolver", "inalpha-dashboard-credential"], "jti": "11111111-1111-4111-8111-111111111111", "operation_id": operation_id, diff --git a/services/evolver/tests/test_approval.py b/services/evolver/tests/test_approval.py index ca8ccb18..1f3bcd31 100644 --- a/services/evolver/tests/test_approval.py +++ b/services/evolver/tests/test_approval.py @@ -37,6 +37,7 @@ def _token( "operation_id": "approval-operation-1", "config_id": "config-1", "provider": "deepseek", + "grant_purpose": "e1_run", "llm_config_digest": _DIGEST, "request_digest": _REQUEST_DIGEST, "iat": now, @@ -59,6 +60,7 @@ def _verify(token: str) -> None: provider="deepseek", llm_config_digest=_DIGEST, request_digest=_REQUEST_DIGEST, + grant_purpose="e1_run", settings=SimpleNamespace( # type: ignore[arg-type] evolution_credential_public_key_b64=_PUBLIC_KEY_B64 ), @@ -105,6 +107,7 @@ def test_approval_rejects_another_owner() -> None: {"operation_id": "another-operation"}, {"config_id": "config-2"}, {"provider": "openai"}, + {"grant_purpose": "event_campaign"}, {"llm_config_digest": "b" * 64}, {"request_digest": "c" * 64}, {"iat": None}, diff --git a/services/evolver/tests/test_campaign_failure_reporting.py b/services/evolver/tests/test_campaign_failure_reporting.py new file mode 100644 index 00000000..85d0da9b --- /dev/null +++ b/services/evolver/tests/test_campaign_failure_reporting.py @@ -0,0 +1,160 @@ +"""Campaign 输入预检与异步失败信息测试。""" +from __future__ import annotations + +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from datetime import UTC, datetime, timedelta +from types import SimpleNamespace +from typing import Any +from uuid import uuid4 + +import pytest +from inalpha_shared.errors import ValidationError + +from inalpha_evolver.config import EvolverSettings +from inalpha_evolver.owner_llm import CredentialTemporarilyUnavailable +from inalpha_evolver.runtime import campaign as campaign_runtime +from inalpha_evolver.runtime import campaign_manager as manager_runtime + + +@pytest.mark.asyncio +async def test_campaign_validates_data_before_redeeming_llm_grant(monkeypatch) -> None: + campaign = { + "campaign_id": uuid4(), + "owner_account_id": uuid4(), + "event_snapshot_id": uuid4(), + "frozen_config": { + "venue": "binance", + "symbol": "BTC/USDT:USDT", + "timeframe": "4h", + "from_ts": datetime.now(UTC) - timedelta(days=30), + "as_of": datetime.now(UTC) - timedelta(hours=1), + }, + } + redeemed = False + + async def fail_load(self, **kwargs: Any) -> None: + raise ValidationError( + "market data upstream unavailable", + code="EVOLUTION_DATA_FRESHNESS_FAILED", + ) + + async def build_mutator(*args: Any, **kwargs: Any) -> None: + nonlocal redeemed + redeemed = True + + monkeypatch.setattr(campaign_runtime.FrozenBarsLoader, "load", fail_load) + monkeypatch.setattr(campaign_runtime, "build_owner_mutator", build_mutator) + + with pytest.raises(ValidationError, match="market data upstream unavailable"): + await campaign_runtime.execute_campaign(campaign, EvolverSettings()) + + assert redeemed is False + + +@pytest.mark.asyncio +async def test_campaign_manager_persists_taskgroup_leaf_error(monkeypatch) -> None: + campaign = { + "campaign_id": uuid4(), + "owner_account_id": uuid4(), + "lease_token": uuid4(), + } + captured: dict[str, Any] = {} + transition: dict[str, Any] = {} + + async def fail_campaign(*args: Any, **kwargs: Any) -> None: + raise ValidationError( + "data-service backfill 502: market data upstream unavailable", + code="EVOLUTION_DATA_FRESHNESS_FAILED", + ) + + @asynccontextmanager + async def fake_conn() -> AsyncIterator[object]: + yield object() + + async def capture_transition(*args: Any, **kwargs: Any) -> None: + transition["to_status"] = kwargs["to_status"] + captured.update(kwargs["values"]) + + monkeypatch.setattr(manager_runtime, "execute_campaign", fail_campaign) + monkeypatch.setattr(manager_runtime, "get_conn", fake_conn) + monkeypatch.setattr(manager_runtime.store, "transition", capture_transition) + settings = SimpleNamespace(campaign_max_concurrent=1, campaign_lease_ttl_s=90) + + await manager_runtime.CampaignManager(settings)._execute(campaign) # type: ignore[arg-type] + + assert captured["failure_code"] == "EVOLUTION_DATA_FRESHNESS_FAILED" + assert captured["failure_message"] == ( + "data-service backfill 502: market data upstream unavailable" + ) + assert transition["to_status"] == "draft" + assert captured["finished_at"] is None + assert captured["lease_token"] is None + + +@pytest.mark.asyncio +async def test_campaign_manager_requeues_temporary_credential_failure(monkeypatch) -> None: + campaign = { + "campaign_id": uuid4(), + "owner_account_id": uuid4(), + "lease_token": uuid4(), + } + transition: dict[str, Any] = {} + + async def fail_campaign(*args: Any, **kwargs: Any) -> None: + raise CredentialTemporarilyUnavailable( + "owner LLM credential service unavailable: ReadTimeout" + ) + + @asynccontextmanager + async def fake_conn() -> AsyncIterator[object]: + yield object() + + async def capture_transition(*args: Any, **kwargs: Any) -> None: + transition["to_status"] = kwargs["to_status"] + transition.update(kwargs["values"]) + + monkeypatch.setattr(manager_runtime, "execute_campaign", fail_campaign) + monkeypatch.setattr(manager_runtime, "get_conn", fake_conn) + monkeypatch.setattr(manager_runtime.store, "transition", capture_transition) + settings = SimpleNamespace(campaign_max_concurrent=1, campaign_lease_ttl_s=90) + + await manager_runtime.CampaignManager(settings)._execute(campaign) # type: ignore[arg-type] + + assert transition["to_status"] == "draft" + assert transition["failure_code"] == "EVOLUTION_CREDENTIAL_UNAVAILABLE" + assert transition["failure_message"].endswith("ReadTimeout") + assert transition["lease_expires_at"] is None + + +@pytest.mark.asyncio +async def test_campaign_dispatcher_recovers_health_after_database_poll(monkeypatch) -> None: + """A successful empty poll clears a prior transient dispatcher failure.""" + settings = SimpleNamespace(campaign_max_concurrent=1, campaign_lease_ttl_s=90) + manager = manager_runtime.CampaignManager(settings) # type: ignore[arg-type] + attempts = 0 + + @asynccontextmanager + async def fake_conn() -> AsyncIterator[object]: + yield object() + + async def claim(*args: Any, **kwargs: Any) -> None: + nonlocal attempts + attempts += 1 + if attempts == 1: + raise RuntimeError("database temporarily unavailable") + manager.closing = True + return None + + async def no_delay(*args: Any, **kwargs: Any) -> None: + return None + + monkeypatch.setattr(manager_runtime, "get_conn", fake_conn) + monkeypatch.setattr(manager_runtime.store, "claim_next_campaign", claim) + monkeypatch.setattr(manager_runtime.asyncio, "sleep", no_delay) + monkeypatch.setattr(manager, "_wait", no_delay) + + await manager._dispatch() + + assert attempts == 2 + assert manager.unhealthy_reason is None diff --git a/services/evolver/tests/test_event_client.py b/services/evolver/tests/test_event_client.py new file mode 100644 index 00000000..7de15ba2 --- /dev/null +++ b/services/evolver/tests/test_event_client.py @@ -0,0 +1,71 @@ +"""Data event snapshot client error-contract tests.""" + +from __future__ import annotations + +from uuid import uuid4 + +import httpx +import pytest +from inalpha_shared.errors import InalphaError, NotFoundError + +from inalpha_evolver import event_client +from inalpha_evolver.config import EvolverSettings + +_REAL_ASYNC_CLIENT = httpx.AsyncClient + + +def _patch_transport( + monkeypatch: pytest.MonkeyPatch, + handler: httpx.MockTransport, +) -> None: + monkeypatch.setattr( + event_client.httpx, + "AsyncClient", + lambda **kwargs: _REAL_ASYNC_CLIENT(transport=handler, timeout=kwargs["timeout"]), + ) + + +@pytest.mark.asyncio +async def test_snapshot_client_preserves_data_not_found_contract(monkeypatch) -> None: + snapshot_id = uuid4() + transport = httpx.MockTransport( + lambda _request: httpx.Response( + 404, + json={ + "detail": { + "code": "EVENT_RECORD_NOT_FOUND", + "message": "snapshot does not exist", + "details": {}, + } + }, + ) + ) + _patch_transport(monkeypatch, transport) + + with pytest.raises(NotFoundError) as error: + await event_client.fetch_event_snapshot( + snapshot_id, + owner_account_id=uuid4(), + settings=EvolverSettings(), + ) + + assert error.value.status_code == 404 + assert error.value.code == "EVENT_RECORD_NOT_FOUND" + + +@pytest.mark.asyncio +async def test_snapshot_client_maps_bad_identity_to_upstream_failure(monkeypatch) -> None: + transport = httpx.MockTransport( + lambda _request: httpx.Response(200, json={"snapshot_id": str(uuid4())}) + ) + _patch_transport(monkeypatch, transport) + + with pytest.raises(InalphaError) as error: + await event_client.fetch_event_snapshot( + uuid4(), + owner_account_id=uuid4(), + settings=EvolverSettings(), + ) + + assert error.value.status_code == 502 + assert error.value.code == "EVENT_SNAPSHOT_IDENTITY_MISMATCH" diff --git a/services/evolver/tests/test_event_evolution.py b/services/evolver/tests/test_event_evolution.py new file mode 100644 index 00000000..f6040888 --- /dev/null +++ b/services/evolver/tests/test_event_evolution.py @@ -0,0 +1,363 @@ +"""Hypothesis DSL, statistical selection, and event-study unit tests.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from uuid import uuid4 + +import pytest +from inalpha_paper.engine.backtest import BacktestEngine +from inalpha_paper.execution.exchange import EventExecutionPolicy +from inalpha_paper.kernel.identifiers import InstrumentId +from inalpha_paper.model.data import Bar +from inalpha_paper.model.market_events import MarketEvent +from inalpha_paper.strategy_authoring import audit_strategy_code, load_strategy_class +from inalpha_shared_llm.types import CacheMetrics, MutationResponse + +from inalpha_evolver.hypothesis.compiler import ( + canonical_spec_hash, + compile_hypothesis, + expand_implementations, +) +from inalpha_evolver.hypothesis.models import HypothesisSpec +from inalpha_evolver.hypothesis.proposer import propose_generation +from inalpha_evolver.hypothesis.seeding import seed_generation_one +from inalpha_evolver.hypothesis.selection import ( + HypothesisScore, + ImplementationScore, + _apply_niche_cap, + benjamini_hochberg, + block_bootstrap_p_value, + credit_hypothesis, + pareto_ranks, + plan_next_generation, +) +from inalpha_evolver.mutator import Mutator +from inalpha_evolver.runtime.campaign import _clone, _mutate + + +def _spec() -> HypothesisSpec: + return HypothesisSpec( + lane="event", + thesis="交易所上币后价格发现可能延迟,成交量确认能够过滤虚假反应。", + evidence_ids=["fact-1:0"], + event_types=["listing"], + assets=["BTC"], + direction="long", + trigger_mode="confirmed", + ) + + +def _bars() -> list[Bar]: + instrument = InstrumentId(symbol="BTC/USDT", venue="binance") + start = datetime(2026, 1, 1, tzinfo=UTC) + return [ + Bar( + instrument_id=instrument, + timeframe="1h", + open=100 + index, + high=101 + index, + low=99 + index, + close=100 + index, + volume=1_000, + ts_open=int((start + timedelta(hours=index)).timestamp() * 1e9), + ts_event=int((start + timedelta(hours=index + 1)).timestamp() * 1e9), + ts_init=int((start + timedelta(hours=index + 1)).timestamp() * 1e9), + ) + for index in range(8) + ] + + +def _event(bars: list[Bar]) -> MarketEvent: + available_at = bars[1].bar_open_at + 30 * 60 * 1_000_000_000 + return MarketEvent( + event_id="listing-1", + event_type="listing", + assets=("BTC",), + action="exchange lists BTC", + severity=1.0, + confidence=1.0, + effective_at=available_at, + available_at=available_at, + ) + + +def _run_compiled(spec: HypothesisSpec) -> int: + bars = _bars() + compiled = compile_hypothesis(spec) + strategy_class = load_strategy_class(compiled.source_code) + engine = BacktestEngine( + fee_rate=0, + event_execution_policy=EventExecutionPolicy(), + ) + strategy = strategy_class( + f"compiled-{spec.trigger_mode}", + engine.clock, + engine.msgbus, + instrument_id=bars[0].instrument_id, + ) + engine.add_strategy(strategy) + return len(engine.run(bars, events=[_event(bars)]).fills) + + +def _updated_spec(**updates: object) -> HypothesisSpec: + payload = _spec().model_dump(mode="python") + payload.update(updates) + return HypothesisSpec.model_validate(payload) + + +def test_strong_event_expands_to_three_auditable_ablation_arms() -> None: + spec = _spec() + arms = expand_implementations(spec) + assert [item.trigger_mode for item in arms] == ["direct", "confirmed", "hybrid"] + for arm in arms: + compiled = compile_hypothesis(arm) + assert audit_strategy_code(compiled.source_code).ok + assert load_strategy_class(compiled.source_code).__name__.startswith("EventHypothesis_") + + +def test_spec_hash_ignores_storage_identity_but_not_mechanism() -> None: + spec = _spec() + clone = spec.model_copy(update={"hypothesis_id": uuid4()}) + changed = clone.model_copy(update={"direction": "short"}) + assert canonical_spec_hash(spec) == canonical_spec_hash(clone) + assert canonical_spec_hash(spec) != canonical_spec_hash(changed) + + +def test_generation_plan_has_fixed_2_4_1_1_topology() -> None: + scores = [ + HypothesisScore( + hypothesis_id=uuid4(), + lane="event" if index < 4 else "regime", + event_family=f"family-{index}", + trigger_mode="confirmed", + credit=float(index), + objectives=(float(index), -index, 0.5, 0.2, 0.8, index / 10), + ) + for index in range(8) + ] + plan = plan_next_generation(scores, seed=7) + assert len(plan.elites) == 2 + assert len(plan.mutation_parents) == 4 + assert len(plan.crossover_parents) == 2 + assert plan.restart_slots == 1 + + +def test_restart_parent_becomes_a_regular_lane_when_inherited() -> None: + restart = HypothesisSpec( + lane="restart", + lineage_kind="restart", + thesis="随机重启探索安全事件冲击后的延迟价格反应与成交量确认机制。", + event_types=["exploit"], + direction="short", + trigger_mode="confirmed", + ) + + elite = _clone( + restart, + lineage_kind="elite", + parent_ids=[restart.hypothesis_id], + ) + mutation = _mutate(restart, 0, lineage_kind="mutation") + + assert elite.lane == "event" + assert elite.lineage_kind == "elite" + assert mutation.lane == "event" + assert mutation.lineage_kind == "mutation" + + +def test_benjamini_hochberg_controls_the_whole_generation() -> None: + assert benjamini_hochberg([0.001, 0.01, 0.04, 0.2], q=0.05) == [True, True, False, False] + + +@pytest.mark.parametrize( + ("trigger_mode", "expected_trades"), + [("direct", 1), ("confirmed", 1), ("hybrid", 2)], +) +def test_compiled_trigger_arms_execute_their_distinct_entry_paths( + trigger_mode: str, + expected_trades: int, +) -> None: + spec = _updated_spec( + trigger_mode=trigger_mode, + confirmation={"min_price_change_pct": 0.1, "min_volume_ratio": 0.1}, + ) + + assert _run_compiled(spec) == expected_trades + + +def test_compiled_confirmation_expires_at_ttl_without_a_trade() -> None: + spec = _updated_spec( + confirmation={"min_price_change_pct": 30.0, "min_volume_ratio": 20.0}, + invalidation={"ttl_bars": 1, "holding_bars": 12, "max_adverse_pct": 4.0}, + ) + + assert _run_compiled(spec) == 0 + + +def test_generation_one_seeds_all_eight_required_direction_lanes() -> None: + snapshot = { + "facts": [ + {"fact_id": "fact-exploit", "event_type": "exploit", "severity": 1.0}, + {"fact_id": "fact-listing", "event_type": "listing", "severity": 0.8}, + {"fact_id": "fact-halt", "event_type": "chain_halt", "severity": 0.9}, + ] + } + + seeds = seed_generation_one(snapshot, "btc") + + assert len(seeds) == 8 + assert [seed.lane for seed in seeds] == [ + "event", + "event", + "event", + "event_regime", + "factor", + "execution_risk", + "regime", + "restart", + ] + assert seeds[0].trigger_mode == "direct" + assert seeds[-1].lineage_kind == "restart" + assert all(seed.assets == ["BTC"] for seed in seeds) + assert {item for seed in seeds[:3] for item in seed.evidence_ids} == { + "fact-exploit:0", + "fact-listing:0", + "fact-halt:0", + } + + +def test_credit_and_pareto_penalize_fragility_and_rank_dominance() -> None: + hypothesis_id = uuid4() + strong = ImplementationScore(2.0, 5.0, 1.0, 0.9, 0.9, 0.8, 0.1) + weak = ImplementationScore(-2.0, 40.0, -1.0, 0.1, 0.1, 0.1, 0.9) + stable = credit_hypothesis( + hypothesis_id, + lane="event", + event_family="listing", + trigger_mode="confirmed", + implementations=[strong, strong], + ) + fragile = credit_hypothesis( + uuid4(), + lane="event", + event_family="listing", + trigger_mode="confirmed", + implementations=[strong, weak], + ) + + assert stable.credit > fragile.credit + dominated = HypothesisScore( + hypothesis_id=uuid4(), + lane=fragile.lane, + event_family=fragile.event_family, + trigger_mode=fragile.trigger_mode, + credit=fragile.credit, + objectives=tuple(value - 1.0 for value in stable.objectives), + ) + assert pareto_ranks([stable, dominated]) == { + stable.hypothesis_id: 0, + dominated.hypothesis_id: 1, + } + + +def test_selection_caps_niches_and_rejects_invalid_statistics() -> None: + same_niche = [ + HypothesisScore( + hypothesis_id=uuid4(), + lane="event", + event_family="listing", + trigger_mode="confirmed", + credit=float(credit), + objectives=(float(credit),) * 6, + ) + for credit in (1, 3, 2) + ] + + capped = _apply_niche_cap(same_niche, cap=2) + + assert [item.credit for item in capped] == [3.0, 2.0] + assert block_bootstrap_p_value([], samples=100) == 1.0 + assert block_bootstrap_p_value([1.0, -0.5, 0.8], samples=100, seed=7) == ( + block_bootstrap_p_value([1.0, -0.5, 0.8], samples=100, seed=7) + ) + with pytest.raises(ValueError, match="q must be within"): + benjamini_hochberg([0.1], q=0) + with pytest.raises(ValueError, match="p-values"): + benjamini_hochberg([-0.1]) + with pytest.raises(ValueError, match="samples"): + block_bootstrap_p_value([1.0], samples=99) + + +class _ProposalClient: + def __init__(self, content: str) -> None: + self.content = content + self.calls = 0 + + async def mutate(self, _request: object) -> MutationResponse: + self.calls += 1 + return MutationResponse( + content=self.content, + cache_metrics=CacheMetrics(input_tokens=100, output_tokens=40), + ) + + async def close(self) -> None: + return None + + +@pytest.mark.asyncio +async def test_agent_proposer_uses_exactly_two_calls_and_preserves_platform_evidence() -> None: + client = _ProposalClient( + """[ + {"thesis":"事件发生后流动性重定价可能形成可证伪的延迟价格反应","trigger_mode":"confirmed"}, + {"thesis":"重大事件冲击可能存在需要成交量确认的延迟反应窗口","trigger_mode":"hybrid"}, + {"thesis":"高置信事件的价格反应持续时间可能显著长于低置信事件"}, + {"thesis":"使用波动状态约束事件触发条件可能减少无效交易和误报"} + ]""" + ) + scaffolds = [ + _spec().model_copy( + update={ + "hypothesis_id": uuid4(), + "evidence_ids": [f"fact-{index}:0"], + "lane": "event" if index < 4 else "event_regime", + } + ) + for index in range(8) + ] + result = await propose_generation( + Mutator( + llm_client=client, # type: ignore[arg-type] + input_usd_per_million=1.0, + output_usd_per_million=2.0, + ), + generation=1, + scaffolds=scaffolds, + feedback=[], + ) + + assert client.calls == 2 + assert len(result.hypotheses) == 8 + assert result.fallback_calls == 0 + assert result.cost_usd == pytest.approx(0.00036) + assert [item.evidence_ids for item in result.hypotheses] == [ + item.evidence_ids for item in scaffolds + ] + assert [item.lane for item in result.hypotheses] == [item.lane for item in scaffolds] + + +@pytest.mark.asyncio +async def test_invalid_agent_batches_fall_back_without_losing_direction_coverage() -> None: + client = _ProposalClient("not-json") + scaffolds = [_spec().model_copy(update={"hypothesis_id": uuid4()}) for _ in range(8)] + + result = await propose_generation( + Mutator(llm_client=client), # type: ignore[arg-type] + generation=2, + scaffolds=scaffolds, + feedback=[{"selected": True}], + ) + + assert client.calls == 2 + assert result.fallback_calls == 2 + assert result.hypotheses == tuple(scaffolds) diff --git a/services/evolver/tests/test_owner_llm.py b/services/evolver/tests/test_owner_llm.py index c6072694..a93bc4b5 100644 --- a/services/evolver/tests/test_owner_llm.py +++ b/services/evolver/tests/test_owner_llm.py @@ -54,6 +54,7 @@ def _settings() -> SimpleNamespace: jwt_secret="test-secret-at-least-32-bytes-long", jwt_algorithm="HS256", evolver_llm_timeout_s=45, + evolver_credential_timeout_s=60, ) @@ -76,6 +77,7 @@ async def test_owner_mutator_uses_frozen_snapshot_and_credential_reference( mutator = await build_owner_mutator(run, settings) # type: ignore[arg-type] assert _CredentialClient.kwargs["trust_env"] is False + assert _CredentialClient.kwargs["timeout"] == 60 assert _CredentialClient.requested_url.endswith("/api/internal/llm-config/config-1") assert _CredentialClient.requested_headers["Authorization"] == "Bearer signed-credential-grant" assert mutator.llm_client.settings.effective_api_key == "owner-test-key" diff --git a/services/paper/src/inalpha_paper/bar_conversion.py b/services/paper/src/inalpha_paper/bar_conversion.py index d1368c39..57888bf2 100644 --- a/services/paper/src/inalpha_paper/bar_conversion.py +++ b/services/paper/src/inalpha_paper/bar_conversion.py @@ -1,4 +1,5 @@ """data-service bar 响应到 paper 内核模型的转换。""" + from __future__ import annotations from datetime import UTC, datetime @@ -6,6 +7,7 @@ from .kernel.clock import datetime_to_ns from .kernel.identifiers import InstrumentId +from .market_evaluation import fixed_timeframe_seconds from .model.data import Bar @@ -17,13 +19,15 @@ def bar_from_dict( """把 data-service ``BarResponse`` 转成内核 ``Bar``。""" raw_ts = data["ts"] timestamp = ( - datetime.fromisoformat(raw_ts.replace("Z", "+00:00")) - if isinstance(raw_ts, str) - else raw_ts + datetime.fromisoformat(raw_ts.replace("Z", "+00:00")) if isinstance(raw_ts, str) else raw_ts ) if timestamp.tzinfo is None: timestamp = timestamp.replace(tzinfo=UTC) - ts_ns = datetime_to_ns(timestamp) + ts_open_ns = datetime_to_ns(timestamp) + timeframe_seconds = fixed_timeframe_seconds(timeframe) + if timeframe_seconds is None: + raise ValueError(f"timeframe {timeframe!r} has no fixed bar-known offset") + ts_known_ns = ts_open_ns + timeframe_seconds * 1_000_000_000 return Bar( instrument_id=instrument_id, timeframe=timeframe, @@ -32,8 +36,9 @@ def bar_from_dict( low=float(data["low"]), close=float(data["close"]), volume=float(data["volume"]), - ts_event=ts_ns, - ts_init=ts_ns, + ts_event=ts_known_ns, + ts_init=ts_known_ns, + ts_open=ts_open_ns, ) diff --git a/services/paper/src/inalpha_paper/data_client.py b/services/paper/src/inalpha_paper/data_client.py index 058ca652..5e08f56d 100644 --- a/services/paper/src/inalpha_paper/data_client.py +++ b/services/paper/src/inalpha_paper/data_client.py @@ -8,6 +8,7 @@ 后续 D-7+ 长任务 / WS 订阅另外加 client。 """ + from __future__ import annotations from datetime import UTC, datetime, timedelta @@ -92,9 +93,7 @@ async def get_ticker( result = r.json() if not isinstance(result, dict): - raise DataServiceError( - f"unexpected ticker response shape: {type(result).__name__}" - ) + raise DataServiceError(f"unexpected ticker response shape: {type(result).__name__}") return result async def get_execution_ticker( @@ -126,12 +125,11 @@ async def get_perp_funding(self, *, venue: str, symbol: str) -> dict[str, Any]: best-effort 兜底(funding=0 + 用 bar close 当 mark + 标注失真)。 """ try: - r = await self._client.get( - "/perp/funding", params={"venue": venue, "symbol": symbol} - ) + r = await self._client.get("/perp/funding", params={"venue": venue, "symbol": symbol}) except httpx.RequestError as e: raise DataServiceError( - f"failed to reach data-service: {e}", code="DATA_SERVICE_UNREACHABLE", + f"failed to reach data-service: {e}", + code="DATA_SERVICE_UNREACHABLE", ) from e if r.status_code >= 400: try: @@ -188,9 +186,7 @@ async def get_fx( result = r.json() if not isinstance(result, dict): - raise DataServiceError( - f"unexpected fx response shape: {type(result).__name__}" - ) + raise DataServiceError(f"unexpected fx response shape: {type(result).__name__}") return result async def get_bars( @@ -318,6 +314,34 @@ async def get_bars_pit( out.append(b) return out + async def get_event_snapshot(self, snapshot_id: str) -> dict[str, Any]: + """Load an immutable point-in-time event snapshot from data-service.""" + try: + response = await self._client.get(f"/events/snapshots/{snapshot_id}") + except httpx.RequestError as exc: + raise DataServiceError( + f"failed to reach data-service event snapshot: {exc}", + code="DATA_SERVICE_UNREACHABLE", + ) from exc + if response.status_code >= 400: + try: + detail = response.json() + except Exception: + detail = {"message": response.text} + raise DataServiceError( + f"data-service event snapshot {response.status_code}: " + f"{detail.get('message', 'unknown')}", + code=detail.get("code", "EVENT_SNAPSHOT_UNAVAILABLE"), + details={"upstream_status": response.status_code, "upstream_body": detail}, + ) + result = response.json() + if not isinstance(result, dict) or not isinstance(result.get("facts"), list): + raise DataServiceError( + "unexpected event snapshot response shape", + code="EVENT_SNAPSHOT_INVALID", + ) + return result + async def backfill_bars( self, *, @@ -362,7 +386,5 @@ async def backfill_bars( result = r.json() if not isinstance(result, dict): - raise DataServiceError( - f"unexpected backfill response shape: {type(result).__name__}" - ) + raise DataServiceError(f"unexpected backfill response shape: {type(result).__name__}") return result diff --git a/services/paper/src/inalpha_paper/engine/backtest.py b/services/paper/src/inalpha_paper/engine/backtest.py index 7fe26932..6f28ec33 100644 --- a/services/paper/src/inalpha_paper/engine/backtest.py +++ b/services/paper/src/inalpha_paper/engine/backtest.py @@ -19,6 +19,7 @@ - 单 strategy 单 instrument(多策略 / 多标的能跑但没专门测试过) - 不收盘强平(最后剩仓位的 PnL 用最后 mark 估) """ + from __future__ import annotations from collections.abc import Callable, Iterable @@ -26,13 +27,14 @@ from datetime import datetime from ..execution import perp_margin -from ..execution.exchange import SimulatedExchange +from ..execution.exchange import EventExecutionPolicy, SimulatedExchange from ..execution.execution_engine import ExecutionEngine from ..execution.risk_engine import RiskEngine from ..execution.risk_rules import LockStore, RiskRule from ..kernel.clock import TestClock from ..kernel.msgbus import MessageBus from ..model.data import Bar +from ..model.market_events import MarketEvent from ..strategy.base import Strategy from .cv import CombinatorialPurgedCV, PurgedKFold, WalkForward from .metrics import max_drawdown_pct, periods_per_year, sharpe_ratio @@ -61,6 +63,7 @@ def __init__( leverage: int = 1, funding_rate: float = 0.0, annualization_periods: int | None = None, + event_execution_policy: EventExecutionPolicy | None = None, ) -> None: """初始化。 @@ -85,7 +88,11 @@ def __init__( self.msgbus = MessageBus() # 执行链(注册顺序:endpoint 先注册,否则 RiskEngine forward 会抛 KeyError) - self.exchange = SimulatedExchange(self.msgbus, self.clock) + self.exchange = SimulatedExchange( + self.msgbus, + self.clock, + event_policy=event_execution_policy, + ) self.execution_engine = ExecutionEngine(self.msgbus, self.exchange) # rules + starting_balance 统一从 BacktestEngine.initial_cash 派生 self.risk_engine = RiskEngine( @@ -96,8 +103,11 @@ def __init__( lock_store=lock_store, ) self.portfolio = Portfolio( - self.msgbus, initial_cash=initial_cash, fee_rate=fee_rate, - trading_mode=trading_mode, leverage=leverage, + self.msgbus, + initial_cash=initial_cash, + fee_rate=fee_rate, + trading_mode=trading_mode, + leverage=leverage, ) # perp 资金费驱动用:每根 bar 在结算时点按此(常数)费率计提。v1 用常数;接历史 # funding 序列见 data.fetch_perp_funding_rate(后续把逐根真 rate 喂进来)。 @@ -141,11 +151,23 @@ def add_strategy(self, strategy: Strategy) -> None: if self._guard is not None: self._guard.bind_strategy(strategy.strategy_id) - def run(self, bars: Iterable[Bar]) -> BacktestReport: - """跑回测,返回 ``BacktestReport``。""" + def run( + self, + bars: Iterable[Bar], + *, + events: Iterable[MarketEvent] | None = None, + ) -> BacktestReport: + """跑回测;在每个 ``bar_known_at`` 决策点先发布已可用事件,再发布 bar。 + + ``events=None`` 与空序列都不新增消息,保持 E1/现有策略的行为不变。相同 + ``event_id`` 只发布一次,输入顺序不影响结果。 + """ bars_list = list(bars) if not bars_list: raise ValueError("backtest needs at least one bar") + ordered_events = sorted(events or (), key=lambda item: (item.available_at, item.event_id)) + event_cursor = 0 + published_event_ids: set[str] = set() # 初始化时间(第一根 bar 之前),便于 strategy.on_start 时拿 clock.now first_ts = bars_list[0].ts_event @@ -172,6 +194,19 @@ def run(self, bars: Iterable[Bar]) -> BacktestReport: bar.instrument_id, self._funding_rate, mark=bar.close ) prev_ts_ns = bar.ts_event + # 3.25 事件事实只在 ``available_at <= bar_known_at`` 后对策略可见。同一 + # 决策点先 event 后 bar,让确认型策略可用当前刚关闭 bar,但订单仍只会在 + # 下一根 process_bar 撮合;不会按“最新价”假成交。 + while ( + event_cursor < len(ordered_events) + and ordered_events[event_cursor].available_at <= bar.bar_known_at + ): + market_event = ordered_events[event_cursor] + event_cursor += 1 + if market_event.event_id in published_event_ids: + continue + published_event_ids.add(market_event.event_id) + self.msgbus.publish("data.market_events", market_event) # 3.5 ADR-0052:框架级持仓保护止损在 mark 更新后判定(与 live session 同点), # 触发的保护性出场单进 pending,下一根 process_bar 撮合(不偷未来)。 # 已知限制(CR #88,仅显式传 rules 的回测受影响、生产 runner rules=None 不触达、 @@ -183,8 +218,7 @@ def run(self, bars: Iterable[Bar]) -> BacktestReport: self._guard.evaluate(bar) # 4. 发布 bar,触发 strategy.on_bar topic = ( - f"data.bars.{bar.instrument_id.venue}." - f"{bar.instrument_id.symbol}.{bar.timeframe}" + f"data.bars.{bar.instrument_id.venue}.{bar.instrument_id.symbol}.{bar.timeframe}" ) self.msgbus.publish(topic, bar) # 5. 记 equity curve(含本根 bar 上策略发单后的最新 mark;下一根 bar 撮合后会再更新一次同 ts 的快照) @@ -320,9 +354,7 @@ def run_cv_backtest( for k in range(1, len(eq)): orig = run_idx[k] if orig in test_set and eq[k - 1] > 0: - returns_by_path.setdefault(sp.path_id, []).append( - (orig, eq[k] / eq[k - 1] - 1.0) - ) + returns_by_path.setdefault(sp.path_id, []).append((orig, eq[k] / eq[k - 1] - 1.0)) sharpe_per_path: list[float] = [] max_dd_per_path: list[float] = [] diff --git a/services/paper/src/inalpha_paper/evaluation_worker.py b/services/paper/src/inalpha_paper/evaluation_worker.py index bd899b62..d0707104 100644 --- a/services/paper/src/inalpha_paper/evaluation_worker.py +++ b/services/paper/src/inalpha_paper/evaluation_worker.py @@ -8,6 +8,7 @@ from inalpha_shared.errors import ValidationError from .engine.backtest import BacktestEngine +from .execution.exchange import EventExecutionPolicy from .strategies import get_strategy_class from .strategy_authoring import ( ContractError, @@ -21,6 +22,7 @@ from .engine.report import BacktestReport from .kernel.identifiers import InstrumentId from .model.data import Bar + from .model.market_events import MarketEvent def run_engine_worker( @@ -42,6 +44,8 @@ def run_engine_worker( leverage: int = 1, funding_rate: float = 0.0, annualization_periods: int | None = None, + events: list[MarketEvent] | None = None, + event_execution_policy: EventExecutionPolicy | None = None, ) -> BacktestReport: """实例化 engine 与策略并执行冻结 bars,不做 IO。""" engine = BacktestEngine( @@ -56,6 +60,7 @@ def run_engine_worker( leverage=leverage, funding_rate=funding_rate, annualization_periods=annualization_periods, + event_execution_policy=event_execution_policy, ) if candidate_code is not None: audit = audit_strategy_code(candidate_code) @@ -106,7 +111,7 @@ def run_engine_worker( **strategy_kwargs, ) engine.add_strategy(strategy) - return engine.run(bars) + return engine.run(bars, events=events) __all__ = ["run_engine_worker"] diff --git a/services/paper/src/inalpha_paper/event_conversion.py b/services/paper/src/inalpha_paper/event_conversion.py new file mode 100644 index 00000000..8bf4bf1d --- /dev/null +++ b/services/paper/src/inalpha_paper/event_conversion.py @@ -0,0 +1,33 @@ +"""Data EventFact wire payload to Paper MarketEvent conversion.""" + +from __future__ import annotations + +from datetime import datetime +from typing import Any + +from .model.market_events import MarketEvent + + +def market_event_from_fact(value: dict[str, Any]) -> MarketEvent: + """Convert without importing Data service models into the Paper kernel.""" + effective_at = datetime.fromisoformat(str(value["effective_at"]).replace("Z", "+00:00")) + available_at = datetime.fromisoformat(str(value["available_at"]).replace("Z", "+00:00")) + evidence = value.get("evidence_spans") or [] + return MarketEvent( + event_id=str(value["fact_id"]), + event_type=str(value["event_type"]), + assets=tuple(str(item).upper() for item in value.get("assets") or []), + action=str(value.get("action") or ""), + severity=float(value.get("severity") or 0), + confidence=float(value.get("confidence") or 0), + effective_at=int(effective_at.timestamp() * 1_000_000_000), + available_at=int(available_at.timestamp() * 1_000_000_000), + evidence_ids=tuple( + str(item.get("quote_hash")) for item in evidence if isinstance(item, dict) + ), + policy_version=str(value.get("policy_version") or ""), + metadata={"raw_event_id": str(value.get("raw_event_id") or "")}, + ) + + +__all__ = ["market_event_from_fact"] diff --git a/services/paper/src/inalpha_paper/execution/exchange.py b/services/paper/src/inalpha_paper/execution/exchange.py index f3bce457..ec35e10e 100644 --- a/services/paper/src/inalpha_paper/execution/exchange.py +++ b/services/paper/src/inalpha_paper/execution/exchange.py @@ -19,15 +19,19 @@ - ``internal.venue.filled`` —— venue 撮合成功 - ``internal.venue.rejected`` —— venue 拒单(如不支持的 OrderType / 现金不足) """ + from __future__ import annotations import logging +from collections import deque +from dataclasses import dataclass from typing import TYPE_CHECKING from ..kernel.clock import Clock from ..kernel.identifiers import ClientOrderId, StrategyId, VenueOrderId from ..kernel.msgbus import MessageBus from ..model.data import Bar +from ..model.market_events import MarketEvent from ..model.orders import Order, OrderSide, OrderType, is_protective_order from .gateway import Gateway @@ -39,10 +43,34 @@ _logger = logging.getLogger(__name__) +@dataclass(frozen=True, slots=True) +class EventExecutionPolicy: + """Versioned conservative fill assumptions for event-driven backtests.""" + + version: str = "event-fill-v1" + max_participation_rate: float = 0.01 + liquidity_floor_bps: float = 8.0 + severity_bps: float = 35.0 + atr_fraction: float = 0.10 + impact_bars: int = 3 + + def __post_init__(self) -> None: + if not 0 < self.max_participation_rate <= 0.1: + raise ValueError("max_participation_rate must be within (0,0.1]") + if self.impact_bars < 1: + raise ValueError("impact_bars must be positive") + + class SimulatedExchange(Gateway): """同时是 Gateway 和 venue 撮合器。""" - def __init__(self, msgbus: MessageBus, clock: Clock) -> None: + def __init__( + self, + msgbus: MessageBus, + clock: Clock, + *, + event_policy: EventExecutionPolicy | None = None, + ) -> None: self._msgbus = msgbus self._clock = clock # 待撮合订单:list of (order, strategy_id) @@ -53,6 +81,13 @@ def __init__(self, msgbus: MessageBus, clock: Clock) -> None: # 本轮 process_bar 内被 portfolio 守门拒的 client_order_id # 用于 process_bar 末把它们从 _pending 移除(避免下一根 bar 重复拒) self._denied_this_round: set[ClientOrderId] = set() + self._event_policy = event_policy + self._active_event_severity = 0.0 + self._event_impact_bars_remaining = 0 + self._previous_volume: float | None = None + self._known_ranges: deque[tuple[float, float]] = deque(maxlen=20) + if event_policy is not None: + self._msgbus.subscribe("data.market_events", self._on_market_event) def bind_portfolio(self, portfolio: Portfolio) -> None: """注入 Portfolio 让 ``_try_fill`` 能在撮合前做现金 / 持仓守门。 @@ -137,6 +172,13 @@ def process_bar(self, bar: Bar) -> int: filled_count += 1 self._pending = remaining + if self._event_impact_bars_remaining > 0: + self._event_impact_bars_remaining -= 1 + if self._event_impact_bars_remaining == 0: + self._active_event_severity = 0.0 + self._previous_volume = bar.volume + if bar.close > 0: + self._known_ranges.append((bar.high - bar.low, bar.close)) return filled_count def flush_protective_at_close(self, bar: Bar) -> int: @@ -167,21 +209,20 @@ def flush_protective_at_close(self, bar: Bar) -> int: # 正常不可达(guard 出场量=持仓);留日志便于未来排查边界异常 _logger.warning( "flush_protective_at_close: 保护单 %s 量 %s 超持仓被守门拒,跳过", - order.client_order_id, order.quantity, + order.client_order_id, + order.quantity, ) remaining.append((order, strategy_id)) continue # perp reduce-only 校验(#117):BUY 保护单必须对应空头持仓,与 live_runner 同口径 - if ( - order.side == OrderSide.BUY - and self._portfolio is not None - ): + if order.side == OrderSide.BUY and self._portfolio is not None: pos = self._portfolio.position(order.instrument_id) cur_qty = pos.quantity if pos is not None else 0.0 if cur_qty >= 0: # 不持空头 → 不允许保护性 BUY(无持仓可平) _logger.warning( "flush_protective_at_close: perp BUY 保护单 %s 无空头持仓(当前 %s),跳过", - order.client_order_id, cur_qty, + order.client_order_id, + cur_qty, ) remaining.append((order, strategy_id)) continue @@ -250,6 +291,24 @@ def _try_fill( if fill_price is None: return None # LIMIT 未触发 + if self._event_policy is not None and self._event_impact_bars_remaining > 0: + if ( + self._previous_volume is not None + and fill_qty > self._previous_volume * self._event_policy.max_participation_rate + ): + participation_limit = ( + f"{self._event_policy.max_participation_rate * 100:g}%" + ) + self._emit_denied( + order, + strategy_id, + "EVENT_CAPACITY_EXCEEDED: order quantity exceeds " + f"{participation_limit} of previously known bar volume", + ) + return None + if order.type == OrderType.MARKET: + fill_price = self._event_adjusted_price(fill_price, order.side) + # spot 守门:portfolio 注入后启用;未注入时退化为旧行为 if self._portfolio is not None: if order.side == OrderSide.BUY: @@ -281,6 +340,32 @@ def _try_fill( return (fill_qty, fill_price) + def _on_market_event(self, message: object) -> None: + """Activate conservative impact assumptions without retaining raw evidence.""" + if not isinstance(message, MarketEvent) or self._event_policy is None: + return + self._active_event_severity = max(self._active_event_severity, message.severity) + self._event_impact_bars_remaining = max( + self._event_impact_bars_remaining, + self._event_policy.impact_bars, + ) + + def _event_adjusted_price(self, open_price: float, side: OrderSide) -> float: + """Apply only information known before the fill bar to adverse slippage.""" + assert self._event_policy is not None + atr_ratio = 0.0 + if self._known_ranges: + atr_ratio = sum(range_ / close for range_, close in self._known_ranges) / len( + self._known_ranges + ) + bps = max( + self._event_policy.liquidity_floor_bps, + self._active_event_severity * self._event_policy.severity_bps + + atr_ratio * 10_000 * self._event_policy.atr_fraction, + ) + direction = 1.0 if side == OrderSide.BUY else -1.0 + return open_price * (1.0 + direction * bps / 10_000) + def _emit_denied(self, order: Order, strategy_id: StrategyId, reason: str) -> None: """守门拒单:emit rejected + 加入 denied 集合让 process_bar drop。""" self._denied_this_round.add(order.client_order_id) @@ -293,3 +378,6 @@ def _emit_denied(self, order: Order, strategy_id: StrategyId, reason: str) -> No "ts": self._clock.now_ns(), }, ) + + +__all__ = ["EventExecutionPolicy", "SimulatedExchange"] diff --git a/services/paper/src/inalpha_paper/model/__init__.py b/services/paper/src/inalpha_paper/model/__init__.py index 6cfa0446..d766a761 100644 --- a/services/paper/src/inalpha_paper/model/__init__.py +++ b/services/paper/src/inalpha_paper/model/__init__.py @@ -3,6 +3,7 @@ 所有事件 / 行情数据用 ``@dataclass(frozen=True, slots=True)`` 强制不可变。 可变状态(``Order``、``Position``)只在 owner 内单线程修改。 """ + from .commands import CancelOrderCommand, ModifyOrderCommand, SubmitOrderCommand from .data import Bar, QuoteTick, TradeTick from .events import ( @@ -17,12 +18,14 @@ PositionEvent, PositionOpened, ) +from .market_events import MarketEvent from .orders import Order, OrderSide, OrderStatus, OrderType from .positions import Position, PositionSide __all__ = [ "Bar", "CancelOrderCommand", + "MarketEvent", "ModifyOrderCommand", "Order", "OrderAccepted", diff --git a/services/paper/src/inalpha_paper/model/data.py b/services/paper/src/inalpha_paper/model/data.py index e867d25b..848242e1 100644 --- a/services/paper/src/inalpha_paper/model/data.py +++ b/services/paper/src/inalpha_paper/model/data.py @@ -3,6 +3,7 @@ ``data_epoch`` 是 [ADR-0013](../../../../docs/decisions/0013-stale-state-detection.md) 的落地: 每次数据连接重连 +1,策略 / 模型在跨 epoch 时必须 reset indicator。 """ + from __future__ import annotations from dataclasses import dataclass @@ -52,7 +53,12 @@ class TradeTick: @dataclass(frozen=True, slots=True) class Bar: - """K 线。``timeframe`` 走 CCXT 风格字符串(``1m`` / ``5m`` / ``1h`` / ``1d``)。""" + """K 线,区分窗口开盘时间与完整 OHLC 已知时间。 + + ``ts_event`` 始终表示本根完整 OHLC 可被策略使用的 ``bar_known_at``。从 + data-service 转换的 bar 同时携带 ``ts_open``;旧测试或内存调用未提供 + ``ts_open`` 时保持原有 ``ts_event`` 语义,避免无事件路径发生行为漂移。 + """ instrument_id: InstrumentId timeframe: str @@ -65,3 +71,14 @@ class Bar: ts_init: int data_epoch: int = 1 is_stale_after_reconnect: bool = False + ts_open: int | None = None + + @property + def bar_open_at(self) -> int: + """Return the candle window open timestamp in nanoseconds.""" + return self.ts_open if self.ts_open is not None else self.ts_event + + @property + def bar_known_at(self) -> int: + """Return the earliest timestamp at which full OHLC is available.""" + return self.ts_event diff --git a/services/paper/src/inalpha_paper/model/market_events.py b/services/paper/src/inalpha_paper/model/market_events.py new file mode 100644 index 00000000..29367fc0 --- /dev/null +++ b/services/paper/src/inalpha_paper/model/market_events.py @@ -0,0 +1,34 @@ +"""Generic normalized market events consumed by deterministic strategies.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + + +@dataclass(frozen=True, slots=True) +class MarketEvent: + """Point-in-time event fact detached from Research and Evolver service models.""" + + event_id: str + event_type: str + assets: tuple[str, ...] + action: str + severity: float + confidence: float + effective_at: int + available_at: int + evidence_ids: tuple[str, ...] = () + policy_version: str = "" + metadata: dict[str, Any] = field(default_factory=dict) + + def __post_init__(self) -> None: + if not self.event_id: + raise ValueError("MarketEvent.event_id is required") + if self.available_at < 0 or self.effective_at < 0: + raise ValueError("MarketEvent timestamps must be non-negative") + if not 0 <= self.severity <= 1 or not 0 <= self.confidence <= 1: + raise ValueError("MarketEvent severity/confidence must be within [0,1]") + + +__all__ = ["MarketEvent"] diff --git a/services/paper/src/inalpha_paper/runner.py b/services/paper/src/inalpha_paper/runner.py index 99117c8d..ccb2437e 100644 --- a/services/paper/src/inalpha_paper/runner.py +++ b/services/paper/src/inalpha_paper/runner.py @@ -9,6 +9,7 @@ ``run_engine_in_subprocess`` 顶层函数,async ``run_backtest`` 通过 ``ProcessPoolExecutor`` ``loop.run_in_executor`` 提交。HTTP I/O / DB 写仍在 main 协程里。 """ + from __future__ import annotations import asyncio @@ -39,9 +40,12 @@ validation_from_report as _validation_from_report, ) from .evaluation_worker import run_engine_worker as run_engine_in_subprocess +from .event_conversion import market_event_from_fact +from .execution.exchange import EventExecutionPolicy from .kernel.identifiers import InstrumentId from .market_evaluation import build_market_evaluation_context from .model.data import Bar +from .model.market_events import MarketEvent from .schemas import ( BacktestRequest, BacktestResponse, @@ -102,10 +106,7 @@ async def run_backtest( ) instrument_id = InstrumentId(symbol=req.symbol, venue=req.venue) - bars = [ - _bar_from_dict(b, instrument_id, market_context.canonical_timeframe) - for b in raw_bars - ] + bars = [_bar_from_dict(b, instrument_id, market_context.canonical_timeframe) for b in raw_bars] if not bars: raise ValidationError( f"data-service returned 0 bars for {req.symbol}@{req.venue} " @@ -122,6 +123,13 @@ async def run_backtest( }, ) + market_events: list[MarketEvent] = [] + event_policy: EventExecutionPolicy | None = None + if req.event_snapshot_id is not None: + event_snapshot = await data_client.get_event_snapshot(str(req.event_snapshot_id)) + market_events = [market_event_from_fact(item) for item in event_snapshot["facts"]] + event_policy = EventExecutionPolicy() + # D-9 · candidate 路径:从 strategy_candidates 表读源码 + 二次审计(defense in depth) candidate_code: str | None = None if req.candidate_id is not None: @@ -140,9 +148,7 @@ async def run_backtest( # 用作落库 / 响应的 strategy_code("candidate:" 与内置 ID 同字段区分) effective_strategy_code = ( - req.strategy_id - if req.strategy_id is not None - else f"candidate:{req.candidate_id}" + req.strategy_id if req.strategy_id is not None else f"candidate:{req.candidate_id}" ) # 2-3. 实例化 engine + strategy + 跑回测(CPU 重活,丢 ProcessPool) @@ -169,6 +175,8 @@ async def run_backtest( trading_mode=req.trading_mode, leverage=req.leverage, funding_rate=req.funding_rate, + events=market_events, + event_execution_policy=event_policy, ), evaluate_buy_and_hold( bars=bars, @@ -196,6 +204,8 @@ async def run_backtest( leverage=req.leverage, funding_rate=req.funding_rate, annualization_periods=market_context.annualization_periods, + events=market_events, + event_execution_policy=event_policy, ) baseline_report = None except (AttributeError, TypeError, ValueError, KeyError, IndexError, ZeroDivisionError) as exc: @@ -252,9 +262,7 @@ async def run_backtest( # 5a'. D-12:holdout 时间切分验证(单次运行按曲线切段,不二次跑引擎)。 # baseline 不切段——alpha 对照仍看全窗。 validation: ValidationBlock | None = ( - source_evaluation.snapshot.validation - if source_evaluation is not None - else None + source_evaluation.snapshot.validation if source_evaluation is not None else None ) if source_evaluation is None and req.validation_split > 0: validation = _validation_from_report( @@ -267,9 +275,7 @@ async def run_backtest( baseline_fitness = ( baseline_evaluation.snapshot.fitness if baseline_evaluation is not None - else _fitness_from_report( - baseline_report, bars_per_year=bars_per_year - ) + else _fitness_from_report(baseline_report, bars_per_year=bars_per_year) ) baseline_snapshot = BaselineSnapshot( strategy_id=BASELINE_BUY_AND_HOLD, @@ -282,9 +288,7 @@ async def run_backtest( ) # 6. 可选落库 + 计算 params_hash(即使不落库也算给响应用) - params_hash = backtest_runs_store.compute_params_hash( - effective_strategy_code, req.params - ) + params_hash = backtest_runs_store.compute_params_hash(effective_strategy_code, req.params) run_id: UUID | None = None if conn is not None: # run 行 + 逐笔成交同一事务写入(只写主候选/内置策略的 fills,**不写 baseline**): @@ -350,6 +354,8 @@ async def run_backtest( params_hash=params_hash, strategy_id=effective_strategy_code, candidate_id=req.candidate_id, + event_snapshot_id=req.event_snapshot_id, + execution_model_version=req.execution_model_version, fitness=fitness_value, baseline=baseline_snapshot, venue=req.venue, @@ -415,6 +421,8 @@ async def _persist_run( "from_ts": req.from_ts.isoformat(), "to_ts": req.to_ts.isoformat(), "initial_cash": req.initial_cash, + "event_snapshot_id": str(req.event_snapshot_id) if req.event_snapshot_id else None, + "execution_model_version": req.execution_model_version, "fee_rate": req.fee_rate, "params": req.params, "candidate_id": str(req.candidate_id) if req.candidate_id else None, @@ -529,9 +537,7 @@ async def run_cv( splitter: CombinatorialPurgedCV | PurgedKFold | WalkForward if req.splitter == "cpcv": - splitter = CombinatorialPurgedCV( - req.n_folds, req.n_test_folds, embargo_pct=req.embargo_pct - ) + splitter = CombinatorialPurgedCV(req.n_folds, req.n_test_folds, embargo_pct=req.embargo_pct) elif req.splitter == "purged_kfold": splitter = PurgedKFold(req.n_folds, embargo_pct=req.embargo_pct) else: @@ -688,6 +694,8 @@ async def _run_engine( leverage: int = 1, funding_rate: float = 0.0, annualization_periods: int | None = None, + events: list[MarketEvent] | None = None, + event_execution_policy: EventExecutionPolicy | None = None, ) -> BacktestReport: """调度 engine 执行:pool 已起则丢 ProcessPool,未起则同进程跑兜底。 @@ -724,6 +732,8 @@ async def _run_engine( leverage=leverage, funding_rate=funding_rate, annualization_periods=annualization_periods, + events=events, + event_execution_policy=event_execution_policy, ) loop = asyncio.get_running_loop() @@ -747,13 +757,13 @@ async def _run_engine( leverage=leverage, funding_rate=funding_rate, annualization_periods=annualization_periods, + events=events, + event_execution_policy=event_execution_policy, ) return await loop.run_in_executor(pool, fn) -def _protective_thresholds() -> tuple[ - float | None, float | None, float | None, float | None, int -]: +def _protective_thresholds() -> tuple[float | None, float | None, float | None, float | None, int]: """从 Settings 读 ADR-0052 框架级持仓保护止损阈值 ``(stop_loss, take_profit, trailing, chandelier_atr_mult, chandelier_atr_period)``。""" s = get_paper_settings() @@ -785,6 +795,8 @@ def _make_pool_call( leverage: int = 1, funding_rate: float = 0.0, annualization_periods: int | None = None, + events: list[MarketEvent] | None = None, + event_execution_policy: EventExecutionPolicy | None = None, ) -> Any: """生成一个无参 callable,丢给 ``run_in_executor``。 @@ -812,4 +824,6 @@ def _make_pool_call( leverage=leverage, funding_rate=funding_rate, annualization_periods=annualization_periods, + events=events, + event_execution_policy=event_execution_policy, ) diff --git a/services/paper/src/inalpha_paper/schemas.py b/services/paper/src/inalpha_paper/schemas.py index 064705a9..644fc8a0 100644 --- a/services/paper/src/inalpha_paper/schemas.py +++ b/services/paper/src/inalpha_paper/schemas.py @@ -1,4 +1,5 @@ """REST API 请求 / 响应 schema。""" + from __future__ import annotations from datetime import UTC, datetime @@ -88,6 +89,14 @@ class BacktestRequest(BaseModel): default=None, description="触发本次回测的原始 strategy_hint(审计用,可空)", ) + event_snapshot_id: UUID | None = Field( + default=None, + description="冻结的 Data EventSnapshot;提供后按 available_at 在 bar 决策点回放。", + ) + execution_model_version: Literal["legacy-bar-v1", "event-fill-v1"] = Field( + default="legacy-bar-v1", + description="事件策略必须显式使用 event-fill-v1 保守冲击/容量模型。", + ) @field_validator("from_ts", "to_ts", mode="after") @classmethod @@ -104,6 +113,11 @@ def _exactly_one_strategy_source(self) -> BacktestRequest: "strategy_source", "must provide exactly one of strategy_id / candidate_id, not both / neither", ) + if self.event_snapshot_id is not None and self.execution_model_version != "event-fill-v1": + raise PydanticCustomError( + "event_execution_model", + "event_snapshot_id requires execution_model_version='event-fill-v1'", + ) return self @@ -127,9 +141,7 @@ class CVBacktestRequest(BacktestRequest): default=0.05, ge=0.0, lt=1.0, description="purge+embargo 占总 bar 比例(按 bar 数)" ) wf_test_size: int = Field(default=21, ge=1, description="walk_forward 每折 test bar 数") - wf_train_size: int = Field( - default=252, ge=1, description="walk_forward train 窗口 bar 数" - ) + wf_train_size: int = Field(default=252, ge=1, description="walk_forward train 窗口 bar 数") @model_validator(mode="after") def _check_cpcv_test_folds(self) -> CVBacktestRequest: @@ -159,9 +171,7 @@ class CVBacktestResponse(BaseModel): sharpe_mean: float dsr: float | None = None dsr_p_value: float | None = None - note: str | None = Field( - default=None, description="回落 / 降级说明(如 cpcv → walk_forward)" - ) + note: str | None = Field(default=None, description="回落 / 降级说明(如 cpcv → walk_forward)") class PositionSnapshot(BaseModel): @@ -373,6 +383,14 @@ class BacktestResponse(BaseModel): default=None, description="D-9 起:若本次回测走候选路径,回填 candidate_id,便于前端串血缘", ) + event_snapshot_id: UUID | None = Field( + default=None, + description="本次回放使用的不可变事件快照;无事件回测为 null。", + ) + execution_model_version: str = Field( + default="legacy-bar-v1", + description="撮合/冲击成本模型版本,缓存和审计必须包含。", + ) fitness: float | None = Field( default=None, description="D-9 起:多目标 fitness(ADR-0020 §适应度函数);裸 Sharpe 排序不可用", @@ -641,7 +659,9 @@ class SubmitOrderRequest(BaseModel): "仅 crypto 永续标的如 BTC/USDT:USDT 生效)", ) leverage: int = Field( - default=1, ge=1, le=20, + default=1, + ge=1, + le=20, description="杠杆倍数(perp 用,1..20);spot 恒 1", ) @@ -653,9 +673,7 @@ def _normalize_and_check_order(self) -> SubmitOrderRequest: if self.order_type == "LIMIT" and self.price is None: raise PydanticCustomError("limit_requires_price", "LIMIT order requires price") if self.order_type == "MARKET" and self.price is not None: - raise PydanticCustomError( - "market_no_price", "MARKET order must not specify price" - ) + raise PydanticCustomError("market_no_price", "MARKET order must not specify price") return self model_config = {"populate_by_name": True} @@ -718,12 +736,9 @@ class PositionRecord(BaseModel): symbol: str quantity: float avg_open_price: float - realized_pnl: float = Field( - description="全历史累计已实现盈亏(毛口径,不含手续费)" - ) + realized_pnl: float = Field(description="全历史累计已实现盈亏(毛口径,不含手续费)") session_realized_pnl: float = Field( - default=0.0, - description="当前持仓相关的已实现盈亏(开仓时清零,平仓时累加)" + default=0.0, description="当前持仓相关的已实现盈亏(开仓时清零,平仓时累加)" ) generation: int currency: str | None = Field( @@ -811,16 +826,18 @@ class DepositRequest(BaseModel): """``POST /accounts/me/deposit`` 请求体:给账户充值(外生资金事件,写流水)。""" amount: float = Field( - ..., gt=0, le=1e9, + ..., + gt=0, + le=1e9, description="充值金额(>0);上限 1e9 防超大值", ) currency: str | None = Field( - default=None, min_length=1, max_length=16, + default=None, + min_length=1, + max_length=16, description="入账币种桶;省略 = 账户 base_currency", ) - note: str | None = Field( - default=None, max_length=500, description="备注(留痕,可空)" - ) + note: str | None = Field(default=None, max_length=500, description="备注(留痕,可空)") class ResetAccountRequest(BaseModel): @@ -831,12 +848,12 @@ class ResetAccountRequest(BaseModel): """ initial_cash: float | None = Field( - default=None, gt=0, le=1e9, + default=None, + gt=0, + le=1e9, description="新一轮初始资金(base_currency 计);省略 = 沿用账户当前 initial_cash", ) - note: str | None = Field( - default=None, max_length=500, description="备注(留痕,可空)" - ) + note: str | None = Field(default=None, max_length=500, description="备注(留痕,可空)") class CashFlowRecord(BaseModel): @@ -964,7 +981,9 @@ class StartStrategyRunRequest(BaseModel): ) leverage: int = Field(default=1, ge=1, le=20, description="杠杆倍数(perp 用,1..20);spot 恒 1") allocation: float | None = Field( - default=None, gt=0, le=1e9, + default=None, + gt=0, + le=1e9, description="本 run 的资金额度(账户 base_currency 计):sizing 与 run 级购买力" "都以它为上限,多 run 共享账户时各自的资金边界。省略时服务端取 " "min(10000, 账户折算可用现金);账户可用 ≤0 时拒绝 start", diff --git a/services/paper/src/inalpha_paper/strategy/base.py b/services/paper/src/inalpha_paper/strategy/base.py index d57f66f3..4e0c8754 100644 --- a/services/paper/src/inalpha_paper/strategy/base.py +++ b/services/paper/src/inalpha_paper/strategy/base.py @@ -3,6 +3,7 @@ 用户子类化 ``Strategy`` 实现交易策略。``submit_order`` 走 ``msgbus.send`` 推到 ``RiskEngine.execute`` endpoint,**不直接调 Gateway**。 """ + from __future__ import annotations from ..kernel.clock import Clock @@ -19,6 +20,7 @@ PositionClosed, PositionOpened, ) +from ..model.market_events import MarketEvent from ..model.orders import Order from .actor import Actor @@ -40,6 +42,7 @@ def __init__(self, name: str, clock: Clock, msgbus: MessageBus) -> None: # 订阅本策略的订单 / 仓位事件 self._msgbus.subscribe(f"events.order.{name}", self._handle_order_event) self._msgbus.subscribe(f"events.position.{name}", self._handle_position_event) + self._msgbus.subscribe("data.market_events", self._handle_market_event) @property def strategy_id(self) -> StrategyId: @@ -108,6 +111,11 @@ def _handle_position_event(self, msg: object) -> None: elif isinstance(msg, PositionClosed): self.on_position_closed(msg) + def _handle_market_event(self, msg: object) -> None: + """Dispatch only normalized market events to untrusted strategy code.""" + if isinstance(msg, MarketEvent): + self.on_market_event(msg) + # ─── 用户覆盖的事件回调 ─── def on_order_submitted(self, event: OrderSubmitted) -> None: ... @@ -118,3 +126,4 @@ def on_order_canceled(self, event: OrderCanceled) -> None: ... def on_position_opened(self, event: PositionOpened) -> None: ... def on_position_changed(self, event: PositionChanged) -> None: ... def on_position_closed(self, event: PositionClosed) -> None: ... + def on_market_event(self, event: MarketEvent) -> None: ... diff --git a/services/paper/src/inalpha_paper/strategy_authoring/dynamic_loader.py b/services/paper/src/inalpha_paper/strategy_authoring/dynamic_loader.py index f4206e94..e984e3cc 100644 --- a/services/paper/src/inalpha_paper/strategy_authoring/dynamic_loader.py +++ b/services/paper/src/inalpha_paper/strategy_authoring/dynamic_loader.py @@ -10,6 +10,7 @@ - 受限 globals 的 ``__builtins__`` 是裁剪过的子集(``ast_audit`` 已拦但 defense in depth) - exec 后从 namespace 里捞出唯一一个 ``Strategy`` 子类 """ + from __future__ import annotations import builtins @@ -31,6 +32,7 @@ PositionClosed, PositionOpened, ) +from ..model.market_events import MarketEvent from ..model.orders import Order, OrderSide, OrderType from ..strategy.base import Strategy @@ -44,17 +46,66 @@ class DynamicLoadError(RuntimeError): _SAFE_BUILTINS: Final[dict[str, Any]] = { name: getattr(builtins, name) for name in ( - "abs", "all", "any", "bool", "bytes", "callable", "chr", "complex", - "dict", "divmod", "enumerate", "filter", "float", "format", "frozenset", - "hash", "hex", "id", "int", "isinstance", "issubclass", "iter", "len", - "list", "map", "max", "min", "next", "object", "oct", "ord", "pow", - "print", "range", "repr", "reversed", "round", "set", "slice", "sorted", - "str", "sum", "tuple", "type", "zip", + "abs", + "all", + "any", + "bool", + "bytes", + "callable", + "chr", + "complex", + "dict", + "divmod", + "enumerate", + "filter", + "float", + "format", + "frozenset", + "hash", + "hex", + "id", + "int", + "isinstance", + "issubclass", + "iter", + "len", + "list", + "map", + "max", + "min", + "next", + "object", + "oct", + "ord", + "pow", + "print", + "range", + "repr", + "reversed", + "round", + "set", + "slice", + "sorted", + "str", + "sum", + "tuple", + "type", + "zip", # 异常类——LLM 写策略时可能 raise ValueError 校验参数 - "Exception", "ValueError", "TypeError", "KeyError", "RuntimeError", - "IndexError", "ZeroDivisionError", "ArithmeticError", "AssertionError", + "Exception", + "ValueError", + "TypeError", + "KeyError", + "RuntimeError", + "IndexError", + "ZeroDivisionError", + "ArithmeticError", + "AssertionError", # 用于 super() 和类相关 - "super", "property", "staticmethod", "classmethod", + "super", + "property", + "staticmethod", + "classmethod", # True/False/None 是关键字不在 builtins,但 NotImplemented 是 "NotImplemented", ) @@ -84,6 +135,7 @@ def _build_restricted_globals() -> dict[str, Any]: "StrategyId": StrategyId, # 数据 / 订单 model "Bar": Bar, + "MarketEvent": MarketEvent, "Order": Order, "OrderSide": OrderSide, "OrderType": OrderType, @@ -129,18 +181,12 @@ def load_strategy_class(code: str) -> type[Strategy]: try: exec(compiled, restricted_globals, namespace) except Exception as exc: - raise DynamicLoadError( - f"exec 策略源码失败:{type(exc).__name__}: {exc}" - ) from exc + raise DynamicLoadError(f"exec 策略源码失败:{type(exc).__name__}: {exc}") from exc # 找 Strategy 子类(必须是 namespace 里**新定义**的,不能是注入的 Strategy 本身) candidates: list[type[Strategy]] = [] for value in namespace.values(): - if ( - isinstance(value, type) - and issubclass(value, Strategy) - and value is not Strategy - ): + if isinstance(value, type) and issubclass(value, Strategy) and value is not Strategy: candidates.append(value) if not candidates: diff --git a/services/paper/src/inalpha_paper/strategy_evaluation.py b/services/paper/src/inalpha_paper/strategy_evaluation.py index fd723fe0..62b03aeb 100644 --- a/services/paper/src/inalpha_paper/strategy_evaluation.py +++ b/services/paper/src/inalpha_paper/strategy_evaluation.py @@ -15,8 +15,10 @@ if TYPE_CHECKING: from .engine.report import BacktestReport + from .execution.exchange import EventExecutionPolicy from .kernel.identifiers import InstrumentId from .model.data import Bar + from .model.market_events import MarketEvent EngineRunner = Callable[..., Awaitable["BacktestReport"]] @@ -42,6 +44,8 @@ async def evaluate_strategy_source( trading_mode: str = "spot", leverage: int = 1, funding_rate: float = 0.0, + events: list[MarketEvent] | None = None, + event_execution_policy: EventExecutionPolicy | None = None, ) -> SourceEvaluation: """审计临时源码并在调用方提供的隔离执行器中评估。""" _validate_bars(bars) @@ -60,6 +64,8 @@ async def evaluate_strategy_source( leverage=leverage, funding_rate=funding_rate, annualization_periods=int(periods), + events=events, + event_execution_policy=event_execution_policy, ) validation, fitness = await asyncio.to_thread( _compute_metrics, diff --git a/services/paper/tests/test_market_event_backtest.py b/services/paper/tests/test_market_event_backtest.py new file mode 100644 index 00000000..614da4d4 --- /dev/null +++ b/services/paper/tests/test_market_event_backtest.py @@ -0,0 +1,147 @@ +"""Point-in-time event ordering and conservative execution regression tests.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +from inalpha_paper.engine.backtest import BacktestEngine +from inalpha_paper.execution.exchange import EventExecutionPolicy +from inalpha_paper.kernel.identifiers import ClientOrderId, InstrumentId +from inalpha_paper.model.data import Bar +from inalpha_paper.model.events import OrderRejected +from inalpha_paper.model.market_events import MarketEvent +from inalpha_paper.model.orders import Order, OrderSide, OrderType +from inalpha_paper.strategy.base import Strategy + + +def _bars() -> list[Bar]: + instrument = InstrumentId(symbol="BTC/USDT", venue="binance") + start = datetime(2026, 1, 1, tzinfo=UTC) + return [ + Bar( + instrument_id=instrument, + timeframe="1h", + open=100 + index, + high=101 + index, + low=99 + index, + close=100 + index, + volume=1_000, + ts_open=int((start + timedelta(hours=index)).timestamp() * 1e9), + ts_event=int((start + timedelta(hours=index + 1)).timestamp() * 1e9), + ts_init=int((start + timedelta(hours=index + 1)).timestamp() * 1e9), + ) + for index in range(5) + ] + + +class EventRecorder(Strategy): + def __init__( + self, + *args: object, + instrument_id: InstrumentId, + order_quantity: float = 0.01, + ) -> None: + super().__init__(*args) # type: ignore[arg-type] + self.instrument_id = instrument_id + self.order_quantity = order_quantity + self.calls: list[tuple[str, str | int]] = [] + self.rejections: list[str] = [] + + def on_start(self) -> None: + self.subscribe_bars(self.instrument_id, "1h") + + def on_market_event(self, event: MarketEvent) -> None: + self.calls.append(("event", event.event_id)) + self.submit_order( + Order( + client_order_id=ClientOrderId(f"event-{event.event_id}"), + instrument_id=self.instrument_id, + side=OrderSide.BUY, + type=OrderType.MARKET, + quantity=self.order_quantity, + ) + ) + + def on_bar(self, bar: Bar) -> None: + self.calls.append(("bar", bar.bar_known_at)) + + def on_order_rejected(self, event: OrderRejected) -> None: + self.rejections.append(event.reason) + + +def test_event_is_stably_deduplicated_before_bar_and_fills_next_open() -> None: + bars = _bars() + available_at = bars[1].bar_open_at + 30 * 60 * 1_000_000_000 + event = MarketEvent( + event_id="event-1", + event_type="listing", + assets=("BTC",), + action="lists BTC pair", + severity=0.8, + confidence=0.9, + effective_at=available_at, + available_at=available_at, + evidence_ids=("fact-1:0",), + metadata={"actor": "exchange", "version": 1}, + ) + engine = BacktestEngine( + fee_rate=0, + event_execution_policy=EventExecutionPolicy(liquidity_floor_bps=10), + ) + strategy = EventRecorder( + "events", + engine.clock, + engine.msgbus, + instrument_id=bars[0].instrument_id, + ) + engine.add_strategy(strategy) + + report = engine.run(bars, events=[event, event]) + + event_index = strategy.calls.index(("event", "event-1")) + assert strategy.calls[event_index + 1] == ("bar", bars[1].bar_known_at) + assert sum(call == ("event", "event-1") for call in strategy.calls) == 1 + assert report.num_trades == 1 + assert report.fills[0].ts_ns == bars[2].bar_known_at + assert report.fills[0].fill_price > bars[2].open + + +def test_no_event_input_is_identical_to_legacy_call() -> None: + bars = _bars() + left = BacktestEngine().run(bars) + right = BacktestEngine().run(bars, events=[]) + assert left == right + + +def test_event_order_above_previous_volume_cap_is_rejected_and_removed() -> None: + bars = _bars() + event = MarketEvent( + event_id="capacity-event", + event_type="exploit", + assets=("BTC",), + action="exploit detected", + severity=1.0, + confidence=1.0, + effective_at=bars[1].bar_open_at, + available_at=bars[1].bar_open_at, + ) + engine = BacktestEngine( + fee_rate=0, + event_execution_policy=EventExecutionPolicy(max_participation_rate=0.01), + ) + strategy = EventRecorder( + "event-capacity", + engine.clock, + engine.msgbus, + instrument_id=bars[0].instrument_id, + order_quantity=10.01, + ) + engine.add_strategy(strategy) + + report = engine.run(bars, events=[event]) + + assert report.num_trades == 0 + assert strategy.rejections == [ + "EVENT_CAPACITY_EXCEEDED: order quantity exceeds 1% of previously known bar volume" + ] + assert engine.exchange.pending_count() == 0 diff --git a/services/research/src/inalpha_research/api/event_facts.py b/services/research/src/inalpha_research/api/event_facts.py new file mode 100644 index 00000000..1c193dfc --- /dev/null +++ b/services/research/src/inalpha_research/api/event_facts.py @@ -0,0 +1,158 @@ +"""Deterministic, tenant-neutral market-event fact extraction.""" + +from __future__ import annotations + +import hashlib +import re +from datetime import UTC, datetime +from typing import Annotated, Any + +from fastapi import APIRouter, Depends, Header +from inalpha_shared.auth import User, get_current_user +from inalpha_shared.errors import UnauthorizedError + +from ..config import ResearchSettings, get_research_settings +from ..data_client import DataClient +from ..schemas import ( + ExtractedEventFact, + ExtractEventFactsRequest, + ExtractEventFactsResponse, +) + +router = APIRouter(prefix="/event-facts", tags=["event-facts"]) + +_EVENT_RULES: tuple[tuple[str, tuple[str, ...], float], ...] = ( + ("delisting", ("delist", "remove trading", "下架", "退市"), 0.95), + ("listing", ("listing", "list on", "new pair", "上线", "上币"), 0.75), + ("exploit", ("exploit", "hack", "drained", "漏洞", "攻击", "被盗"), 1.0), + ("chain_halt", ("chain halt", "network halt", "paused chain", "停链", "暂停出块"), 1.0), + ("regulatory", ("regulator", "sec ", "lawsuit", "ban", "监管", "诉讼"), 0.8), + ("upgrade", ("upgrade", "mainnet", "hard fork", "升级", "主网"), 0.55), + ("unlock", ("token unlock", "vesting", "解锁"), 0.65), + ("burn", ("token burn", "burned", "销毁"), 0.5), + ("partnership", ("partnership", "integration", "合作", "集成"), 0.4), + ("macro", ("interest rate", "inflation", "cpi", "利率", "通胀"), 0.55), +) + + +@router.post("/extract", response_model=ExtractEventFactsResponse) +async def extract_event_facts( + body: ExtractEventFactsRequest, + settings: Annotated[ResearchSettings, Depends(get_research_settings)], + _user: Annotated[User, Depends(get_current_user)], + authorization: Annotated[str | None, Header()] = None, +) -> ExtractEventFactsResponse: + """Extract bounded facts without sending untrusted news text to any LLM.""" + if not authorization or not authorization.startswith("Bearer "): + raise UnauthorizedError("missing Authorization header") + token = authorization.removeprefix("Bearer ").strip() + facts: list[ExtractedEventFact] = [] + failed: list[Any] = [] + async with DataClient(settings.data_service_url, token) as data: + for raw_event_id in body.raw_event_ids: + try: + raw = await data.get_raw_event(str(raw_event_id)) + payload = _extract(raw, body.policy_version) + result = await data.write_event_fact(payload) + fact = result["fact"] + facts.append( + ExtractedEventFact( + fact_id=fact["fact_id"], + raw_event_id=fact["raw_event_id"], + event_type=fact["event_type"], + assets=fact["assets"], + action=fact["action"], + effective_at=fact["effective_at"], + available_at=fact["available_at"], + evidence_ids=[ + f"{fact['fact_id']}:{index}" + for index, _span in enumerate(fact["evidence_spans"]) + ], + created=bool(result["created"]), + ) + ) + except Exception: + failed.append(raw_event_id) + return ExtractEventFactsResponse(facts=facts, failed_event_ids=failed) + + +def _extract(raw: dict[str, Any], policy_version: str) -> dict[str, Any]: + """Map curated provider text to a conservative fact and hashed evidence span.""" + title = str(raw.get("title") or "") + content = str(raw.get("content") or "") + evidence = f"{title}\n{content}".strip() + lowered = evidence.lower() + event_type = "other" + severity = 0.3 + for candidate, terms, candidate_severity in _EVENT_RULES: + if any(term in lowered for term in terms): + event_type = candidate + severity = candidate_severity + break + raw_payload = raw.get("raw_payload") if isinstance(raw.get("raw_payload"), dict) else {} + assets = _assets(raw_payload, evidence) + effective_at = raw.get("source_valid_at") or raw.get("claimed_published_at") + effective_at = effective_at or raw["first_seen_at"] + available_at = raw["accepted_at"] + end = min(len(evidence), 2_000) + spans = [] + if end: + spans.append( + { + "start": 0, + "end": end, + "quote_hash": hashlib.sha256(evidence[:end].encode()).hexdigest(), + } + ) + fact_key = hashlib.sha256( + f"{raw['source']}\0{raw['source_event_id']}\0{event_type}".encode() + ).hexdigest()[:48] + return { + "raw_event_id": raw["event_id"], + "fact_key": fact_key, + "event_type": event_type, + "assets": assets, + "actor": str(raw_payload.get("exchange") or raw.get("source") or "")[:500] or None, + "action": title[:2_000] or f"{event_type} event", + "severity": severity, + "confidence": 0.85 if event_type != "other" else 0.35, + "effective_at": _iso(effective_at), + "available_at": _iso(available_at), + "evidence_spans": spans, + "extractor_version": "deterministic-event-extractor-v1", + "policy_version": policy_version, + "retracted": bool(raw.get("retracted")), + } + + +def _assets(payload: dict[str, Any], evidence: str) -> list[str]: + """Extract normalized crypto symbols from structured fields, then bounded text.""" + values: list[str] = [] + for key in ("coins", "symbols", "coin", "symbol"): + value = payload.get(key) + if isinstance(value, list): + for item in value: + if isinstance(item, dict): + values.append(str(item.get("symbol") or item.get("code") or "")) + else: + values.append(str(item)) + elif value: + values.append(str(value)) + values.extend( + re.findall(r"\b(?:BTC|ETH|SOL|XRP|ADA|DOGE|BNB|AVAX|DOT|LINK)\b", evidence.upper()) + ) + return sorted({item.strip().upper().split("/")[0] for item in values if item.strip()})[:64] + + +def _iso(value: Any) -> str: + """Return one UTC ISO timestamp accepted by Data's strict schema.""" + if isinstance(value, datetime): + parsed = value + else: + parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00")) + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=UTC) + return parsed.astimezone(UTC).isoformat() + + +__all__ = ["router"] diff --git a/services/research/src/inalpha_research/data_client.py b/services/research/src/inalpha_research/data_client.py index 3aee8cc6..7a6ab983 100644 --- a/services/research/src/inalpha_research/data_client.py +++ b/services/research/src/inalpha_research/data_client.py @@ -4,6 +4,7 @@ 避免 services 之间互相 import([docs/miro/03 §模块依赖图](../../../docs/miro/03-kernel-design.md) 强约束:service 间只能通过 HTTP,不互相 import)。 """ + from __future__ import annotations from datetime import datetime @@ -47,6 +48,32 @@ async def __aexit__(self, *args: Any) -> None: async def close(self) -> None: await self._client.aclose() + async def get_raw_event(self, event_id: str) -> dict[str, Any]: + """Load raw evidence inside the trusted platform extraction boundary.""" + response = await self._client.get(f"/events/raw/{event_id}") + if response.status_code >= 400: + raise DataServiceError( + f"raw event request failed with upstream {response.status_code}", + code="RAW_EVENT_UNAVAILABLE", + ) + payload = response.json() + if not isinstance(payload, dict): + raise DataServiceError("raw event response must be an object") + return payload + + async def write_event_fact(self, payload: dict[str, Any]) -> dict[str, Any]: + """Persist one normalized fact while keeping raw text out of the response path.""" + response = await self._client.post("/events/facts", json=payload) + if response.status_code >= 400: + raise DataServiceError( + f"event fact write failed with upstream {response.status_code}", + code="EVENT_FACT_WRITE_FAILED", + ) + result = response.json() + if not isinstance(result, dict): + raise DataServiceError("event fact response must be an object") + return result + async def get_bars( self, *, @@ -179,8 +206,11 @@ async def get_news( r = await self._client.get("/news", params=params) except Exception as exc: return { - "items": [], "providers": [], "is_partial": True, - "coverage_complete": False, "error": str(exc), + "items": [], + "providers": [], + "is_partial": True, + "coverage_complete": False, + "error": str(exc), } if 400 <= r.status_code < 500: raise DataServiceError( @@ -200,13 +230,23 @@ async def get_news( payload = r.json() except Exception: return { - "items": [], "providers": [], "is_partial": True, - "coverage_complete": False, "error": "invalid json", + "items": [], + "providers": [], + "is_partial": True, + "coverage_complete": False, + "error": "invalid json", } - return payload if isinstance(payload, dict) else { - "items": [], "providers": [], "is_partial": True, - "coverage_complete": False, "error": "invalid payload", - } + return ( + payload + if isinstance(payload, dict) + else { + "items": [], + "providers": [], + "is_partial": True, + "coverage_complete": False, + "error": "invalid payload", + } + ) async def get_fundamentals( self, venue: str, symbol: str, as_of: datetime | None = None @@ -233,9 +273,7 @@ async def get_fundamentals( except Exception: return {"available": False, "reason": "invalid json"} - async def get_web_search( - self, query: str, max_results: int = 5 - ) -> list[dict[str, Any]]: + async def get_web_search(self, query: str, max_results: int = 5) -> list[dict[str, Any]]: """``GET /web/search`` —— web 搜索。 失败时返空 list(不阻断整条链路)。 diff --git a/services/research/src/inalpha_research/main.py b/services/research/src/inalpha_research/main.py index 93282359..4755a978 100644 --- a/services/research/src/inalpha_research/main.py +++ b/services/research/src/inalpha_research/main.py @@ -2,6 +2,7 @@ 启动:``uvicorn inalpha_research.main:app --port 8003`` """ + from __future__ import annotations from collections.abc import AsyncIterator @@ -15,7 +16,7 @@ ) from . import __version__ -from .api import deep_dive, health +from .api import deep_dive, event_facts, health from .config import get_research_settings _settings = get_research_settings() @@ -39,3 +40,4 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]: app.include_router(health.router) app.include_router(deep_dive.router) +app.include_router(event_facts.router) diff --git a/services/research/src/inalpha_research/schemas.py b/services/research/src/inalpha_research/schemas.py index 72969e5b..0b59460c 100644 --- a/services/research/src/inalpha_research/schemas.py +++ b/services/research/src/inalpha_research/schemas.py @@ -1,17 +1,60 @@ """REST API + 内部数据契约。""" + from __future__ import annotations from datetime import UTC, datetime from typing import Any, Literal from uuid import UUID, uuid4 -from pydantic import BaseModel, Field, field_validator +from pydantic import BaseModel, ConfigDict, Field, field_validator def _assume_utc_if_naive(v: datetime) -> datetime: return v.replace(tzinfo=UTC) if v.tzinfo is None else v +class ExtractEventFactsRequest(BaseModel): + """Platform-level raw-event extraction request; no owner model is involved.""" + + model_config = ConfigDict(extra="forbid") + + raw_event_ids: list[UUID] = Field(min_length=1, max_length=500) + policy_version: str = Field(default="event-time-policy-v1", min_length=1, max_length=120) + + +class ExtractedEventFact(BaseModel): + """Safe extraction result containing evidence IDs and hashes, never raw content.""" + + fact_id: UUID + raw_event_id: UUID + event_type: Literal[ + "listing", + "delisting", + "exploit", + "chain_halt", + "regulatory", + "upgrade", + "unlock", + "burn", + "partnership", + "macro", + "other", + ] + assets: list[str] + action: str + effective_at: datetime + available_at: datetime + evidence_ids: list[str] + created: bool + + +class ExtractEventFactsResponse(BaseModel): + """Batch extraction summary used to build event snapshots.""" + + facts: list[ExtractedEventFact] + failed_event_ids: list[UUID] + + # ──────────────────────────────────────────────────────────────────── # 输入 # ──────────────────────────────────────────────────────────────────── @@ -90,9 +133,7 @@ def _validate_personas(cls, v: list[str] | None) -> list[str] | None: Horizon = Literal["intraday", "swing", "position"] -StrategyFamily = Literal[ - "trend", "mean_reversion", "buy_hold", "breakout", "volatility", "none" -] +StrategyFamily = Literal["trend", "mean_reversion", "buy_hold", "breakout", "volatility", "none"] """策略族。docs/miro/11 M4 起加 breakout(Donchian 通道突破)/ volatility(ATR 通道)。 ``none`` 表示因子不支持任何已注册策略族,由 compose 引擎拒绝。""" @@ -173,11 +214,20 @@ class AnalystBrief(BaseModel): """单个 analyst 的输出 —— 1 视角研究简报。""" analyst: Literal[ - "technical", "fundamental", "sentiment", "risk", "macro", "valuation", + "technical", + "fundamental", + "sentiment", + "risk", + "macro", + "valuation", # ADR-0037 §A:投资大师人格 persona(可选启用)。runner 的合法类型集从本 # Literal 动态派生(typing.get_args),新增 persona 只需在这里加值。 - "persona_buffett", "persona_lynch", "persona_wood", - "persona_burry", "persona_druckenmiller", "persona_marks", + "persona_buffett", + "persona_lynch", + "persona_wood", + "persona_burry", + "persona_druckenmiller", + "persona_marks", ] = Field( ..., description="哪种分析师产出",