|
1 | 1 | # ResearchSignalContextPipelines |
2 | 2 |
|
3 | | -<!-- qsl-doc-overview:start --> |
| 3 | +[Chinese README](README.zh-CN.md) |
4 | 4 |
|
5 | | -> ⚠️ 投资有风险,不构成投资建议,仅供学习交流用途。 |
6 | 5 | > ⚠️ Investing involves risk. This project does not provide investment advice and is for educational and research purposes only. |
7 | 6 |
|
8 | | -## Open-source overview / 开源项目入口 |
| 7 | +## What this project does |
9 | 8 |
|
10 | | -| Item | Description | |
11 | | -| --- | --- | |
12 | | -| Project type | research pipeline | |
13 | | -| What it does | Builds research signal context artifacts for medium-horizon themes and AI shadow context. | |
14 | | -| 中文说明 | 研究信号上下文管线,输出主题、价格、波动和回撤等可审计上下文 artifact。 | |
15 | | -| Current status | Research-only. Outputs are context inputs, not orders or allocation instructions. | |
| 9 | +ResearchSignalContextPipelines is a **Research context pipeline** in the QuantStrategyLab ecosystem. It builds research signal context artifacts for medium-horizon themes and long-horizon AI shadow context. |
16 | 10 |
|
17 | | -### Quick start |
| 11 | +## Who this is for |
18 | 12 |
|
19 | | -- `python -m pip install -e '.[test]'` |
20 | | -- `python -m pytest -q` |
| 13 | +- Engineers and researchers who want to inspect, reproduce, or extend this part of the QuantStrategyLab stack. |
| 14 | +- Operators who need a clear entry point before reading the deeper runbooks or workflow files. |
| 15 | +- Reviewers who need to understand the repository purpose, safety boundary, and evidence requirements before enabling automation. |
21 | 16 |
|
22 | | -### Deploy / operate safely |
| 17 | +## Current status |
23 | 18 |
|
24 | | -Use scheduled/report workflows only after confirming source freshness; do not connect outputs directly to execution. |
| 19 | +Research-only artifact pipeline; outputs are context, not trade instructions. |
25 | 20 |
|
26 | | -### Strategy performance / evidence boundary |
| 21 | +## Repository layout |
27 | 22 |
|
28 | | -See `docs/architecture.md` and overlay backtest tests for drawdown and overlay evaluation mechanics. |
| 23 | +- `src/`: main library and runtime code. |
| 24 | +- `tests/`: unit and contract tests. |
| 25 | +- `docs/`: detailed design notes, runbooks, and evidence docs. |
| 26 | +- `.github/workflows/`: CI, scheduled jobs, and deployment workflows. |
| 27 | +- `scripts/`: operator scripts and local helpers. |
29 | 28 |
|
30 | | -> Detailed runbooks, migration notes, workflow internals, and historical decisions are kept below. Start with this overview before using the lower-level operational sections. |
| 29 | +## Quick start |
31 | 30 |
|
32 | | -<!-- qsl-doc-overview:end --> |
33 | | - |
34 | | -> ⚠️ 投资有风险,不构成投资建议,仅供学习交流用途。 |
35 | | -
|
36 | | - |
37 | | -## 中文摘要 |
38 | | - |
39 | | -- 完整中文版见 [`README.zh-CN.md`](README.zh-CN.md);本节保留在英文文件顶部,方便从当前文件直接找到中文入口。 |
40 | | -- 用途:本文档围绕 `ResearchSignalContextPipelines`,用于理解 `ResearchSignalContextPipelines` 的配置、运行、部署、研究或验收边界。 |
41 | | -- 主要覆盖:`Repository Role`、`Boundary`、`Current Status`、`Operating Model`、`Name and Horizon Boundary`。 |
42 | | -- 阅读顺序:先确认边界、输入输出和权限要求,再执行文档里的命令、CI、dry-run、发布或切换步骤。 |
43 | | -- 风险提示:涉及实盘、密钥、权限、Cloud Run、交易所或券商 API 的变更,必须先在测试环境或 dry-run 验证;不要只凭示例直接修改生产。 |
44 | | -- 英文正文保留更完整的命令、字段名和配置键;如果摘要和正文不一致,以正文中的实际命令和配置为准。 |
45 | | -[English](README.md) | [简体中文](README.zh-CN.md) |
46 | | - |
47 | | -Research-only signal context artifact repository for QuantStrategyLab. |
48 | | - |
49 | | -Current schema validation requires newly generated and promoted shadow signal |
50 | | -artifacts to use the `1-3 years` horizon. Older pre-contract artifacts may keep |
51 | | -their original point-in-time horizon text, but they should be treated as legacy |
52 | | -records and not promoted as the current `latest_signal.json`. |
53 | | - |
54 | | -This repository does not place trades, store broker credentials, or own live |
55 | | -allocation policy. It prepares and validates shadow signal artifacts that can |
56 | | -later be consumed by sidecar plugins after a separate review and promotion |
57 | | -process. |
58 | | - |
59 | | -## Repository Role |
60 | | - |
61 | | -This is a research artifact repository, not an agent runner, model gateway, |
62 | | -execution service, or strategy plugin repository. |
63 | | - |
64 | | -Its job is to keep point-in-time research evidence reproducible: |
65 | | - |
66 | | -- build current-market context bundles |
67 | | -- create dated GitHub Issues for operator review |
68 | | -- store schema-valid shadow AI signal artifacts |
69 | | -- preserve `signal_history` for future walk-forward replay |
70 | | -- provide deterministic replay tooling around saved artifacts |
71 | | - |
72 | | -`CodexAuditBridge` remains the only bridge/runner for model providers and |
73 | | -cross-repository write automation. Future live or notification behavior belongs |
74 | | -in a separate deterministic plugin after the shadow artifacts have enough |
75 | | -evidence. |
76 | | - |
77 | | -## Boundary |
78 | | - |
79 | | -This repo owns: |
80 | | - |
81 | | -- long-horizon AI context bundle examples |
82 | | -- shadow signal JSON schema expectations |
83 | | -- validation tooling for `latest_signal.json` |
84 | | -- issue/workflow handoff to `QuantStrategyLab/CodexAuditBridge` |
85 | | -- replay-ready artifact records for later research review |
86 | | - |
87 | | -This repo does not own: |
88 | | - |
89 | | -- broker API access |
90 | | -- order placement |
91 | | -- live portfolio allocation |
92 | | -- deterministic strategy rules in `UsEquityStrategies` |
93 | | -- runtime plugin execution in `QuantStrategyPlugins` |
94 | | -- API keys for model providers |
95 | | -- Codex/OpenAI/Anthropic provider routing |
96 | | -- GitHub App token minting for source repository writes |
97 | | -- Telegram or broker-facing runtime notifications |
98 | | - |
99 | | -## Current Status |
100 | | - |
101 | | -This repository is in shadow research accumulation mode. The first saved |
102 | | -point-in-time artifact is `data/output/signal_history/2026-05-28.json`; the |
103 | | -current promoted artifact is `data/output/signal_history/2026-05-31.json` and |
104 | | -uses the long-horizon contract. |
105 | | - |
106 | | -Near-term work should focus on: |
107 | | - |
108 | | -- keeping the monthly workflow healthy |
109 | | -- accumulating saved `signal_history/*.json` artifacts |
110 | | -- replaying only saved artifacts, not regenerated historical AI judgments |
111 | | -- improving context quality before any downstream plugin integration |
112 | | - |
113 | | -Do not promote the output into runtime allocation or notifications until the |
114 | | -saved artifact history has enough walk-forward evidence to justify a separate |
115 | | -plugin contract. |
116 | | - |
117 | | -## Operating Model |
118 | | - |
119 | | -1. A monthly workflow builds a point-in-time context bundle from current market |
120 | | - prices. |
121 | | -2. The workflow creates or updates a dated long-horizon shadow-signal issue and |
122 | | - embeds the context bundle as review evidence. |
123 | | -3. The issue is dispatched to `QuantStrategyLab/CodexAuditBridge` with task |
124 | | - `long_horizon_signal_shadow`. |
125 | | -4. `CodexAuditBridge` tries self-hosted Codex first and uses its own OpenAI or |
126 | | - Anthropic API fallback only when configured. |
127 | | -5. Any AI-generated artifact must remain `mode=shadow` and pass local schema |
128 | | - validation. |
129 | | -6. Downstream runtimes must treat the artifact as advisory context only until a |
130 | | - separate deterministic policy engine explicitly consumes it. |
131 | | - |
132 | | - |
133 | | -## Name and Horizon Boundary |
134 | | - |
135 | | -`ResearchSignalContextPipelines` is the canonical repository name for this layer. |
136 | | -The name reflects its actual responsibility: maintaining reusable research |
137 | | -context artifacts, including medium-horizon theme momentum and long-horizon AI |
138 | | -shadow context. Short/medium/long final recommendations are still produced by |
139 | | -`QuantAdvisorResearch`, not by this repository. |
140 | | - |
141 | | - |
142 | | -## Horizon Boundary |
143 | | - |
144 | | -This repository does not directly produce short-term buy/sell recommendations. |
145 | | -The horizon split is: |
146 | | - |
147 | | -- Short term (`1-10 trading days`): handled by event evidence from `PoliticalEventTrackingResearch` plus deterministic Advisor rules. AI can explain context but should not decide short-term recommendations. |
148 | | -- Medium term (`2-12 weeks`): represented by `theme_momentum_snapshot.json` as `medium_horizon_theme_context`, including theme momentum, breadth, and strong members inside each theme. |
149 | | -- Long term (`1-3 years`): represented by `latest_signal.json` and `signal_history/*.json` as AI shadow context. |
150 | | - |
151 | | -`QuantAdvisorResearch` remains the only layer that combines these inputs into final short/medium/long recommendations. |
152 | | - |
153 | | -## GitHub Configuration |
154 | | - |
155 | | -The model API keys are centralized in `CodexAuditBridge`; do not add |
156 | | -`OPENAI_API_KEY` or `ANTHROPIC_API_KEY` to this repository. |
157 | | - |
158 | | -This repository needs only dispatch credentials for the bridge workflow: |
159 | | - |
160 | | -- preferred: `CROSS_REPO_GITHUB_APP_ID` variable and |
161 | | - `CROSS_REPO_GITHUB_APP_PRIVATE_KEY` secret with Actions write permission on |
162 | | - `CodexAuditBridge` |
163 | | -- fallback: `CODEX_AUDIT_DISPATCH_TOKEN` secret with permission to dispatch the |
164 | | - bridge workflow |
165 | | - |
166 | | -Configured non-secret variables: |
167 | | - |
168 | | -- `SELFHOSTED_CODEX_REVIEW_REPOSITORY=QuantStrategyLab/CodexAuditBridge` |
169 | | -- `SELFHOSTED_CODEX_REVIEW_PROVIDER=auto` |
170 | | -- `CROSS_REPO_GITHUB_APP_ID=3250578` |
171 | | - |
172 | | -## Notification Policy |
173 | | - |
174 | | -The GitHub issue created by `.github/workflows/dispatch_shadow_signal.yml` is the |
175 | | -initial operator notification channel. It is labeled `long-horizon-shadow`, |
176 | | -deduplicated by date, and receives the CodexAuditBridge result as comments or a |
177 | | -focused PR. |
178 | | - |
179 | | -Do not add Telegram, broker, or runtime plugin notifications at this stage. Those |
180 | | -belong downstream only after the signal graduates from shadow research to a |
181 | | -deterministic plugin contract. |
182 | | - |
183 | | -## Local Validation |
184 | | - |
185 | | -Validate the example artifact: |
| 31 | +From a fresh clone: |
186 | 32 |
|
187 | 33 | ```bash |
188 | | -python scripts/validate_latest_signal.py examples/latest_signal.example.json |
| 34 | +python -m pip install -e . |
| 35 | +python -m pytest -q |
189 | 36 | ``` |
190 | 37 |
|
191 | | -Build a context bundle from a local price file: |
| 38 | +If a command requires credentials, run it only after reading the relevant workflow or runbook and configuring secrets outside Git. |
192 | 39 |
|
193 | | -```bash |
194 | | -python scripts/build_context_bundle.py \ |
195 | | - --prices examples/price_history.example.csv \ |
196 | | - --symbols QQQ \ |
197 | | - --output data/output/context_bundle/latest_context_bundle.json |
198 | | -``` |
| 40 | +## Deployment and operation |
199 | 41 |
|
200 | | -Without `--prices`, the script downloads recent daily prices for the default |
201 | | -universe through Yahoo's chart endpoint and writes a point-in-time context bundle |
202 | | -for the monthly shadow issue. The scheduled workflow uses |
203 | | -`--allow-download-errors`, so external data-source failures still create an |
204 | | -operator issue with the failure recorded instead of silently skipping the run. |
| 42 | +Run the configured workflows or local commands to refresh artifacts. Review source coverage and artifact diffs before publishing downstream. |
205 | 43 |
|
206 | | -Validate the promoted latest artifact when it exists: |
| 44 | +Prefer manual or dry-run execution first. Enable schedules or live execution only after logs, artifacts, permissions, and rollback steps are reviewed. |
207 | 45 |
|
208 | | -```bash |
209 | | -python scripts/validate_latest_signal.py |
210 | | -``` |
211 | | - |
212 | | -Run the synthetic overlay replay: |
213 | | - |
214 | | -```bash |
215 | | -python scripts/backtest_signal_overlay.py \ |
216 | | - --prices examples/price_history.example.csv \ |
217 | | - --signals examples/signal_history \ |
218 | | - --symbol QQQ |
219 | | -``` |
220 | | - |
221 | | -The replay tests a deterministic risk-reducing overlay only. It does not call |
222 | | -AI models and does not treat the example as production evidence. |
| 46 | +## Strategy performance and evidence |
223 | 47 |
|
224 | | -Extract compact real-price input from an existing QuantStrategyLab price file: |
| 48 | +Not a trading strategy repository. Evidence quality is measured by source traceability, freshness, and downstream review usefulness. |
225 | 49 |
|
226 | | -```bash |
227 | | -python scripts/extract_price_history.py \ |
228 | | - --source ../UsEquitySnapshotPipelines/data/output/tqqq_growth_income_real_full_archive_2026-05-26/price_history.csv \ |
229 | | - --target data/input/qqq_price_history.csv \ |
230 | | - --symbols QQQ |
231 | | -``` |
232 | | - |
233 | | -Then replay stored shadow signals against those prices: |
| 50 | +README files are intentionally not a source of dated performance promises. Re-run the relevant tests, backtests, or pipeline jobs before relying on any result. |
234 | 51 |
|
235 | | -```bash |
236 | | -python scripts/backtest_signal_overlay.py \ |
237 | | - --prices data/input/qqq_price_history.csv \ |
238 | | - --signals data/output/signal_history \ |
239 | | - --symbol QQQ \ |
240 | | - --output data/output/tmp/replay_summary.json |
241 | | -``` |
| 52 | +## Safety notes |
242 | 53 |
|
243 | | -The price loader accepts both this repository's compact `date,symbol,close` |
244 | | -schema and the existing QuantStrategyLab `symbol,as_of,close` schema. |
| 54 | +- Never commit API keys, broker credentials, OAuth tokens, cookies, or account identifiers. |
| 55 | +- Run new strategies and platform changes in dry-run or paper mode before any live execution. |
| 56 | +- Review generated orders, artifacts, and logs manually before enabling schedules. |
245 | 57 |
|
246 | | -## Artifact Contract |
| 58 | +## Contributing |
247 | 59 |
|
248 | | -The latest artifact path is: |
249 | | - |
250 | | -```text |
251 | | -data/output/latest_signal.json |
252 | | -``` |
253 | | - |
254 | | -Historical generated copies can be stored under: |
255 | | - |
256 | | -```text |
257 | | -data/output/signal_history/YYYY-MM-DD.json |
258 | | -``` |
259 | | - |
260 | | -All promoted latest artifacts must use `horizon: "1-3 years"`. To support |
261 | | -Advisor long-horizon buckets, current artifacts should also include versioned |
262 | | -theme context through `theme_bias`, `symbol_theme_exposure`, and optional |
263 | | -`symbol_bias` for symbols that need explicit long-context coverage. |
264 | | - |
265 | | -All artifacts must remain shadow-only. They cannot encode broker orders, target |
266 | | -quantities, or live allocation overrides. |
267 | | - |
268 | | -`candidate_bias` and `theme_bias` may use either the legacy compact form: |
269 | | - |
270 | | -```json |
271 | | -{"MU": "watch"} |
272 | | -``` |
273 | | - |
274 | | -or the structured audit form: |
275 | | - |
276 | | -```json |
277 | | -{ |
278 | | - "MU": { |
279 | | - "bias": "watch", |
280 | | - "confidence": 0.55, |
281 | | - "linked_themes": ["hbm_memory"], |
282 | | - "rationale": "Shadow context only; not a trade instruction." |
283 | | - } |
284 | | -} |
285 | | -``` |
286 | | - |
287 | | -`symbol_bias` is optional and uses the same structured shape for symbol-specific |
288 | | -long-horizon context. Downstream Advisor code treats these fields as context and |
289 | | -still blocks orders, target quantities, and portfolio weights. |
290 | | - |
291 | | -## Replay Contract |
292 | | - |
293 | | -Historical validation should replay stored signal artifacts instead of asking a |
294 | | -model to re-create old judgments. The current example policy is intentionally |
295 | | -conservative: |
296 | | - |
297 | | -- no active signal: keep baseline exposure |
298 | | -- `confidence < 0.55`: no-op |
299 | | -- `risk_off`: reduce exposure to `0.5` |
300 | | -- `mixed`: reduce exposure to `0.8` |
301 | | -- severe risk flags such as `liquidity_stress` cap exposure at `0.6` |
302 | | -- the overlay never increases exposure above the baseline |
| 60 | +Keep changes small, reproducible, and covered by the narrowest useful tests. For strategy-facing changes, include the evidence artifact or command used to validate behavior. |
303 | 61 |
|
304 | 62 | ## License |
305 | 63 |
|
306 | | -This repository is licensed under the MIT License. See [LICENSE](LICENSE). |
307 | | - |
308 | | -## Cross-Sector Theme Universe |
309 | | - |
310 | | -The repository now keeps a static, versioned cross-sector theme universe instead |
311 | | -of limiting long-horizon research to the current AI trade: |
312 | | - |
313 | | -```text |
314 | | -config/theme_taxonomy.csv |
315 | | -config/symbol_theme_exposure.csv |
316 | | -``` |
317 | | - |
318 | | -The taxonomy covers AI compute, HBM/memory, foundry policy, AI servers, |
319 | | -data-center power, cybersecurity, defense/aerospace, energy security, clean grid, |
320 | | -financial infrastructure, healthcare policy, consumer platforms, industrial |
321 | | -automation, crypto infrastructure, and EV/auto transition. |
322 | | - |
323 | | -Theme membership is research context, not a hot-list override. Monthly context |
324 | | -bundles embed `theme_context`; downstream use remains shadow-only and must replay |
325 | | -saved artifacts rather than regenerating historical AI judgments. |
326 | | - |
327 | | -## Scheduled Theme Momentum Refresh |
328 | | - |
329 | | -`.github/workflows/theme_momentum_snapshot.yml` runs weekly before the advisor |
330 | | -publication workflow. It builds `data/output/theme_momentum_snapshot.json` as |
331 | | -`medium_horizon_theme_context` for the `2-12 weeks` horizon and, |
332 | | -on scheduled runs, commits the changed snapshot back to the repository so |
333 | | -`QuantAdvisorResearch` can consume a point-in-time artifact. Manual runs can pass |
334 | | -`prices_path` to use an audited local CSV instead of Yahoo chart downloads. |
335 | | - |
336 | | -Yahoo chart downloads remain a temporary fallback only. The snapshot records |
337 | | -source metadata and file hashes when a local price CSV is used. Random free proxy |
338 | | -pools should not be used in this stable pipeline because they make replay, data |
339 | | -quality, and compliance review harder. |
340 | | - |
341 | | -## Theme Momentum Snapshot |
342 | | - |
343 | | -The static taxonomy only defines long-horizon theme membership. Ranking is done |
344 | | -by a separate theme momentum snapshot so the universe and weights are not changed |
345 | | -just because a currently popular symbol has moved. |
346 | | - |
347 | | -Build from a local price CSV: |
348 | | - |
349 | | -```bash |
350 | | -python scripts/build_theme_momentum_snapshot.py \ |
351 | | - --prices data/input/theme_price_history.csv \ |
352 | | - --symbols MU,INTC,DELL,NVDA,VRT,UNH,XOM,JPM,LMT \ |
353 | | - --output data/output/theme_momentum_snapshot.json |
354 | | -``` |
355 | | - |
356 | | -If `--prices` is omitted, the script downloads Yahoo chart data. Partial symbol |
357 | | -failures are recorded in `data_quality.missing_price_symbols` by default; |
358 | | -`--strict-downloads` turns those into hard failures. |
359 | | - |
360 | | -The snapshot records `artifact_type=medium_horizon_theme_context`, horizon metadata, fixed 12-1m, 6-1m, and 3m momentum windows, breadth, risk |
361 | | -penalties, top symbols per theme, source metadata, and a policy block that keeps |
362 | | -the artifact research-only. `data_quality.coverage` now records configured |
363 | | -symbol count, priced symbol count, price coverage ratio, and symbols with |
364 | | -insufficient price history. |
| 64 | +See [LICENSE](LICENSE) if present in this repository. |
0 commit comments