Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions docs/evidence_portfolio/DEEPSEEK_PAPER_WINDOW_PREREG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
# DeepSeek sentiment-macro paper evidence window

**Status:** LOCKED — pre-registered 2026-08-31, before any post-T0 P&L review
**What this is:** a prospective paper evidence protocol for the deployed
sentiment-macro agent on funded DeepSeek.
**What this is not:** a live-promotion gate, a strategy retune, or a score of the
new window. This document does **not** authorize Deploy, provider change, or
live orders.
**Machine lock:** `research/paper_windows/deepseek-sentiment-macro-v1/lock.json`

---

> Fields below were frozen from (1) the exact deployed SHA, (2) the baked
> sentiment-macro config, (3) existing event/trade schemas, and (4) **pre-T0**
> sentiment cadence only. No post-T0 winners, losses, P&L, profit factor, or
> trade outcomes were inspected before this lock.

## Frozen identity

| Field | Value |
|-------|--------|
| Protocol ID | `deepseek-sentiment-macro-paper-window` |
| Version | `1` |
| Deployed SHA | `bc6ea9e1c62b36c82d27b96f2fb2c28d99f2f316` |
| Provider | `deepseek` |
| Model | `deepseek-v4-pro` |
| Window start (T0) | `2026-08-31T15:26:51Z` (container start) |
| Agent ID | `sentiment-macro-bot` |
| Service | `agent_sentiment_macro` |
| Config | `config/settings.sentiment_macro.yaml` |
| Event log | `data/event_log_sentiment-macro-bot.jsonl` |
| Decision-policy SHA-256 | `lock.json` `decision_policy_sha256` (canonical JSON of `decision_policy` only) |

Taxonomy and every decision-affecting field live inside `decision_policy`. Metadata
(`locked_at`, pre-T0 cadence notes) is outside the digest. Missing or extra
decision fields, or a digest mismatch, are `LockTamper`.

## Hypothesis under measurement

The deployed paper agent, answering on funded DeepSeek (`deepseek-v4-pro`), can be
observed for **operational** health after T0 without mixing pre-T0 xAI/DeepSeek
history or the three historical paper positions.

This window does **not** claim an edge and **cannot** complete a performance
verdict until a separately approved trade denominator exists. Live promotion is
prohibited.

## Pre-T0 cadence (denominator source)

Measured from `sentiment_score` events with `ts < T0` only:

| Statistic | Value |
|-----------|--------|
| Observations | 1970 |
| First / last | 2026-03-27T12:32:24Z → 2026-08-31T14:28:11Z |
| Median gap | 3605 s (~1h, matches `evaluation_interval_seconds: 3600`) |
| Rate | 12.541 obs/day |
| No-answer | 8 / 1970 = 0.406% |
| Historical paper positions | 3, all `entry_time < T0` — **excluded**, not validation |

## Horizons and operational denominator

Chosen from that pre-T0 rate, not from post-T0 results:

- **Operational review:** 14 days. Expected ~175 observations at 12.541/day.
`min_n_observations = 140` (14 × 10 obs/day floor under the measured 12.5).
- **Strategy review:** 30 days. **No approved performance trade count.**
- **Degradation (BUY-block, not health success):** rolling **10** observations,
no-answer ≥ **50%** (`SentimentScorer.degradation_error_pct`). ~10 hours at
the pre-T0 median gap. This is an emergency stop, not operational completion.
- **Window-level operational health:** `min_answered_pct = 99.1187234014`.
One-sided 99% Clopper-Pearson lower bound on frozen pre-T0 answered rate
**1962/1970 = 99.593909%**. On a 140-obs floor that requires ≥139 answered
(138/140 = 98.57% fails). Transient outages **remain in the denominator**.
Counters must be exact nonnegative integers with
`answered_n + no_answer_n == n_observations`.

If a horizon arrives under the observation minimum, the only allowed non-failure
decision is `CONTINUE_COLLECTING`.

## Performance denominator — not approved

Three pre-T0 paper positions cannot support a terminal PF/P&L sample (n=3, no
concentration/outlier distribution). The sparse-WFO engineering minimum of 4 is
**not** a prospective paper-performance denominator and is not used here.

Until a **separately approved** denominator is written into `decision_policy`
(new digest, new review):

- `STOP_PERFORMANCE_FAILURE` is unreachable
- `EVIDENCE_COMPLETE` is unreachable
- strategy horizon + operational pass → `INSUFFICIENT_EVIDENCE`
- any later performance-complete state must still require a concentration/outlier
check (`performance_denominator.concentration_check_required`)

Zero-loss profit factor is **undefined, not valid**. Infinity, NaN, None, or
`gross_loss == 0` cannot pass or fail performance.

## Inclusion — observations

A JSONL event is **in-window** iff:

- `type == sentiment_score`
- `ts >= 2026-08-31T15:26:51Z`
- `agent_id == sentiment-macro-bot` exactly (missing agent → fail closed)

A **valid DeepSeek answer** additionally requires:

- `payload.provider == deepseek`
- `payload.model == deepseek-v4-pro`
- `payload.source` in lock `valid_deepseek_sources` (`deepseek_fallback` only)

Source class uses **lock lists only** (not `src.sentiment_sources`):

- Historical answered labels: `xai_live`, `deepseek_fallback`, `zai_live`
- No-answer: `xai_error_fallback`, `error_fallback`, `neutral_fallback`
- Unknown source → no-answer
- `xai_live` / `zai_live` after T0 are **not** valid DeepSeek answers; they are
provider mismatches and terminate (`interruptions.provider_model_mismatch`)
- Missing provider or model → identity failure, fail closed

Every pre-T0 `sentiment_score` is excluded.

## Inclusion — eligible closed paper trades

The `positions` table does **not** store executor provenance. An unverified
position is **not** paper.

Include iff all of:

- `agent_id == sentiment-macro-bot` exactly (missing → ineligible)
- `entry_time >= T0`
- `status == closed` and `exit_time` present
- `executor` is not a live/Binance marker
- caller passes `paper_runtime_verified=True` from overlapping
`system_startup` / startup-diagnostics for this agent covering `entry_time`
(`executor=paper`, “Paper mode: using internal PaperExecutor”)

If provenance cannot be proven from existing event-log evidence, the row is
ineligible. This PR does not add a DB column or change runtime.

**Exclude** any position entered before T0 even if it closes afterward.
**Exclude** the three historical paper positions. **Exclude** open positions.

## Decision-state semantics

Evaluate in this order. Missing/invalid metrics never yield `EVIDENCE_COMPLETE`.

| Decision | When |
|----------|------|
| `STOP_OPERATIONAL_FAILURE` | Emergency safety, invariant break, identity failure, provider/model mismatch, config/strategy interruption, unproven paper runtime, `n >= 140` with answered% below the pre-T0 Clopper-Pearson 99% floor (99.1187234014%), all-no-answer window, 50/50 split, active rolling-10 degradation at the operational horizon, non-integer/negative counters, or missing operational aggregates at that horizon |
| `CONTINUE_COLLECTING` | Before operational pass |
| `OPERATIONAL_EVIDENCE_COMPLETE` | Operational horizon + `n >= 140` + exact nonnegative integer counts that add up + answered% ≥ 99.1187234014 + not degraded + invariants + no interruption. **Not** a performance verdict |
| `INSUFFICIENT_EVIDENCE` | Strategy horizon after operational pass while `performance_denominator.approved == false`, or required performance aggregates invalid |
| `STOP_PERFORMANCE_FAILURE` | Only if a future approved denominator exists and valid finite PF/P&L fail the frozen bars **and** concentration check passes as an input |
| `EVIDENCE_COMPLETE` | Only if that future approved denominator exists **and** operational pass **and** strategy horizon **and** valid finite aggregates **and** concentration_ok **and** no degradation/interruption. Unreachable in this lock |

`promote: false`. `live_go: false`. This window **cannot** authorize live trading.

## Safety invariants

The window is valid only while all remain true:

- `mode: paper`
- `PaperExecutor` is the order path
- no Binance order executor is constructed
- degraded sentiment blocks new BUY entries

## Interruption / censoring

| Event | Rule |
|-------|------|
| Provider, config, strategy, threshold, or sizing change | **Terminate** |
| Provider/model mismatch on in-window observations | **Terminate** |
| Restart or outage | **Record**; do not erase; outages count in the no-answer denominator |
| Operational failures | Count with the frozen lock taxonomy |
| Emergency safety action | **Annotate and terminate** |

## Scope limits

- Documentation, this lock, and reporting tests only.
- No strategy, runtime provider, config, sizing, Docker, Deploy, or migration
edits in the lock PR.
- Do not score the new window here.

## Sign-off

Parameters locked by: protocol PR against `bc6ea9e1c62b36c82d27b96f2fb2c28d99f2f316`.
Post-T0 performance was not reviewed before freeze.
1 change: 1 addition & 0 deletions docs/evidence_portfolio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ reviewed, to prevent open-ended strategy research and post-hoc rationalization.
| NFP forward gate | [NFP_FORWARD_GATE.md](NFP_FORWARD_GATE.md) | Forward-confirmation protocol bought by probe #1's YES. Signed 2026-07-21; measurement only, no build, no capital. First clean print 2026-08-07. |
| Day-30 fallback | [PORTFOLIO_KILL_GATE.md](PORTFOLIO_KILL_GATE.md) | If everything fails: no new public-data lane. Fallback = input acquisition or redeploy agent-hours. |
| Named-changed-input (post-portfolio) | [CVD_ABSORPTION_PREREG.md](CVD_ABSORPTION_PREREG.md) | Binance spot CVD absorption v1. Locked 2026-08-20 **before** develop fetch. Not probe #3 of the closed 30-day pair. `promote=no`. |
| DeepSeek paper window | [DEEPSEEK_PAPER_WINDOW_PREREG.md](DEEPSEEK_PAPER_WINDOW_PREREG.md) | Sentiment-macro paper evidence on funded DeepSeek after deploy `bc6ea9e`. T0 `2026-08-31T15:26:51Z`. Pre-T0 history excluded. Performance denominator not approved. `live_go=no`. |

## Metrics, reporting, kill gates

Expand Down
153 changes: 153 additions & 0 deletions research/paper_windows/deepseek-sentiment-macro-v1/lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
{
"decision_policy": {
"protocol_id": "deepseek-sentiment-macro-paper-window",
"version": 1,
"deployed_sha": "bc6ea9e1c62b36c82d27b96f2fb2c28d99f2f316",
"provider": "deepseek",
"model": "deepseek-v4-pro",
"window_start": "2026-08-31T15:26:51Z",
"agent_id": "sentiment-macro-bot",
"service": "agent_sentiment_macro",
"settings_path": "config/settings.sentiment_macro.yaml",
"observation_event_type": "sentiment_score",
"trade_close_event_type": "order_filled",
"review": {
"operational_horizon_days": 14,
"strategy_horizon_days": 30,
"min_n_observations": 140
},
"operational_health": {
"min_answered_pct": 99.1187234014,
"confidence_level": 0.99,
"interval": "clopper_pearson_one_sided_lower",
"pre_t0_answered_n": 1962,
"pre_t0_n": 1970,
"derivation": "Window-level health is the one-sided 99% Clopper-Pearson lower bound on the frozen pre-T0 answered rate 1962/1970 (99.593909%). Bound = 99.1187234014%. Rolling-10 50% remains the emergency BUY-block degradation rule only, not operational-health success. Transient outages stay in the denominator. Not calibrated from post-T0 data."
},
"degradation": {
"window": 10,
"no_answer_pct": 0.5,
"blocks_new_buy": true
},
"answered_sources": [
"xai_live",
"deepseek_fallback",
"zai_live"
],
"no_answer_sources": [
"xai_error_fallback",
"error_fallback",
"neutral_fallback"
],
"valid_deepseek_sources": [
"deepseek_fallback"
],
"unknown_source_is_no_answer": true,
"require_exact_agent_id": true,
"require_deepseek_provider_model": true,
"exclusions": [
"every sentiment_score with ts < window_start",
"every position with entry_time < window_start, including if closed after T0",
"open positions (status != closed or missing exit_time)",
"the three historical paper positions already in the sentiment-macro book",
"live / Binance-executor fills",
"observations missing agent_id, provider, or model",
"positions without proven PaperExecutor provenance"
],
"inclusion_observations": {
"agent_id": "sentiment-macro-bot",
"type": "sentiment_score",
"ts_gte": "window_start",
"provider": "deepseek",
"model": "deepseek-v4-pro"
},
"inclusion_trades": {
"agent_id": "sentiment-macro-bot",
"entry_time_gte": "window_start",
"status": "closed",
"exit_time_required": true,
"executor": "paper",
"paper_runtime_verified_required": true
},
"operational_metrics": [
"answered_live_pct",
"no_answer_error_pct",
"source_provider_model_attribution",
"rolling_10_degradation_state",
"alert_transitions"
],
"strategy_metrics": [
"eligible_trades",
"realized_pnl",
"profit_factor",
"win_rate",
"drawdown",
"symbol_breakdown",
"exit_reason_breakdown"
],
"safety_invariants": {
"mode": "paper",
"executor": "PaperExecutor",
"no_binance_order_executor": true,
"degraded_blocks_new_buy": true
},
"interruptions": {
"provider_config_strategy_threshold_sizing_change": "terminate",
"restart_or_outage": "record_do_not_erase",
"operational_failures": "count_per_frozen_taxonomy",
"emergency_safety_action": "annotate_and_terminate",
"provider_model_mismatch": "terminate"
},
"permitted_decisions": [
"CONTINUE_COLLECTING",
"STOP_OPERATIONAL_FAILURE",
"STOP_PERFORMANCE_FAILURE",
"INSUFFICIENT_EVIDENCE",
"OPERATIONAL_EVIDENCE_COMPLETE",
"EVIDENCE_COMPLETE"
],
"performance_denominator": {
"approved": false,
"status": "not_approved",
"concentration_check_required": true,
"reason": "Three pre-T0 paper positions cannot justify a terminal PF/P&L denominator. Sparse-WFO n=4 is an engineering gate, not a prospective paper-performance sample. STOP_PERFORMANCE_FAILURE and EVIDENCE_COMPLETE from trade performance are prohibited until a separately approved denominator exists."
},
"performance_failure_when_denominator_met": {
"realized_pnl_lte": 0,
"profit_factor_lt": 1.1,
"zero_loss_profit_factor": "undefined_not_valid",
"finite_profit_factor_required": true
},
"promote": false,
"live_go": false,
"prohibit_live_promotion": true,
"not_in_scope": [
"live promotion from this window",
"counting pre-T0 trades as validation",
"scoring the window in this lock PR",
"provider change",
"strategy/threshold/sizing edits",
"Deploy / Docker / migrations",
"terminal performance verdict without an approved denominator"
]
},
"decision_policy_sha256": "51c2f5103684a27314119b65bbe7d65b142e28ed7097fb0abca7f6597604c64e",
"metadata": {
"locked_at": "2026-08-31T16:00:00Z",
"locked_before_post_t0_performance_review": true,
"event_log": "data/event_log_sentiment-macro-bot.jsonl",
"pre_t0_cadence": {
"measured_from": "event_log sentiment_score rows with ts < window_start",
"n_observations": 1970,
"first_ts": "2026-03-27T12:32:24.727187Z",
"last_ts": "2026-08-31T14:28:11.543308Z",
"median_gap_sec": 3605,
"obs_per_day": 12.541,
"answered_n": 1962,
"no_answer_n": 8,
"historical_paper_positions_excluded": 3,
"note": "Cadence only. Pre-T0 trades are excluded from validation and were not scored for this lock."
},
"denominator_notes": "min_n_observations = 14 days * 10 obs/day floor versus pre-T0 12.541 obs/day (median gap 3605s). min_answered_pct = 99.1187234014 is the one-sided 99% Clopper-Pearson lower bound on pre-T0 1962/1970. Rolling-10 50% is degradation-only, not window health."
}
}
Loading
Loading