Skip to content

Commit ba1f00a

Browse files
committed
Add shadow rollout gate command
1 parent 9f9878a commit ba1f00a

8 files changed

Lines changed: 567 additions & 7 deletions

File tree

README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,20 @@ to collect a single shadow trace where production actions stay live and
379379
candidate actions are appended as `shadow_result` evidence for
380380
`tether policy diff --shadow`.
381381

382+
For the self-serve rollout decision, run the shadow gate directly:
383+
384+
```bash
385+
tether policy shadow-gate ./traces/shadow.jsonl.gz \
386+
--packet-dir /tmp/tether-shadow-rollout \
387+
--profile lab-shadow \
388+
--min-compared 100 \
389+
--wait-timeout-s 5
390+
```
391+
392+
It writes a hashed packet with `policy-diff.json` and
393+
`promotion-decision.json`, then exits `0` for `PROMOTE`, `1` for `HOLD`, and
394+
`4` for `ROLLBACK`.
395+
382396
```bash
383397
tether prove ./p0 \
384398
--embodiment franka \

docs/cli_reference.md

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -144,22 +144,35 @@ path exported by `tether profiles init`.
144144

145145
## `tether policy`
146146

147-
Policy rollout gates that operate on recorded traces. Today the main command is
148-
`policy diff`: compare a baseline trace with a candidate trace, or compare a
149-
single shadow trace's recorded production actions against shadow evidence
150-
(`shadow_result` rows, with legacy `routing.shadow_actions` still supported).
147+
Policy rollout gates that operate on recorded traces. The common self-serve
148+
path is `policy shadow-gate`: turn a recorded shadow rollout into a proof
149+
packet plus one operator decision. Use `policy diff` when you only need the raw
150+
baseline/candidate or shadow comparison report.
151151

152152
```bash
153153
# Offline promotion check: same observations, candidate policy output
154154
tether policy diff ./traces/v1.jsonl.gz ./traces/v2.jsonl.gz --fail-on any
155155

156156
# Shadow rollout check: live actions vs shadow actions in one trace
157157
tether policy diff ./traces/shadow.jsonl.gz --shadow --output policy-diff.json
158+
159+
# Self-serve shadow promotion gate
160+
tether policy shadow-gate ./traces/shadow.jsonl.gz \
161+
--packet-dir ./shadow-rollout-packet \
162+
--profile lab-shadow \
163+
--min-compared 100 \
164+
--wait-timeout-s 5
158165
```
159166

160-
The report includes action cosine/max-delta, latency regressions, shape failures,
161-
guard regressions, request mismatches, metadata warnings, and a pass/warn/fail
162-
verdict. Exit code `3` means the selected `--fail-on` gate tripped.
167+
`policy shadow-gate` writes `deployment-proof.json`, `policy-diff.json`,
168+
`promotion-decision.json`, and `MANIFEST.json`. Exit codes: `0` means
169+
`PROMOTE`, `1` means `HOLD`, `4` means `ROLLBACK`, and `2` means the trace,
170+
packet, or profile could not be loaded.
171+
172+
The underlying policy-diff report includes action cosine/max-delta, latency
173+
regressions, shape failures, guard regressions, request mismatches, metadata
174+
warnings, and a pass/warn/fail verdict. Exit code `3` from `policy diff` means
175+
the selected `--fail-on` gate tripped.
163176

164177
---
165178

docs/deploy_proof.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,25 @@ tether profiles show warehouse-safe
5252
tether profiles init warehouse-safe --output warehouse-safe.yml
5353
```
5454

55+
## Shadow rollout gate
56+
57+
For a candidate already mirrored with `tether serve --shadow-policy --record`,
58+
use `policy shadow-gate` to build the rollout packet and promotion decision in
59+
one step:
60+
61+
```bash
62+
tether policy shadow-gate ./traces/shadow.jsonl.gz \
63+
--packet-dir /tmp/tether-shadow-rollout \
64+
--profile lab-shadow \
65+
--min-compared 100 \
66+
--wait-timeout-s 5
67+
```
68+
69+
The command waits for pending `shadow_result` rows, writes `policy-diff.json`,
70+
verifies a hashed packet, and writes `promotion-decision.json`. Its top-level
71+
decision is `PROMOTE`, `HOLD`, or `ROLLBACK`; the embedded promotion report
72+
still uses `BLOCK` for a failed inactive candidate.
73+
5574
## What it checks
5675

5776
- Deploy diagnostics via the same checks as `tether doctor --json --model`.

src/tether/cli.py

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5595,6 +5595,112 @@ def policy_diff_cmd(
55955595
raise typer.Exit(3)
55965596

55975597

5598+
@policy_app.command("shadow-gate")
5599+
def policy_shadow_gate_cmd(
5600+
trace: str = typer.Argument(
5601+
...,
5602+
help="Shadow trace file recorded by `tether serve --shadow-policy --record`.",
5603+
),
5604+
packet_dir: str = typer.Option(
5605+
"./shadow-rollout-packet",
5606+
"--packet-dir",
5607+
help="Output packet directory for deployment-proof, policy-diff, and promotion decision artifacts.",
5608+
),
5609+
profile: str = typer.Option(
5610+
"lab-shadow",
5611+
"--profile",
5612+
help="Promotion profile name or YAML/JSON path. Default: lab-shadow.",
5613+
),
5614+
candidate_active: bool = typer.Option(
5615+
False,
5616+
"--candidate-active",
5617+
help="Return ROLLBACK instead of HOLD when gates fail for an active candidate.",
5618+
),
5619+
min_compared: int = typer.Option(
5620+
1,
5621+
"--min-compared",
5622+
help="Minimum compared shadow requests required before a PROMOTE decision is allowed.",
5623+
),
5624+
wait_timeout_s: float = typer.Option(
5625+
0.0,
5626+
"--wait-timeout-s",
5627+
help="Seconds to wait for pending background shadow_result rows to flush.",
5628+
),
5629+
poll_s: float = typer.Option(
5630+
0.25,
5631+
"--poll-s",
5632+
help="Polling interval while waiting for shadow_result rows.",
5633+
),
5634+
fail_on: str = typer.Option(
5635+
"any",
5636+
"--fail-on",
5637+
help="Policy diff gate: none/actions/latency/guard/shape/any.",
5638+
),
5639+
min_action_cos: float = typer.Option(
5640+
0.995,
5641+
"--min-action-cos",
5642+
help="Minimum cosine similarity before the action diff fails.",
5643+
),
5644+
max_action_delta: float = typer.Option(
5645+
0.10,
5646+
"--max-action-delta",
5647+
help="Max absolute action delta before the action diff fails.",
5648+
),
5649+
max_latency_regression_pct: float = typer.Option(
5650+
0.10,
5651+
"--max-latency-regression-pct",
5652+
help="Max shadow latency regression as a fraction, e.g. 0.10 = 10%.",
5653+
),
5654+
use_existing_packet: bool = typer.Option(
5655+
False,
5656+
"--use-existing-packet",
5657+
help="Use an existing deployment-proof.json in --packet-dir instead of writing a shadow-only proof packet.",
5658+
),
5659+
json_output: bool = typer.Option(
5660+
False,
5661+
"--json",
5662+
help="Print the full JSON report instead of a compact summary.",
5663+
),
5664+
) -> None:
5665+
"""Turn shadow trace evidence into PROMOTE, HOLD, or ROLLBACK."""
5666+
from tether.shadow_rollout import (
5667+
ShadowRolloutError,
5668+
format_shadow_rollout_human,
5669+
run_shadow_rollout_gate,
5670+
)
5671+
5672+
try:
5673+
report = run_shadow_rollout_gate(
5674+
trace=trace,
5675+
packet_dir=packet_dir,
5676+
profile=profile,
5677+
candidate_active=candidate_active,
5678+
min_compared=min_compared,
5679+
wait_timeout_s=wait_timeout_s,
5680+
poll_s=poll_s,
5681+
fail_on=fail_on,
5682+
min_action_cos=min_action_cos,
5683+
max_action_delta=max_action_delta,
5684+
max_latency_regression_pct=max_latency_regression_pct,
5685+
use_existing_packet=use_existing_packet,
5686+
)
5687+
except ShadowRolloutError as exc:
5688+
err_console.print(f"[red]Shadow rollout gate failed:[/red] {exc}")
5689+
raise typer.Exit(2)
5690+
5691+
if json_output:
5692+
typer.echo(json.dumps(report, indent=2, sort_keys=True))
5693+
else:
5694+
console.print(format_shadow_rollout_human(report), markup=False)
5695+
5696+
decision = report.get("decision")
5697+
if decision == "PROMOTE":
5698+
raise typer.Exit(0)
5699+
if decision == "ROLLBACK":
5700+
raise typer.Exit(4)
5701+
raise typer.Exit(1)
5702+
5703+
55985704
app.add_typer(models_app, name="models")
55995705
app.add_typer(train_app, name="train")
56005706
app.add_typer(validate_app, name="validate")

src/tether/promote.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@
3737
"max_policy_guard_regressions": 0,
3838
"max_policy_shape_failures": 0,
3939
"max_policy_missing_candidate": 0,
40+
"max_policy_shadow_pending": 0,
41+
"max_policy_shadow_errors": 0,
4042
"max_roundtrip_p95_ms": None,
4143
"max_warm_roundtrip_p95_ms": None,
4244
"max_deadline_misses": None,
@@ -492,6 +494,8 @@ def _evaluate_policy_diff(
492494
("max_policy_guard_regressions", "guard_regressions"),
493495
("max_policy_shape_failures", "shape_failures"),
494496
("max_policy_missing_candidate", "missing_candidate"),
497+
("max_policy_shadow_pending", "shadow_pending"),
498+
("max_policy_shadow_errors", "shadow_errors"),
495499
)
496500
for threshold_key, summary_key in threshold_map:
497501
limit = _threshold(profile, threshold_key)

0 commit comments

Comments
 (0)