M4 epic: production hardening + E2E + load testing#97
Open
jean-neiverth wants to merge 42 commits into
Open
Conversation
Drops the duplicated helpers in `modules/twap-monitor` and `modules/ethflow-watcher` in favour of `shepherd_sdk::cow` / `shepherd_sdk::chain`: - `gpv2_to_order_data` (was duplicated verbatim) - `RetryAction` + `classify_api_error` + `try_decode_api_error` - `PollOutcome` enum (was duplicated verbatim — Ready boxed, TryAtEpoch / TryOnBlock / TryNextBlock / DontTryAgain) - `IConditionalOrder` sol! errors + `decode_revert` - `eth_call_params` + `parse_eth_call_result` + `decode_revert_hex` Kept module-side: - `abi::Params` + `getTradeableOrderWithSignatureCall` in twap-monitor (TWAP-specific selector source — EthFlow does not poll). - `decode_conditional_order_created` / `decode_order_placement` in their respective modules (each is bound to a specific event signature on a specific contract). - `watch:` / `next_block:` / `next_epoch:` key conventions in twap-monitor and `submitted:` / `backoff:` / `dropped:` in ethflow-watcher (per-module persistence policies, not shared). - `to_signature` (OnchainSignature → Signature) in ethflow- watcher (single consumer; will move to SDK if a second emerges). - `BuildError` / `WatchUpdate` / lifecycle plumbing in their modules (strategy-specific). LOC: -387 in twap-monitor (1058 → 671), -101 in ethflow-watcher (537 → 436). Tests: 13 host tests stay in twap-monitor (was 34 — the 21 that moved live in shepherd-sdk now), 7 stay in ethflow-watcher (was 10). .wasm size delta: - twap-monitor: 300 K → 305 K (+5 K — SDK re-exports + slight link-table growth; alloy + cowprotocol deduped). - ethflow-watcher: 272 K → 275 K (+3 K). Same wire behaviour — the SDK migration is a pure refactor; no new dispatch paths, no new key conventions.
QA pass against the team's rust-idiomatic skill ahead of M4. All
mandatory rules now hold; the cleanup is mostly mechanical with a
handful of small typing improvements where the rule asked for one
thiserror enum per error type.
Replaced every U+2014 with " - " across .rs / .toml / .md:
- 51 source-file occurrences
- 5 Cargo.toml comments
- 366 occurrences across docs/*.md (most in ADRs and the
deployment / tutorial / sdk landings)
Grep gate: `grep -rn '—' crates/ modules/ docs/` returns 0.
Added to every crate root that previously lacked it:
- crates/shepherd-sdk/src/lib.rs
- crates/shepherd-sdk-test/src/lib.rs
- modules/{example,twap-monitor,ethflow-watcher}/src/lib.rs
- modules/examples/{price-alert,balance-tracker}/src/lib.rs
`crates/nexum-engine/src/main.rs` already had it.
- shepherd-sdk dropped `serde` (only `serde_json` is actually
imported; cowprotocol re-exports carry their own serde derive
transitively).
- balance-tracker dropped its direct `alloy-primitives` dep —
now goes through `shepherd_sdk::prelude::{Address, U256,
address}`. Tests adapt.
- `shepherd_sdk::host::HostError` gains `#[derive(thiserror::
Error)]` + `#[error("{domain}: {message} (code={code},
kind={kind:?})")]`. Was a plain struct without Display.
Added `thiserror = "2"` as a dep.
- `modules/twap-monitor::BuildError`: hand-rolled Display impl
replaced with `#[derive(thiserror::Error)]` + per-variant
`#[error(...)]` + `#[from] cowprotocol::Error`. The map_err
at the call site collapses to `?`.
- `modules/ethflow-watcher::BuildError`: same conversion (4
variants, one of them `#[from]`).
Both modules add `thiserror = "2"` as a direct dep.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo test --workspace`: 121 tests pass.
- nexum-engine 41, shepherd-sdk 27, shepherd-sdk-test 8 + 1
doctest, twap-monitor 13, ethflow-watcher 7, price-alert
11, balance-tracker 13.
- `#[non_exhaustive]` is *not* applied to public enums
(`HostErrorKind`, `LogLevel`, `RetryAction`, `PollOutcome`).
The first two mirror the WIT 0.2 enums (locked at the WIT
contract layer); the last two are intentional 3- and 5-arm
contracts with no expected growth. If a future kind shows
up, the rule applies then.
- `parse_config` / `parse_settings` in the example modules
return `Result<T, String>` rather than a typed enum. The
rule's "no string-wrapping" applies to error variants that
*wrap* an upstream `std::error::Error`; one-shot config
parsers with bespoke per-field messages are pragmatic. The
error surface is internal to the module's `init` and not
part of the orderbook retry contract.
Mirrors what BLEU-851 (price-alert) and BLEU-852 (stop-loss) did for
the M3 example modules. Closes the parallel M2 gap.
Before: the entire dispatch pipeline (indexer / poll / submit /
retry / lifecycle) lived in `lib.rs` alongside the wit-bindgen
glue, calling `chain::request`, `local_store::*`, `cow_api::submit_order`,
and `logging::log` directly. The 13 existing tests covered only
parsers and encoders - the state machine itself was unverified in
unit.
After:
1. `strategy.rs` (new) - pure logic against `shepherd_sdk::host::Host`.
Defines `LogView<'a>` and `BlockInfo` so the strategy stays
wit-independent; exposes `on_logs` / `on_block` entry points.
2. `lib.rs` (rewritten, 665 -> 165 lines) - wit-bindgen `generate!`,
`WitBindgenHost` adapter implementing all four SDK host traits,
`Guest` impl that destructures `types::Event` and delegates to
`strategy`.
3. Tests against `shepherd_sdk_test::MockHost` (7 new) cover the
dispatch matrix that was previously hand-verified only:
- `index_records_new_watch_on_conditional_order_created`
- `index_overwrites_in_place_on_redelivered_log` (re-org
replay guard, BLEU-826 invariant)
- `poll_skips_when_next_block_gate_is_in_future`
- `poll_ready_submits_order_and_persists_submitted_uid`
- `submit_transient_error_leaves_state_unchanged_for_next_block`
- `submit_permanent_error_drops_watch`
- `poll_dont_try_again_drops_watch_and_gates` (uses a real
`OrderNotValid` selector via the SDK-exported sol! interface)
4. All 13 original pure tests preserved unchanged. Total: 20 tests
(was 13).
Numbers:
- `.wasm` 313,926 bytes (release wasm32-wasip2).
- 20 tests passing; clippy clean on host + wasm32-wasip2.
- 0 em-dashes in the module tree.
Stacks on PR #23 (BLEU-852) so reviewers can compare strategy /
lib.rs split side-by-side with the price-alert and stop-loss
references.
…855) Same shape as BLEU-854 (twap-monitor / PR #24). Closes the M2-side gap on ethflow-watcher. Before: `submit_placement`, `prior_outcome`, `apply_submit_retry`, and the `submitted:` / `dropped:` / `backoff:` bookkeeping called `local_store::*` and `cow_api::submit_order` directly, with all the state-machine bits unverified in unit (only 7 decoder / encoder tests). After: 1. `strategy.rs` (new) - pure logic against `shepherd_sdk::host::Host`. `LogView<'a>` keeps the strategy wit- independent; `on_logs` is the entry point. 2. `lib.rs` (rewritten, 427 -> 157 lines) - wit-bindgen `generate!`, `WitBindgenHost` adapter, `Guest` impl that destructures `types::Event::Logs` into `LogView`s and delegates to `strategy::on_logs`. 3. Tests against `shepherd_sdk_test::MockHost` (5 new) cover the dispatch + idempotency matrix: - `placement_log_submits_order_and_persists_submitted_uid` - `redelivered_placement_is_skipped_via_submitted_uid_dedup` (PR #10 / commit c5e4d7d regression guard) - `submit_transient_error_writes_backoff_marker_and_returns` - `submit_permanent_error_persists_dropped_uid_and_clears_backoff` - `eip1271_signature_shape_round_trips_through_submit_body` (decodes the JSON body MockCowApi received and asserts `signingScheme=eip1271`, signature blob verbatim, `from` = EthFlow contract) 4. All 7 original pure tests preserved unchanged. Total: 12 tests (was 7). Numbers: - `.wasm` 281,518 bytes (release wasm32-wasip2). - 12 tests passing; clippy clean on host + wasm32-wasip2. - 0 em-dashes in the module tree. Stacks on PR #24 (BLEU-854) so reviewers can compare both M2 strategy / lib.rs splits in one stack with the M3 examples.
Applies forward-compatibility hardening to the two SDK enums that mirror the WIT type system. Both `HostErrorKind` (7 variants) and `LogLevel` (5 variants) now carry `#[non_exhaustive]`. The WIT contract drives the variant set: once a new variant is added in WIT (e.g. a future `WasmTrap` HostErrorKind, or a `Critical` LogLevel), the SDK side mirrors it without breaking downstream `match` sites. `RetryAction` (3 variants) and `PollOutcome` (5 variants) stay exhaustive - they're domain-locked to the cowprotocol `OrderPostErrorKind::is_retriable()` contract and the `IConditionalOrder` Solidity interface respectively. The `#[non_exhaustive]` issue body (COW-1029 / BLEU-853) explicitly exempts them. The 4 modules that ported to the M3 Host trait pattern (price-alert, stop-loss, twap-monitor, ethflow-watcher) match on SDK `HostErrorKind` and `LogLevel` in their `sdk_err_into_wit` and `convert_level` adapters. Each gains a wildcard arm: _ => HostErrorKind::Internal // safest catch-all _ => logging::Level::Info // most neutral default balance-tracker is unaffected (it predates the host-trait refactor and matches against wit-bindgen types directly, not SDK types). The balance-tracker port is tracked as the COW-1063 QA matrix optional follow-up. Issue body originally proposed "open as tracking ticket; trigger when WIT adds the 8th HostErrorKind". Rejected for M4: production hardening means stable API contracts BEFORE shipping. Applying non_exhaustive proactively trades 10 wildcard arms (5 modules x 2 match blocks, minus 1 module that doesn't have the adapter) for forward-compat stability. - `crates/shepherd-sdk/src/host.rs`: rustdoc on each enum cites the COW-1029 decision + wildcard-arm guidance for module adapters. - `docs/adr/0009-host-trait-surface.md`: Consequences section updated. The "ADR-0009 follow-up COW-1029" stub is replaced with the applied state. - `cargo test --workspace` -> 151 host tests + 6 doctests passing (no change in count). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. - 5 wasm modules build under `wasm32-wasip2 --release`. - 0 em-dashes in changed files. Linear: COW-1029. First M4 issue landed.
…1036)
Locks the M1 fuel + memory wiring (BLEU-818) against regression with
two evil-by-design wasm fixtures and three supervisor integration
tests that exercise the full trap path: dispatch -> wasmtime trap ->
supervisor catches -> module marked dead -> engine continues.
## New fixtures
`modules/fixtures/fuel-bomb/` (66 KB wasm)
on_event runs an unbounded `wrapping_add` loop with
`std::hint::black_box` so the optimiser cannot elide it.
wasmtime exhausts the per-event DEFAULT_FUEL_PER_EVENT (1B) and
traps with OutOfFuel.
`modules/fixtures/memory-bomb/` (67 KB wasm)
on_event allocates 128 MiB which exceeds the per-store
DEFAULT_MEMORY_LIMIT (64 MiB). wasmtime rejects the memory.grow
and traps the module.
Both fixtures live under `modules/fixtures/` so they are obviously
test-only - the M2 / M3 testnet configs never reference them. Both
declare only the `logging` capability + a single block subscription.
## New supervisor integration tests
`resource_limit_fuel_bomb_traps_and_marks_module_dead`
Boots fuel-bomb alone, dispatches a block, asserts:
- dispatched == 0 (trap, not delivery)
- alive_count() == 0 (module marked dead)
- second dispatch returns 0 (dead module excluded)
-> proves the fuel limit fires + the supervisor catches the
trap without panicking.
`resource_limit_memory_bomb_traps_and_marks_module_dead`
Same shape for the 64 MiB cap. The wasm32 allocator surfaces
"memory allocation of 134217728 bytes failed" (the trap firing).
`resource_limit_dead_bomb_does_not_starve_healthy_module`
Strongest isolation test: loads fuel-bomb + the M1 example
module side-by-side, dispatches a block, asserts:
- dispatched == 1 (example survived + accepted the dispatch
even though the bomb trapped on the same block)
- alive_count() == 1 (only example alive)
- second dispatch == 1 (dead bomb skipped, example continues)
-> proves a rogue module cannot starve the supervisor or
starve sibling modules.
## Validation
- `cargo test -p nexum-engine resource_limit` -> 3 passed.
- `cargo test --workspace` -> 154 host tests + 6 doctests passing
(was 151 + 6; +3 from the new tests).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release -p {fuel-bomb,memory-bomb}`
clean.
- 0 em-dashes in new files.
## Out of scope
- Fuel + memory limits made configurable per-module via `engine.toml`
(today they are workspace constants in `runtime/limits.rs`).
Already noted in the source comments as "configurable in 0.3";
acknowledged not addressed here.
- Adversarial fuzz of the resource-limit defaults under sustained
load. That is COW-1065 (security review) territory.
- CI integration of the fixtures into the build matrix (PR #27).
Not needed - `cargo test --workspace` already builds them in the
test profile, and the `module_wasm_or_skip` guard means CI does
not need a separate fixture-build job.
Linear: COW-1036. Second M4 issue landed; stacks on #35 (COW-1029).
Closes the COW-1035 structured-logging audit. The engine now ships
JSON-formatted logs by default, with consistent field shapes across
every dispatch, host call, and order submission. A single `jq` /
Loki / Grafana stream over the engine output reconstructs the full
timeline of any module event.
- New `--pretty-logs` CLI flag (parsed in `cli.rs`). When set, the
engine uses the historical 0.1 human-readable formatter; otherwise
emits JSON with flattened event fields + no current-span noise.
- `main.rs` `tracing_subscriber::fmt` builder picks the formatter from
the flag. `EnvFilter` (`RUST_LOG` / `engine.toml::[engine].log_level`)
applies to both.
- `tracing-subscriber` feature set gains `json`.
Every `dispatch_block` and `dispatch_log` invocation now emits a
structured log line:
- `dispatch ok` (DEBUG) on success with `module`, `chain_id`,
`event_kind` ("block" / "log"), `block_number`, `latency_ms`.
- Existing host-error WARN + trap ERROR paths gain the same fields
for cross-correlation.
Live Sepolia validation:
```json
{"timestamp":"2026-06-18T13:56:48.587125Z","level":"DEBUG","message":"dispatch ok",
"module":"price-alert","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":134,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:48.857911Z","level":"DEBUG","message":"dispatch ok",
"module":"balance-tracker","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":270,"target":"nexum_engine::supervisor"}
{"timestamp":"2026-06-18T13:56:50.193531Z","level":"DEBUG","message":"dispatch ok",
"module":"stop-loss","chain_id":11155111,"event_kind":"block",
"block_number":11087508,"latency_ms":1335,"target":"nexum_engine::supervisor"}
```
The latency-per-module distribution is now observable per-block:
price-alert ~135 ms (1 eth_call), balance-tracker ~270 ms (2
eth_getBalance), stop-loss ~1.3 s (oracle read + OrderCreation +
cow-api submit + retry classify).
The M1 host backends already emit structured DEBUG with `chain_id`,
`method`, `bytes`, `latency_ms` on every `chain::request` /
`cow-api::submit-order` / `cow-api::request`. The audit confirmed
they already satisfy the COW-1035 contract; no changes needed.
`just run-m2` and `just run-m3` pass `--pretty-logs` so the
runbook output samples (M2 + M3 runbooks) keep matching what the
operator sees locally. Production deploys (`cargo run -p
nexum-engine -- --engine-config engine.toml` directly, e.g. from
a Docker entrypoint) get JSON by default.
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: JSON output captured; per-module dispatch
latencies plausible against expected work per module.
- Pretty-logs flag verified to opt back into the 0.1 formatter.
- Per-order timeline aggregation (would need a `uid` field on the
stop-loss / twap-monitor submit logs). Currently the order UID
is logged by each module's `host.log` call (module-side) but
not joined with the supervisor dispatch line. Acceptable today;
the JSON shape supports adding the field later without breaking
consumers.
- Trace IDs / spans across the dispatch -> host-call boundary.
Worth a follow-up once Prometheus (COW-1034) lands and we know
the metric labels we want to correlate against.
Linear: COW-1035. Third M4 issue landed; stacks on #36 (COW-1036).
…1034)
Wires `metrics` + `metrics-exporter-prometheus` into the engine.
When `engine.toml::[engine.metrics].enabled = true` the engine binds
a Prometheus HTTP exporter on the configured `bind_addr` (default
`127.0.0.1:9100`) and serves `/metrics`. When disabled, the recorder
is still installed so call sites stay live but no port binds. The
same binary now ships into CI / tests (recorder no-ops) and into
production (full exporter) by flipping one config flag.
## Recording sites instrumented
| Metric | Type | Labels | Site |
|---|---|---|---|
| `shepherd_event_latency_seconds` | summary | module, event_kind | supervisor::dispatch_{block,log} on the OK path |
| `shepherd_module_errors_total` | counter | module, error_kind | supervisor::dispatch_{block,log} on host-error WARN + trap ERROR paths (error_kind = `{:?}` of HostErrorKind, or `"trap"`) |
| `shepherd_chain_request_total` | counter | chain_id, method, outcome | host::impls::chain after every `chain::request` |
| `shepherd_cow_api_submit_total` | counter | chain_id, outcome | host::impls::cow_api after every `cow-api::submit-order` |
Sites match the structured logging audit (COW-1035) so each metric
event has a sibling log line with the same labels for cross-
correlation.
## Live Sepolia scrape
```
# TYPE shepherd_cow_api_submit_total counter
shepherd_cow_api_submit_total{chain_id="11155111",outcome="err"} 2
# TYPE shepherd_chain_request_total counter
shepherd_chain_request_total{chain_id="11155111",method="eth_getBalance",outcome="ok"} 4
shepherd_chain_request_total{chain_id="11155111",method="eth_call",outcome="ok"} 4
# TYPE shepherd_event_latency_seconds summary
shepherd_event_latency_seconds{module="price-alert", event_kind="block",quantile="0.5"} 0.1394
shepherd_event_latency_seconds{module="stop-loss", event_kind="block",quantile="0.5"} 0.9091
shepherd_event_latency_seconds{module="balance-tracker",event_kind="block",quantile="0.5"} 0.2823
```
Quantiles (p50/p90/p95/p99) appear automatically via the default
reservoir sampling. Production SRE can write SLO alerts against
these without any further wiring.
## Config schema
```toml
[engine.metrics]
enabled = false # default
bind_addr = "127.0.0.1:9100"
```
Disabled by default so M3 runbook smoke runs do not bind a port
unintentionally. Operators flip `enabled = true` for production.
## Deferred from issue scope
| Metric | Why deferred |
|---|---|
| `shepherd_module_uptime_seconds` | Needs a per-module start-time tracker on LoadedModule. Easy to add but cluttered the supervisor.rs diff. Worth a follow-up. |
| `shepherd_fuel_consumed` | Requires reading `store.get_fuel()` after each dispatch to compute consumed = DEFAULT - remaining. Mechanical addition, not in this PR to keep scope tight. |
| `shepherd_memory_peak_bytes` | Requires inspecting the wasmtime component's exported Memory instance. Harder; defer until the memory-bomb fixture surfaces a useful baseline. |
| `shepherd_module_restarts_total` | Depends on COW-1033 (supervisor auto-restart) which is not built yet. Wired into restart sites when that lands. |
| `shepherd_module_poisoned` | Depends on COW-1032 (poison pill) for the same reason. |
The 4 metrics shipped here cover the load-bearing observability use
cases (dispatch latency SLO + error rate by kind + per-RPC method
volume + per-orderbook submit outcome). The 5 deferred ones lock in
on top of M4 work that isn't built yet.
## Validation
- `cargo test --workspace` -> 154 host tests + 6 doctests passing.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: enabled the exporter on port 9101, scraped
/metrics via `curl`, confirmed all 4 metrics flowing with correct
labels.
- Recorder no-op path verified: `[engine.metrics].enabled = false`
(the default) does not bind a port; call sites stay silent but
do not panic.
Linear: COW-1034. Fourth M4 issue landed; stacks on #37 (COW-1035).
…tiation (COW-1033)
When a module traps in `on_event` (OutOfFuel, MemoryOutOfBounds,
unhandled host error), the supervisor now:
1. Marks the module `alive = false` and increments `failure_count`.
2. Schedules a `next_attempt` instant via the new
`runtime::restart_policy::backoff_for` (1s → 2s → 4s → ... cap
5 min). All dispatches before that instant skip the module.
3. On the first dispatch past the backoff window, the supervisor
tears down the trapped wasmtime Store + component instance and
re-instantiates from the cached `Component`. The instance state
resets but host-side persistent state (local-store) survives
so a module's progress counters live across restarts.
4. On a successful `on_event` after recovery, `failure_count` resets
to 0 + `next_attempt = None`.
A wasmtime trap leaves the component instance poisoned: subsequent
`call_on_event` returns "wasm trap: cannot enter component instance".
Just refueling the Store does not recover. The supervisor caches
the `Component`, `init_config`, and `http_allowlist` on
`LoadedModule` at boot so a restart only needs a fresh Store +
re-instantiation - the compiled component bytes are reused.
- `crates/nexum-engine/src/runtime/restart_policy.rs`: `backoff_for(failure_count) -> Duration` with the 1s → 5min schedule. 4 unit tests covering the steady-state, first-failure, doubling, and cap arms.
- `Supervisor` gains four cached backends (`engine`, `cow_pool`, `provider_pool`, `local_store`) so `reinstantiate_one(idx)` can rebuild the wasi Linker + HostState + Store + bindings on demand.
- `LoadedModule` gains `component: Component`, `init_config: Config`, `http_allowlist: Vec<String>` (all cloned at boot), plus `failure_count: u32` and `next_attempt: Option<Instant>` for the schedule.
`dispatch_block` and `dispatch_log` now restructure into two
phases:
1. **Phase 1 (restart sweep)**: walk modules, collect indices of
dead-but-due modules, call `reinstantiate_one` on each. Failed
restarts bump the backoff again. Successful restarts flip
`alive = true` so phase 2 dispatches the next event to them.
2. **Phase 2 (steady-state dispatch)**: unchanged from before -
walk modules, dispatch where subscribed + alive. Trap path
sets `next_attempt` + bumps `failure_count`; success path
resets both.
The structured logs from COW-1035 gain `failure_count` + `backoff_ms`
on trap + `restart attempt` info lines on each restart. The
`shepherd_module_restarts_total{module}` Prometheus counter from
COW-1034 increments on every restart attempt.
`modules/fixtures/flaky-bomb/` (test-only): traps via OutOfFuel on
the first N events (N from `[config].fail_first_n`) and recovers
afterwards. Uses local-store for the attempt counter because the
wasm instance state resets on each reinstantiation; the counter
persists in the host-side store so the module deterministically
recovers after the configured N.
`supervisor::tests::restart_flaky_module_recovers_after_backoff`
(new): boots flaky-bomb with fail_first_n=1, dispatches, observes:
- Dispatch 1: trap. alive=false, failure_count=1, next_attempt=+1s.
- Immediate redispatch: skipped (still in backoff).
- Sleep 1.1s.
- Dispatch 3: restart fires, fresh instance attempts again. With
attempt=2 > N=1, returns Ok. alive=true, failure_count=0,
next_attempt=None.
- Dispatch 4: steady-state, dispatches normally.
Test wall-clock ~1.4s.
- `cargo test --workspace` -> 159 host tests + 6 doctests passing.
+4 from `restart_policy` unit tests + 1 from the new integration
test (was 154 + 6).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing resource-limit tests (COW-1036) still pass against
the new dispatch shape: their assertions are against state
*immediately* after the trap (before backoff elapses), so the
restart machinery is transparent.
- The `init_failure_marks_module_dead_and_excludes_from_dispatch`
test (COW-1070) still passes: init-failed modules carry
`next_attempt = None` so the restart sweep never picks them up.
- Persistence of `failure_count` / `next_attempt` across full
engine restarts. The schedule resets on every boot; cross-engine
persistence is a 0.3 follow-up.
- WS reconnect-with-backoff for upstream RPC drops - that is
COW-1071, a separate axis.
- Operator-tunable backoff via `engine.toml::[engine.restart]`.
The current constants are workspace literals in
`runtime::restart_policy`; configurable in 0.3.
- Module-side `on_restart` hook. Modules just see a fresh `init`
call after a restart, same as boot.
Linear: COW-1033. Fifth M4 issue landed; stacks on #38 (COW-1034).
…OW-1071)
Replaces the previous "bail on WS drop" semantic (flagged as the
"0.3 fix" in the source) with per-stream reconnect-aware tasks. Each
chain's block subscription and each (module, chain) log subscription
gets a dedicated task that:
1. Opens the subscription via `ProviderPool`.
2. Pumps items to an mpsc channel until the underlying stream
yields `None` (WS drop) or `Err` (transport-level error).
3. Logs the drop + sleeps for `restart_policy::backoff_for(attempt)`
(1s -> 2s -> 4s -> ... cap 5 min, reusing the COW-1033 policy).
4. Reopens. The first event after a reopen emits an `INFO ...
reopened` line + increments `shepherd_stream_reconnects_total`.
5. Resets `attempt = 0` once the stream has been healthy for the
`HEALTHY_WINDOW` (60 s of uninterrupted events) so a flaky-but-
then-stable connection reverts to fast retries on the next drop.
The event loop reads the channel as a regular `Stream` (wrapped
with `futures::stream::unfold` to avoid pulling in `tokio-stream`
just for `ReceiverStream`). A bare `None` from the merged stream
now indicates the reconnect task itself exited (panic or channel
closed); that path still bails the engine as before, but the log
message updated to reflect the new semantic.
## Key behavioural change
Public Sepolia (`wss://ethereum-sepolia-rpc.publicnode.com`) drops
WS connections after ~20 min of sustained load. Pre-fix: engine
bailed within seconds of the first drop, an operator restart
re-opened the subscription but the engine had missed every event in
between. Post-fix: the reconnect task waits 1s and reopens; only
events that arrived during the 1s gap are missed. Multi-minute drops
get progressively longer waits, capped at 5 min.
## New metric (consumed via COW-1034)
`shepherd_stream_reconnects_total{kind, chain_id, module}` counter,
incremented on every successful reopen. Operators write SLO alerts
against this for "stream churn" (e.g. > 5 reconnects per 10 min on
the same chain).
## Channel buffer + back-pressure
Buffer is 64 events per task. Real-time dispatch usually drains in
~12 s (Sepolia block time) so the buffer is overkill for normal
operation; it absorbs a brief dispatch-side stall (e.g. a stop-loss
cow-api submit that takes 2 s) without dropping events at the WS
boundary.
## Tests
- `cargo test --workspace` -> 159 host tests + 6 doctests passing
(unchanged shape - all existing tests still pass, including the
`run_does_not_bail_when_both_stream_kinds_are_empty` regression
guard which verifies the empty-stream path).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia happy path: `just run-m3` boots, all 3 modules
dispatch normally, `subscription open` log line emitted, no
reconnect activity in 60 s window (network was stable). Clean
SIGTERM shutdown.
## Out of scope
- WS endpoint failover (swap Alchemy <-> publicnode on failure).
Operator concern; track via `[engine.chains.<id>]` schema if
demand arises.
- Backfill of events missed during the drop window. Live-stream
semantic only; backfill is an indexer concern outside the M4
engine scope.
- Operator-tunable backoff / healthy-window via `engine.toml`. The
current constants are workspace literals; configurable in 0.3.
- Per-chain isolation across reconnects (COW-1073). The current
patch already gives partial isolation: each chain's task drops
+ reconnects independently and one task's failure does not
starve the others. COW-1073 covers the supervisor-side
multi-chain coordination.
Linear: COW-1071. Sixth M4 issue landed; stacks on #39 (COW-1033).
Escalates the COW-1033 restart policy: when a module traps more
than `PoisonPolicy.max_failures` times within a sliding
`PoisonPolicy.window`, the supervisor marks it **poisoned**:
- Dispatch path skips poisoned modules forever (no further restart
attempts, no fuel + RPC cost on no-ops).
- A WARN log emits the module name + last error class with a hint
to remove it from `engine.toml::[[modules]]` + restart.
- `shepherd_module_poisoned{module}` gauge flips to 1.
Production thresholds: 5 traps inside 10 minutes -> quarantine.
Aggressive enough to catch a deterministically broken module
without burning every restart slot from the COW-1033 backoff
schedule; lenient enough that a one-off RPC blip during a real
cow-api submit does not get a module quarantined.
Recovery requires an operator action: remove the entry from
`engine.toml::[[modules]]` + restart the engine. There is no
automatic recovery on the production schedule; the assumption is
that 5 traps inside 10 min is a structural failure, not a
transient that would self-heal.
## New file
`crates/nexum-engine/src/runtime/poison_policy.rs`:
- `POISON_MAX_FAILURES = 5`, `POISON_WINDOW = 600 s` consts.
- `PoisonPolicy { max_failures, window }` struct with `Default`
pointing at production + `::new(...)` for tests.
- `should_poison(policy, recent_failures) -> bool` helper.
- 2 unit tests covering the threshold edge cases.
## supervisor.rs changes
- `Supervisor` gains `poison_policy: PoisonPolicy` (defaults to
production; tests override via `with_poison_policy`).
- `LoadedModule` gains `failure_timestamps: VecDeque<Instant>` +
`poisoned: bool`.
- New free-function `record_failure_and_maybe_poison` is called
from every trap arm in `dispatch_block` + `dispatch_log`. It
prunes old entries beyond the window, pushes the current
timestamp, and flips `poisoned = true` if the window holds
>= `policy.max_failures` entries.
- Restart sweep + dispatch fast-path both check `poisoned` first,
excluding quarantined modules from any further work.
- New `poisoned_count()` accessor for metrics + tests.
## New integration test
`poison_pill_quarantines_module_after_threshold` (real-time,
~3.5 s wall clock):
1. Boot fuel-bomb (always-trapping fixture from COW-1036) with a
tight policy: `PoisonPolicy::new(3, Duration::from_secs(60))`.
2. Dispatch 1 -> trap. failure_count=1, next_attempt=+1s, poisoned=0.
3. Sleep 1.1s, dispatch 2 -> trap. failure_count=2, poisoned=0.
4. Sleep 2.1s, dispatch 3 -> trap. failure_count=3. **3 failures
inside the 60-s window crosses the threshold -> poisoned=1.**
5. Dispatch 4 (no wait) -> returns 0, no restart attempt, no
dispatch entered. The module is silently excluded.
## Workspace impact
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(was 159 + 6; +2 from `poison_policy` units + 1 from the
integration test).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- All existing tests pass against the new dispatch shape: the
`restart_flaky_module_recovers_after_backoff` test (COW-1033)
uses fail_first_n=1 with the default production policy, so the
module recovers well before the 5-trap threshold.
- `resource_limit_dead_bomb_does_not_starve_healthy_module`
(COW-1036) dispatches the bomb twice; both with the default
policy, well under 5 traps -> no quarantine.
## Out of scope
- Operator-tunable thresholds via `engine.toml::[engine.poison]`.
The current constants live in `runtime::poison_policy`;
configurable in 0.3.
- Auto-recovery via slow decay (e.g. "after 1 h of being
poisoned, try one more time"). The spec is explicit: poisoned
modules need operator action.
- Per-module poison policies. One workspace-wide threshold today.
Linear: COW-1032. Seventh M4 issue landed; stacks on #40 (COW-1071).
…ce (COW-1072)
Two coupled changes that make operator-driven shutdowns observable
and recoverable:
## 1. Event loop: dispatch outside `select!`
`run()` previously had its `call_on_event().await` inside the
`tokio::select!`. A shutdown signal arriving mid-dispatch
cancelled the in-flight wasmtime call, leaving the wasm store in
an indeterminate state.
The refactor splits the loop into two phases:
- **Phase 1**: a small `tokio::select!` picks the next event OR
observes shutdown OR reports an upstream-task panic. Each branch
resolves into a `NextEvent` value; the select drops without
cancelling anything *outside* itself.
- **Phase 2**: `match next` dispatches the event to the supervisor
via a fully-awaited call, OR exits cleanly on the shutdown
variant.
The shutdown signal is now only observed *between* dispatches.
In-flight wasmtime calls always finish naturally.
## 2. Per-module last-dispatched-block persistence
Every successful `dispatch_block` writes a host-side marker to the
module's own local-store namespace:
```
namespace = module.name
key = "last_dispatched_block:{chain_id}"
value = block.number.to_le_bytes()
```
The marker survives engine restarts (it lives in the redb file
under `state_dir`). Operators can confirm at-which-block an engine
last ran without trawling the logs; modules that care about block-
gap detection can read it back on their next `init`.
Write failures are best-effort (a `WARN` log; the dispatch is not
considered failed).
## 3. Graceful shutdown log
The event loop now emits a structured exit line:
```
INFO graceful shutdown complete
dispatched_blocks=N dispatched_logs=M uptime_secs=K
```
Visible live on Sepolia after a `kill -TERM`:
```
INFO shutdown signal received signal=SIGTERM
INFO graceful shutdown complete dispatched_blocks=1 dispatched_logs=0 uptime_secs=13
```
## Out of scope
- 30s drain timeout via `tokio::time::timeout`. The current
dispatch path always terminates (fuel cap caps wall time to <1s
in practice); a 30s drain timer is dead code today. Worth
adding once a module ever needs longer-running host calls (HTTP
capability, etc).
- `engine.toml::[engine.shutdown]` config knobs. The internal
default is "wait as long as the in-flight dispatch takes";
configurable in 0.3.
- Module-side `shutdown` hook. Modules just see the dispatch
complete normally; the supervisor exits without invoking
anything new.
## Tests
- `cargo test --workspace` -> 161 host tests + 6 doctests passing
(unchanged shape; the dispatch refactor is transparent to the
existing test suite).
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo fmt --all --check` clean.
- Live Sepolia smoke: ran the engine, observed the graceful
shutdown log + last_dispatched_block markers in
`data/m3/local-store.redb`.
Linear: COW-1072. Eighth M4 issue landed; stacks on #41 (COW-1032).
The supervisor's dispatch path is per-chain by construction (`dispatch_block(block)` filters modules by `block.chain_id` matching their `[[subscription]]` table), and the COW-1071 WS reconnect tasks own one per-chain backoff timer each. Multi-chain isolation is therefore structural, not derived. This PR locks the guarantee into the test suite with two new integration tests + a supervisor.rs docstring stating the invariant explicitly. ## New tests `multi_chain_dispatch_isolates_modules_by_chain`: - Boot two `example` modules with different `[[subscription]]` chain_ids (1 + 100). - Dispatch a block on chain 1 -> only module-a receives it (dispatched=1, alive_count=2 unchanged). - Dispatch a block on chain 100 -> only module-b receives it. - Validates: subscription filter is per-chain; a block on one chain does not even enter modules subscribed to a different chain. `multi_chain_poisoned_module_does_not_affect_other_chains`: - Boot fuel-bomb (always-traps) on chain 1 + example (healthy) on chain 100, with `PoisonPolicy::new(2, 60s)`. - Trap bomb #1 on chain 1 -> bomb dies, poisoned=0, example untouched. - Dispatch on chain 100 -> example receives (1/1). - Wait 1.1 s (bomb backoff window), trap bomb #2 -> poisoned=1. - Dispatch on chain 100 again -> example STILL receives. - Validates: a permanently-poisoned module on one chain does not consume restart slots, fuel, or scheduling attention from modules on any other chain. Total wall-clock ~1.2 s for the second test (one backoff window). ## supervisor.rs docstring The module-level comment now articulates the multi-chain isolation invariant explicitly so a future reader of the dispatch path knows the property is load-bearing. ## What this proves Supervisor side (dispatch fast-path): - Per-module `alive`, `failure_count`, `next_attempt`, `poisoned` are independent of which chain triggered the event. - Subscription filter excludes mismatched modules before any dispatch / restart logic runs. Upstream side (already proven by COW-1071's architecture): - `open_block_streams` spawns one task per chain; tasks share no state. A chain-A WS drop changes only chain-A's task state. - `open_log_streams` is per-(module, chain) -> even tighter isolation than block streams. ## Out of scope - A unit test that "fakes a WS drop" on chain A while chain B keeps yielding. Requires mocking `ProviderPool::subscribe_blocks` which today goes through real alloy / tokio infrastructure. The COW-1064 (E2E 4-6h testnet) and COW-1031 (7-day soak) will exercise this path against live RPCs. - Per-chain configurable backoff / health-window. Today the reconnect policy is workspace-wide; per-chain tuning is a 0.3 follow-up. ## Workspace impact - `cargo test --workspace` -> 163 host tests + 6 doctests passing (was 161 + 6; +2 from the new integration tests). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. Linear: COW-1073. Ninth M4 issue landed; stacks on #42 (COW-1072).
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This is the integration step between unit-test
coverage (MockHost, per-module strategy tests) and the COW-1031
7-day soak; the soak validates stability, this validates
correctness in a live dispatch context.
## What this PR ships (scaffold, not run)
- `engine.e2e.toml` — unified Sepolia config loading all 5
modules (twap-monitor + ethflow-watcher + price-alert +
balance-tracker + stop-loss), separate `state_dir = ./data/e2e`,
Prometheus `/metrics` enabled on 127.0.0.1:9100. Operator
swaps in their Alchemy/Infura WS URL before launching the run.
- `justfile` targets `build-e2e` + `run-e2e`. `build-e2e`
reuses `build-m2 + build-m3` so the 5 wasm artefacts are
produced in one go; `run-e2e` boots the engine pointed at
`engine.e2e.toml` (no `--pretty-logs` so production-shape
JSON logs are emitted, ready for jq mining).
- `docs/operations/e2e-testnet-runbook.md` — full operator
runbook mirroring the M2 + M3 shape. Sections cover RPC
selection, on-chain prep (test EOA + Safe + stop-loss
pre-sign), boot sequence + expected log shape, the three
on-chain triggers that satisfy the per-module terminal-state
markers, metrics capture, red flags to watch, and report
filing. Acceptance bar from COW-1064 reproduced verbatim.
- `docs/operations/e2e-reports/e2e-report.template.md` — empty
report skeleton operator copies to
`e2e-report-YYYY-MM-DD.md` at the start of each run and
fills in as the run progresses. Sections: run metadata,
chain coverage, on-chain actions submitted, per-module
terminal-state markers, error-count deltas from Prometheus,
anomalies, acceptance checklist, sign-off.
## What this PR explicitly does NOT do
The 4-6 h run itself is operator-driven and cannot be
exercised from CI:
1. Real Sepolia RPC keys (rate-limited public node will not
survive a multi-hour run with 4+ eth_call per block).
2. Funded test EOA + ComposableCoW Safe access to submit a
real conditional order (twap-monitor's only path to a
`submitted:` marker).
3. EthFlow swap from a real EOA on Sepolia
(ethflow-watcher's only path to a `submitted:` marker).
4. `setPreSignature` + sell-token allowance from the stop-loss
`owner` EOA (stop-loss's only path to a `submitted:`
marker that is not a typed `TransferSimulationFailed` warn).
5. 4-6 h wall clock + metrics-start.txt / metrics-end.txt
capture.
The runbook is unambiguous about what each step requires; the
report template's section 8 is the gating sign-off for
COW-1031 (7-day soak).
## Smoke-validation done before commit
Booted `engine.e2e.toml` end-to-end against live Sepolia for
60+ s (kill -INT-style early shutdown):
```
INFO supervisor ready modules=5 chains=1
INFO log subscription open module=twap-monitor chain_id=11155111
INFO block subscription open chain_id=11155111
INFO log subscription open module=ethflow-watcher chain_id=11155111
DEBUG dispatch ok module=twap-monitor block_number=11088259 latency_ms=1
WARN price-alert: TRIGGERED answer=168110190000 threshold=250000000000 (Below)
DEBUG dispatch ok module=balance-tracker block_number=11088259 latency_ms=271
WARN stop-loss retry on next block (0): orderbook error
(TransferSimulationFailed): sell token cannot be transferred
DEBUG dispatch ok module=stop-loss block_number=11088259 latency_ms=1802
```
This proves: 5/5 modules init successfully, both log
subscriptions + the block subscription open, the dispatch loop
ticks against real Sepolia blocks, every module that has a
block subscription dispatches on every block, and the real
RPC + Chainlink decode + cow-api submit path is exercised
inside seconds. The remaining acceptance bar (terminal
markers on twap-monitor + ethflow-watcher, 1500-block run,
0-ERROR supervisor log) only the operator can produce.
## Workspace impact
- No production-code changes (this is pure ops scaffolding).
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release` produces all
5 module artefacts named exactly as `engine.e2e.toml`
references them (twap_monitor.wasm, ethflow_watcher.wasm,
price_alert.wasm, balance_tracker.wasm, stop_loss.wasm).
Linear: COW-1064. Tenth M4 issue landed; stacks on #43 (COW-1073).
Adds `docs/production.md` — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to `docs/06-production-hardening.md`: the existing
file is the architecture / design rationale (resource model,
restart policy, RPC resilience, logging + metrics design); this
new one is the concrete operator handbook (unit files, backup
recipes, alert rules, runbook procedures). Cross-referenced
both ways.
## Sections
1. **Pre-flight checklist** — every box you need ticked before
the first start: release-mode binary, persistent state dir,
metrics on loopback, paid RPC, Prometheus + log pipeline,
on-call runbook reference.
2. **systemd unit** — full `/etc/systemd/system/shepherd.service`
with: dedicated `shepherd` user, SIGINT for graceful
shutdown (30 s timeout — covers the COW-1072 last-block
persistence path), `NoNewPrivileges`/`ProtectSystem=strict`,
2 G memory cap (defence in depth on top of wasmtime's 64
MiB / module), restart-on-failure with 5 s backoff. Install
recipe + journalctl tail snippet.
3. **Docker Compose** — interim Dockerfile (multi-stage Rust
build + Debian slim runtime, non-root, EXPOSE 9100,
tini PID 1) + compose stack with bundled Prometheus, host
loopback port mapping only, `stop_signal: SIGINT`,
`stop_grace_period: 30s`, /metrics-based healthcheck.
Marked interim because the official Dockerfile is a
separate tracking issue.
4. **redb backup** — three operationally-supported paths:
cold backup (systemctl stop + cp + start; byte-identical
on graceful shutdown), hot backup (SIGSTOP + cp + SIGCONT
pattern, safe because the on-disk format is consistent at
any commit boundary), restore + `Database::check_integrity()`.
Honest about the redb 2.6 surface — no in-process snapshot
API today; flagged the roadmap. Retention policy:
7 daily / 4 weekly / 12 monthly = ~2.3 GiB on a 100 MiB
store.
5. **Logs** — JSON shape on stdout, two-tier retention model
(7 d hot full debug, 90 d cold INFO-only on S3 Glacier),
Vector config sample for journald -> Loki/hot +
journald -> S3/cold split. Sizing estimate based on the
E2E run shape (5 modules × 1 dispatch/12 s ≈ 200 MiB/wk
INFO+DEBUG combined).
6. **RPC selection** — provider plan recommendations
(Alchemy/Infura/QuickNode tiers), capacity sizing per
chain (1 block sub + N log subs + M `eth_call`/block where
M grows with TWAP active orders), why public nodes are
non-starters.
7. **Metrics + scraping** — complete metric surface table
from `grep metrics::counter/histogram/gauge`:
`shepherd_event_latency_seconds`,
`shepherd_module_errors_total`,
`shepherd_module_restarts_total`,
`shepherd_module_poisoned`,
`shepherd_chain_request_total`,
`shepherd_cow_api_submit_total`,
`shepherd_stream_reconnects_total`. Label set verified
against the source. 15 s scrape interval recommended.
8. **Workload-class tuning** — light indexer / TWAP-style
polling / multi-chain swarm classes with concrete
fuel + memory numbers. Honest that the limits are
compile-time constants today and per-module overrides via
`[engine.limits]` are a 0.3 follow-up — operator can
change `runtime/limits.rs` constants and rebuild, or
ensure load fits within defaults.
9. **Alert rules** — full `prometheus-rules.yml` covering
the seven alerts that map to the metric surface:
- `ShepherdModulePoisoned` (page) — production module
quarantined, needs operator action.
- `ShepherdModuleTraps` (ticket) — pre-poison signal.
- `ShepherdRpcErrorRate` (ticket) — > 5% RPC errors.
- `ShepherdReconnectStorm` (ticket) — WS flapping.
- `ShepherdCowApiErrorRate` (ticket) — > 20% orderbook
errors over 15 min.
- `ShepherdDispatchLatency` (ticket) — p95 > 5 s
sustained.
- `ShepherdDown` (page) — engine absent for 2 min.
10. **Operational runbook** — five common tasks:
tail-per-module, reset poisoned module, add module to
running deploy, inspect local-store, bump log level.
Each task carries a concrete shell snippet.
11. **Pre-upgrade checklist** — CHANGELOG read, cold backup,
stage binary, validate `supervisor ready modules=N
chains=M`, swap binary, restart, watch 5 min.
12. **References** — links to the architecture doc, ADRs,
runbooks, sister Linear issues.
## Drive-by fix
The COW-1064 e2e report template (committed in PR #44) had two
metric-label mistakes I caught while writing the metric
surface table in section 7:
- `result="ok|err"` -> `outcome="ok|err"` (the actual label
name in `cow_api.rs` + `chain.rs`).
- `reason="trap"` -> `error_kind="trap"` (the actual label
name in `supervisor.rs`).
Both labels appear in 4 places in the template (section 5
metric delta table + section 7 acceptance checklist). Fixed
in-place rather than as a separate PR.
## Workspace impact
- No code changes.
- No new build dependencies.
- `cargo fmt --all --check` clean.
Linear: COW-1030. Eleventh M4 issue landed; stacks on #44
(COW-1064).
…-1064 dry run
Reconfigures the M3 example modules' manifests to the pinned
identities for the 2026-06-18 COW-1064 E2E dry run (Bruno's test
EOA + Safe on Sepolia) and adds a `docs/operations/e2e-cow-1064-
prep.md` companion to the runbook that captures every
copy-paste-able value the operator needs to drive the on-chain
side of the run without re-deriving any UID, address, or
calldata.
## Module config pinning
`modules/examples/stop-loss/module.toml`:
- owner -> 0x7bF140727D27ea64b607E042f1225680B40ECa6A (test EOA)
- sell_token -> WETH9 Sepolia (was a mainnet KNC address — bug
that would have failed the orderbook accept regardless)
- buy_token -> COW Sepolia (verified on-chain: name="CoW
Protocol Token", symbol="COW", decimals=18)
- sell_amount -> 0.005 WETH (fits 0.01 WETH wrap budget)
- buy_amount -> 20 COW (conservative quote)
- trigger_price -> $2000 (above the Sepolia Chainlink mocked
answer ~$1681 so the strategy fires on the first block)
`modules/examples/balance-tracker/module.toml`:
- addresses -> EOA + Safe (was the hardhat default accounts)
- change_threshold -> 0.001 ETH (was 0.1; lower so the small
E2E gas-side transfers show as Warn diffs)
## OrderUid pinning + regression test
`modules/examples/stop-loss/src/strategy.rs` gains
`cow_1064_e2e_settings_yield_expected_uid`: an integration
test that constructs `Settings` from the exact same constants
as the new manifest and asserts the resulting `build_creation`
UID against:
0xc2b9cb4ea1ee5a86d8049ac09d8f494bf04cca0a68407285f31e2e6379800be8
7bf140727d27ea64b607e042f1225680b40eca6a
ffffffff
(orderDigest || owner || validTo per packOrderUidParams.)
If anything drifts — manifest values, EIP-712 type-hash,
domain separator — the test fires before the run starts, not
during the run.
## Run-prep punch list
`docs/operations/e2e-cow-1064-prep.md` (~ 280 lines):
1. **Pinned identities table** — every address the runbook
references (EOA, Safe, ComposableCoW, TWAP handler,
EthFlow, GPv2Settlement, GPv2VaultRelayer, WETH, COW
token, domain separator). All verified via `eth_getCode`
on Sepolia before commit.
2. **Per-module config pinning** — stop-loss + balance-
tracker effective values in table form.
3. **OrderUid decomposition** — orderDigest (32) + owner (20)
+ validTo (4) breakdown so an operator reading
`setPreSignature` calldata can sanity-check the UID
without redoing the EIP-712 math.
4. **Four on-chain actions** — each as a numbered step with
the exact contract + function + arguments + Etherscan
write-UI URL:
- Action 1: wrap 0.01 ETH -> 0.01 WETH9 (optional, only
for `submitted:` path; `backoff:` works without).
- Action 2: setPreSignature + WETH allowance to
GPv2VaultRelayer (optional, paired with action 1).
- Action 3: TWAP create() via Safe TX Builder; the full
516-byte calldata pinned verbatim (selector 0x6bfae1ca
+ tuple-encoded ConditionalOrderParams + dispatch=true).
- Action 4: EthFlow swap via cow-swap UI on Sepolia
(UI-driven for the quote endpoint hit; calldata
fallback link if UI flakes).
5. **Validation snippets** — `cast` invocations to check EOA
+ Safe balances, WETH balance, allowance,
`preSignature(bytes)` lookup, and a `journalctl + jq`
one-liner that tails per-module terminal markers in real
time.
6. **Re-derivation recipes** — Python + `cargo test`
commands to regenerate every pinned value if config drift
ever forces a re-run with different identities.
7. **Per-run acceptance checklist** — 9 box-checks that
double-pin section 7 of the e2e-report template, scoped
to THIS specific run.
## Workspace impact
- `cargo test -p stop-loss --lib` -> 8 passed (was 7; +1
for the new pinning test).
- `cargo fmt --all --check` clean.
- No production-code changes outside the test module.
Linear: COW-1064. Twelfth M4 deliverable; stacks on #45.
Three-step automation that wraps the COW-1064 runbook +
prep punch list into shell scripts. Operator workflow
collapses to:
cp scripts/env-template scripts/.env && $EDITOR scripts/.env
scripts/e2e-run.sh
scripts/e2e-onchain.sh
## … 4-6 h …
scripts/e2e-finish.sh
Secrets stay on disk. `scripts/.env` is gitignored; the
engine config with embedded RPC URL is rendered into a
gitignored `engine.e2e.local.toml` at boot time; the
PK is read from `scripts/.env` by `cast send` and never
echoed.
## Files
- `scripts/env-template` (committed): every variable the
scripts read, with comments. Operator copies to
`scripts/.env` and fills in.
- `scripts/lib.sh`: shared bash helpers — `log/warn/die`,
`load_env`, `render_engine_config`, `state_value`, and
the pinned address constants (EOA, Safe, ComposableCoW,
TWAP handler, EthFlow, GPv2Settlement, GPv2VaultRelayer,
WETH, COW, expected OrderUid).
- `scripts/e2e-run.sh`: renders engine config, cleans
data/e2e, builds 5 modules + engine in release, launches
via nohup, waits ≤ 60 s for `supervisor ready modules=5
chains=1`, snapshots `metrics-start-<ts>.txt`. Persists
PID + log path + start-ISO into `scripts/.state`.
- `scripts/e2e-onchain.sh`: derives EOA from
`OPERATOR_PRIVATE_KEY` + asserts it matches the pinned
test EOA + asserts balance ≥ 0.02 ETH; then `cast send`s:
1. ComposableCoW.create() with the 516-byte pinned
TWAP calldata → `ConditionalOrderCreated` →
twap-monitor `watch:`.
2. EthFlow.createOrder() with the tuple built from the
cow.fi `/api/v1/quote` response (via
`_ethflow_quote.py`) → `OrderPlacement` →
ethflow-watcher `submitted:`.
If `RUN_OPTIONAL_PRESIGN=1` also runs WETH wrap +
setPreSignature + GPv2VaultRelayer approval (for
stop-loss on-chain settlement; the `submitted:{uid}`
marker is produced regardless). Each tx hash appended to
`scripts/.state` as `TX_<KIND>=<hash>`.
- `scripts/_ethflow_quote.py`: small Python helper that
POSTs to cow.fi Sepolia, gets feeAmount + quoteId +
validTo + buyAmount, ABI-encodes the EthFlowOrder.Data
tuple, and prints the calldata + msg.value for the
shell script to consume.
- `scripts/e2e-finish.sh`: snapshots
`metrics-end-<ts>.txt`, sends SIGINT, waits ≤ 30 s for
`graceful shutdown complete` in the log (COW-1072 path),
escalates to SIGKILL after 30 s, then invokes the report
generator.
- `scripts/e2e-report-gen.sh`: parses the JSON-formatted
engine log + metrics snapshots + state file into the
e2e-report template's 9 sections. Auto-derives chain
delta, per-module first marker, every `shepherd_*`
counter / histogram delta, ERROR/trapped/poisoned
tallies, and the per-row acceptance checklist
(block-delta ≥ 1500, all-5-markers, zero traps, zero
poison, zero ERROR, TWAP+EthFlow txs present). Writes
`docs/operations/e2e-reports/e2e-report-<date>.md` ready
for operator review.
- `scripts/README.md`: one-time setup, run sequence,
troubleshooting table, re-run recipe.
## .gitignore additions
```
*.local.toml # engine config rendered with embedded RPC key
scripts/.state # run-state cache (PIDs, paths, tx hashes)
scripts/.env # operator secrets (redundant with the existing .env / .env.* rules but explicit)
docs/operations/e2e-reports/engine-*.log
docs/operations/e2e-reports/metrics-*.txt
```
The auto-generated `e2e-report-<date>.md` is NOT gitignored
— operator reviews + commits manually with `git add -f` (the
report belongs in history; the raw log + metrics dumps
don't).
## Why a separate render step
`engine.e2e.toml` is committed with a public-placeholder RPC URL
(`wss://ethereum-sepolia-rpc.publicnode.com`); `e2e-run.sh`
substitutes `RPC_URL_SEPOLIA` from `.env` into a local file
`engine.e2e.local.toml` (gitignored via `*.local.toml`) and
points the engine at the local file. This means:
- No secret ever lands in `git diff`.
- The committed config still boots cleanly (against the public
endpoint) for anyone cloning the repo who doesn't have a paid
RPC key.
- The render step is idempotent — re-running `e2e-run.sh`
always overwrites the local file.
## Verification
- `bash -n` syntax check on all 5 shell scripts: clean.
- `python3 -c "ast.parse(...)"` on `_ethflow_quote.py`: clean.
- `render_engine_config` smoke: produced `engine.e2e.local.toml`
with the rpc_url line correctly substituted; diff showed
exactly one line changed.
Linear: COW-1064. Stacks on the existing PR #46.
`scripts/e2e-run.sh` was grepping for the pretty-printed
`supervisor ready modules=5 chains=1` flat string. Without
`--pretty-logs` (which production-shape JSON deliberately
omits) the engine emits
{"message":"supervisor ready","modules":5,"chains":1,...}
so the grep never matched and the script died at the 60 s
deadline even though the engine was already healthy and
dispatching blocks (the nohup'd engine stayed alive
detached; the wrapper just couldn't see it).
Fix: extended the grep to two JSON-field-order alternatives
(`modules` before `chains` and vice versa, since the JSON
serialiser does not guarantee field order across releases).
Bumped the deadline to 90 s because cold-start of the wasm
component compile + first RPC handshake on a paid endpoint
can comfortably take 30-40 s on a fresh checkout.
Linear: COW-1064 (run-prep regression caught live during the
2026-06-18 dry run).
macOS ships /usr/bin/bash at version 3.2.57 due to GPLv3
licensing; `${var,,}` lowercase expansion is bash 4+ only.
The EOA-match check died with `bad substitution` on first
invocation against the live Sepolia run.
Routed both sides of the comparison through `tr '[:upper:]'
'[:lower:]'` which is POSIX-portable.
Grepped the rest of scripts/ for other `${var,,}` / `${var^^}`
constructs — none found, so this was the only impacted site.
Linear: COW-1064 (run-prep regression caught live).
… (COW-1064) Two fixes caught live during the 2026-06-18 dry run: 1. `_ethflow_quote.py` imports eth_abi + eth_utils + eth_hash; these are not in the Python stdlib and the script was failing with `ModuleNotFoundError: eth_abi` after the TWAP tx had already landed on Sepolia. Added a pre-flight `python3 -c 'import eth_abi, eth_utils, eth_hash.auto'` at the top of e2e-onchain.sh that fails loudly with the exact `pip3 install` command the operator needs. 2. Re-running e2e-onchain.sh after a partial failure would re-submit the TWAP create() (different nonce → new tx → same salt → ComposableCoW reverts) AND re-fetch a new EthFlow quote (new feeAmount/quoteId → new tx → wastes ETH on duplicate orders). Added idempotency: each action is wrapped in `if existing="$(state_value TX_*)"; then skip`, so the script picks up exactly where it left off using the tx hashes already persisted in scripts/.state. Acceptance: the dry run had TX_TWAP already in .state (manual recovery write); re-running now skips TWAP and only attempts EthFlow. Linear: COW-1064 (run-prep regressions caught live).
The CoW orderbook's `/quote` endpoint rejects the native-ETH sentinel `0xEeee…EEeE` with `InvalidNativeSellToken`. EthFlow orders are still quoted with the *wrapped* form (WETH9 Sepolia) as the sell side; the EthFlow contract itself does the wrap from msg.value on `createOrder`. Verified end-to-end: `python3 _ethflow_quote.py <EOA> 5000000000000000` returns a 292-byte calldata + VALUE_WEI on the live Sepolia orderbook (fee_amount ≈ 0.000308 ETH, buy_amount ≈ 0.192 COW, quote_id 1519204). Linear: COW-1064.
…ns (COW-1064)
tracing-subscriber's JSON formatter writes `message` /
`module` / `block_number` / `target` at the top level of the
event object (no nested `fields`); the parser was looking
inside `fields` and finding nothing. Added an
`event_field(ev, key)` helper that checks both top-level and
nested-fields shapes.
Replaced the single substring list with a per-module pattern
map, derived from the host.log() call sites inside
modules/*/src/strategy.rs. Specifically:
twap-monitor -> "watch:", "indexed watch:", "poll watch:"
ethflow-watcher -> "ethflow submitted", "ethflow backoff",
"ethflow dropped", "already submitted"
price-alert -> "TRIGGERED"
balance-tracker -> "changed +", "changed -" (per-block
"0x<addr> changed +N wei ..." diff log)
stop-loss -> "TRIGGERED", "retry on next block",
"stop-loss submitted", "stop-loss dropped",
"already submitted", "submitted:"
Verified against the live 2026-06-18 dry run's engine log:
all 5 modules surface ≥ 1 terminal marker.
Linear: COW-1064 (run-prep regression caught live during the
T+12-min mark of the run).
…okup (COW-1074)
Closes the gap surfaced by the COW-1064 dry run (2026-06-18):
TWAP orders created through cow-swap UI sign with a non-empty
`appData` hash pointing at a richer JSON document (partner-id,
slippage settings, quote-id). twap-monitor hard-coded
`EMPTY_APP_DATA_JSON` when assembling `OrderCreation`, so the
orderbook rejected every submit with `invalid OrderCreation:
app_data JSON digest does not match signed app_data hash` and
the watch sat in retry-loop forever.
The WIT already exposes `cow-api::request(method, path, body)`
as a generic REST passthrough. We surface that capability on
the SDK trait, wrap it in a typed helper, and use the helper
from the strategy. No new host imports, no WIT ABI change, no
forced rebuild of unrelated modules.
Extended `CowApiHost` with:
```rust
fn cow_api_request(
&self,
chain_id: u64,
method: &str,
path: &str,
body: Option<&str>,
) -> Result<String, HostError>;
```
404 responses surface as `HostError { code: 404, kind:
Unavailable }` so callers can distinguish "orderbook does not
have this resource" from genuine upstream failures without
introducing a new `HostErrorKind` variant (the existing enum
is `non_exhaustive`, but adding a variant on the WIT side
would still need an ABI bump).
`resolve_app_data(host, chain_id, hash) -> Result<String,
HostError>` with:
- Short-circuits `EMPTY_APP_DATA_HASH` (`keccak256("{}")`)
to `EMPTY_APP_DATA_JSON` (`"{}"`) — no host call needed.
- Otherwise GETs `/api/v1/app_data/{hex_hash}` and pulls
the `fullAppData` field out of the orderbook's envelope
shape (`{"fullAppData": "<JSON string>", ...}`).
- 5 unit tests pinning the short-circuit, the success path,
the unexpected-shape fall-through, the 404 propagation,
and the hex encoder.
Extended `MockCowApi` with:
- `respond_to_request_for(method, path, result)`: per-key
programmable response.
- `respond_to_request(result)`: catch-all default.
- `request_calls()`: records the (chain_id, method, path,
body) tuple for every invocation.
The existing `respond` / `calls()` / `submit_order` surface
is unchanged.
`modules/twap-monitor/src/lib.rs`,
`modules/ethflow-watcher/src/lib.rs`,
`modules/examples/price-alert/src/lib.rs`, and
`modules/examples/stop-loss/src/lib.rs` each gained the
trivial 8-line forwarder to the generated
`cow_api::request` binding. Example modules implement
`CowApiHost` purely for the `Host` blanket-impl supertrait
even though some don't actively submit orders — the impl
is symmetrically extended.
`build_order_creation` now takes the resolved
`app_data_json` as an explicit parameter (was hard-coded to
`EMPTY_APP_DATA_JSON`). The resolution itself is lifted into
the caller `submit_ready`, which calls
`shepherd_sdk::cow::resolve_app_data` before assembling the
`OrderCreation`. Two graceful-fallback branches:
- `err.code == 404` → log Warn "appData hash not mirrored
on orderbook" + leave the watch in place. Operators can
re-trigger by pinning the document via a future
orderbook PUT or by re-creating the order with empty
appData.
- Any other resolver error → log Warn "appData resolve
failed" + leave the watch. Future retry on the next
block re-attempts the lookup.
Two new strategy tests:
- `poll_ready_resolves_non_empty_app_data_then_submits`:
programs MockHost with a known JSON + its hash on the
order, asserts the full resolve → submit → `submitted:`
marker flow.
- `poll_ready_skips_submit_when_app_data_hash_not_mirrored`:
programs MockHost to 404, asserts no submit attempt, no
`submitted:` / `dropped:` markers, Warn log line present.
Plus one updated test
(`build_order_creation_accepts_matching_non_empty_app_data`)
that pins the new "matching hash → JSON" success path
directly on `build_order_creation`.
- `cargo test --workspace` → 13 + 12 + 16 + 32 + 8 + 8 +
61 (engine) + 23 (twap-monitor) + 7 doctests + 1
integration = 181 tests passing (was 174; +5 SDK +2
twap-monitor).
- `cargo clippy --all-targets --workspace -- -D warnings`
clean.
- `cargo fmt --all --check` clean.
- All 4 production module .wasm artefacts build cleanly
with the new SDK trait.
- No WIT changes. Modules built against the prior SDK
trait will fail to compile (the new method is required),
but the WIT-generated wasm-side surface is bit-identical.
- No host-impl changes (`crates/nexum-engine/src/host/
impls/cow_api.rs`). The host already implements `request`
for the wit-bindgen binding.
- No metric surface drift. The orderbook lookup goes
through the same `shepherd_cow_api_*` counters via the
existing `request` path.
Linear: COW-1074. Stacks on the COW-1064 run-config branch
(#46). Validated locally end-to-end via `cargo test
--workspace`; live validation against the running engine
will happen on the next COW-1064 dry run (engine restart
required to pick up the rebuilt modules).
…-1074)
Symmetric extension of the twap-monitor fix in this PR. The
ethflow-watcher strategy's `build_eth_flow_creation` hard-coded
`EMPTY_APP_DATA_JSON` exactly like twap-monitor did; any
OrderPlacement event whose embedded `GPv2OrderData.appData`
hash differs from `keccak256("{}")` (i.e. every cow-swap UI
EthFlow swap) would hit "app_data JSON digest does not match
signed app_data hash" and be silently skipped client-side.
The COW-1064 dry run didn't surface this for the EthFlow tx I
fired via `scripts/e2e-onchain.sh` — because that script's
helper sets `appData = EMPTY_APP_DATA_HASH` — but a cow-swap UI
EthFlow swap (which is the realistic production path) would.
## Changes
- `build_eth_flow_creation` now takes `app_data_json: String`
alongside `chain_id` and `placement`. Docstring updated to
reference COW-1074.
- `submit_placement` calls `shepherd_sdk::cow::resolve_app_data`
before `build_eth_flow_creation`; on 404 logs a Warn
"ethflow submit skipped (sender=...): appData hash not
mirrored on orderbook" and returns Ok (no marker written, no
submit attempt).
- 6 test call sites updated to pass
`cowprotocol::EMPTY_APP_DATA_JSON.to_string()` explicitly,
preserving the existing assertions verbatim.
- 2 new integration tests:
`placement_with_non_empty_app_data_resolves_then_submits`
`placement_skips_submit_when_app_data_hash_not_mirrored`
mirror the twap-monitor pair, programming MockHost with a
synthetic appData JSON + hash, asserting the resolve →
build → submit chain produces a `submitted:{uid}` marker
and that 404 produces a Warn-only skip.
## Workspace impact
- `cargo test -p ethflow-watcher` → 14 tests passing
(was 12; +2 from this commit).
- `cargo test --workspace` → 183 tests passing total
(was 181 after the twap-monitor commit; +2 ethflow-watcher).
- `cargo clippy --all-targets --workspace -- -D warnings`
clean.
- `cargo fmt --all --check` clean.
Linear: COW-1074 (extended scope — same gap in ethflow-watcher).
Captures the 2026-06-18 COW-1064 dry run + live in-flight validation of PR #47 (resolve_app_data fix). ## Acceptance summary 5 of 6 rows green; the only [ ] is `block delta ≥ 1500` (got 415) because the run was intentionally interrupted twice to validate PR #47 against the same data/e2e local-store across pre-PR-47 + PR-47-twap-monitor + PR-47-ethflow-watcher commits. | Row | Result | |---|---| | block delta ≥ 1500 | [ ] (got 415; 3 engine restarts for PR #47 mid-run validation) | | all 5 modules have a terminal marker | [x] | | shepherd_module_errors_total{trap} == 0 | [x] | | no module poisoned | [x] | | 0 ERROR lines from nexum_engine | [x] | | TWAP + EthFlow tx submitted | [x] | ## 4 anomalies filed in Linear, fully documented in §6 - COW-1074 — twap-monitor + ethflow-watcher hardcoded EMPTY_APP_DATA_JSON. **Fixed in-run via PR #47**; live-validated for both modules (§6.5). - COW-1075 — SDK classify_api_error should map `DuplicatedOrder` -> `Drop` (stop-loss retry loop). - COW-1076 — ethflow on-chain `validTo=uint32::MAX` rejected by Sepolia orderbook (`ExcessiveValidTo`; upstream issue). - COW-1077 — scripts/e2e-onchain.sh TWAP `t0=0` produces permanently-finished order (caller-side encoding bug). ## Live PR #47 validation (§6.5 — the key methodology note) Three engine binaries exercised on the same redb local-store: 1. `5bcd47b` (pre-PR-47): surfaces the digest-mismatch client-side skip for both twap-monitor + ethflow-watcher on non-empty appData orders. 2. `acc9654` (PR #47 twap-monitor): existing cow-swap UI TWAP re-polls to Ready -> resolve_app_data resolves the JSON from `/api/v1/app_data/{hash}` -> submit reaches orderbook -> DuplicatedOrder (server-side reject only). Client-side digest check bypassed. 3. `cd68de0` (PR #47 ethflow-watcher): new cow-swap UI EthFlow swap (`0x82da5ced...`) observed -> appData = `0xe46e7d0c...` (NON-empty rich JSON: appCode="CoW Swap", slippageBips=857, smartSlippage=true) -> resolve_app_data calls orderbook -> JSON extracted from `fullAppData` field -> build produces matching-digest body -> submit reaches orderbook -> ExcessiveValidTo (server-side reject only, tracked separately in COW-1076). The PR #47 fix is therefore live-validated end-to-end against the real Sepolia orderbook in **both** affected modules. ## What this report unblocks COW-1031 (7-day soak) is technically unblocked: the engine + 5-module dispatch is proven correct under live conditions; PR #47 closes the only blocking SDK gap for the soak's TWAP + EthFlow coverage. The remaining 3 follow-ups (COW-1075/1076/1077) are quality-of-output rather than correctness regressions and do not block the soak. Operator sign-off pending in §8. Linear: COW-1064 (closes).
…COW-1075)
`OrderBookPool::submit_order_json` returns `CowApiError::Orderbook(cowprotocol::Error::OrderbookApi { status, api })` for any 4xx with a typed `{"errorType": "...", ...}` body (see `cowprotocol::transport::HttpResponse::into_status_error`). The WIT adapter was dropping `api` on the floor (`data: None`), so the guest's `shepherd_sdk::cow::classify_api_error` always saw `None` and fell back to its safe-default `TryNextBlock`. Permanent rejections like `DuplicatedOrder`, `InvalidSignature`, or `ExcessiveValidTo` therefore looped forever, masquerading as transient failures.
Root cause of the stop-loss infinite-retry behaviour observed in the 2026-06-18 COW-1064 dry run (e2e-report-2026-06-18.md §6.3): 76 retries of an already-submitted order in 170 blocks because the host never let the guest see what the orderbook actually said.
Fix is in the WIT adapter (`crates/nexum-engine/src/host/impls/cow_api.rs`), not the SDK classifier. The classifier already handles `Unknown(_)` -> `Drop` correctly via its `Some(_) => Drop` branch; it just needed the envelope to dispatch on. Extracted the projection into a testable `orderbook_to_host_error` helper that:
- serialises `ApiError` into `HostError.data` as JSON when the variant is `OrderbookApi { status, api }` (the only variant carrying a structured payload),
- sets `code` to the HTTP status so guests can disambiguate 4xx vs 5xx,
- leaves `data: None` for other `cowprotocol::Error` variants (transport, serde, unexpected-status) since they have no envelope and `TryNextBlock` is the correct safe default for them.
Tests:
- `orderbook_to_host_error` unit tests cover the envelope-forward, the optional inner `data` round-trip, and the non-envelope `UnexpectedStatus` branch (3 cases).
- New wiremock integration test `submit_order_propagates_orderbook_envelope` confirms a 400 with `errorType: "DuplicatedOrder"` surfaces the `OrderbookApi` variant end-to-end through `OrderBookPool::submit_order_json`.
All 13 cow-api-adjacent tests pass; workspace tests untouched.
…1076)
EthFlow on-chain orders use `validTo = u32::MAX` by design (see `cowprotocol::eth_flow`). The Sepolia orderbook's max-validTo cap rejects this shape with `errorType = "ExcessiveValidTo"`, and after the COW-1075 host fix the strategy already classifies it correctly as Drop. The remaining gap was operator ergonomics: every EthFlow placement on Sepolia produced a Warn-level "ethflow dropped" line, which would dominate a 7-day soak dashboard with non-anomalous traffic.
Change: in `apply_submit_retry`'s Drop arm, peek at the decoded ApiError. If the orderbook's `errorType == "ExcessiveValidTo"`, log at Info instead of Warn. All other Drop reasons (InvalidSignature, WrongOwner, etc.) keep Warn so real anomalies still page the operator. Dispatch (write `dropped:{uid}`, clear stale `backoff:{uid}`) is unchanged.
Why not gate on more (e.g. inspect the order's validTo field): the strategy already filters logs to EthFlow contract addresses; ExcessiveValidTo from the orderbook for an EthFlow placement is unambiguously the documented constraint. Keeping the gate narrow avoids accidentally suppressing other-cause Warns.
Tests (3 new in `modules/ethflow-watcher/src/strategy.rs`):
- `submit_excessive_valid_to_logs_at_info_not_warn`: end-to-end through `on_logs`; confirms exactly one drop line at Info level and zero Warn drops for this case.
- `submit_other_permanent_error_still_logs_at_warn`: regression guard - InvalidSignature stays at Warn.
- `submit_drop_without_envelope_keeps_warn_level`: predicate-level unit test confirming `is_expected_excessive_valid_to` returns false when `HostError.data` is None (e.g. transport failure).
Docs: added "Known upstream constraints on Sepolia" section to `docs/operations/e2e-testnet-runbook.md` documenting this gap, the post-fix operator-visible behaviour, the Prometheus signal (`shepherd_cow_api_submit_total{outcome=\"err\"}` grows by the EthFlow placement count then stops), and a pointer to COW-1076 for the upstream-confirmation status.
Soak impact: the COW-1031 7-day run on Sepolia will now show ExcessiveValidTo drops as Info-level traffic. The soak's "0 unexpected errors" acceptance bar is preserved because Warn-level drops only fire on real anomalies.
All 17 ethflow-watcher tests pass (+3 new); workspace tests untouched. clippy + fmt clean.
The previous `e2e-onchain.sh` pinned a 516-byte hex blob with `t0 = 0` in the ComposableCoW.create() static-input tuple. TWAP handler's `validateData` does NOT reject `t0 = 0` (it only checks `t0 >= type(uint32).max`), so the `create()` tx succeeded but `TWAPOrderMathLib.calculateValidTo` then computed `part = (block.timestamp - 0) / t = ~3.3M`, which is far above the configured `n = 2` and triggers `AFTER_TWAP_FINISHED` on every `getTradeableOrderWithSignature` poll. Surfaced in the COW-1064 dry run (2026-06-18 report §6.4): supervisor logged the `0xc8fc2725...after twap finished` revert per block. Fix: - New `scripts/_twap_calldata.py` encodes the calldata fresh on every invocation with `t0 = int(time.time()) - 60` (backdated 60s so part 0 is Ready as soon as the order is on-chain). Module docstring explicitly warns against re-hardcoding t0. - `scripts/e2e-onchain.sh` Action 1 now shells out to the helper rather than carrying the hex inline. Validates the output is hex-shaped before passing to `cast send`. - `docs/operations/e2e-cow-1064-prep.md` section 2.3 step 3 replaces the pinned blob with a `python3 scripts/_twap_calldata.py` recipe and a historical note pointing at COW-1077. - `docs/operations/e2e-cow-1064-prep.md` section 4.2 recipe gets `import time` + `int(time.time()) - 60` for `t0` so the re-derivation flow does not re-introduce the bug. - `scripts/README.md` Action 1 description updated to mention the helper. Constants in the helper (sell/buy tokens, amounts, n, t, salt) mirror the prep doc's section 4.2; both must change in lockstep if the TWAP shape is retargeted. Validation: `python3 scripts/_twap_calldata.py` produces 516-byte calldata (1034 hex chars) starting with the correct selector `0x6bfae1ca`; the t0 word reflects current epoch (verified against `0x00...006a3537b5` on the smoke run). `bash -n scripts/e2e-onchain.sh` passes. No engine-side changes; this is a script-and-docs PR.
mfw78 review of PR #8 (nullislabs#8) flagged "we already pull alloy, so pulling hex via there is really not much of a deal". The PR #47 (COW-1074) commit acc9654 then introduced two new custom hex helpers that recreate the same antipattern at a different scope: - `crates/shepherd-sdk/src/cow/app_data.rs::encode_hex` - 32-byte hash → `0x...`. Used by `resolve_app_data` to format the orderbook lookup path. - `modules/twap-monitor/src/strategy.rs::hex_short` - 8-byte prefix → `0x...…`. Used to format `appData` hashes in INFO log lines. Both crates already depend on `alloy-primitives` (sdk: 1.6, twap-monitor: 1.5), so the swap is a one-liner per call site: - `encode_hex(b)` → `format!("0x{}", alloy_primitives::hex::encode(b))` - `hex_short(b)` → `format!("0x{}…", alloy_primitives::hex::encode(&b[..8]))` Both functions keep their old signature so callers (`resolve_app_data` in the SDK, every `host.log` line in twap-monitor strategy) need no changes. Comments on both helpers now explicitly reference mfw78's PR #8 guidance so the next person tempted to hand-roll a `0123456789abcdef` table has a hook. Validation: cargo test -p shepherd-sdk -p twap-monitor: 32 + 23 passed; cargo clippy --all-targets -- -D warnings: clean; cargo fmt --check: clean; zero em-dash drift. Why this PR sits in a separate branch rather than amending PR #47: PR #47 is already In Review, and #48/#49/#50 stack on top of it. Amending would require force-pushing 4 branches. A small follow-up PR keeps each one bisectable and lets mfw78 review the alloy alignment in isolation.
Synthetic load test for shepherd's M4 stack. Distinct from: - COW-1064 (real Sepolia E2E, correctness, 90 min, 5 modules) - COW-1078 (backtest of 7d historical events, replay) - COW-1031 (7-day soak, wall-clock stability) This issue answers one question the others do not: how many events per block can the supervisor dispatch before something breaks? lgahdl's PR #9 review thread flagged sequential per-module dispatch as a potential bottleneck; this PR is how we measure it. Components added: 1. `tools/orderbook-mock` (new crate, axum-based) - HTTP server serving the two endpoints shepherd's cow-api host hits per submission. POST /api/v1/orders returns a synthetic 56-byte OrderUid; GET /api/v1/app_data/{hash} returns the empty appData document. CLI knobs: --port, --latency-ms, --error-rate (alternates InsufficientFee / InvalidSignature to exercise both TryNextBlock and Drop paths). 3 unit tests covering the happy path, the empty appData path, and the error-rate envelope. 2. `tools/load-gen` (new crate, alloy-based) - connects to Anvil, impersonates the pinned Sepolia test EOA via anvil_impersonateAccount + anvil_setBalance, then on every new block fires N ComposableCoW.create(...) + M CoWSwapEthFlow.createOrder(...) calls. Each create uses a fresh salt counter so submissions do not collide on the dedup check. 3 unit tests covering pinned address parsing, salt uniqueness, and calldata selector shape. 3. Engine config: ChainConfig gains optional `orderbook_url` (per chain). OrderBookPool::from_config honours the override using cowprotocol::OrderBookApi::new_with_base_url; absent overrides fall back to canonical api.cow.fi URLs. main.rs switches from ::default() to ::from_config(&engine_cfg). Useful long-term for staging/barn targets, immediately needed to point at the mock. 4. `engine.load.toml` - chain 11155111 -> ws://localhost:8545, cow base URL -> http://localhost:9999, metrics on 127.0.0.1:9100, state_dir = ./data/load (wiped per run). 5. Scripts: - `scripts/load-bootstrap.sh` brings up Anvil + orderbook-mock, tracks PIDs in /tmp/shepherd-load.pids, exposes a teardown helper. - `scripts/load-teardown.sh` idempotent cleanup. - `scripts/load-run.sh` orchestrates one scenario end-to-end: bootstrap, build modules, start engine, snapshot /metrics, run load-gen for --duration-min, snapshot /metrics again, tear down, drop a report skeleton at docs/operations/load-reports/load-NxM-YYYY-MM-DD.md. 6. `docs/operations/load-testnet-runbook.md` - operator runbook covering the three scenarios (baseline 5x5, medium 20x20, saturation 50x50), expected acceptance bars, what the test does NOT prove (WS reconnect / drift / real-orderbook fidelity), troubleshooting. Validation: - cargo test --workspace --exclude <wasm-only-modules>: 196 passed. - cargo clippy --workspace --all-targets --tests -- -D warnings: clean. - cargo fmt --all --check: clean. - bash -n scripts/load-{bootstrap,run,teardown}.sh: clean. - Live orderbook-mock smoke: POST returns valid 56-byte hex UID, GET returns {"fullAppData":"{}"}, /_stats reflects counters. Pending (not in this PR): - Baseline 5x5 report against a real Anvil fork - requires Bruno's RPC_URL_SEPOLIA_HTTP from scripts/.env; once that runs, the report lands in docs/operations/load-reports/. - Metrics-delta auto-generation in scripts/load-run.sh (left as TBD in the script; e2e-report-gen.sh has the delta logic we can adapt). - Saturation scenario - run after the baseline lands so the bottleneck has a clean baseline to compare against.
…tion (COW-1079)
First COW-1079 run on a real Anvil fork of Sepolia. The engine-side
acceptance bar is cleared with wide margin:
- Per-block dispatch latency p50/p95/p99 = 4/6/7 ms (bar was < 2 s).
- Zero traps, zero poisoned modules, zero shepherd_module_errors_total.
- EthFlow strategy submitted 1 OrderPlacement end-to-end through the
mock orderbook in 10 ms; submitted:{uid} marker written cleanly.
- 63 Anvil blocks dispatched flawlessly.
The honest finding: load-gen's transactions get into Anvil's mempool
(twap_ok=270, ethflow_ok=270 per the eth_sendTransaction response),
but only 5 ConditionalOrderCreated + 1 OrderPlacement events
actually fired - the rest reverted at the contract level
(ComposableCoW.create + EthFlow.createOrder run preconditions the
load-gen-crafted bodies don't pass).
So this run stressed the engine with ~6 events over 60 s, not
5+5 per block. The bar criterion that depends on the load-gen
(events-per-block delivered) is the only one that doesn't pass;
filing a follow-up to calibrate the revert rate before re-running.
Report at docs/operations/load-reports/load-5x5-2026-06-19.md
mirrors the COW-1064 e2e-report shape and signs off as
"conditional pass" - engine meets the bar; load-gen needs work.
scripts/lib.sh exports REPORTS_DIR=e2e-reports/ unconditionally. load-run.sh used to set REPORTS_DIR=load-reports/ BEFORE sourcing load-bootstrap.sh (which transitively sources lib.sh), so the override was lost and the auto-generated skeleton ended up under e2e-reports/ next to the COW-1064 reports. Move the assignment after the source so the load-reports/ path wins, with a comment explaining the ordering trap. Drive-by: removed the misplaced e2e-reports/load-5x5-2026-06-19.md from the first run; the committed report at load-reports/load-5x5-2026-06-19.md (commit 59fe714) is the canonical copy.
COW-1079 baseline's 5/270 + 1/270 revert rate had two distinct root causes, both contract-side, neither shepherd's fault: 1. **Nonce race in burst submissions.** Anvil's `eth_sendTransaction` against an impersonated account auto-assigns a nonce when none is provided, but the assignment racts with the caller's burst submission. When load-gen fired 5 TWAP + 5 EthFlow per block without waiting for individual receipts, most txs landed in the mempool sharing the same nonce, and Anvil's miner included only one per block - the rest reverted as nonce-too-low. Fix: read the EOA's current nonce at boot, increment locally per successful submission, pin `tx.nonce` explicitly on every `TransactionRequest`. Lock-step with cargo build cache so the nonce counter never crosses async-boundary corruption. 2. **EthFlow OrderUid dedup on identical GPv2 OrderData.** The CoWSwapEthFlow contract dedups by the GPv2 `OrderUid` which is keccak over (buyToken, receiver, sellAmount, buyAmount, appData, feeAmount, validTo, partiallyFillable, kind, sellTokenSource, buyTokenDestination). quoteId is NOT part of that hash. The prior load-gen varied only `quoteId` per call, so all 270 EthFlow submissions produced the same UID and the contract rejected 269/270 as `OrderIsAlreadyOwned`. Fix: vary `sellAmount` by 1 wei per call (`BASE_SELL_AMOUNT + seq`) and pass that same value as `msg.value` so the contract's `msg.value == order.sellAmount` invariant holds. Re-ran baseline 5x5 after both fixes: 130/130 TWAP + 130/130 EthFlow delivered, 130 ConditionalOrderCreated + 130 OrderPlacement events on-chain, 130 cow_api submits OK to mock, 130 ethflow markers written, zero shepherd_module_errors_total. Updated baseline report at docs/operations/load-reports/load-5x5-2026-06-19.md from 'conditional pass' to 'full PASS' with the post-calibration numbers (TWAP block p99 = 49 ms, EthFlow log p99 = 11 ms, 40x margin on the < 2 s bar). Medium 20x20 and saturation 50x50 are now unblocked per the COW-1079 acceptance roadmap.
…(COW-1079) Closes the COW-1079 three-scenario sweep with the COW-1080 calibration in place. All three scenarios pass: baseline 5x5 - 130/130 each, TWAP block p99=49ms medium 20x20 - 280/280 each, TWAP block p99=67ms saturation 50x50 - 300/300 each, TWAP block p99=78ms Latency growth across the watch-count range (130 -> 280 -> 300) is sub-linear: 49 -> 67 -> 78 ms. The lgahdl PR #9 concern about sequential per-module dispatch saturating under load is NOT surfaced at this scale. Zero shepherd_module_errors_total, zero traps, zero EthFlow submit errors across all three runs. The unexpected finding from saturation: the engine did not saturate. The bottleneck is load-gen's sequential eth_sendTransaction submission (each tx ~200 ms RTT, so 100 tx/iteration = ~20 s, vs. Anvil's 1 s block time). To genuinely saturate the engine we would need parallel load-gens against different impersonated EOAs, a sub-second block-time, or thousands of pre-seeded watches. EthFlow log p99 stayed flat at ~9 ms across all three scenarios (it is dominated by the cow-api submit roundtrip, not engine state), confirming the submit path scales independently of the watch count. The cold-start outlier (~500 ms on the first watch-heavy block) appears consistently across runs and is independent of the steady- state watch count - it is a one-shot first-block redb/eth_call warmup cost, NOT a saturation symptom. What this proves: - Shepherd M4 supervisor handles >= 300 concurrent watches + >= 138 block dispatch cycles in 2 min with p99 < 80 ms. - cow-api submit path is steady at ~9 ms p99 regardless of watch count. - Zero error/trap/poison across all three scenarios. What it does NOT prove (and is not in scope here): - Behaviour at 3000+ watches. - WS reconnect resilience (COW-1031 soak). - Multi-day memory drift (COW-1031). - Real-orderbook 4xx variety (COW-1078 backtest). COW-1079 ready to move to In Review.
…079) The single-EOA saturation 50x50 report identified the per-EOA nonce serialisation as the bottleneck before the engine had a chance to saturate. This commit removes that bottleneck: load-gen: - New --parallel N flag. Each worker impersonates a synthetic EOA (0x57...01..0a), gets its own WS connection + nonce stream, runs its own per-block submission loop. Total events per block scales linearly with N. - Disjoint salt space per worker via 96-bit prefix. - Disjoint EthFlow sellAmount space via a 10_000-wide per-worker window (the first attempt shifted by 96 bits, blowing past the 1M ETH funded balance with 7.9e28 wei sellAmounts; fixed). scripts/load-bootstrap.sh + scripts/load-run.sh: - Accept --block-time (passes to anvil) and --parallel (passes to load-gen). Defaults preserve historic behaviour: --block-time 1, --parallel 1. - Auto-report filename now includes scenario label (load-NxM-SCENARIO-date.md) so saturation-parallel does not overwrite the baseline 5x5 report. Saturation-parallel run (10 workers x 5 TWAP + 5 EthFlow per block, --block-time 0.5, 2 min): - load-gen: 895/895 TWAP + 895/895 EthFlow acks, 0 errors. - engine saw 381 ConditionalOrderCreated + 343 OrderPlacement events (43% / 38% delivery vs load-gen acks - Anvil + WS dropping under the heavier load). - shepherd_module_errors_total = 0, zero traps. - All 343 EthFlow submissions reached the mock orderbook 1:1. - TWAP block dispatch: histogram p50/p99 = 145 ms, max = 101 593 ms (101 s outlier on one block when 380+ watches polled against a stressed Anvil JSON-RPC). - Engine-log dispatch_block: n=586, p50=4ms, p95=46ms, p99=74ms, max=101 593 ms - same outlier. Saturation knee identified: 380+ active watches + 0.5s block-time + 10 concurrent WS subscribers produces a 101-second worst-case dispatch + 38-43% event delivery loss. Both symptoms point at the surrounding system (Anvil + WS transport), not at shepherd; engine continues to scale sub-linearly with watch count and never produces a module error, trap, or panic under any tested configuration. For the 7-day COW-1031 soak: this implies the operator should use a paid Sepolia archive endpoint (Alchemy / drpc / QuickNode), not publicnode, OR accept event drops and rely on supervisor reconnect + eth_getLogs re-indexing. Documented in the new report. Report at docs/operations/load-reports/load-50x50-parallel-2026-06-19.md.
Squash of PR #66 - applies 5 blockers + 8 majors from M4 audit.
…a doc link
Rebase fallout from the M4 compliance pass:
- `chain/chainlink.rs` defines `StubHost<Result<String, HostError>>` and
manually implements every `*Host` trait. When the M4 conflict resolution
added the `cow_api_request` forwarder into the macro's `CowApiHost`
impl, this local StubHost was missed, producing `E0046: not all trait
items implemented`. Add a parallel `unreachable!("not used in this
test")` body; the test never exercises the cow-api surface.
- `cow/app_data.rs`'s module-level doc referred to `EMPTY_APP_DATA_JSON`
as an unqualified intra-doc link, but the symbol is only used as
`cowprotocol::EMPTY_APP_DATA_JSON` inside the function body (no `use`
at module scope). `RUSTDOCFLAGS=-D warnings` rejects the unresolved
link. Qualify the path so it resolves while keeping the prose intent.
- `wit_bindgen_macro.rs` fmt drift: cargo fmt collapses the
`shepherd::cow::cow_api::request(...).map_err(convert_err)` chain to
a single line. Apply the canonical format.
Brings dev/m4-base back to fmt/clippy/test/doc green.
…face Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #3 (`[u8; 32]` for protocol hash across SDK public boundary). The rubric explicitly calls out: "Newtypes for protocol IDs (no raw `[u8; 32]` across module boundaries)." `B256` is already in `shepherd_sdk::prelude` so the swap costs callers nothing - both twap-monitor and ethflow-watcher were holding the appData as `B256` already and reaching through `.0` to satisfy the prior signature. Changes: - `resolve_app_data(host, chain_id, &B256)` (was `&[u8; 32]`) - `encode_hex(&B256)` internal helper - Doctest + 5 unit tests rewritten against `B256::from(bytes)` and `B256::from_slice(EMPTY_APP_DATA_HASH.as_slice())`. Coverage stays identical. - Call sites in twap-monitor and ethflow-watcher drop the `.0` reach-through; pass `&order.appData` directly. No public surface beyond `shepherd-sdk` consumes this function; external module crates in the workspace are the only consumers and both land in the same commit.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md,
duplication finding "Canonical CoW chain set
[Mainnet, Gnosis, Sepolia, ArbitrumOne, Base]" duplicated at
`crates/nexum-engine/src/host/cow_orderbook.rs:39-43` and `:66-70`.
`from_config` was added in the M4 multi-chain pass and reproduced the
same 5-element array `Default::default` already used. Adding a sixth
chain previously needed touching both arrays in lock-step; pull the
list into a single `const DEFAULT_CHAINS: &[Chain]` so the
single-source-of-truth property is structural.
Also drops the redundant `use cowprotocol::OrderBookApi;` inside
`from_config` (already in scope from the module-top `use cowprotocol::
{Chain, OrderBookApi, ...}` line). Behaviour identical.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #6. Rubric forbids em-dashes in operator-facing config files; while .toml is technically a grey zone the comment surfaces verbatim when operators `cat engine.e2e.toml` during e2e runbook execution.
- Add memory_limit field to LoadedModule for reinstantiation - Fix reinstantiate_one to use ModuleStore instead of LocalStore - Fix COW-1072 progress marker to use ModuleStore API - Add missing ModuleLimits argument to test boot_single calls - Add missing limits field to test EngineConfig initializations - Add missing orderbook_url field to test ChainConfig
- Add strum dep to shepherd-sdk, twap-monitor, ethflow-watcher - Add proptest dev-dep to shepherd-sdk - Add cow_api_request impl to twap-monitor and ethflow-watcher - Add wildcard arms for #[non_exhaustive] HostErrorKind and LogLevel - Add ProviderError::Timeout variant for chain JSON-RPC timeout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on M3 (
dev/m3). M4 takes the SDK + modules from M3 and hardens the runtime around them so a single Shepherd instance is operable as a production daemon: bounded resource use, supervised crash recovery, structured observability, and an end-to-end testnet harness.Core deliverable
crates/nexum-engineenforces per-module fuel + memory budgets; SDK error enums made#[non_exhaustive]so SDK bumps don't silently drop arms in module code (COW-1029, COW-1036).Supervisorrestarts crashed module instances behind a backoff; SIGTERM/SIGINT drain in-flight work; a module that crashes N times in a row is parked rather than restart-spammed (COW-1033, COW-1072, COW-1032).runtime/event_loop.rsrebuilds WS subscriptions after disconnect;tracing+ JSON output across the engine;/metricsscrape endpoint with per-module counters (COW-1071, COW-1035, COW-1034).docs/operations/e2e-testnet-runbook.mdwalks a full Sepolia round-trip;docs/production.mdcovers operator setup, env vars, log shipping, scrape config (COW-1064, COW-1030).shepherd-sdk::cowresolves app-data digests through the orderbook resolver endpoint, removing the IPFS hard dependency on submit paths (COW-1074).HostError.datanow carries the orderbook's structured error envelope so module code can decodeOrderPostErrorKindwithout re-parsing JSON (COW-1075).modules/ethflow-watcherdowngrades the known-benignExcessiveValidTodrop toInfo;scripts/_twap_calldata.pyproduces a fresh-t0TWAP fixture for the e2e harness (COW-1076, COW-1077).tools/load-gen+tools/orderbook-mock+scripts/load-run.shdrive baseline / medium / saturation runs against an Anvil fork; reports underdocs/operations/load-reports/(COW-1079, COW-1080).Validation
cargo fmt --all -- --checkclean.cargo clippy --workspace --all-targets -- -D warningsclean.cargo test --workspace— 230 tests passing; load-test harness has its owntools/test pass.wasm32-wasip2 --release) green for all modules in CI.docs/operations/load-reports/document engine behaviour at 5x5 / 20x20 / 50x50 grids.-D warningsCI gate still clean.M4-specific paths for review
crates/nexum-engine/src/{runtime,supervisor,host}/**(resource limits, supervisor restart, WS reconnect, multi-chain isolation, error envelope forwarding)crates/nexum-engine/src/engine_config.rs(Prometheus + log config)modules/ethflow-watcher/src/strategy.rs(ExcessiveValidTodowngrade)modules/twap-monitor/src/strategy.rs(AppData resolver consumption)crates/shepherd-sdk/src/cow/*.rs(AppData resolver, structured error envelope)docs/operations/{e2e-testnet-runbook,production}.mddocs/operations/load-reports/*.mdscripts/{e2e-onchain,load-run,load-bootstrap,load-teardown,_twap_calldata}.{sh,py}tools/{load-gen,orderbook-mock}/**Closes COW-1029, COW-1030, COW-1032, COW-1033, COW-1034, COW-1035, COW-1036, COW-1064, COW-1071, COW-1072, COW-1073, COW-1074, COW-1075, COW-1076, COW-1077, COW-1079, COW-1080.
Linear milestone: M4 - production hardening + E2E. Companion: M3 (
dev/m3).