M5 epic: multi-chain deploy + packaging + docs reconciliation#98
Open
jean-neiverth wants to merge 22 commits into
Open
M5 epic: multi-chain deploy + packaging + docs reconciliation#98jean-neiverth wants to merge 22 commits into
jean-neiverth wants to merge 22 commits into
Conversation
Mirrors what BLEU-851 (price-alert) and BLEU-852 (stop-loss) did for
the M3 example modules. Closes the parallel M2 gap.
Before: the entire dispatch pipeline (indexer / poll / submit /
retry / lifecycle) lived in `lib.rs` alongside the wit-bindgen
glue, calling `chain::request`, `local_store::*`, `cow_api::submit_order`,
and `logging::log` directly. The 13 existing tests covered only
parsers and encoders - the state machine itself was unverified in
unit.
After:
1. `strategy.rs` (new) - pure logic against `shepherd_sdk::host::Host`.
Defines `LogView<'a>` and `BlockInfo` so the strategy stays
wit-independent; exposes `on_logs` / `on_block` entry points.
2. `lib.rs` (rewritten, 665 -> 165 lines) - wit-bindgen `generate!`,
`WitBindgenHost` adapter implementing all four SDK host traits,
`Guest` impl that destructures `types::Event` and delegates to
`strategy`.
3. Tests against `shepherd_sdk_test::MockHost` (7 new) cover the
dispatch matrix that was previously hand-verified only:
- `index_records_new_watch_on_conditional_order_created`
- `index_overwrites_in_place_on_redelivered_log` (re-org
replay guard, BLEU-826 invariant)
- `poll_skips_when_next_block_gate_is_in_future`
- `poll_ready_submits_order_and_persists_submitted_uid`
- `submit_transient_error_leaves_state_unchanged_for_next_block`
- `submit_permanent_error_drops_watch`
- `poll_dont_try_again_drops_watch_and_gates` (uses a real
`OrderNotValid` selector via the SDK-exported sol! interface)
4. All 13 original pure tests preserved unchanged. Total: 20 tests
(was 13).
Numbers:
- `.wasm` 313,926 bytes (release wasm32-wasip2).
- 20 tests passing; clippy clean on host + wasm32-wasip2.
- 0 em-dashes in the module tree.
Stacks on PR #23 (BLEU-852) so reviewers can compare strategy /
lib.rs split side-by-side with the price-alert and stop-loss
references.
…855) Same shape as BLEU-854 (twap-monitor / PR #24). Closes the M2-side gap on ethflow-watcher. Before: `submit_placement`, `prior_outcome`, `apply_submit_retry`, and the `submitted:` / `dropped:` / `backoff:` bookkeeping called `local_store::*` and `cow_api::submit_order` directly, with all the state-machine bits unverified in unit (only 7 decoder / encoder tests). After: 1. `strategy.rs` (new) - pure logic against `shepherd_sdk::host::Host`. `LogView<'a>` keeps the strategy wit- independent; `on_logs` is the entry point. 2. `lib.rs` (rewritten, 427 -> 157 lines) - wit-bindgen `generate!`, `WitBindgenHost` adapter, `Guest` impl that destructures `types::Event::Logs` into `LogView`s and delegates to `strategy::on_logs`. 3. Tests against `shepherd_sdk_test::MockHost` (5 new) cover the dispatch + idempotency matrix: - `placement_log_submits_order_and_persists_submitted_uid` - `redelivered_placement_is_skipped_via_submitted_uid_dedup` (PR #10 / commit c5e4d7d regression guard) - `submit_transient_error_writes_backoff_marker_and_returns` - `submit_permanent_error_persists_dropped_uid_and_clears_backoff` - `eip1271_signature_shape_round_trips_through_submit_body` (decodes the JSON body MockCowApi received and asserts `signingScheme=eip1271`, signature blob verbatim, `from` = EthFlow contract) 4. All 7 original pure tests preserved unchanged. Total: 12 tests (was 7). Numbers: - `.wasm` 281,518 bytes (release wasm32-wasip2). - 12 tests passing; clippy clean on host + wasm32-wasip2. - 0 em-dashes in the module tree. Stacks on PR #24 (BLEU-854) so reviewers can compare both M2 strategy / lib.rs splits in one stack with the M3 examples.
…okup (COW-1074)
Closes the gap surfaced by the COW-1064 dry run (2026-06-18):
TWAP orders created through cow-swap UI sign with a non-empty
`appData` hash pointing at a richer JSON document (partner-id,
slippage settings, quote-id). twap-monitor hard-coded
`EMPTY_APP_DATA_JSON` when assembling `OrderCreation`, so the
orderbook rejected every submit with `invalid OrderCreation:
app_data JSON digest does not match signed app_data hash` and
the watch sat in retry-loop forever.
The WIT already exposes `cow-api::request(method, path, body)`
as a generic REST passthrough. We surface that capability on
the SDK trait, wrap it in a typed helper, and use the helper
from the strategy. No new host imports, no WIT ABI change, no
forced rebuild of unrelated modules.
Extended `CowApiHost` with:
```rust
fn cow_api_request(
&self,
chain_id: u64,
method: &str,
path: &str,
body: Option<&str>,
) -> Result<String, HostError>;
```
404 responses surface as `HostError { code: 404, kind:
Unavailable }` so callers can distinguish "orderbook does not
have this resource" from genuine upstream failures without
introducing a new `HostErrorKind` variant (the existing enum
is `non_exhaustive`, but adding a variant on the WIT side
would still need an ABI bump).
`resolve_app_data(host, chain_id, hash) -> Result<String,
HostError>` with:
- Short-circuits `EMPTY_APP_DATA_HASH` (`keccak256("{}")`)
to `EMPTY_APP_DATA_JSON` (`"{}"`) — no host call needed.
- Otherwise GETs `/api/v1/app_data/{hex_hash}` and pulls
the `fullAppData` field out of the orderbook's envelope
shape (`{"fullAppData": "<JSON string>", ...}`).
- 5 unit tests pinning the short-circuit, the success path,
the unexpected-shape fall-through, the 404 propagation,
and the hex encoder.
Extended `MockCowApi` with:
- `respond_to_request_for(method, path, result)`: per-key
programmable response.
- `respond_to_request(result)`: catch-all default.
- `request_calls()`: records the (chain_id, method, path,
body) tuple for every invocation.
The existing `respond` / `calls()` / `submit_order` surface
is unchanged.
`modules/twap-monitor/src/lib.rs`,
`modules/ethflow-watcher/src/lib.rs`,
`modules/examples/price-alert/src/lib.rs`, and
`modules/examples/stop-loss/src/lib.rs` each gained the
trivial 8-line forwarder to the generated
`cow_api::request` binding. Example modules implement
`CowApiHost` purely for the `Host` blanket-impl supertrait
even though some don't actively submit orders — the impl
is symmetrically extended.
`build_order_creation` now takes the resolved
`app_data_json` as an explicit parameter (was hard-coded to
`EMPTY_APP_DATA_JSON`). The resolution itself is lifted into
the caller `submit_ready`, which calls
`shepherd_sdk::cow::resolve_app_data` before assembling the
`OrderCreation`. Two graceful-fallback branches:
- `err.code == 404` → log Warn "appData hash not mirrored
on orderbook" + leave the watch in place. Operators can
re-trigger by pinning the document via a future
orderbook PUT or by re-creating the order with empty
appData.
- Any other resolver error → log Warn "appData resolve
failed" + leave the watch. Future retry on the next
block re-attempts the lookup.
Two new strategy tests:
- `poll_ready_resolves_non_empty_app_data_then_submits`:
programs MockHost with a known JSON + its hash on the
order, asserts the full resolve → submit → `submitted:`
marker flow.
- `poll_ready_skips_submit_when_app_data_hash_not_mirrored`:
programs MockHost to 404, asserts no submit attempt, no
`submitted:` / `dropped:` markers, Warn log line present.
Plus one updated test
(`build_order_creation_accepts_matching_non_empty_app_data`)
that pins the new "matching hash → JSON" success path
directly on `build_order_creation`.
- `cargo test --workspace` → 13 + 12 + 16 + 32 + 8 + 8 +
61 (engine) + 23 (twap-monitor) + 7 doctests + 1
integration = 181 tests passing (was 174; +5 SDK +2
twap-monitor).
- `cargo clippy --all-targets --workspace -- -D warnings`
clean.
- `cargo fmt --all --check` clean.
- All 4 production module .wasm artefacts build cleanly
with the new SDK trait.
- No WIT changes. Modules built against the prior SDK
trait will fail to compile (the new method is required),
but the WIT-generated wasm-side surface is bit-identical.
- No host-impl changes (`crates/nexum-engine/src/host/
impls/cow_api.rs`). The host already implements `request`
for the wit-bindgen binding.
- No metric surface drift. The orderbook lookup goes
through the same `shepherd_cow_api_*` counters via the
existing `request` path.
Linear: COW-1074. Stacks on the COW-1064 run-config branch
(#46). Validated locally end-to-end via `cargo test
--workspace`; live validation against the running engine
will happen on the next COW-1064 dry run (engine restart
required to pick up the rebuilt modules).
Squash of PR #66 - applies 5 blockers + 8 majors from M4 audit.
…W-1084) Adds `tools/baseline-latency/baseline_latency.py`, a per-chain script that pairs every on-chain `EthFlow.OrderPlacement` event in a trailing window with the orderbook's record for the same UID and reports `(creationDate - block.timestamp)`. Matching is rigorous: the script ABI-decodes the GPv2OrderData from each event, computes the EIP-712 order digest against the chain's GPv2Settlement domain, and looks up the resulting UID against the orderbook's bulk `/account/.../orders` fetch (single-UID fallback if missed). No temporal-FIFO approximation. For EthFlow orders the orderbook indexer sets `creationDate := block.timestamp` (not the indexer's ingest time), so the historical delta is structurally 0s on every chain. This is intentional back-fill-style behaviour, not a measurement bug. **Implication**: EthFlow indexer latency cannot be derived from historical orderbook data — the meaningful relayer-latency baseline lives on the TWAP lane (where the orderbook records the indexer's `now()` per child order PUT). TWAP child-latency is a follow-up; it needs per-part UID derivation from each parent `ConditionalOrderCreated` static input. Sepolia ran clean: 256 events scanned, 200 UID-derived pairs, all 200 matched against the bulk fetch (`bulk_hit=200`). Median = p95 = 0.0s, exactly as the finding predicts. Public-tier RPCs (drpc.org free, 1rpc.io, ankr w/o key, llamarpc, cloudflare-eth) all refuse / throttle `eth_getLogs` at any usable chunk size on the production chains. The script halves down to 50-block chunks and gives up after 3 consecutive failures, marking the cell `RPC-LIMITED` with a pointer to the `RPC_URL_*` env override. This is the same constraint the M5 soak (COW-1031) will face and independently confirms the paid-endpoint requirement for any serious log-scanning workload. - `tools/baseline-latency/baseline_latency.py` (~520 lines): argparse CLI, per-chain `Chain` dataclass, JSON-RPC helper with halving retry + `RpcLimited` sentinel, EIP-712 order digest + UID derivation, UID-keyed orderbook matching, markdown report renderer. - `tools/baseline-latency/data/*.json`: per-chain raw dump (events, pairs, deltas, diagnostics) for auditability. - `docs/operations/baselines/baseline-latency-2026-06-19.md`: the first run's report. Pinning the orderbook's `creationDate` semantics matters because the COW-1079 and COW-1031 KPIs reference "watchtower latency" — the M4 report needs to be honest about which lane the latency lives on (TWAP relayer PUT, not EthFlow indexer ingest). The Sepolia data set also gives the M4 e2e harness ground-truth UID ↔ block pairings to cross-check against.
…W-1082) The chain backend previously dropped alloy's structured `RpcError::ErrorResp` payload on the floor — the formatted error string went into `HostError.message`, but `HostError.data` stayed `None` and `HostError.code` was hard-coded to `-32603`. That made the twap-monitor's poll-time revert classifier inert on real traffic: `OrderNotValid` / `PollNever` / `PollTryAtBlock` / `PollTryAtEpoch` all fell through to `TryNextBlock` because `decode_revert_hex` only fires on a non-empty `err.data`. This change wires the structured payload through end-to-end. - `crates/nexum-engine/src/host/provider_pool.rs`: when alloy's `provider.raw_request` fails with an `RpcError::ErrorResp`, the pool now captures both `payload.code` (as `Option<i64>` so we can distinguish "no ErrorResp" from "ErrorResp with code 0") and `payload.data` (as `Option<String>`, the JSON-encoded revert hex) and surfaces them on `ProviderError::Rpc`. Transport-side failures (timeout, websocket disconnect) leave both `None`. The two subscribe paths (`subscribe_blocks`, `subscribe_logs`) keep `code: None, data: None` since they don't carry an ErrorResp. - `crates/nexum-engine/src/host/impls/chain.rs`: extract the `ProviderError -> HostError` projection into a free helper `provider_error_to_host_error`. The `Rpc` arm forwards the structured `data` verbatim, preserves the node-reported code (saturating out-of-`i32` values to `-32603`), and falls back to `-32603` only when no `ErrorResp` was present. Five unit tests cover: revert with data, transport failure with `None`, out-of-range code, unknown-chain, and invalid-params. - `modules/twap-monitor/src/strategy.rs`: update the stale comment on the `decode_revert_hex` branch — that branch is now live on real traffic, the only `None` path is transport-level failures (which keep the safe `TryNextBlock` default). No incorrect order is ever submitted (the contract reverts; the orderbook never sees a bad body). The issue is pruning efficiency: a permanently dead TWAP watch was re-polled every block until a submit eventually failed for an unrelated reason, and the local-store filled with `watch:` entries the strategy could otherwise drop on the first revert. With this fix the SDK-side classifier dispatches `Drop` / gate on the first revert, matching the documented expectation in `docs/adr/0007-upstream-protocol-logic-to-cow-rs.md`. - 70/70 nexum-engine tests pass - 23/23 twap-monitor tests pass - 5/5 new chain.rs projection tests pass (revert-with-data, transport-fail, out-of-range-code, unknown-chain, invalid-params) - `cargo clippy -p nexum-engine -p twap-monitor --all-targets -- -D warnings` clean jeffersonBastos's PR #55 (M3 mirror) review, thread on `modules/twap-monitor/src/strategy.rs:189`. The mirror of this fix on the cow-api side is COW-1075 (already merged via PR #48).
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
Closes the M5 packaging gap surfaced by the audit: the Dockerfile +
compose recipe lived inside `docs/production.md` but neither was at
the repo root, so `docker build .` didn't work and there was no
published image. This change makes the deploy path one-line on a
fresh VM.
- **`Dockerfile`** — multi-stage build (rust:1.96-slim-bookworm →
debian:bookworm-slim). Builds the engine in release + the 5
production modules to wasm32-wasip2. Runtime stage strips down to
`tini` (PID 1 for graceful shutdown / SIGINT forwarding per
COW-1072) + `ca-certificates` (TLS to cow.fi + paid RPCs) + a
non-root `shepherd` user owning `/var/lib/shepherd`. Final image:
**198 MB** (engine + 5 wasm modules + Debian slim).
- **`.dockerignore`** — excludes `target/`, `data/`, the heavy
backtest / baseline JSON fixtures, and local-only engine configs,
while keeping `modules/fixtures/*-bomb` (workspace members; Cargo
rejects the manifest if they're missing) and the source markdown
docs (so `docker exec` can grep them in place).
- **`docker-compose.yml`** — two profiles. Default boots just the
engine with a `shepherd-state` named volume + the operator's
`./engine.toml` mounted ro at `/etc/shepherd/engine.toml`, metrics
on the host loopback (`127.0.0.1:9100`). The `observability`
profile (`docker compose --profile observability up`) layers a
Prometheus container pre-wired to scrape `shepherd:9100`. Graceful
shutdown via `stop_signal: SIGINT` + `stop_grace_period: 30s` per
the production runbook. Healthcheck hits `/metrics`.
- **`engine.docker.toml`** — pre-baked config that matches the
paths the image bakes (`/opt/shepherd/modules/*.wasm`,
`/opt/shepherd/manifests/*.toml`, `/var/lib/shepherd` state
dir). Operator workflow: `cp engine.docker.toml engine.toml`,
swap `<RPC_KEY>` placeholders, `docker compose up -d`.
- **`docs/deployment/docker.md`** — operator runbook. Covers
first-boot, engine.toml configuration, upgrade / rollback,
local-build path, post-deploy verification, cross-links to
`docs/production.md` for the full hardening surface.
- **`docs/deployment/prometheus.yml`** — scrape config consumed by
the observability compose profile.
- **`.github/workflows/docker.yml`** — build + push to
`ghcr.io/bleu/nullis-shepherd` on every push to `main` and every
`v*` tag. PR builds run the build for smoke (no push). Tags
produced: `latest` (main HEAD), `v<tag>` (releases),
`sha-<short>` (every event for exact pinning), `manual-<run_id>`
(workflow_dispatch). Registry-side layer cache via
`:buildcache` keeps incremental rebuilds fast. linux/amd64 only —
the soak VM is x86_64; add arm64 once an operator surfaces a
real need. Action SHAs pinned to match `.github/workflows/ci.yml`
style.
Build runs locally end-to-end in ~10 min on a clean Docker daemon:
$ docker build -t shepherd:smoke .
$ docker run --rm shepherd:smoke --help
usage: nexum-engine [<wasm-path> [<manifest-path>]] \
[--engine-config <path>] [--pretty-logs]
$ docker run --rm -v "$PWD/engine.docker.toml:/etc/shepherd/engine.toml:ro" \
shepherd:smoke
{"level":"INFO","message":"nexum-engine starting",...}
{"level":"INFO","message":"metrics exporter listening at /metrics",...}
{"level":"INFO","message":"opening chain RPC provider","chain_id":1,...}
Error: connect chain 1: HTTP format error: invalid uri character
^- expected: <RPC_KEY> placeholder not a real URL
Proves: image builds, entrypoint forwards CMD, engine loads
`/etc/shepherd/engine.toml`, metrics exporter binds, provider pool
iterates the configured chains, graceful error path works.
- [x] Local `docker build .` succeeds (rust:1.96 base — wasmtime 45
requires rustc >= 1.93, the docs/production.md `1.86` pin was
stale)
- [x] Image size: 198 MB
- [x] `docker run ... --help` works
- [x] `docker run ... -v engine.docker.toml:...` reads config + binds
metrics + iterates chains
- [x] `cargo test --workspace` clean (18 groups, 203 passed, 0 failed)
On a fresh Debian/Ubuntu VM with Docker installed:
```bash
git clone https://github.com/bleu/nullis-shepherd /opt/shepherd
cd /opt/shepherd
cp engine.docker.toml engine.toml
$EDITOR engine.toml # add real RPC URL
docker compose pull # once ghcr.io image is published
docker compose up -d
docker compose logs -f shepherd
curl -s http://127.0.0.1:9100/metrics | head -50
```
- `docs/deployment/multi-chain-guide.md` — dedicated walkthrough
configuring 4 chains together (Mainnet + Gnosis + Arbitrum + Base)
with per-chain module subscriptions
- Example module declaring multi-chain support (every current
example pins Sepolia)
- Optional automated CD trigger (workflow_dispatch SSH'ing to the
soak VM to pull + restart) — gated on SSH_PRIVATE_KEY repo secret
Companion to the M5 Docker packaging — the operator workflow is `cp engine.docker.toml engine.toml` then drop in a paid RPC URL. Without this rule a clumsy `git add -A` could commit the key. The committed sibling templates (engine.example/docker/m2/m3/e2e/load.toml) stay trackable. Validated against a live smoke run: drpc Sepolia WSS endpoint pasted into engine.toml, `docker compose up`, engine subscribed to newHeads + logs, 6 sequential blocks dispatched (11117171..76), metrics `shepherd_event_latency_seconds` p99 = 0.14ms. Tear-down clean. No engine.toml ever staged.
Closes the footgun surfaced by the M5 smoke run on drpc Sepolia:
configuring `rpc_url = "https://..."` for a chain that the modules
subscribe to silently degrades to an infinite WARN-with-backoff loop
(COW-1071's reconnect retries forever because `eth_subscribe` is
WS-only in the JSON-RPC spec). Three coordinated changes:
`EngineConfig::validate_transports()` walks every `[chains.<id>]`
entry, and for any `rpc_url` not starting with `ws://` / `wss://`
emits one loud ERROR-level structured log line with:
- the chain id
- the redacted offending URL
- the redacted suggested `wss://` swap
- actionable copy explaining the WS requirement and the escape
hatch (`[chains.<id>] require_ws = false` for poll-only chains
that never subscribe)
The validator is invoked from `main.rs` AFTER the tracing
subscriber is initialised (calling it inside `load_or_default`
silently dropped the log).
A `require_ws: bool` field is added to `ChainConfig` with
`#[serde(default = "default_require_ws")]` = `true`. Operators who
genuinely need an HTTP endpoint (poll-only modules, no block / log
subscriptions on this chain) opt out explicitly per chain.
The pre-existing `opening chain RPC provider` log in
`provider_pool::from_config` was emitting the full URL — API key
included — at INFO level. Log aggregators (Loki / Datadog / Splunk)
routinely retain weeks of these lines; the key has no business
sitting in cold storage. The new `engine_config::redact_url` helper
(public so other call sites can adopt it) replaces any path segment
longer than 20 chars that doesn't contain `.` or `:` with `<KEY>`.
Matches Alchemy / drpc / Infura / QuickNode key shapes.
Same helper is used for both the validation ERROR's `rpc_url` and
`suggested` fields and the provider-pool boot log.
- `engine.example.toml`: every chain entry switched to `wss://`,
with a header block explaining the WS requirement + the
`require_ws = false` escape hatch. The previous mix of `https://`
+ `wss://` would have tripped the new validator on its own example.
- `docs/production.md §6`: blockquote callout pointing operators at
the WS requirement, redaction behaviour, and the escape hatch.
Smoke 1 (HTTP, expected to ERROR):
{"level":"ERROR","message":"rpc_url uses HTTP transport but the engine subscribes to blocks/logs via eth_subscribe (WS-only). [...]","chain_id":11155111,"rpc_url":"https://lb.drpc.live/sepolia/<KEY>","suggested":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
Smoke 2 (WSS, expected to pass + redacted):
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,"url":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
- 9 new unit tests in `engine_config::tests`:
* `validate_accepts_wss_url`, `validate_accepts_ws_url`
* `validate_is_silent_when_require_ws_is_false`
* `validate_runs_without_panicking_on_http_url`
* `suggest_swaps_https_to_wss`, `suggest_swaps_http_to_ws`,
`suggest_passes_through_already_ws_url`
* `redact_replaces_long_path_segments`,
`redact_keeps_short_segments_intact`
- Workspace: 18 groups, **212 passed, 0 failed** (was 203 → +9)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
Operator workflow before this change forced the paid-RPC URL to
live in a file (`engine.toml`), which is fine for systemd but
awkward for Docker/compose: the URL had to be hand-edited inside a
volume-mounted file, secrets and config got tangled, and the
internal drpc test key was at risk of slipping into a committed
example. This change makes the engine treat `${VAR_NAME}` tokens
inside `engine.toml` as environment-variable references, resolved
at config-load time:
[chains.11155111]
rpc_url = "${SEPOLIA_RPC_URL}"
The `engine.docker.toml` and `engine.example.toml` templates ship
with `${VAR}` placeholders for all five chains, so the committed
files stay secret-free regardless of deployment path.
cp .env.example .env
$EDITOR .env # paste real wss:// URLs
docker compose up -d
`docker compose` reads the repo-root `.env` automatically (already
the compose default) and forwards the named variables into the
container via the new `environment:` block; the engine substitutes
them when parsing `/etc/shepherd/engine.toml`.
- `engine_config.rs::substitute_env_vars` — hand-rolled parser
(no regex dep) that walks the raw TOML text, matches `${NAME}`
tokens against `[A-Z_][A-Z0-9_]*`, and looks each up via
`std::env::var`. Three error variants via `thiserror`:
* `Missing { name }` — variable referenced but unset; message
includes the exact name and a pointer to the `.env` workflow.
* `InvalidName { name }` — typo (lowercase, leading digit);
suggests the upper-cased variant.
* `Unclosed { offset }` — `${` without matching `}`.
- Called from `load_or_default` before `toml::from_str`, so the
substitution layer never sees parsed TOML — a missing env var
surfaces with the exact variable name, not a downstream
"invalid URI character" several layers deep.
- Substitution runs over the whole file (comments included; harmless).
- `.env.example` — committed template with placeholders for all 5
chain `*_RPC_URL` variables + the optional `SHEPHERD_IMAGE` and
`SHEPHERD_ENGINE_CONFIG` overrides.
- `.gitignore` — adds `!.env.example` exception so the template
stays trackable while `.env` and `.env.local` etc. stay ignored.
- `docker-compose.yml` — passes the five `*_RPC_URL` env vars
through to the container; the engine config bind-mount now
defaults to `engine.docker.toml` (the committed template) and
honours `SHEPHERD_ENGINE_CONFIG` for operators who prefer a
bespoke file.
- `engine.docker.toml` + `engine.example.toml` — every `[chains.*]`
entry switched to `${*_RPC_URL}` placeholders. Header comments
spell out the workflow.
- `docs/deployment/docker.md` — first-boot section now leads with
`cp .env.example .env` (was `cp engine.example.toml engine.toml
&& edit`). §2 explains the bind-mount + the
`SHEPHERD_ENGINE_CONFIG` escape hatch.
Smoke 1 (compose end-to-end):
$ cp .env.example .env
$ echo "SEPOLIA_RPC_URL=wss://lb.drpc.live/sepolia/<real-key>" >> .env
$ echo "SHEPHERD_ENGINE_CONFIG=./engine.local.toml" >> .env
$ docker compose up -d
...
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,
"url":"wss://lb.drpc.live/sepolia/<KEY>",...} ← env-resolved, key redacted
{"level":"INFO","message":"supervisor up","loaded":2,"alive":2,...}
{"level":"INFO","message":"block subscription open","chain_id":11155111,...}
{"level":"INFO","message":"log subscription open","module":"twap-monitor",...}
{"level":"INFO","message":"log subscription open","module":"ethflow-watcher",...}
$ docker compose logs | grep -c <real-key>
0 ← zero leaks
$ curl -s http://127.0.0.1:9100/metrics | grep latency_seconds_count
shepherd_event_latency_seconds_count{module="twap-monitor",event_kind="block"} 4
Smoke 2 (missing env var, expected fail-fast):
$ unset SEPOLIA_RPC_URL
$ docker compose up
Error: engine config env-var substitution failed: environment variable
`SEPOLIA_RPC_URL` referenced via ${SEPOLIA_RPC_URL} in engine.toml but
not set. Export it before launching the engine (e.g. via a `.env`
file consumed by `docker compose`).
- 7 new unit tests in `engine_config::tests`:
* `substitute_replaces_known_variable`
* `substitute_errors_on_missing_variable`
* `substitute_errors_on_invalid_name`
* `substitute_errors_on_unclosed_brace`
* `substitute_passes_text_with_no_placeholders_through`
* `substitute_handles_multiple_placeholders_in_one_line`
* `substitute_preserves_utf8_around_placeholder`
- Workspace: 18 groups, **219 passed, 0 failed** (was 212 → +7)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
VM smoke surfaced a false-negative `(unhealthy)`: the compose healthcheck called `wget` but the runtime image is built on debian:bookworm-slim which doesn't include it (only ca-certificates + tini, intentionally minimal). `wget: not found` → exit 127 → unhealthy mark, despite the engine actually working (21 blocks dispatched in 3 min, p99 latency 0.09ms, zero errors). Swap to bash's `/dev/tcp` builtin (always present in bookworm-slim's `/bin/bash`). Successful TCP open on the metrics port proves the exporter bound, which only happens after the supervisor finishes boot — same semantic, no image growth.
First fix attempt swapped wget for `/dev/tcp` but kept `CMD-SHELL`, which routes through `/bin/sh` (dash on debian:bookworm-slim). dash doesn't have the `/dev/tcp/<host>/<port>` builtin — it's bash- only. Probes failed with "cannot create /dev/tcp/...: Directory nonexistent". Switch to `CMD ["bash", "-c", ...]` so the bash builtin actually resolves. `bash` ships in the slim base; verified via `docker exec shepherd which bash` → `/usr/bin/bash`.
Cherry-pick of PR #62 + PR #63's redesign onto the M5 host runtime (env-var substitution in engine.toml, healthcheck fixes, etc) for the Sepolia soak VM. The PR review continues on the proper layered branches: - PR #62 — M2/BLEU-833 layer observe design - PR #63 — M3/M4 BLEU-855 split + COW-1074 cow_api_request integration This branch is deploy-only: it lets the soak run on the redesigned ethflow-watcher with the latest host runtime while review iterates on the layered PRs. After merge, this branch can be deleted; CI will republish ghcr.io/bleu/nullis-shepherd:latest with the merged design and the VM rolls forward to the official image. See COW-1076 for the full empirical evidence.
…econnects (COW-1086)
Adds a positive-recovery Info log when the block subscription
resumes after a silence ≥ 60 s, covering the observability gap
identified in the 2026-06-23 Sepolia soak.
## Background
The 2026-06-23 soak surfaced this sequence:
09:05:43 ERROR WS connection error: WebSocket protocol error:
Connection reset without closing handshake
target=alloy_transport_ws::native
(no further WS-related lines for ~1 h)
10:02:24 INFO indexed watch:... ← twap-monitor activity resumes
10:05:24 INFO ethflow observed ... ← ethflow-watcher activity resumes
`docker ps` showed 0 restarts and the container stayed healthy
throughout — alloy's transport layer reconnected internally without
the engine's `reconnecting_block_task` ever observing
`inner.next().await -> None`. So the engine never entered its
"stream ended → backoff → subscription reopened" path, and the
existing `block subscription reopened` Info log (COW-1071) never
fired. The transport-layer ERROR followed by silence is
indistinguishable from a hung engine on a soak dashboard.
## What changes
In `reconnecting_block_task`, on every yielded item compare
`now.duration_since(last_event)` against `BLOCK_GAP_LOG_THRESHOLD`
(60 s, 5× Sepolia block time). When the gap meets or exceeds the
threshold, emit:
INFO chain_id=... gap_s=... kind="block"
"stream gap closed - first event after silence
(likely an alloy-internal transport reconnect)"
The gap-detection logic is factored into a small synchronous helper
`block_stream_gap_to_log(now, last_event, threshold) -> Option<Duration>`
so it can be unit-tested without spinning up an async runtime or a
real provider.
## Why blocks only (not logs)
Block subscriptions have predictable cadence — Sepolia produces a
new block every ~12 s, mainnet every ~12 s. A 60 s silence is
therefore anomalous and worth surfacing. Log subscriptions, by
contrast, are inherently sparse (driven by on-chain user activity),
so the same threshold would fire false positives on quiet windows.
The existing `log subscription reopened` log already handles the
engine-detectable reconnect for log streams.
## Tests
4 new unit tests on the gap-detection helper:
* `block_stream_gap_to_log_returns_none_when_no_prior_event`
* `block_stream_gap_to_log_returns_none_when_under_threshold`
* `block_stream_gap_to_log_returns_some_at_threshold_boundary`
* `block_stream_gap_to_log_returns_some_when_well_over_threshold`
All 90 nexum-engine tests pass (86 existing + 4 new). Clippy strict
clean, fmt clean. Wasm build untouched.
## Out of scope
* End-to-end test of `reconnecting_block_task` against a mock
provider — no existing scaffolding for that path, and the gap
helper covers the decision logic deterministically.
* Suppressing or downgrading the `alloy_transport_ws::native` ERROR
itself — it is a legitimate transport-layer event, just one whose
recovery wasn't previously observable. The new Info line closes
that loop without losing the original signal.
## Live validation
The next time alloy auto-reconnects internally on the soak VM, the
new line will surface as a structured JSON event with
`gap_s=<seconds>` so the soak dashboard can correlate it with the
preceding transport ERROR.
) Squash of PR #68 - 9 markdown files reconciled (5 vapor items rephrased as future direction + capability-gating diagrams aligned to link-time enforcement). Verified: cargo doc --workspace --no-deps clean.
…terError Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #1 (remaining enums introduced on the M5 multi-chain pass). - `EnvVarError` (engine_config.rs): introduced with the COW-1071 env-var substitution path. Snake_case variant labels feed the boot-time `tracing::error!(error_kind = ...)` call sites in `main.rs`. - `FilterError` (supervisor.rs): introduced with the M5 multi-chain log-filter parsing. Snake_case variant labels feed the `tracing::warn!(error_kind = ...)` log emitted when a `[[subscription]]` address or topic fails to parse. The audit's M3 / M4 derives landed on the milestones that introduced the enums; these two complete the workspace-wide IntoStaticStr pass flagged in audit Major #1 on the milestones that own them.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #6. The rubric forbids em-dashes in "code, rustdoc, commit messages, PR bodies, or review comments". `.toml` is technically a grey zone but these comments surface verbatim when operators `cat engine.docker.toml` or `engine.example.toml` during deployment onboarding. Mechanical find/replace to ` - ` (ASCII hyphen with spaces). Files touched: - engine.example.toml: 2 em-dashes (lines 20, 38) - engine.docker.toml: 4 em-dashes (lines 4, 5, 6, 31)
…rse (audit JC5) shepherd-backtest's offline replay harness carried its own `AddressParseError` enum (hex-decode + length check). The shape overlaps directly with the `AddressParse` typed error introduced into `shepherd-sdk` by the audit JC5 pass. Extend `shepherd_sdk::address` with a single-address `parse_address` helper alongside the existing `parse_address_list` (the `InvalidAddress` variant covers both call sites via the `index` field). Replay's `fixtures::parse_address` becomes a thin wrapper that calls the SDK and converts the `Address` to the `[u8; 20]` shape the strategy consumes via `LogView::address`. Drops the now-unused `thiserror` dependency from shepherd-backtest; `hex` stays for topic/data decoding.
Restored files from pre-rebase M5 (c50e47c) that had unresolvable conflict markers from the cascaded M3→M4→M5 rebase. The M2 fixes (cow-api HTTP status, timeouts, dangerous method warning) flow through the engine crate which rebased cleanly; the module-side files needed the M5-specific versions (observe+verify redesign, ErrorResp.data forwarding) restored wholesale.
This was referenced Jun 30, 2026
Closed
Closed
Closed
This was referenced Jun 30, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on M4 (
dev/m4). M5 closes out the grant: packages the M4 daemon for operators (Docker + ghcr CI), adds the pre-soak backtest harness + baseline-latency tooling, lands the small protocol-side hardening items surfaced during M4 soak runs, and reconciles the docs across M2-M5 so the on-disk story matches the shipped behaviour.Core deliverable
Dockerfile(multi-stage rust build),docker-compose.ymlfor the daemon + scrape stack, GHCR push on tag.shepherd-backtestcrate +tools/backtest-collect/replay a 7-day Sepolia EthFlow window before soak runs, giving an offline regression bar for module behaviour (COW-1078).tools/baseline-latency/measures TWAP-relayer PUT and EthFlow indexer ingest latencies across 5 chains so soak reports can attribute regressions to the right lane (COW-1084).engine_config.rs+host/impls/chain.rsforwardeth_callErrorResp.dataintoHostError.dataso module code can decodeIConditionalOrderreverts the same way it decodes orderbook errors (COW-1082).ethflow-watchercapsbackoff:{uid}retries atMAX_BACKOFF_RETRIESso a permanently-failing orderbook submission stops eating fuel (COW-1083).twap-monitorconsultssubmitted:{uid}before re-submitting, preventing duplicate orderbook posts on supervisor restart (COW-1085).runtime/event_loop.rslogs the WS-reconnect gap atInfo, giving operators a visible signal during reconnects (COW-1086).engine_config.rsresolves${VAR}placeholders inengine.toml; HTTPrpc_urlconfigs fail-fast at boot; API keys in RPC URLs are redacted from boot logs.docs/updated end-to-end: ADR statuses, deployment guide, e2e runbook, operations guides re-flowed to match shipped behaviour.Validation
cargo fmt --all -- --checkclean.cargo clippy --workspace --all-targets -- -D warningsclean.cargo test --workspace— 252 tests passing; backtest harness has its own integration tests.docker build .; compose stack boots end-to-end against a local Anvil + RPC + orderbook-mock.M5-specific paths for review
Dockerfile,docker-compose.yml,.github/workflows/docker.ymlcrates/shepherd-backtest/tools/backtest-collect/,tools/baseline-latency/crates/nexum-engine/src/engine_config.rs(env-var substitution, fail-fast HTTP, key redaction)crates/nexum-engine/src/host/impls/chain.rs(forwardeth_callErrorResp.data)modules/ethflow-watcher/src/strategy.rs(observe+verify redesign, backoff cap)modules/twap-monitor/src/strategy.rs(skip-submitted-uid)crates/nexum-engine/src/runtime/event_loop.rs(WS reconnect gap log)docs/reconciliation sweep across the M2-M5 surfaceCloses COW-1078, COW-1082, COW-1083, COW-1084, COW-1085, COW-1086.
Linear milestone: M5 - multi-chain deploy + docs. Companion: M4 (
dev/m4).