Dev/m5 rebase#93
Closed
jean-neiverth wants to merge 20 commits into
Closed
Conversation
…W-1084) Adds `tools/baseline-latency/baseline_latency.py`, a per-chain script that pairs every on-chain `EthFlow.OrderPlacement` event in a trailing window with the orderbook's record for the same UID and reports `(creationDate - block.timestamp)`. Matching is rigorous: the script ABI-decodes the GPv2OrderData from each event, computes the EIP-712 order digest against the chain's GPv2Settlement domain, and looks up the resulting UID against the orderbook's bulk `/account/.../orders` fetch (single-UID fallback if missed). No temporal-FIFO approximation. For EthFlow orders the orderbook indexer sets `creationDate := block.timestamp` (not the indexer's ingest time), so the historical delta is structurally 0s on every chain. This is intentional back-fill-style behaviour, not a measurement bug. **Implication**: EthFlow indexer latency cannot be derived from historical orderbook data — the meaningful relayer-latency baseline lives on the TWAP lane (where the orderbook records the indexer's `now()` per child order PUT). TWAP child-latency is a follow-up; it needs per-part UID derivation from each parent `ConditionalOrderCreated` static input. Sepolia ran clean: 256 events scanned, 200 UID-derived pairs, all 200 matched against the bulk fetch (`bulk_hit=200`). Median = p95 = 0.0s, exactly as the finding predicts. Public-tier RPCs (drpc.org free, 1rpc.io, ankr w/o key, llamarpc, cloudflare-eth) all refuse / throttle `eth_getLogs` at any usable chunk size on the production chains. The script halves down to 50-block chunks and gives up after 3 consecutive failures, marking the cell `RPC-LIMITED` with a pointer to the `RPC_URL_*` env override. This is the same constraint the M5 soak (COW-1031) will face and independently confirms the paid-endpoint requirement for any serious log-scanning workload. - `tools/baseline-latency/baseline_latency.py` (~520 lines): argparse CLI, per-chain `Chain` dataclass, JSON-RPC helper with halving retry + `RpcLimited` sentinel, EIP-712 order digest + UID derivation, UID-keyed orderbook matching, markdown report renderer. - `tools/baseline-latency/data/*.json`: per-chain raw dump (events, pairs, deltas, diagnostics) for auditability. - `docs/operations/baselines/baseline-latency-2026-06-19.md`: the first run's report. Pinning the orderbook's `creationDate` semantics matters because the COW-1079 and COW-1031 KPIs reference "watchtower latency" — the M4 report needs to be honest about which lane the latency lives on (TWAP relayer PUT, not EthFlow indexer ingest). The Sepolia data set also gives the M4 e2e harness ground-truth UID ↔ block pairings to cross-check against.
…W-1082) The chain backend previously dropped alloy's structured `RpcError::ErrorResp` payload on the floor — the formatted error string went into `HostError.message`, but `HostError.data` stayed `None` and `HostError.code` was hard-coded to `-32603`. That made the twap-monitor's poll-time revert classifier inert on real traffic: `OrderNotValid` / `PollNever` / `PollTryAtBlock` / `PollTryAtEpoch` all fell through to `TryNextBlock` because `decode_revert_hex` only fires on a non-empty `err.data`. This change wires the structured payload through end-to-end. - `crates/nexum-engine/src/host/provider_pool.rs`: when alloy's `provider.raw_request` fails with an `RpcError::ErrorResp`, the pool now captures both `payload.code` (as `Option<i64>` so we can distinguish "no ErrorResp" from "ErrorResp with code 0") and `payload.data` (as `Option<String>`, the JSON-encoded revert hex) and surfaces them on `ProviderError::Rpc`. Transport-side failures (timeout, websocket disconnect) leave both `None`. The two subscribe paths (`subscribe_blocks`, `subscribe_logs`) keep `code: None, data: None` since they don't carry an ErrorResp. - `crates/nexum-engine/src/host/impls/chain.rs`: extract the `ProviderError -> HostError` projection into a free helper `provider_error_to_host_error`. The `Rpc` arm forwards the structured `data` verbatim, preserves the node-reported code (saturating out-of-`i32` values to `-32603`), and falls back to `-32603` only when no `ErrorResp` was present. Five unit tests cover: revert with data, transport failure with `None`, out-of-range code, unknown-chain, and invalid-params. - `modules/twap-monitor/src/strategy.rs`: update the stale comment on the `decode_revert_hex` branch — that branch is now live on real traffic, the only `None` path is transport-level failures (which keep the safe `TryNextBlock` default). No incorrect order is ever submitted (the contract reverts; the orderbook never sees a bad body). The issue is pruning efficiency: a permanently dead TWAP watch was re-polled every block until a submit eventually failed for an unrelated reason, and the local-store filled with `watch:` entries the strategy could otherwise drop on the first revert. With this fix the SDK-side classifier dispatches `Drop` / gate on the first revert, matching the documented expectation in `docs/adr/0007-upstream-protocol-logic-to-cow-rs.md`. - 70/70 nexum-engine tests pass - 23/23 twap-monitor tests pass - 5/5 new chain.rs projection tests pass (revert-with-data, transport-fail, out-of-range-code, unknown-chain, invalid-params) - `cargo clippy -p nexum-engine -p twap-monitor --all-targets -- -D warnings` clean jeffersonBastos's PR #55 (M3 mirror) review, thread on `modules/twap-monitor/src/strategy.rs:189`. The mirror of this fix on the cow-api side is COW-1075 (already merged via PR #48).
…OW-1083)
The strategy's `apply_submit_retry` previously wrote an empty
`backoff:{uid}` marker on every retriable submit failure (including
the `TryNextBlock` fallback for unparseable orderbook envelopes). The
marker was a presence flag with no payload, so on every supervisor
reconnect / engine restart the same dead placement would retry
indefinitely — bounded only by log re-delivery frequency.
This change persists a per-UID retry count in the marker's value
(ASCII `u32`) and upgrades to `dropped:` after `MAX_BACKOFF_RETRIES =
5` consecutive retries. The upgrade emits a Warn-level log line so
the operator sees the structural issue (flaky CDN, indexer hiccup,
poisoned envelope) rather than silently accumulating retries.
- `modules/ethflow-watcher/src/strategy.rs`:
- New const `MAX_BACKOFF_RETRIES = 5`.
- New helper `read_backoff_count` that reads + parses the marker
payload; pre-COW-1083 empty markers decode to 0 so previously-set
backoff: rows still get a fresh attempt (no premature drop on
rollout).
- `apply_submit_retry`'s retriable branch now reads the prior
count, increments, and either writes the new count or upgrades
to `dropped:` (clearing the stale `backoff:`) at the cap.
- Cap-upgrade log line carries the retry-count and message: "...
after 5 retries on transient/unparseable rejection ...".
- 19/19 ethflow-watcher tests pass.
- New `submit_transient_error_at_cap_upgrades_to_dropped_warn`:
seeds `backoff:{uid} = "4"`, triggers a `data: None` rejection
(the unparseable case the issue names explicitly), asserts:
* `dropped:{uid}` is now set
* `backoff:{uid}` is cleared (single outcome marker at rest)
* exactly one Warn log line containing "ethflow dropped" +
"retries"
- New `submit_transient_error_with_legacy_empty_marker_resets_counter`:
backwards-compat — a pre-COW-1083 empty `b""` marker is treated
as count 0, bumped to "1" on first retry rather than prematurely
dropping. Protects in-flight backoffs across the rollout.
- Existing `submit_transient_error_writes_backoff_marker_and_returns`
extended with an assertion that the first retry persists
`backoff:{uid} = "1"`.
- `cargo clippy -p ethflow-watcher --all-targets -- -D warnings`
clean.
Surfaced by jeffersonBastos's PR #55 (M3 mirror) review, thread on
`crates/shepherd-sdk/src/cow/error.rs:82`. Latent in normal
operation (the host forwards parseable envelopes after COW-1075, so
`classify_api_error` returns `Drop` for permanent rejections), but
the gap fires when the orderbook returns a non-JSON 4xx body
(e.g. an HTML error page from a CDN) or if a future host change
accidentally drops the envelope again. Bounded retry semantics
close the latent risk without changing the safe-default
classification (still `TryNextBlock` on `None` data — that part is
explicitly out of scope per the issue).
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
Closes the M5 packaging gap surfaced by the audit: the Dockerfile +
compose recipe lived inside `docs/production.md` but neither was at
the repo root, so `docker build .` didn't work and there was no
published image. This change makes the deploy path one-line on a
fresh VM.
- **`Dockerfile`** — multi-stage build (rust:1.96-slim-bookworm →
debian:bookworm-slim). Builds the engine in release + the 5
production modules to wasm32-wasip2. Runtime stage strips down to
`tini` (PID 1 for graceful shutdown / SIGINT forwarding per
COW-1072) + `ca-certificates` (TLS to cow.fi + paid RPCs) + a
non-root `shepherd` user owning `/var/lib/shepherd`. Final image:
**198 MB** (engine + 5 wasm modules + Debian slim).
- **`.dockerignore`** — excludes `target/`, `data/`, the heavy
backtest / baseline JSON fixtures, and local-only engine configs,
while keeping `modules/fixtures/*-bomb` (workspace members; Cargo
rejects the manifest if they're missing) and the source markdown
docs (so `docker exec` can grep them in place).
- **`docker-compose.yml`** — two profiles. Default boots just the
engine with a `shepherd-state` named volume + the operator's
`./engine.toml` mounted ro at `/etc/shepherd/engine.toml`, metrics
on the host loopback (`127.0.0.1:9100`). The `observability`
profile (`docker compose --profile observability up`) layers a
Prometheus container pre-wired to scrape `shepherd:9100`. Graceful
shutdown via `stop_signal: SIGINT` + `stop_grace_period: 30s` per
the production runbook. Healthcheck hits `/metrics`.
- **`engine.docker.toml`** — pre-baked config that matches the
paths the image bakes (`/opt/shepherd/modules/*.wasm`,
`/opt/shepherd/manifests/*.toml`, `/var/lib/shepherd` state
dir). Operator workflow: `cp engine.docker.toml engine.toml`,
swap `<RPC_KEY>` placeholders, `docker compose up -d`.
- **`docs/deployment/docker.md`** — operator runbook. Covers
first-boot, engine.toml configuration, upgrade / rollback,
local-build path, post-deploy verification, cross-links to
`docs/production.md` for the full hardening surface.
- **`docs/deployment/prometheus.yml`** — scrape config consumed by
the observability compose profile.
- **`.github/workflows/docker.yml`** — build + push to
`ghcr.io/bleu/nullis-shepherd` on every push to `main` and every
`v*` tag. PR builds run the build for smoke (no push). Tags
produced: `latest` (main HEAD), `v<tag>` (releases),
`sha-<short>` (every event for exact pinning), `manual-<run_id>`
(workflow_dispatch). Registry-side layer cache via
`:buildcache` keeps incremental rebuilds fast. linux/amd64 only —
the soak VM is x86_64; add arm64 once an operator surfaces a
real need. Action SHAs pinned to match `.github/workflows/ci.yml`
style.
Build runs locally end-to-end in ~10 min on a clean Docker daemon:
$ docker build -t shepherd:smoke .
$ docker run --rm shepherd:smoke --help
usage: nexum-engine [<wasm-path> [<manifest-path>]] \
[--engine-config <path>] [--pretty-logs]
$ docker run --rm -v "$PWD/engine.docker.toml:/etc/shepherd/engine.toml:ro" \
shepherd:smoke
{"level":"INFO","message":"nexum-engine starting",...}
{"level":"INFO","message":"metrics exporter listening at /metrics",...}
{"level":"INFO","message":"opening chain RPC provider","chain_id":1,...}
Error: connect chain 1: HTTP format error: invalid uri character
^- expected: <RPC_KEY> placeholder not a real URL
Proves: image builds, entrypoint forwards CMD, engine loads
`/etc/shepherd/engine.toml`, metrics exporter binds, provider pool
iterates the configured chains, graceful error path works.
- [x] Local `docker build .` succeeds (rust:1.96 base — wasmtime 45
requires rustc >= 1.93, the docs/production.md `1.86` pin was
stale)
- [x] Image size: 198 MB
- [x] `docker run ... --help` works
- [x] `docker run ... -v engine.docker.toml:...` reads config + binds
metrics + iterates chains
- [x] `cargo test --workspace` clean (18 groups, 203 passed, 0 failed)
On a fresh Debian/Ubuntu VM with Docker installed:
```bash
git clone https://github.com/bleu/nullis-shepherd /opt/shepherd
cd /opt/shepherd
cp engine.docker.toml engine.toml
$EDITOR engine.toml # add real RPC URL
docker compose pull # once ghcr.io image is published
docker compose up -d
docker compose logs -f shepherd
curl -s http://127.0.0.1:9100/metrics | head -50
```
- `docs/deployment/multi-chain-guide.md` — dedicated walkthrough
configuring 4 chains together (Mainnet + Gnosis + Arbitrum + Base)
with per-chain module subscriptions
- Example module declaring multi-chain support (every current
example pins Sepolia)
- Optional automated CD trigger (workflow_dispatch SSH'ing to the
soak VM to pull + restart) — gated on SSH_PRIVATE_KEY repo secret
Companion to the M5 Docker packaging — the operator workflow is `cp engine.docker.toml engine.toml` then drop in a paid RPC URL. Without this rule a clumsy `git add -A` could commit the key. The committed sibling templates (engine.example/docker/m2/m3/e2e/load.toml) stay trackable. Validated against a live smoke run: drpc Sepolia WSS endpoint pasted into engine.toml, `docker compose up`, engine subscribed to newHeads + logs, 6 sequential blocks dispatched (11117171..76), metrics `shepherd_event_latency_seconds` p99 = 0.14ms. Tear-down clean. No engine.toml ever staged.
Closes the footgun surfaced by the M5 smoke run on drpc Sepolia:
configuring `rpc_url = "https://..."` for a chain that the modules
subscribe to silently degrades to an infinite WARN-with-backoff loop
(COW-1071's reconnect retries forever because `eth_subscribe` is
WS-only in the JSON-RPC spec). Three coordinated changes:
`EngineConfig::validate_transports()` walks every `[chains.<id>]`
entry, and for any `rpc_url` not starting with `ws://` / `wss://`
emits one loud ERROR-level structured log line with:
- the chain id
- the redacted offending URL
- the redacted suggested `wss://` swap
- actionable copy explaining the WS requirement and the escape
hatch (`[chains.<id>] require_ws = false` for poll-only chains
that never subscribe)
The validator is invoked from `main.rs` AFTER the tracing
subscriber is initialised (calling it inside `load_or_default`
silently dropped the log).
A `require_ws: bool` field is added to `ChainConfig` with
`#[serde(default = "default_require_ws")]` = `true`. Operators who
genuinely need an HTTP endpoint (poll-only modules, no block / log
subscriptions on this chain) opt out explicitly per chain.
The pre-existing `opening chain RPC provider` log in
`provider_pool::from_config` was emitting the full URL — API key
included — at INFO level. Log aggregators (Loki / Datadog / Splunk)
routinely retain weeks of these lines; the key has no business
sitting in cold storage. The new `engine_config::redact_url` helper
(public so other call sites can adopt it) replaces any path segment
longer than 20 chars that doesn't contain `.` or `:` with `<KEY>`.
Matches Alchemy / drpc / Infura / QuickNode key shapes.
Same helper is used for both the validation ERROR's `rpc_url` and
`suggested` fields and the provider-pool boot log.
- `engine.example.toml`: every chain entry switched to `wss://`,
with a header block explaining the WS requirement + the
`require_ws = false` escape hatch. The previous mix of `https://`
+ `wss://` would have tripped the new validator on its own example.
- `docs/production.md §6`: blockquote callout pointing operators at
the WS requirement, redaction behaviour, and the escape hatch.
Smoke 1 (HTTP, expected to ERROR):
{"level":"ERROR","message":"rpc_url uses HTTP transport but the engine subscribes to blocks/logs via eth_subscribe (WS-only). [...]","chain_id":11155111,"rpc_url":"https://lb.drpc.live/sepolia/<KEY>","suggested":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
Smoke 2 (WSS, expected to pass + redacted):
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,"url":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
- 9 new unit tests in `engine_config::tests`:
* `validate_accepts_wss_url`, `validate_accepts_ws_url`
* `validate_is_silent_when_require_ws_is_false`
* `validate_runs_without_panicking_on_http_url`
* `suggest_swaps_https_to_wss`, `suggest_swaps_http_to_ws`,
`suggest_passes_through_already_ws_url`
* `redact_replaces_long_path_segments`,
`redact_keeps_short_segments_intact`
- Workspace: 18 groups, **212 passed, 0 failed** (was 203 → +9)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
Operator workflow before this change forced the paid-RPC URL to
live in a file (`engine.toml`), which is fine for systemd but
awkward for Docker/compose: the URL had to be hand-edited inside a
volume-mounted file, secrets and config got tangled, and the
internal drpc test key was at risk of slipping into a committed
example. This change makes the engine treat `${VAR_NAME}` tokens
inside `engine.toml` as environment-variable references, resolved
at config-load time:
[chains.11155111]
rpc_url = "${SEPOLIA_RPC_URL}"
The `engine.docker.toml` and `engine.example.toml` templates ship
with `${VAR}` placeholders for all five chains, so the committed
files stay secret-free regardless of deployment path.
cp .env.example .env
$EDITOR .env # paste real wss:// URLs
docker compose up -d
`docker compose` reads the repo-root `.env` automatically (already
the compose default) and forwards the named variables into the
container via the new `environment:` block; the engine substitutes
them when parsing `/etc/shepherd/engine.toml`.
- `engine_config.rs::substitute_env_vars` — hand-rolled parser
(no regex dep) that walks the raw TOML text, matches `${NAME}`
tokens against `[A-Z_][A-Z0-9_]*`, and looks each up via
`std::env::var`. Three error variants via `thiserror`:
* `Missing { name }` — variable referenced but unset; message
includes the exact name and a pointer to the `.env` workflow.
* `InvalidName { name }` — typo (lowercase, leading digit);
suggests the upper-cased variant.
* `Unclosed { offset }` — `${` without matching `}`.
- Called from `load_or_default` before `toml::from_str`, so the
substitution layer never sees parsed TOML — a missing env var
surfaces with the exact variable name, not a downstream
"invalid URI character" several layers deep.
- Substitution runs over the whole file (comments included; harmless).
- `.env.example` — committed template with placeholders for all 5
chain `*_RPC_URL` variables + the optional `SHEPHERD_IMAGE` and
`SHEPHERD_ENGINE_CONFIG` overrides.
- `.gitignore` — adds `!.env.example` exception so the template
stays trackable while `.env` and `.env.local` etc. stay ignored.
- `docker-compose.yml` — passes the five `*_RPC_URL` env vars
through to the container; the engine config bind-mount now
defaults to `engine.docker.toml` (the committed template) and
honours `SHEPHERD_ENGINE_CONFIG` for operators who prefer a
bespoke file.
- `engine.docker.toml` + `engine.example.toml` — every `[chains.*]`
entry switched to `${*_RPC_URL}` placeholders. Header comments
spell out the workflow.
- `docs/deployment/docker.md` — first-boot section now leads with
`cp .env.example .env` (was `cp engine.example.toml engine.toml
&& edit`). §2 explains the bind-mount + the
`SHEPHERD_ENGINE_CONFIG` escape hatch.
Smoke 1 (compose end-to-end):
$ cp .env.example .env
$ echo "SEPOLIA_RPC_URL=wss://lb.drpc.live/sepolia/<real-key>" >> .env
$ echo "SHEPHERD_ENGINE_CONFIG=./engine.local.toml" >> .env
$ docker compose up -d
...
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,
"url":"wss://lb.drpc.live/sepolia/<KEY>",...} ← env-resolved, key redacted
{"level":"INFO","message":"supervisor up","loaded":2,"alive":2,...}
{"level":"INFO","message":"block subscription open","chain_id":11155111,...}
{"level":"INFO","message":"log subscription open","module":"twap-monitor",...}
{"level":"INFO","message":"log subscription open","module":"ethflow-watcher",...}
$ docker compose logs | grep -c <real-key>
0 ← zero leaks
$ curl -s http://127.0.0.1:9100/metrics | grep latency_seconds_count
shepherd_event_latency_seconds_count{module="twap-monitor",event_kind="block"} 4
Smoke 2 (missing env var, expected fail-fast):
$ unset SEPOLIA_RPC_URL
$ docker compose up
Error: engine config env-var substitution failed: environment variable
`SEPOLIA_RPC_URL` referenced via ${SEPOLIA_RPC_URL} in engine.toml but
not set. Export it before launching the engine (e.g. via a `.env`
file consumed by `docker compose`).
- 7 new unit tests in `engine_config::tests`:
* `substitute_replaces_known_variable`
* `substitute_errors_on_missing_variable`
* `substitute_errors_on_invalid_name`
* `substitute_errors_on_unclosed_brace`
* `substitute_passes_text_with_no_placeholders_through`
* `substitute_handles_multiple_placeholders_in_one_line`
* `substitute_preserves_utf8_around_placeholder`
- Workspace: 18 groups, **219 passed, 0 failed** (was 212 → +7)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
VM smoke surfaced a false-negative `(unhealthy)`: the compose healthcheck called `wget` but the runtime image is built on debian:bookworm-slim which doesn't include it (only ca-certificates + tini, intentionally minimal). `wget: not found` → exit 127 → unhealthy mark, despite the engine actually working (21 blocks dispatched in 3 min, p99 latency 0.09ms, zero errors). Swap to bash's `/dev/tcp` builtin (always present in bookworm-slim's `/bin/bash`). Successful TCP open on the metrics port proves the exporter bound, which only happens after the supervisor finishes boot — same semantic, no image growth.
First fix attempt swapped wget for `/dev/tcp` but kept `CMD-SHELL`, which routes through `/bin/sh` (dash on debian:bookworm-slim). dash doesn't have the `/dev/tcp/<host>/<port>` builtin — it's bash- only. Probes failed with "cannot create /dev/tcp/...: Directory nonexistent". Switch to `CMD ["bash", "-c", ...]` so the bash builtin actually resolves. `bash` ships in the slim base; verified via `docker exec shepherd which bash` → `/usr/bin/bash`.
Cherry-pick of PR #62 + PR #63's redesign onto the M5 host runtime (env-var substitution in engine.toml, healthcheck fixes, etc) for the Sepolia soak VM. The PR review continues on the proper layered branches: - PR #62 — M2/BLEU-833 layer observe design - PR #63 — M3/M4 BLEU-855 split + COW-1074 cow_api_request integration This branch is deploy-only: it lets the soak run on the redesigned ethflow-watcher with the latest host runtime while review iterates on the layered PRs. After merge, this branch can be deleted; CI will republish ghcr.io/bleu/nullis-shepherd:latest with the merged design and the VM rolls forward to the official image. See COW-1076 for the full empirical evidence.
…store (COW-1085)
`getTradeableOrderWithSignature` returns the same Ready tuple in every
poll-tick during a TWAP child's validity window — the on-chain
conditional order has no way to know shepherd already POSTed it. The
strategy already wrote a `submitted:{uid}` marker after a successful
submit, but the next poll-tick polled the chain and submitted again,
producing a wasted orderbook call and a misleading
`DuplicatedOrder` Warn line in every soak that runs a TWAP.
Live evidence (2026-06-23 Sepolia soak):
10:02:36.784 INFO poll watch:0x8fab71c0...:0x93b1626c... -> Ready
10:02:37.190 INFO submitted submitted:0xd7116bd2...
10:02:48.870 INFO poll watch:0x8fab71c0...:0x93b1626c... -> Ready
10:02:49.855 WARN submit dropped watch (400): orderbook error
(DuplicatedOrder): order already exists
The first submission succeeded (`GET /api/v1/orders/0xd7116bd2...`
returns `status: fulfilled`); the second was wasted work.
The fix: at the top of `submit_ready`, compute the client-side UID
deterministically from the on-chain `(order, owner, chain)` tuple via
`OrderData::uid` and check `submitted:{uid}` in local-store; skip the
submit (and the appData resolve that precedes it) when the marker
exists. The marker write site is also updated to use the
client-computed UID for the key so the read and write paths agree
(in production the server-returned UID is the same value — both sides
derive it from the canonical `digest || owner || valid_to` layout —
and a divergence is now surfaced via a Warn).
Tests (24, all green natively + wasm32-wasip2):
* Existing `poll_ready_submits_order_and_persists_submitted_uid` and
`poll_ready_resolves_non_empty_app_data_then_submits` updated to
compute the expected marker key via `compute_uid_hex` instead of
hardcoding `submitted:0xfeedface` (the mock orderbook's stub UID,
which now triggers the divergence Warn so we also assert that).
* New `poll_ready_skips_submit_when_submitted_uid_already_in_store`:
seeds the marker, dispatches a block tick, asserts
`submit_order` (and the preceding appData resolve) are NOT called
and that the expected Info log appears.
Out of scope (deferred): the same idempotency pattern could be
applied to ethflow-watcher's `observed:{uid}` marker (already correct
there — the GET-not-POST design makes this naturally idempotent).
…econnects (COW-1086)
Adds a positive-recovery Info log when the block subscription
resumes after a silence ≥ 60 s, covering the observability gap
identified in the 2026-06-23 Sepolia soak.
## Background
The 2026-06-23 soak surfaced this sequence:
09:05:43 ERROR WS connection error: WebSocket protocol error:
Connection reset without closing handshake
target=alloy_transport_ws::native
(no further WS-related lines for ~1 h)
10:02:24 INFO indexed watch:... ← twap-monitor activity resumes
10:05:24 INFO ethflow observed ... ← ethflow-watcher activity resumes
`docker ps` showed 0 restarts and the container stayed healthy
throughout — alloy's transport layer reconnected internally without
the engine's `reconnecting_block_task` ever observing
`inner.next().await -> None`. So the engine never entered its
"stream ended → backoff → subscription reopened" path, and the
existing `block subscription reopened` Info log (COW-1071) never
fired. The transport-layer ERROR followed by silence is
indistinguishable from a hung engine on a soak dashboard.
## What changes
In `reconnecting_block_task`, on every yielded item compare
`now.duration_since(last_event)` against `BLOCK_GAP_LOG_THRESHOLD`
(60 s, 5× Sepolia block time). When the gap meets or exceeds the
threshold, emit:
INFO chain_id=... gap_s=... kind="block"
"stream gap closed - first event after silence
(likely an alloy-internal transport reconnect)"
The gap-detection logic is factored into a small synchronous helper
`block_stream_gap_to_log(now, last_event, threshold) -> Option<Duration>`
so it can be unit-tested without spinning up an async runtime or a
real provider.
## Why blocks only (not logs)
Block subscriptions have predictable cadence — Sepolia produces a
new block every ~12 s, mainnet every ~12 s. A 60 s silence is
therefore anomalous and worth surfacing. Log subscriptions, by
contrast, are inherently sparse (driven by on-chain user activity),
so the same threshold would fire false positives on quiet windows.
The existing `log subscription reopened` log already handles the
engine-detectable reconnect for log streams.
## Tests
4 new unit tests on the gap-detection helper:
* `block_stream_gap_to_log_returns_none_when_no_prior_event`
* `block_stream_gap_to_log_returns_none_when_under_threshold`
* `block_stream_gap_to_log_returns_some_at_threshold_boundary`
* `block_stream_gap_to_log_returns_some_when_well_over_threshold`
All 90 nexum-engine tests pass (86 existing + 4 new). Clippy strict
clean, fmt clean. Wasm build untouched.
## Out of scope
* End-to-end test of `reconnecting_block_task` against a mock
provider — no existing scaffolding for that path, and the gap
helper covers the decision logic deterministically.
* Suppressing or downgrading the `alloy_transport_ws::native` ERROR
itself — it is a legitimate transport-layer event, just one whose
recovery wasn't previously observable. The new Info line closes
that loop without losing the original signal.
## Live validation
The next time alloy auto-reconnects internally on the soak VM, the
new line will surface as a structured JSON event with
`gap_s=<seconds>` so the soak dashboard can correlate it with the
preceding transport ERROR.
) Squash of PR #68 - 9 markdown files reconciled (5 vapor items rephrased as future direction + capability-gating diagrams aligned to link-time enforcement). Verified: cargo doc --workspace --no-deps clean.
`observe_placement` matches on the `Result<String, HostError>` returned
by `host.cow_api_request(...)`. The M5 conflict resolution (COW-1082
ErrorResp data forwarding) accidentally pasted a wildcard arm that
belongs to `apply_submit_retry`'s `match classify_api_error(...)` (which
matches a `RetryAction` enum). On a `Result`, the wildcard is both
unreachable (`Ok(_)` and `Err(_)` already cover everything) and
references an `err` binding that doesn't exist in its scope:
error[E0425]: cannot find value `err` in this scope
--> modules/ethflow-watcher/src/strategy.rs:205:21
--> modules/ethflow-watcher/src/strategy.rs:205:31
The `Err(err) if err.code == 404` arm + bare `Err(err)` arm already
classify every error case. Drop the spurious `_ =>` arm; bring
`observe_placement` back to fmt/clippy/test green on dev/m5-base.
…terError Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #1 (remaining enums introduced on the M5 multi-chain pass). - `EnvVarError` (engine_config.rs): introduced with the COW-1071 env-var substitution path. Snake_case variant labels feed the boot-time `tracing::error!(error_kind = ...)` call sites in `main.rs`. - `FilterError` (supervisor.rs): introduced with the M5 multi-chain log-filter parsing. Snake_case variant labels feed the `tracing::warn!(error_kind = ...)` log emitted when a `[[subscription]]` address or topic fails to parse. The audit's M3 / M4 derives landed on the milestones that introduced the enums; these two complete the workspace-wide IntoStaticStr pass flagged in audit Major #1 on the milestones that own them.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #6. The rubric forbids em-dashes in "code, rustdoc, commit messages, PR bodies, or review comments". `.toml` is technically a grey zone but these comments surface verbatim when operators `cat engine.docker.toml` or `engine.example.toml` during deployment onboarding. Mechanical find/replace to ` - ` (ASCII hyphen with spaces). Files touched: - engine.example.toml: 2 em-dashes (lines 20, 38) - engine.docker.toml: 4 em-dashes (lines 4, 5, 6, 31)
…rse (audit JC5) shepherd-backtest's offline replay harness carried its own `AddressParseError` enum (hex-decode + length check). The shape overlaps directly with the `AddressParse` typed error introduced into `shepherd-sdk` by the audit JC5 pass. Extend `shepherd_sdk::address` with a single-address `parse_address` helper alongside the existing `parse_address_list` (the `InvalidAddress` variant covers both call sites via the `index` field). Replay's `fixtures::parse_address` becomes a thin wrapper that calls the SDK and converts the `Address` to the `[u8; 20]` shape the strategy consumes via `LogView::address`. Drops the now-unused `thiserror` dependency from shepherd-backtest; `hex` stays for topic/data decoding.
jean-neiverth
force-pushed
the
dev/m5-rebase
branch
from
08445c5 to
0a71333
Compare
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspaceNotes for reviewer