feat(twap-monitor): consume composable-cow batched contract surface#76
Closed
brunota20 wants to merge 98 commits into
Closed
feat(twap-monitor): consume composable-cow batched contract surface#76brunota20 wants to merge 98 commits into
brunota20 wants to merge 98 commits into
Conversation
brunota20
force-pushed
the
feat/composable-cow-m2-integration
branch
3 times, most recently
from
adf6d7e to
82a5adc
Compare
Add modules/twap-monitor/ as a workspace member. Cargo.toml declares [lib] crate-type = ["cdylib"] for WASM Component output, and pulls the deps the TWAP module path needs: cowprotocol (default-features off — only typed primitives and OrderCreation surface needed), alloy-sol-types (event/return decoding lands in BLEU-826/827), and wit-bindgen. src/lib.rs binds against the shepherd:cow/shepherd world (event- module imports + cow-api). generate_all is required because the world include pulls nexum:host/types across packages — without it, wit-bindgen panics on the missing cross-package mapping. init and on_event are stubbed: init logs once; on_event is a no-op until the Event::Log / Event::Block dispatch lands in BLEU-826 / BLEU-827. Verification: cargo build --target wasm32-wasip2 --release -p twap-monitor emits a 65 KB .wasm. Engine load is gated on module.toml (BLEU-834).
…-826)
`on_event(Event::Logs)` decodes each log against
`ComposableCoW.ConditionalOrderCreated` via `alloy_sol_types`,
extracts `(owner, params)`, and writes `watch:{owner}:{params_hash}`
to local-store with the abi-encoded `ConditionalOrderParams` as
the value. BLEU-827 reads this back via `list-keys("watch:")` and
the value is exactly the `(handler, salt, staticInput)` tuple the
poll path passes to `getTradeableOrderWithSignature`.
Idempotency: `local_store::set` overwrites in place, so re-org
replay or overlapping subscription windows produce no observable
side effect.
Resilience: `decode_conditional_order_created` returns `None`
when topic0 does not match the event signature or the payload
fails ABI decoding. Adjacent events on the same subscription
(MerkleRootSet, SwapGuardSet) are silently skipped instead of
short-circuiting the batch. The fn is on plain slices so the
host-free unit tests cover well-formed / wrong-topic / empty-
topics without wit-bindgen scaffolding.
Block, Tick, and Message variants of `Event` are left unhandled
in this PR — `Event::Block` dispatch lands in BLEU-827 (poll
path); the other two are not used by this module.
Adds `alloy-primitives` as a direct dep so the topic/data plumbing
does not rely on alloy types leaking through `cowprotocol`'s
re-exports.
`cargo build --target wasm32-wasip2 --release -p twap-monitor`
emits a 96 KB .wasm (up from the 65 KB skeleton because of the
alloy + cowprotocol composable types now linked in).
`on_event(Event::Block)` walks every persisted watch, skips the
ones gated by a future `next_block:` / `next_epoch:` entry, and
dispatches the ready ones via `chain::request("eth_call",
[{to: COMPOSABLE_COW, data}, "latest"])` to
`ComposableCoW.getTradeableOrderWithSignature(owner, params,
"", [])`.
Returns:
- Successful return data → `<(GPv2OrderData, Bytes)>::abi_decode_params`
→ `PollOutcome::Ready { order, signature }`.
- Revert payload → `decode_revert` matches the four-byte selector
against the five `IConditionalOrder` errors:
OrderNotValid → DontTryAgain
PollNever → DontTryAgain
PollTryNextBlock → TryNextBlock
PollTryAtBlock(n) → TryOnBlock(n)
PollTryAtEpoch(t) → TryAtEpoch(t)
- Anything else falls back to TryNextBlock so a flaky RPC or
unmodelled require-revert is retried instead of dropped.
Decoder ABI: a local `abi::Params` struct mirrors the wire format
of `cowprotocol::ConditionalOrderParams` because sol! cannot cross
crate boundaries; the resulting call selector is byte-equal to the
real contract. The successful return path decodes into the
canonical `cowprotocol::GPv2OrderData` directly, so the 12-field
struct is not duplicated. `Ready` boxes the order to keep
`PollOutcome` cache-friendly (clippy::large_enum_variant).
Storage conventions (shared with BLEU-830, which writes these):
- `next_block:{owner}:{params_hash}` -> u64 LE — block number gate
- `next_epoch:{owner}:{params_hash}` -> u64 LE — Unix-seconds gate
Either / both / neither may be set; the watch polls when both pass.
`block.timestamp` is milliseconds per WIT, so we divide by 1000 to
compare against the `TryAtEpoch` (seconds) convention.
Host follow-up: the chain backend currently swallows alloy's
`RpcError::ErrorResp.data` (it becomes `host-error.message`,
unstructured). `poll_one` is wired to consume structured revert
hex via `host-error.data` once that lands — the `decode_revert_hex`
test locks the path. Until then, every revert defaults to
TryNextBlock, which is the safe choice.
Tests: 14 new (return round-trip, all five revert variants, hex
plumbing, eth_call JSON shape, watch-key round-trip, U256
saturation), keeping the 3 BLEU-826 regressions. `.wasm` grows
from 96 KB to 215 KB (serde_json + IConditionalOrder ABI + the
GPv2OrderData decode path linked in).
Linear: BLEU-827. Ref ADR-0006.
…828)
On `PollOutcome::Ready { order, signature }`, convert the
`GPv2OrderData` to the typed `OrderData` (maps the on-chain
bytes32 markers `kind` / `sellTokenBalance` / `buyTokenBalance`
via cowprotocol's `from_contract_bytes`), wrap the signature as
`Signature::Eip1271` (ComposableCoW returns the orderbook wire
form: raw verifier bytes, the orderbook re-prepends `from`
before settlement), and feed everything through
`OrderCreation::from_signed_order_data`. The body is then
serde-encoded and pushed to `cow_api::submit_order(chain_id,
body)`.
On success, persist `submitted:{uid}` in local-store as an
empty marker — presence of the key is the receipt; BLEU-830
may later attach metadata but the bare flag is enough to
suppress double submits.
Scope notes (deliberately deferred):
- `app_data` is hard-coded to `EMPTY_APP_DATA_JSON`.
Conditional orders that pin a real document on IPFS get
rejected by `from_signed_order_data` (digest mismatch) and
skipped with a Warn log instead of submitting a corrupt body.
Resolving the document is its own concern.
- Submission errors are logged. BLEU-829 wires
`OrderPostError::retry_hint` into this site so the backoff /
drop decision is data-driven.
- `from` is set to the watch owner (the address that emitted
`ConditionalOrderCreated`). The orderbook prepends this to
the EIP-1271 blob during settlement.
Tests: 7 new (gpv2_to_order_data marker mapping incl. zero-
receiver normalisation, unknown kind / balance marker
rejection; build_order_creation happy path with serde round-
trip; rejection of non-empty app_data and `from = ZERO`).
Total 24 host tests. `.wasm` 273 KB (was 215 KB; serde for
OrderCreation, the OrderData/Signature/SigningScheme modules,
and serde_with's runtime ride along).
Linear: BLEU-828. Ref ADR-0006 (modules build orders
themselves).
After \`cow_api::submit_order\` returns Err, decode the orderbook's
typed \`ApiError\` JSON from \`host-error.data\` and dispatch on
\`OrderPostErrorKind::is_retriable()\`:
- retriable (InsufficientFee, TooManyLimitOrders,
PriceExceedsMarketPrice) -> RetryAction::TryNextBlock — leave
the watch in place so the next block re-attempts.
- permanent (InvalidSignature, WrongOwner, DuplicateOrder,
UnsupportedToken, InvalidAppData, ...) -> RetryAction::Drop —
delete watch:{owner}:{params_hash} and any stale next_block /
next_epoch entries the lifecycle layer may have written.
- typed payload missing or unparseable -> TryNextBlock (safe
default: a flaky orderbook should not poison a still-valid
watch).
A \`RetryAction::Backoff { seconds }\` variant is defined for the
BLEU-829 contract but has no producer: cowprotocol's surface today
is bool-only (no server-supplied delay). The variant is kept so the
dispatcher can grow into it once a hint shows up (e.g. server
\`Retry-After\` header or a richer typed error).
## Host follow-up
\`cow_api::submit_order\` in nullislabs/shepherd PR #8 stuffs the
formatted error string into \`host-error.message\` with \`data: None\`
and \`code: 0\`. \`try_decode_api_error\` reads from \`host-error.data\`
already, so once the host forwards the upstream JSON the dispatch
becomes data-driven without further module changes. Test
\`classify_missing_data_defaults_to_try_next_block\` documents the
current fallback; the four other classify tests lock the
intended semantics for when the host catches up.
Tests: 5 new (retriable / permanent / unknown / missing-data /
malformed-data). Total 29 host tests. \`.wasm\` 298 KB (was 273 KB;
adds the typed ApiError decode + the small dispatcher).
Note: this branch also picks up the dev/m2-base bump to
bleu/cow-rs main (\`57f5f55\`), which lands BLEU-822
(\`OrderPostErrorKind\`) + BLEU-823 — both now visible through the
\`cowprotocol\` re-exports.
Linear: BLEU-829.
After poll_one, the non-Ready arms now reach a typed lifecycle
step instead of dead-ending in the log:
- TryNextBlock -> NoOp — re-poll next block
- TryOnBlock(n) -> SetNextBlock(n) -> persist next_block:{...}
- TryAtEpoch(t) -> SetNextEpoch(t) -> persist next_epoch:{...}
- DontTryAgain -> DropWatch -> delete watch:{...} +
best-effort delete of the
stale next_block: /
next_epoch: gates
The decision is split out as a pure `outcome_to_update` returning
a `WatchUpdate` enum, with the impure `apply_watch_update`
performing the local-store writes. That partition lets the four
host-free tests assert the mapping exhaustively without
wit-bindgen scaffolding.
`Ready` is deliberately mapped to `NoOp` here as a safety net —
poll_all_watches routes Ready to submit_ready, which owns the
post-submit book-keeping (submitted: marker + retry / drop). If
a future refactor accidentally pipes Ready through the lifecycle
path, the watch must NOT be erased.
Wire-format conventions (u64 LE bytes, key shape watch:{owner}:
{params_hash} and parallel next_block: / next_epoch:) stay the
same as BLEU-827; no consumer changes required.
Tests: 5 new (Ready, TryNextBlock, TryOnBlock, TryAtEpoch,
DontTryAgain). Total 34 host tests.
Linear: BLEU-830.
Mirror of the twap-monitor skeleton (BLEU-825) for the EthFlow path. Adds modules/ethflow-watcher/ as a workspace member, with [lib] crate-type = ["cdylib"] for WASM Component output, and the same dep set (cowprotocol no-default-features, alloy-primitives, alloy-sol-types, wit-bindgen) pre-pulled so BLEU-832 (event decode) and BLEU-833 (EIP-1271 submit + retry) can layer in without churning Cargo.toml. src/lib.rs binds against shepherd:cow/shepherd, init logs once, on_event logs Event::Logs as a placeholder until BLEU-832 decodes the CoWSwapEthFlow OrderPlacement payload. cargo build --target wasm32-wasip2 --release -p ethflow-watcher emits a 67 KB .wasm (within ~3 KB of twap-monitor's skeleton — identical world + deps, identical link footprint). Engine load is gated on module.toml (BLEU-834).
\`on_event(Event::Logs)\` matches each log against
\`CoWSwapOnchainOrders.OrderPlacement\` and keeps the four
fields BLEU-833's submission path will consume:
(sender, order, signature, data)
where \`order\` is the 12-field \`GPv2OrderData\` the
settlement contract verifies, and \`signature\` is the typed
\`OnchainSignature { scheme, data }\` pair (EIP-1271 or
PreSign).
Guardrails: \`decode_order_placement\` rejects the log when
the contract address is not one of the canonical EthFlow
deployments (production or staging — both share the same
address on every chain). topic0 must match the event
signature hash and the body must round-trip through
\`SolEvent::decode_raw_log\`. The decoder is on plain slices so
the seven host-free tests cover the happy path, the alternate
staging address, an unrelated contract, a wrong topic, a
truncated address, a truncated body, and an empty topic list.
\`DecodedPlacement\` boxes \`GPv2OrderData\` (~300 bytes); the
struct is kept private and \`#[allow(dead_code)]\` until
BLEU-833 wires the submit path.
\`cargo build --target wasm32-wasip2 --release -p
ethflow-watcher\` -> 96 KB .wasm (was 67 KB skeleton; the
\`CoWSwapOnchainOrders\` ABI + GPv2OrderData decode pull in
~30 KB of alloy sol-types runtime).
Linear: BLEU-832. Ref ADR-0006.
…(BLEU-833)
\`on_event(Event::Logs)\` now ends in a complete pipeline:
1. \`decode_order_placement\` (BLEU-832) lifts the log to a
\`DecodedPlacement\` carrying the contract, sender, order,
onchain signature and refund pointer.
2. \`build_eth_flow_creation\` translates that into a typed
\`(OrderCreation, OrderUid)\`:
- \`gpv2_to_order_data\` maps the on-chain \`bytes32\` markers
to the typed \`OrderKind\` / balance enums; same logic as
the TWAP module, kept inline because the two crates are
independent.
- \`to_signature\` lifts \`OnchainSignature\` into
\`Signature::Eip1271(bytes)\` or \`Signature::PreSign\`.
The hidden \`__Invalid\` sol! variant is surfaced as
\`Option::None\` so a malformed event skips the placement
instead of panicking.
- \`OrderData::uid(domain, contract)\` computes the canonical
56-byte order UID locally; the orderbook returns the same
value from POST /api/v1/orders and a Warn fires if they
drift (domain or owner divergence).
- \`from\` = EthFlow contract (the EIP-1271 verifier), NOT
the user's \`sender\` — matches the on-chain signing scheme.
- \`app_data\` is fixed to \`EMPTY_APP_DATA_JSON\` for now;
placements pinning a real IPFS document are rejected by
\`from_signed_order_data\` (digest mismatch) and skipped.
3. Serialise + \`cow_api::submit_order(chain_id, body)\`.
4. Persist the outcome:
- success -> \`submitted:{uid}\`
- retriable -> \`backoff:{uid}\` (same OrderPostError
classification path as
BLEU-829)
- permanent -> \`dropped:{uid}\`
\`apply_submit_retry\` mirrors BLEU-829's
\`classify_submit_error\` — when the host forwards the orderbook
JSON via \`host-error.data\`, the dispatch is data-driven;
absent data, the safe default is \`backoff:\` (retry next event)
rather than \`dropped:\`.
The \`Backoff { seconds }\` variant of \`RetryAction\` is parked:
cowprotocol's surface today is bool-only, so until a server
hint shows up (Retry-After or a typed delay) the variant
remains intentionally producer-less.
Tests: 10 host tests covering BLEU-832 (2 decode regressions)
and BLEU-833 (5 order-build edges + 3 error-classification
arms). \`.wasm\` 268 KB (was 96 KB; the OrderCreation +
serde_json + DomainSeparator + OrderUid surface get linked
in). Same scope-knot on app-data resolution as the TWAP
module; same host follow-up on \`host-error.data\` forwarding.
Linear: BLEU-833. Ref ADR-0006.
\`submit_placement\` now checks for a prior terminal marker before
calling \`cow_api::submit_order\`. Re-delivered \`OrderPlacement\`
logs (engine restart with replay, host reconnect, indexer
back-fill) would otherwise re-submit the same body, the orderbook
would reject \`DuplicateOrder\` (permanent), and the module would
end up with BOTH \`submitted:{uid}\` AND \`dropped:{uid}\` written
for the same key.
The guard is a typed `prior_outcome(uid_hex)` lookup:
- \`Submitted\` -> skip (the most common re-delivery cause)
- \`Dropped\` -> skip (orderbook permanently rejected previously)
- \`Backoff\` -> proceed: a transient failure deserves a fresh
attempt on re-delivery; the new outcome
overrides.
- \`None\` -> proceed: a clean first try.
On a successful submit, any previous \`backoff:\` marker is also
cleared so the local store carries at most one outcome flag per
UID at rest. Same cleanup happens on a permanent drop in
\`apply_submit_retry\`.
Linear: BLEU-833 (fix on the same PR — review identified the
re-delivery gap).
New library crate at crates/shepherd-sdk/ added as a workspace
member. The crate is a regular library (no cdylib), built against
both the host target and wasm32-wasip2 so helpers in BLEU-840 can
stay host-free unit-testable.
Deps follow the same default-features-off / 1.x-pinned pattern as
the modules:
- cowprotocol = "1.0.0-alpha.3", default-features = false
- alloy-primitives 1.6, alloy-sol-types 1.5
- serde 1, serde_json 1 (no_std-compat, alloc-only)
src/lib.rs lays out three placeholder modules (cow, chain, store)
that BLEU-840 will populate with the helpers currently duplicated
between twap-monitor and ethflow-watcher.
src/prelude.rs is the BLEU-835 deliverable proper: a single
`use shepherd_sdk::prelude::*` covers alloy primitives (Address,
B256, Bytes, U256, keccak256) and cowprotocol's order / signing /
orderbook-error surface (OrderCreation, OrderData, OrderUid,
OrderKind, Signature, Chain, GPv2OrderData,
EMPTY_APP_DATA_{HASH,JSON}, ApiError, OrderPostErrorKind).
The wit-bindgen-generated host types (Guest, HostError, Event,
Block, Log, …) deliberately stay *out* of the prelude — those
live in each module's own crate via the per-module
`wit_bindgen::generate!` invocation. Helpers added in BLEU-840
take primitive arguments (`&[u8]`, `Option<&str>`) so the SDK
remains world-neutral. Trade-off documented inline in lib.rs.
Builds + tests + clippy clean on host and wasm32-wasip2 (1 host
test locks the prelude surface).
Lifts the helpers currently duplicated between twap-monitor and
ethflow-watcher into shepherd-sdk so BLEU-843 can collapse the
duplication, and so future strategy modules consume them straight
from the SDK.
Layout:
crates/shepherd-sdk/src/
├── cow/
│ ├── order.rs gpv2_to_order_data
│ ├── composable.rs sol! IConditionalOrder + PollOutcome
│ │ + decode_revert
│ └── error.rs RetryAction + classify_api_error
│ + try_decode_api_error
└── chain/
└── eth_call.rs eth_call_params
+ parse_eth_call_result
+ decode_revert_hex
Every helper takes primitive arguments (`&[u8]`, `&str`,
`Option<&str>`, slices) so the SDK stays world-neutral — modules
unpack their wit-bindgen `HostError` / `Log` into primitives on
the way in. That keeps the SDK testable without a wasm toolchain
and re-usable across worlds (M3 examples, future strategies).
Notable shape:
- `cow::composable::PollOutcome::Ready` boxes `GPv2OrderData`
(~300 B) so the enum stays cache-friendly when the lifecycle
handler in BLEU-830 routes outcomes around.
- `cow::error::RetryAction::Backoff { seconds }` is parked
(`#[allow(dead_code)]`) for the future server-supplied hint;
cowprotocol's `retry_hint()` is bool-only today.
- `cow::error::classify_api_error(None) -> TryNextBlock` is the
safe default — a flaky orderbook should not be treated as a
permanent rejection.
Tests: 26 host tests covering every helper (6 gpv2 marker
mapping, 7 revert decode, 6 retry classification, 5 eth-call
plumbing, 1 SolError selector). Clippy clean on host and
wasm32-wasip2.
The modules in `modules/twap-monitor` and `modules/ethflow-
watcher` still carry their own copies; BLEU-843 deletes them.
Drops the duplicated helpers in `modules/twap-monitor` and `modules/ethflow-watcher` in favour of `shepherd_sdk::cow` / `shepherd_sdk::chain`: - `gpv2_to_order_data` (was duplicated verbatim) - `RetryAction` + `classify_api_error` + `try_decode_api_error` - `PollOutcome` enum (was duplicated verbatim — Ready boxed, TryAtEpoch / TryOnBlock / TryNextBlock / DontTryAgain) - `IConditionalOrder` sol! errors + `decode_revert` - `eth_call_params` + `parse_eth_call_result` + `decode_revert_hex` Kept module-side: - `abi::Params` + `getTradeableOrderWithSignatureCall` in twap-monitor (TWAP-specific selector source — EthFlow does not poll). - `decode_conditional_order_created` / `decode_order_placement` in their respective modules (each is bound to a specific event signature on a specific contract). - `watch:` / `next_block:` / `next_epoch:` key conventions in twap-monitor and `submitted:` / `backoff:` / `dropped:` in ethflow-watcher (per-module persistence policies, not shared). - `to_signature` (OnchainSignature → Signature) in ethflow- watcher (single consumer; will move to SDK if a second emerges). - `BuildError` / `WatchUpdate` / lifecycle plumbing in their modules (strategy-specific). LOC: -387 in twap-monitor (1058 → 671), -101 in ethflow-watcher (537 → 436). Tests: 13 host tests stay in twap-monitor (was 34 — the 21 that moved live in shepherd-sdk now), 7 stay in ethflow-watcher (was 10). .wasm size delta: - twap-monitor: 300 K → 305 K (+5 K — SDK re-exports + slight link-table growth; alloy + cowprotocol deduped). - ethflow-watcher: 272 K → 275 K (+3 K). Same wire behaviour — the SDK migration is a pure refactor; no new dispatch paths, no new key conventions.
…841)
Two-part deliverable:
1. New `shepherd_sdk::host` module exposing the trait seam between
strategy logic and the wit-bindgen shims a module generates per-
cdylib:
- `ChainHost` — request(chain_id, method, params)
- `LocalStoreHost`— get / set / delete / list_keys
- `CowApiHost` — submit_order(chain_id, body)
- `LoggingHost` — log(level, message)
- `Host` — supertrait bundling all four (blanket impl
so callers only need the supertrait bound)
The traits ride on a host-neutral `HostError` (same field shape
as wit-bindgen's), with `HostErrorKind` and `LogLevel` mirroring
the WIT enums verbatim. Modules bridge their own wit-bindgen
`HostError` to the SDK's with a one-liner `From` impl on each
side; the M3 tutorial (BLEU-848) documents the adapter pattern.
2. New `shepherd-sdk-test` crate (dev-only, host-only) supplying
in-memory implementations for every trait + assertion helpers:
- `MockHost { chain, store, cow_api, logging }`
- `MockChain`: programmable `(method, params)` -> result map;
records every call with `chain_id`, `method`, `params`.
- `MockLocalStore`: HashMap-backed; `list_keys` does a prefix
scan (sorted output for stable assertions).
- `MockCowApi`: single programmable response shared across
calls; records each submission's `chain_id` + body bytes;
`last_body_as_json` helper for inline assertions.
- `MockLogging`: buffers all lines with their level; `contains`
/ `count_at` helpers.
Unconfigured calls return `HostErrorKind::Unsupported` so an
unprogrammed test fails fast instead of silently passing on a
default value.
Tests: 8 host tests on `shepherd-sdk-test` + 1 module-level doctest
locking the recommended usage pattern. Workspace + wasm32-wasip2
check still clean.
Adoption is opt-in: existing M2 modules keep their pure-function
tests for now. BLEU-848 (tutorial) will demonstrate the new
strategy-takes-Host pattern with `MockHost` end-to-end.
- Tightened the crate-root rustdoc on `shepherd-sdk/src/lib.rs`:
switched the inline `[Type](path)` link form to top-of-file
reference-style link definitions so the rustdoc target is
unambiguous and the source stays readable.
- Removed the placeholder `pub mod store {}` (out-of-scope until a
second strategy module needs the same key conventions).
- New `crates/shepherd-sdk/README.md` covering: quick tour table,
host-free testing recipe with `shepherd-sdk-test`, the
no-wit-bindgen-in-SDK rationale, layout map, and how to generate
docs with the strict flags.
- New `docs/sdk.md` repo-level landing page that lists the four
host capabilities the SDK mirrors and links into the rustdoc per
module.
Gate: `cargo doc -p shepherd-sdk -p shepherd-sdk-test --no-deps`
runs clean under `RUSTDOCFLAGS="-D warnings -D missing-docs"`.
Every public item carries a doc comment; intra-doc links resolve.
Tests + clippy unchanged.
New `modules/examples/price-alert/` — first canonical SDK example.
A Shepherd module that polls a Chainlink AggregatorV3 price oracle
on every block (throttled by `every_n_blocks`) and emits a Warn-
level log when the answer crosses a config-supplied threshold.
Demonstrates the three load-bearing patterns of a Shepherd module:
- `chain::request` + ABI decode via `alloy_sol_types` (sol!
interface AggregatorV3 declares `latestRoundData`, decode via
`abi_decode_returns`).
- shepherd-sdk helpers (`chain::eth_call_params` +
`chain::parse_eth_call_result`; the SDK's prelude is *not*
used here because the module needs none of the CoW types).
- `[config]` driven behaviour parsed once in `init` and stored
in `OnceLock<Settings>` for read-only access on every event.
Module-internal:
- `Settings` (renamed from `Config` to avoid clashing with the
wit-bindgen-generated `Config` type alias for the `init` arg).
- `Direction { Above, Below }` deciding which side of the
threshold fires.
- `scale_threshold(decimal, decimals)` hand-rolled because alloy
does not ship a `Decimal::parse_units`-style helper; handles
optional sign, missing decimal point, short / long fractional,
rejects non-digit garbage. Locked by 5 unit tests.
- `classify(answer, threshold, direction)` pure 1-liner with 2
edge tests (at-or-above vs. at-or-below behaviour at the
boundary).
- `parse_config(entries)` returns `Result<Settings, String>` with
human-readable errors; 4 unit tests cover happy path, defaults,
unknown direction, missing key.
module.toml:
- `capabilities = ["logging", "chain"]` (no local-store; no
cow-api).
- `[[subscription]]` block on Sepolia (chain_id 11155111).
- `[config]` ships defaults pointing at the canonical Sepolia
ETH/USD feed with a 2500.00 USD threshold + "below" direction.
11 host tests; clippy clean on host + wasm32-wasip2. .wasm is
206 KB optimised — comparable to the M2 modules (twap 305 KB,
ethflow 275 KB) and dominated by alloy-sol-types + wit-bindgen
runtime.
New `modules/examples/balance-tracker/` — second canonical SDK
example. Subscribes to blocks, reads `eth_getBalance(addr)` for a
configured address list, persists each reading under
`balance:{addr}` in local-store, and emits a Warn-level log when
the delta against the prior reading exceeds `change_threshold`
wei.
Demonstrates:
- `chain::request` with a non-`eth_call` method (raw JSON-RPC
with hand-built params), to balance the price-alert example's
sol! / `eth_call` flow.
- `local-store` `get` / `set` per-key persistence with U256 LE
serialisation as the wire format.
- The "diff against last seen" pattern reusable across indexer
modules (transfer monitors, allowance trackers, …).
Module-internal:
- `Settings { addresses: Vec<Address>, change_threshold: U256 }`
parsed from `[config]` once at `init` and stored in
`OnceLock<Settings>`.
- `parse_balance_hex(json)` — strips JSON quotes and the `0x`
prefix, decodes the remaining hex into a U256. Handles `"0x"`
(zero balance), rejects unquoted / non-hex bodies.
- `parse_addresses(raw)` — comma-separated list with whitespace
tolerance and empty-segment skipping; rejects empty lists.
- `abs_diff` + `parse_u256_le` + `u256_to_le_bytes` — pure utilities
with edge-case coverage.
module.toml:
- `capabilities = ["logging", "chain", "local-store"]` (the
superset that distinguishes this example from price-alert,
which only needs chain + logging).
- `[[subscription]]` block on Sepolia (chain_id 11155111).
- `[config]` ships defaults pointing at two anvil-style EOAs and
a 0.1 ETH change threshold.
13 host tests; clippy clean on host + wasm32-wasip2. `.wasm` is
99 KB optimised — about half of price-alert's 206 KB because it
does not pull `alloy-sol-types` into the link tree (no ABI work;
all decoding is hex/U256).
End-to-end cold-start guide that takes an external developer from
"I cloned the repo" to "I see my module's first event in the
engine log" in under four hours.
Scenario: stop-loss order — combines every load-bearing pattern in
the SDK (block subscription, chain::request + ABI decode, local-
store dedup, cow_api::submit_order, host-free tests via MockHost).
The tutorial walks through each pattern via the four worked
examples already in the repo (price-alert, balance-tracker,
twap-monitor, shepherd-sdk-test) and stitches them into the stop-
loss module.
Sections + rough budgets:
0. Prerequisites (15m) — toolchain check; verify the
example module runs.
1. Scaffold workspace (15m) — Cargo.toml template + workspace
members entry.
2. Manifest (10m) — module.toml with the four
capabilities + Sepolia
[[subscription]] + [config]
schema.
3. Strategy (60m)
3a. Pure logic — on_block<H: Host>(...) using
shepherd-sdk's chain helpers
and AggregatorV3 sol! interface.
3b. Guest adapter — wit_bindgen::generate! + the
WitBindgenHost struct that
bridges to shepherd_sdk::host
(one-time boilerplate per
module).
3c. Unit tests — two MockHost tests: idle-above-
trigger + triggers-and-dedups.
4. Build (5m) — cargo build --target
wasm32-wasip2 --release +
size sanity.
5. Run (10m) — engine.toml WS RPC for Sepolia
+ cargo run -p nexum-engine.
6. Where to go (10m) — production hardening + real
order assembly (twap-monitor
cross-ref) + multi-chain.
Pure docs change — no module added (the stop-loss in §3 is the
reader's exercise; build_order_body deliberately ends in a `todo!`
with a cross-reference to twap-monitor's canonical assembly path).
Worked artefacts referenced in the tutorial are the existing
examples landed in #18 / #19 plus shepherd-sdk + shepherd-sdk-test.
Cross-links: docs/sdk.md (BLEU-844), docs/deployment.md
(BLEU-836), ADR-0001 / 0006 / 0007.
Acceptance per the issue: the tutorial is reviewer-validatable.
Time-budget callout at the end asks for a tag `docs/tutorial` if
a section drags, so we tighten on feedback.
QA pass against the team's rust-idiomatic skill ahead of M4. All
mandatory rules now hold; the cleanup is mostly mechanical with a
handful of small typing improvements where the rule asked for one
thiserror enum per error type.
Replaced every U+2014 with " - " across .rs / .toml / .md:
- 51 source-file occurrences
- 5 Cargo.toml comments
- 366 occurrences across docs/*.md (most in ADRs and the
deployment / tutorial / sdk landings)
Grep gate: `grep -rn '—' crates/ modules/ docs/` returns 0.
Added to every crate root that previously lacked it:
- crates/shepherd-sdk/src/lib.rs
- crates/shepherd-sdk-test/src/lib.rs
- modules/{example,twap-monitor,ethflow-watcher}/src/lib.rs
- modules/examples/{price-alert,balance-tracker}/src/lib.rs
`crates/nexum-engine/src/main.rs` already had it.
- shepherd-sdk dropped `serde` (only `serde_json` is actually
imported; cowprotocol re-exports carry their own serde derive
transitively).
- balance-tracker dropped its direct `alloy-primitives` dep —
now goes through `shepherd_sdk::prelude::{Address, U256,
address}`. Tests adapt.
- `shepherd_sdk::host::HostError` gains `#[derive(thiserror::
Error)]` + `#[error("{domain}: {message} (code={code},
kind={kind:?})")]`. Was a plain struct without Display.
Added `thiserror = "2"` as a dep.
- `modules/twap-monitor::BuildError`: hand-rolled Display impl
replaced with `#[derive(thiserror::Error)]` + per-variant
`#[error(...)]` + `#[from] cowprotocol::Error`. The map_err
at the call site collapses to `?`.
- `modules/ethflow-watcher::BuildError`: same conversion (4
variants, one of them `#[from]`).
Both modules add `thiserror = "2"` as a direct dep.
- `cargo clippy --all-targets --workspace -- -D warnings` clean.
- `cargo test --workspace`: 121 tests pass.
- nexum-engine 41, shepherd-sdk 27, shepherd-sdk-test 8 + 1
doctest, twap-monitor 13, ethflow-watcher 7, price-alert
11, balance-tracker 13.
- `#[non_exhaustive]` is *not* applied to public enums
(`HostErrorKind`, `LogLevel`, `RetryAction`, `PollOutcome`).
The first two mirror the WIT 0.2 enums (locked at the WIT
contract layer); the last two are intentional 3- and 5-arm
contracts with no expected growth. If a future kind shows
up, the rule applies then.
- `parse_config` / `parse_settings` in the example modules
return `Result<T, String>` rather than a typed enum. The
rule's "no string-wrapping" applies to error variants that
*wrap* an upstream `std::error::Error`; one-shot config
parsers with bespoke per-field messages are pragmatic. The
error surface is internal to the module's `init` and not
part of the orderbook retry contract.
Validates the host-trait pattern from the M3 tutorial end-to-end on
a real module. The price-alert example now matches the recipe the
tutorial recommends:
modules/examples/price-alert/
├── Cargo.toml adds shepherd-sdk-test as dev-dep
└── src/
├── lib.rs wit_bindgen::generate! + WitBindgenHost
│ adapter + From conversions + Guest impl
└── strategy.rs pure logic against `&impl Host`
+ parse_config + scale_threshold + tests
Strategy logic now takes `&impl shepherd_sdk::host::Host` and never
calls `nexum::host::*` free functions directly. The wit-bindgen
boilerplate (WitBindgenHost struct, ChainHost / LocalStoreHost /
CowApiHost / LoggingHost impls, convert_err / sdk_err_into_wit /
convert_level helpers) lives in lib.rs - mechanical and identical
across modules, a future declarative macro in shepherd-sdk will
elide it.
parse_config now returns `Result<Settings, shepherd_sdk::host::
HostError>` instead of `Result<T, String>`. Carrying the SDK error
through the strategy / adapter / Guest seam means the same domain /
kind / code / message / data fields surface to the operator
verbatim.
Tests: 16 (was 11) - all strategy tests now run against
shepherd_sdk_test::MockHost rather than calling wit-bindgen
directly. The 5 new ones lock the on_block behaviour end-to-end:
- idle when price is on the safe side of the trigger
- triggers below threshold (Direction::Below)
- triggers above threshold (Direction::Above)
- warns + continues on RPC timeout (no propagation into the
supervisor)
- warns on undecodable oracle response
- respects `every_n_blocks` throttle
cargo clippy --all-targets --workspace -- -D warnings clean. .wasm
210 KB (was 206 KB; +4 KB for the adapter boilerplate, which
deduplicates against shepherd-sdk so future modules add no extra
cost).
Closes the loop opened by BLEU-848 (tutorial). The tutorial used to walk through a stop-loss scenario but left `build_order_body` as a `todo!()` cross-referencing twap-monitor. Now: 1. `modules/examples/stop-loss/` ships as a real workspace member, shaped the same way as the price-alert refactor (BLEU-851 / PR #22): pure logic in `strategy.rs` against `&impl Host`, wit-bindgen adapter + Guest impl in `lib.rs`. 2. The strategy is complete - reads a Chainlink oracle, builds an `OrderCreation` with `Signature::PreSign` (owner pre-signs via setPreSignature on-chain ahead of the trigger; module ships zero ECDSA), dedups via `submitted:{uid}`, persists `dropped:{uid}` on permanent submit errors. 3. Tests (7 total) cover the dispatch matrix end-to-end against `shepherd_sdk_test::MockHost`: - idle_when_price_above_trigger - triggers_and_submits_once_then_dedups - permanent_submit_error_marks_dropped (+ dedup on the next block) - transient_submit_error_leaves_state_unchanged - oracle_rpc_error_is_warn_and_continue - parse_config_round_trips_settings - parse_config_rejects_missing_owner 4. `docs/tutorial-first-module.md` rewritten as a guided tour instead of inlined snippets. The tutorial now reads the real `modules/examples/stop-loss/` source top-to-bottom and explains *why* each piece is shaped the way it is - sections on the wit-bindgen adapter, the `OrderCreation` assembly with PreSign, the dedup matrix, and the test recipe against MockHost. No more `todo!()`. Numbers: - `.wasm` 304 KB optimised (release build). - 7 host tests passing; clippy clean on host + wasm32-wasip2. - Tutorial is 449 lines (was 580 with the duplicated inline code); shorter because it points at real files instead of transcribing. Stacks on PR #22 (price-alert host-trait refactor) so both modules land alongside the wit-bindgen adapter recipe the tutorial documents.
Mirrors what BLEU-851 (price-alert) and BLEU-852 (stop-loss) did for
the M3 example modules. Closes the parallel M2 gap.
Before: the entire dispatch pipeline (indexer / poll / submit /
retry / lifecycle) lived in `lib.rs` alongside the wit-bindgen
glue, calling `chain::request`, `local_store::*`, `cow_api::submit_order`,
and `logging::log` directly. The 13 existing tests covered only
parsers and encoders - the state machine itself was unverified in
unit.
After:
1. `strategy.rs` (new) - pure logic against `shepherd_sdk::host::Host`.
Defines `LogView<'a>` and `BlockInfo` so the strategy stays
wit-independent; exposes `on_logs` / `on_block` entry points.
2. `lib.rs` (rewritten, 665 -> 165 lines) - wit-bindgen `generate!`,
`WitBindgenHost` adapter implementing all four SDK host traits,
`Guest` impl that destructures `types::Event` and delegates to
`strategy`.
3. Tests against `shepherd_sdk_test::MockHost` (7 new) cover the
dispatch matrix that was previously hand-verified only:
- `index_records_new_watch_on_conditional_order_created`
- `index_overwrites_in_place_on_redelivered_log` (re-org
replay guard, BLEU-826 invariant)
- `poll_skips_when_next_block_gate_is_in_future`
- `poll_ready_submits_order_and_persists_submitted_uid`
- `submit_transient_error_leaves_state_unchanged_for_next_block`
- `submit_permanent_error_drops_watch`
- `poll_dont_try_again_drops_watch_and_gates` (uses a real
`OrderNotValid` selector via the SDK-exported sol! interface)
4. All 13 original pure tests preserved unchanged. Total: 20 tests
(was 13).
Numbers:
- `.wasm` 313,926 bytes (release wasm32-wasip2).
- 20 tests passing; clippy clean on host + wasm32-wasip2.
- 0 em-dashes in the module tree.
Stacks on PR #23 (BLEU-852) so reviewers can compare strategy /
lib.rs split side-by-side with the price-alert and stop-loss
references.
…855) Same shape as BLEU-854 (twap-monitor / PR #24). Closes the M2-side gap on ethflow-watcher. Before: `submit_placement`, `prior_outcome`, `apply_submit_retry`, and the `submitted:` / `dropped:` / `backoff:` bookkeeping called `local_store::*` and `cow_api::submit_order` directly, with all the state-machine bits unverified in unit (only 7 decoder / encoder tests). After: 1. `strategy.rs` (new) - pure logic against `shepherd_sdk::host::Host`. `LogView<'a>` keeps the strategy wit- independent; `on_logs` is the entry point. 2. `lib.rs` (rewritten, 427 -> 157 lines) - wit-bindgen `generate!`, `WitBindgenHost` adapter, `Guest` impl that destructures `types::Event::Logs` into `LogView`s and delegates to `strategy::on_logs`. 3. Tests against `shepherd_sdk_test::MockHost` (5 new) cover the dispatch + idempotency matrix: - `placement_log_submits_order_and_persists_submitted_uid` - `redelivered_placement_is_skipped_via_submitted_uid_dedup` (PR #10 / commit c5e4d7d regression guard) - `submit_transient_error_writes_backoff_marker_and_returns` - `submit_permanent_error_persists_dropped_uid_and_clears_backoff` - `eip1271_signature_shape_round_trips_through_submit_body` (decodes the JSON body MockCowApi received and asserts `signingScheme=eip1271`, signature blob verbatim, `from` = EthFlow contract) 4. All 7 original pure tests preserved unchanged. Total: 12 tests (was 7). Numbers: - `.wasm` 281,518 bytes (release wasm32-wasip2). - 12 tests passing; clippy clean on host + wasm32-wasip2. - 0 em-dashes in the module tree. Stacks on PR #24 (BLEU-854) so reviewers can compare both M2 strategy / lib.rs splits in one stack with the M3 examples.
Pre-upstream QA pass against the M2 + M3 + M2-host-trait stacks.
Two findings applied here as a single tip-level commit instead of
rewriting each stacked PR (mfw78 prefers history preservation over
amended PRs):
1. `cargo fmt --all` across the workspace. Bulk of the churn is in
M1 `crates/nexum-engine/src/supervisor/tests.rs` (386 line diff,
pre-existing drift); the rest is M2/M3 leaf modules my own
recent PRs introduced. No semantic changes.
2. One em-dash slipped past the rust-idiomatic sweep in
`modules/examples/price-alert/src/strategy.rs:4` (a module-level
doc comment). Replaced with ASCII ` - `.
Three em-dashes remain in `wit/**.wit` files, all in mfw78's M1
prose. Intentionally left alone - the rust-idiomatic skill is a
Bleu-internal preference and should not rewrite his upstream
authoring style. Tracked as a separate question for him in the QA
sign-off report.
QA matrix on this commit:
- `cargo fmt --all --check`: clean
- `cargo clippy --all-targets --workspace -- -D warnings`: clean
- `cargo test --workspace`: 145 host tests + 1 doctest passing
(twap 20, ethflow 12, balance 13, price 16, stop-loss 7,
shepherd-sdk 27, shepherd-sdk-test 8, nexum-engine 41, doctest 1)
- `cargo build --target wasm32-wasip2 --release -p <module>`:
clean for all 5 modules. Sizes:
twap-monitor 313,926 B
ethflow-watcher 281,518 B
stop-loss 311,290 B
price-alert 215,080 B
balance-tracker 101,518 B
- Em-dashes in `crates/` + `modules/` + `docs/`: 0
- `warn(unused_crate_dependencies)` on every crate root: present
(sdk, sdk-test, nexum-engine, twap, ethflow, price-alert,
balance-tracker, stop-loss)
Outstanding (deferred):
- BLEU-853 / COW-1029: `#[non_exhaustive]` batch on SDK public
enums (HostErrorKind, LogLevel, PollOutcome, RetryAction). Held
until just before upstream cut so wit-bindgen stays bridge-able.
- WIT-file em-dashes in upstream prose - ask mfw78.
Captures the result of the pre-upstream QA pass. Two non-blocking follow-ups surfaced for mfw78's call before the consolidated PR: 1. `docs/05-sdk-design.md` describes a 2-layer SDK with `nexum-sdk` + proc macros + alloy Provider + Signer that M3 did not ship. M3 actually delivered the thinner Host-trait + helpers + MockHost surface. Doc and code need to agree (either trim doc to M3 scope or expand M4/M5 to match doc). 2. No ADR captures the M3 Host trait + strategy/lib split decision. ADR-0009 candidate. Everything else is green: 145 tests + 1 doctest, clippy clean, 0 em-dashes in our code, all 5 modules build for wasm32-wasip2, warn(unused_crate_dependencies) on every crate root. The 3 WIT-file em-dashes are mfw78's M1 prose - left alone. Optional follow-ups (none gating): - balance-tracker host-trait refactor for shape consistency. - mfw78 PR description template adoption on existing PR bodies.
Addresses the two non-blocking architectural items surfaced in
COW-1063's sign-off matrix before the consolidated upstream PR:
(a) `docs/05-sdk-design.md` -> add a "Current implementation
status (M3, 2026-06-17)" callout at the top with a per-feature
table mapping every section to its actual state. The doc
itself stays as the M5+ north-star (it's mfw78's design
document); the callout tells readers what is shipped vs
deferred so they don't read the proc-macro / Provider /
Signer sections as API reference for code that exists today.
Status table covers:
✅ shipped: shepherd-sdk, shepherd-sdk-test, 4-trait host
surface + supertrait Host, HostError mirror, chain +
cow helpers, MockHost, strategy/lib split recipe,
block.timestamp in ms.
❌ deferred (M5): nexum-sdk crate split, #[nexum::module]
/ #[shepherd::module] proc macros, named event handlers,
async fn dispatch, full alloy Provider via HostTransport,
TypedState (postcard), Signer (identity), Cow typed
client, MockIdentity / MockProvider / WasmTestHarness,
cargo nexum CLI.
(b) `docs/adr/0009-host-trait-surface.md` (new) -> captures the
three coupled M3 architectural decisions:
1. Four per-capability traits (ChainHost, LocalStoreHost,
CowApiHost, LoggingHost) + supertrait Host with a
blanket impl.
2. SDK-side HostError mirroring the wit struct
field-for-field, bridged via per-module one-liner
From impls. World-neutral so shepherd-sdk-test compiles
without wasm.
3. Per-module strategy.rs (pure, &impl Host) + lib.rs
(wit-bindgen adapter) split, applied uniformly across
price-alert, stop-loss, twap-monitor, ethflow-watcher.
Considered alternatives section explicitly rejects: single
fat Host trait, #[nexum::module] proc macro now (M5 work),
re-exporting wit-bindgen HostError, strategy colocated with
wit-bindgen adapter.
Marks the COW-1029 / BLEU-853 #[non_exhaustive] batch as the
follow-up that protects the field-equivalence assumption.
Doc 05 and ADR-0009 cross-reference each other, so readers landing
on either find the other. Both files are em-dash clean.
…face Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #3 (`[u8; 32]` for protocol hash across SDK public boundary). The rubric explicitly calls out: "Newtypes for protocol IDs (no raw `[u8; 32]` across module boundaries)." `B256` is already in `shepherd_sdk::prelude` so the swap costs callers nothing - both twap-monitor and ethflow-watcher were holding the appData as `B256` already and reaching through `.0` to satisfy the prior signature. Changes: - `resolve_app_data(host, chain_id, &B256)` (was `&[u8; 32]`) - `encode_hex(&B256)` internal helper - Doctest + 5 unit tests rewritten against `B256::from(bytes)` and `B256::from_slice(EMPTY_APP_DATA_HASH.as_slice())`. Coverage stays identical. - Call sites in twap-monitor and ethflow-watcher drop the `.0` reach-through; pass `&order.appData` directly. No public surface beyond `shepherd-sdk` consumes this function; external module crates in the workspace are the only consumers and both land in the same commit.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md,
duplication finding "Canonical CoW chain set
[Mainnet, Gnosis, Sepolia, ArbitrumOne, Base]" duplicated at
`crates/nexum-engine/src/host/cow_orderbook.rs:39-43` and `:66-70`.
`from_config` was added in the M4 multi-chain pass and reproduced the
same 5-element array `Default::default` already used. Adding a sixth
chain previously needed touching both arrays in lock-step; pull the
list into a single `const DEFAULT_CHAINS: &[Chain]` so the
single-source-of-truth property is structural.
Also drops the redundant `use cowprotocol::OrderBookApi;` inside
`from_config` (already in scope from the module-top `use cowprotocol::
{Chain, OrderBookApi, ...}` line). Behaviour identical.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #6. Rubric forbids em-dashes in operator-facing config files; while .toml is technically a grey zone the comment surfaces verbatim when operators `cat engine.e2e.toml` during e2e runbook execution.
…W-1084) Adds `tools/baseline-latency/baseline_latency.py`, a per-chain script that pairs every on-chain `EthFlow.OrderPlacement` event in a trailing window with the orderbook's record for the same UID and reports `(creationDate - block.timestamp)`. Matching is rigorous: the script ABI-decodes the GPv2OrderData from each event, computes the EIP-712 order digest against the chain's GPv2Settlement domain, and looks up the resulting UID against the orderbook's bulk `/account/.../orders` fetch (single-UID fallback if missed). No temporal-FIFO approximation. For EthFlow orders the orderbook indexer sets `creationDate := block.timestamp` (not the indexer's ingest time), so the historical delta is structurally 0s on every chain. This is intentional back-fill-style behaviour, not a measurement bug. **Implication**: EthFlow indexer latency cannot be derived from historical orderbook data — the meaningful relayer-latency baseline lives on the TWAP lane (where the orderbook records the indexer's `now()` per child order PUT). TWAP child-latency is a follow-up; it needs per-part UID derivation from each parent `ConditionalOrderCreated` static input. Sepolia ran clean: 256 events scanned, 200 UID-derived pairs, all 200 matched against the bulk fetch (`bulk_hit=200`). Median = p95 = 0.0s, exactly as the finding predicts. Public-tier RPCs (drpc.org free, 1rpc.io, ankr w/o key, llamarpc, cloudflare-eth) all refuse / throttle `eth_getLogs` at any usable chunk size on the production chains. The script halves down to 50-block chunks and gives up after 3 consecutive failures, marking the cell `RPC-LIMITED` with a pointer to the `RPC_URL_*` env override. This is the same constraint the M5 soak (COW-1031) will face and independently confirms the paid-endpoint requirement for any serious log-scanning workload. - `tools/baseline-latency/baseline_latency.py` (~520 lines): argparse CLI, per-chain `Chain` dataclass, JSON-RPC helper with halving retry + `RpcLimited` sentinel, EIP-712 order digest + UID derivation, UID-keyed orderbook matching, markdown report renderer. - `tools/baseline-latency/data/*.json`: per-chain raw dump (events, pairs, deltas, diagnostics) for auditability. - `docs/operations/baselines/baseline-latency-2026-06-19.md`: the first run's report. Pinning the orderbook's `creationDate` semantics matters because the COW-1079 and COW-1031 KPIs reference "watchtower latency" — the M4 report needs to be honest about which lane the latency lives on (TWAP relayer PUT, not EthFlow indexer ingest). The Sepolia data set also gives the M4 e2e harness ground-truth UID ↔ block pairings to cross-check against.
…W-1082) The chain backend previously dropped alloy's structured `RpcError::ErrorResp` payload on the floor — the formatted error string went into `HostError.message`, but `HostError.data` stayed `None` and `HostError.code` was hard-coded to `-32603`. That made the twap-monitor's poll-time revert classifier inert on real traffic: `OrderNotValid` / `PollNever` / `PollTryAtBlock` / `PollTryAtEpoch` all fell through to `TryNextBlock` because `decode_revert_hex` only fires on a non-empty `err.data`. This change wires the structured payload through end-to-end. - `crates/nexum-engine/src/host/provider_pool.rs`: when alloy's `provider.raw_request` fails with an `RpcError::ErrorResp`, the pool now captures both `payload.code` (as `Option<i64>` so we can distinguish "no ErrorResp" from "ErrorResp with code 0") and `payload.data` (as `Option<String>`, the JSON-encoded revert hex) and surfaces them on `ProviderError::Rpc`. Transport-side failures (timeout, websocket disconnect) leave both `None`. The two subscribe paths (`subscribe_blocks`, `subscribe_logs`) keep `code: None, data: None` since they don't carry an ErrorResp. - `crates/nexum-engine/src/host/impls/chain.rs`: extract the `ProviderError -> HostError` projection into a free helper `provider_error_to_host_error`. The `Rpc` arm forwards the structured `data` verbatim, preserves the node-reported code (saturating out-of-`i32` values to `-32603`), and falls back to `-32603` only when no `ErrorResp` was present. Five unit tests cover: revert with data, transport failure with `None`, out-of-range code, unknown-chain, and invalid-params. - `modules/twap-monitor/src/strategy.rs`: update the stale comment on the `decode_revert_hex` branch — that branch is now live on real traffic, the only `None` path is transport-level failures (which keep the safe `TryNextBlock` default). No incorrect order is ever submitted (the contract reverts; the orderbook never sees a bad body). The issue is pruning efficiency: a permanently dead TWAP watch was re-polled every block until a submit eventually failed for an unrelated reason, and the local-store filled with `watch:` entries the strategy could otherwise drop on the first revert. With this fix the SDK-side classifier dispatches `Drop` / gate on the first revert, matching the documented expectation in `docs/adr/0007-upstream-protocol-logic-to-cow-rs.md`. - 70/70 nexum-engine tests pass - 23/23 twap-monitor tests pass - 5/5 new chain.rs projection tests pass (revert-with-data, transport-fail, out-of-range-code, unknown-chain, invalid-params) - `cargo clippy -p nexum-engine -p twap-monitor --all-targets -- -D warnings` clean jeffersonBastos's PR #55 (M3 mirror) review, thread on `modules/twap-monitor/src/strategy.rs:189`. The mirror of this fix on the cow-api side is COW-1075 (already merged via PR #48).
…OW-1083)
The strategy's `apply_submit_retry` previously wrote an empty
`backoff:{uid}` marker on every retriable submit failure (including
the `TryNextBlock` fallback for unparseable orderbook envelopes). The
marker was a presence flag with no payload, so on every supervisor
reconnect / engine restart the same dead placement would retry
indefinitely — bounded only by log re-delivery frequency.
This change persists a per-UID retry count in the marker's value
(ASCII `u32`) and upgrades to `dropped:` after `MAX_BACKOFF_RETRIES =
5` consecutive retries. The upgrade emits a Warn-level log line so
the operator sees the structural issue (flaky CDN, indexer hiccup,
poisoned envelope) rather than silently accumulating retries.
- `modules/ethflow-watcher/src/strategy.rs`:
- New const `MAX_BACKOFF_RETRIES = 5`.
- New helper `read_backoff_count` that reads + parses the marker
payload; pre-COW-1083 empty markers decode to 0 so previously-set
backoff: rows still get a fresh attempt (no premature drop on
rollout).
- `apply_submit_retry`'s retriable branch now reads the prior
count, increments, and either writes the new count or upgrades
to `dropped:` (clearing the stale `backoff:`) at the cap.
- Cap-upgrade log line carries the retry-count and message: "...
after 5 retries on transient/unparseable rejection ...".
- 19/19 ethflow-watcher tests pass.
- New `submit_transient_error_at_cap_upgrades_to_dropped_warn`:
seeds `backoff:{uid} = "4"`, triggers a `data: None` rejection
(the unparseable case the issue names explicitly), asserts:
* `dropped:{uid}` is now set
* `backoff:{uid}` is cleared (single outcome marker at rest)
* exactly one Warn log line containing "ethflow dropped" +
"retries"
- New `submit_transient_error_with_legacy_empty_marker_resets_counter`:
backwards-compat — a pre-COW-1083 empty `b""` marker is treated
as count 0, bumped to "1" on first retry rather than prematurely
dropping. Protects in-flight backoffs across the rollout.
- Existing `submit_transient_error_writes_backoff_marker_and_returns`
extended with an assertion that the first retry persists
`backoff:{uid} = "1"`.
- `cargo clippy -p ethflow-watcher --all-targets -- -D warnings`
clean.
Surfaced by jeffersonBastos's PR #55 (M3 mirror) review, thread on
`crates/shepherd-sdk/src/cow/error.rs:82`. Latent in normal
operation (the host forwards parseable envelopes after COW-1075, so
`classify_api_error` returns `Drop` for permanent rejections), but
the gap fires when the orderbook returns a non-JSON 4xx body
(e.g. an HTML error page from a CDN) or if a future host change
accidentally drops the envelope again. Bounded retry semantics
close the latent risk without changing the safe-default
classification (still `TryNextBlock` on `None` data — that part is
explicitly out of scope per the issue).
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
Closes the M5 packaging gap surfaced by the audit: the Dockerfile +
compose recipe lived inside `docs/production.md` but neither was at
the repo root, so `docker build .` didn't work and there was no
published image. This change makes the deploy path one-line on a
fresh VM.
- **`Dockerfile`** — multi-stage build (rust:1.96-slim-bookworm →
debian:bookworm-slim). Builds the engine in release + the 5
production modules to wasm32-wasip2. Runtime stage strips down to
`tini` (PID 1 for graceful shutdown / SIGINT forwarding per
COW-1072) + `ca-certificates` (TLS to cow.fi + paid RPCs) + a
non-root `shepherd` user owning `/var/lib/shepherd`. Final image:
**198 MB** (engine + 5 wasm modules + Debian slim).
- **`.dockerignore`** — excludes `target/`, `data/`, the heavy
backtest / baseline JSON fixtures, and local-only engine configs,
while keeping `modules/fixtures/*-bomb` (workspace members; Cargo
rejects the manifest if they're missing) and the source markdown
docs (so `docker exec` can grep them in place).
- **`docker-compose.yml`** — two profiles. Default boots just the
engine with a `shepherd-state` named volume + the operator's
`./engine.toml` mounted ro at `/etc/shepherd/engine.toml`, metrics
on the host loopback (`127.0.0.1:9100`). The `observability`
profile (`docker compose --profile observability up`) layers a
Prometheus container pre-wired to scrape `shepherd:9100`. Graceful
shutdown via `stop_signal: SIGINT` + `stop_grace_period: 30s` per
the production runbook. Healthcheck hits `/metrics`.
- **`engine.docker.toml`** — pre-baked config that matches the
paths the image bakes (`/opt/shepherd/modules/*.wasm`,
`/opt/shepherd/manifests/*.toml`, `/var/lib/shepherd` state
dir). Operator workflow: `cp engine.docker.toml engine.toml`,
swap `<RPC_KEY>` placeholders, `docker compose up -d`.
- **`docs/deployment/docker.md`** — operator runbook. Covers
first-boot, engine.toml configuration, upgrade / rollback,
local-build path, post-deploy verification, cross-links to
`docs/production.md` for the full hardening surface.
- **`docs/deployment/prometheus.yml`** — scrape config consumed by
the observability compose profile.
- **`.github/workflows/docker.yml`** — build + push to
`ghcr.io/bleu/nullis-shepherd` on every push to `main` and every
`v*` tag. PR builds run the build for smoke (no push). Tags
produced: `latest` (main HEAD), `v<tag>` (releases),
`sha-<short>` (every event for exact pinning), `manual-<run_id>`
(workflow_dispatch). Registry-side layer cache via
`:buildcache` keeps incremental rebuilds fast. linux/amd64 only —
the soak VM is x86_64; add arm64 once an operator surfaces a
real need. Action SHAs pinned to match `.github/workflows/ci.yml`
style.
Build runs locally end-to-end in ~10 min on a clean Docker daemon:
$ docker build -t shepherd:smoke .
$ docker run --rm shepherd:smoke --help
usage: nexum-engine [<wasm-path> [<manifest-path>]] \
[--engine-config <path>] [--pretty-logs]
$ docker run --rm -v "$PWD/engine.docker.toml:/etc/shepherd/engine.toml:ro" \
shepherd:smoke
{"level":"INFO","message":"nexum-engine starting",...}
{"level":"INFO","message":"metrics exporter listening at /metrics",...}
{"level":"INFO","message":"opening chain RPC provider","chain_id":1,...}
Error: connect chain 1: HTTP format error: invalid uri character
^- expected: <RPC_KEY> placeholder not a real URL
Proves: image builds, entrypoint forwards CMD, engine loads
`/etc/shepherd/engine.toml`, metrics exporter binds, provider pool
iterates the configured chains, graceful error path works.
- [x] Local `docker build .` succeeds (rust:1.96 base — wasmtime 45
requires rustc >= 1.93, the docs/production.md `1.86` pin was
stale)
- [x] Image size: 198 MB
- [x] `docker run ... --help` works
- [x] `docker run ... -v engine.docker.toml:...` reads config + binds
metrics + iterates chains
- [x] `cargo test --workspace` clean (18 groups, 203 passed, 0 failed)
On a fresh Debian/Ubuntu VM with Docker installed:
```bash
git clone https://github.com/bleu/nullis-shepherd /opt/shepherd
cd /opt/shepherd
cp engine.docker.toml engine.toml
$EDITOR engine.toml # add real RPC URL
docker compose pull # once ghcr.io image is published
docker compose up -d
docker compose logs -f shepherd
curl -s http://127.0.0.1:9100/metrics | head -50
```
- `docs/deployment/multi-chain-guide.md` — dedicated walkthrough
configuring 4 chains together (Mainnet + Gnosis + Arbitrum + Base)
with per-chain module subscriptions
- Example module declaring multi-chain support (every current
example pins Sepolia)
- Optional automated CD trigger (workflow_dispatch SSH'ing to the
soak VM to pull + restart) — gated on SSH_PRIVATE_KEY repo secret
Companion to the M5 Docker packaging — the operator workflow is `cp engine.docker.toml engine.toml` then drop in a paid RPC URL. Without this rule a clumsy `git add -A` could commit the key. The committed sibling templates (engine.example/docker/m2/m3/e2e/load.toml) stay trackable. Validated against a live smoke run: drpc Sepolia WSS endpoint pasted into engine.toml, `docker compose up`, engine subscribed to newHeads + logs, 6 sequential blocks dispatched (11117171..76), metrics `shepherd_event_latency_seconds` p99 = 0.14ms. Tear-down clean. No engine.toml ever staged.
Closes the footgun surfaced by the M5 smoke run on drpc Sepolia:
configuring `rpc_url = "https://..."` for a chain that the modules
subscribe to silently degrades to an infinite WARN-with-backoff loop
(COW-1071's reconnect retries forever because `eth_subscribe` is
WS-only in the JSON-RPC spec). Three coordinated changes:
`EngineConfig::validate_transports()` walks every `[chains.<id>]`
entry, and for any `rpc_url` not starting with `ws://` / `wss://`
emits one loud ERROR-level structured log line with:
- the chain id
- the redacted offending URL
- the redacted suggested `wss://` swap
- actionable copy explaining the WS requirement and the escape
hatch (`[chains.<id>] require_ws = false` for poll-only chains
that never subscribe)
The validator is invoked from `main.rs` AFTER the tracing
subscriber is initialised (calling it inside `load_or_default`
silently dropped the log).
A `require_ws: bool` field is added to `ChainConfig` with
`#[serde(default = "default_require_ws")]` = `true`. Operators who
genuinely need an HTTP endpoint (poll-only modules, no block / log
subscriptions on this chain) opt out explicitly per chain.
The pre-existing `opening chain RPC provider` log in
`provider_pool::from_config` was emitting the full URL — API key
included — at INFO level. Log aggregators (Loki / Datadog / Splunk)
routinely retain weeks of these lines; the key has no business
sitting in cold storage. The new `engine_config::redact_url` helper
(public so other call sites can adopt it) replaces any path segment
longer than 20 chars that doesn't contain `.` or `:` with `<KEY>`.
Matches Alchemy / drpc / Infura / QuickNode key shapes.
Same helper is used for both the validation ERROR's `rpc_url` and
`suggested` fields and the provider-pool boot log.
- `engine.example.toml`: every chain entry switched to `wss://`,
with a header block explaining the WS requirement + the
`require_ws = false` escape hatch. The previous mix of `https://`
+ `wss://` would have tripped the new validator on its own example.
- `docs/production.md §6`: blockquote callout pointing operators at
the WS requirement, redaction behaviour, and the escape hatch.
Smoke 1 (HTTP, expected to ERROR):
{"level":"ERROR","message":"rpc_url uses HTTP transport but the engine subscribes to blocks/logs via eth_subscribe (WS-only). [...]","chain_id":11155111,"rpc_url":"https://lb.drpc.live/sepolia/<KEY>","suggested":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
Smoke 2 (WSS, expected to pass + redacted):
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,"url":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
- 9 new unit tests in `engine_config::tests`:
* `validate_accepts_wss_url`, `validate_accepts_ws_url`
* `validate_is_silent_when_require_ws_is_false`
* `validate_runs_without_panicking_on_http_url`
* `suggest_swaps_https_to_wss`, `suggest_swaps_http_to_ws`,
`suggest_passes_through_already_ws_url`
* `redact_replaces_long_path_segments`,
`redact_keeps_short_segments_intact`
- Workspace: 18 groups, **212 passed, 0 failed** (was 203 → +9)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
Operator workflow before this change forced the paid-RPC URL to
live in a file (`engine.toml`), which is fine for systemd but
awkward for Docker/compose: the URL had to be hand-edited inside a
volume-mounted file, secrets and config got tangled, and the
internal drpc test key was at risk of slipping into a committed
example. This change makes the engine treat `${VAR_NAME}` tokens
inside `engine.toml` as environment-variable references, resolved
at config-load time:
[chains.11155111]
rpc_url = "${SEPOLIA_RPC_URL}"
The `engine.docker.toml` and `engine.example.toml` templates ship
with `${VAR}` placeholders for all five chains, so the committed
files stay secret-free regardless of deployment path.
cp .env.example .env
$EDITOR .env # paste real wss:// URLs
docker compose up -d
`docker compose` reads the repo-root `.env` automatically (already
the compose default) and forwards the named variables into the
container via the new `environment:` block; the engine substitutes
them when parsing `/etc/shepherd/engine.toml`.
- `engine_config.rs::substitute_env_vars` — hand-rolled parser
(no regex dep) that walks the raw TOML text, matches `${NAME}`
tokens against `[A-Z_][A-Z0-9_]*`, and looks each up via
`std::env::var`. Three error variants via `thiserror`:
* `Missing { name }` — variable referenced but unset; message
includes the exact name and a pointer to the `.env` workflow.
* `InvalidName { name }` — typo (lowercase, leading digit);
suggests the upper-cased variant.
* `Unclosed { offset }` — `${` without matching `}`.
- Called from `load_or_default` before `toml::from_str`, so the
substitution layer never sees parsed TOML — a missing env var
surfaces with the exact variable name, not a downstream
"invalid URI character" several layers deep.
- Substitution runs over the whole file (comments included; harmless).
- `.env.example` — committed template with placeholders for all 5
chain `*_RPC_URL` variables + the optional `SHEPHERD_IMAGE` and
`SHEPHERD_ENGINE_CONFIG` overrides.
- `.gitignore` — adds `!.env.example` exception so the template
stays trackable while `.env` and `.env.local` etc. stay ignored.
- `docker-compose.yml` — passes the five `*_RPC_URL` env vars
through to the container; the engine config bind-mount now
defaults to `engine.docker.toml` (the committed template) and
honours `SHEPHERD_ENGINE_CONFIG` for operators who prefer a
bespoke file.
- `engine.docker.toml` + `engine.example.toml` — every `[chains.*]`
entry switched to `${*_RPC_URL}` placeholders. Header comments
spell out the workflow.
- `docs/deployment/docker.md` — first-boot section now leads with
`cp .env.example .env` (was `cp engine.example.toml engine.toml
&& edit`). §2 explains the bind-mount + the
`SHEPHERD_ENGINE_CONFIG` escape hatch.
Smoke 1 (compose end-to-end):
$ cp .env.example .env
$ echo "SEPOLIA_RPC_URL=wss://lb.drpc.live/sepolia/<real-key>" >> .env
$ echo "SHEPHERD_ENGINE_CONFIG=./engine.local.toml" >> .env
$ docker compose up -d
...
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,
"url":"wss://lb.drpc.live/sepolia/<KEY>",...} ← env-resolved, key redacted
{"level":"INFO","message":"supervisor up","loaded":2,"alive":2,...}
{"level":"INFO","message":"block subscription open","chain_id":11155111,...}
{"level":"INFO","message":"log subscription open","module":"twap-monitor",...}
{"level":"INFO","message":"log subscription open","module":"ethflow-watcher",...}
$ docker compose logs | grep -c <real-key>
0 ← zero leaks
$ curl -s http://127.0.0.1:9100/metrics | grep latency_seconds_count
shepherd_event_latency_seconds_count{module="twap-monitor",event_kind="block"} 4
Smoke 2 (missing env var, expected fail-fast):
$ unset SEPOLIA_RPC_URL
$ docker compose up
Error: engine config env-var substitution failed: environment variable
`SEPOLIA_RPC_URL` referenced via ${SEPOLIA_RPC_URL} in engine.toml but
not set. Export it before launching the engine (e.g. via a `.env`
file consumed by `docker compose`).
- 7 new unit tests in `engine_config::tests`:
* `substitute_replaces_known_variable`
* `substitute_errors_on_missing_variable`
* `substitute_errors_on_invalid_name`
* `substitute_errors_on_unclosed_brace`
* `substitute_passes_text_with_no_placeholders_through`
* `substitute_handles_multiple_placeholders_in_one_line`
* `substitute_preserves_utf8_around_placeholder`
- Workspace: 18 groups, **219 passed, 0 failed** (was 212 → +7)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
VM smoke surfaced a false-negative `(unhealthy)`: the compose healthcheck called `wget` but the runtime image is built on debian:bookworm-slim which doesn't include it (only ca-certificates + tini, intentionally minimal). `wget: not found` → exit 127 → unhealthy mark, despite the engine actually working (21 blocks dispatched in 3 min, p99 latency 0.09ms, zero errors). Swap to bash's `/dev/tcp` builtin (always present in bookworm-slim's `/bin/bash`). Successful TCP open on the metrics port proves the exporter bound, which only happens after the supervisor finishes boot — same semantic, no image growth.
First fix attempt swapped wget for `/dev/tcp` but kept `CMD-SHELL`, which routes through `/bin/sh` (dash on debian:bookworm-slim). dash doesn't have the `/dev/tcp/<host>/<port>` builtin — it's bash- only. Probes failed with "cannot create /dev/tcp/...: Directory nonexistent". Switch to `CMD ["bash", "-c", ...]` so the bash builtin actually resolves. `bash` ships in the slim base; verified via `docker exec shepherd which bash` → `/usr/bin/bash`.
Cherry-pick of PR #62 + PR #63's redesign onto the M5 host runtime (env-var substitution in engine.toml, healthcheck fixes, etc) for the Sepolia soak VM. The PR review continues on the proper layered branches: - PR #62 — M2/BLEU-833 layer observe design - PR #63 — M3/M4 BLEU-855 split + COW-1074 cow_api_request integration This branch is deploy-only: it lets the soak run on the redesigned ethflow-watcher with the latest host runtime while review iterates on the layered PRs. After merge, this branch can be deleted; CI will republish ghcr.io/bleu/nullis-shepherd:latest with the merged design and the VM rolls forward to the official image. See COW-1076 for the full empirical evidence.
…store (COW-1085)
`getTradeableOrderWithSignature` returns the same Ready tuple in every
poll-tick during a TWAP child's validity window — the on-chain
conditional order has no way to know shepherd already POSTed it. The
strategy already wrote a `submitted:{uid}` marker after a successful
submit, but the next poll-tick polled the chain and submitted again,
producing a wasted orderbook call and a misleading
`DuplicatedOrder` Warn line in every soak that runs a TWAP.
Live evidence (2026-06-23 Sepolia soak):
10:02:36.784 INFO poll watch:0x8fab71c0...:0x93b1626c... -> Ready
10:02:37.190 INFO submitted submitted:0xd7116bd2...
10:02:48.870 INFO poll watch:0x8fab71c0...:0x93b1626c... -> Ready
10:02:49.855 WARN submit dropped watch (400): orderbook error
(DuplicatedOrder): order already exists
The first submission succeeded (`GET /api/v1/orders/0xd7116bd2...`
returns `status: fulfilled`); the second was wasted work.
The fix: at the top of `submit_ready`, compute the client-side UID
deterministically from the on-chain `(order, owner, chain)` tuple via
`OrderData::uid` and check `submitted:{uid}` in local-store; skip the
submit (and the appData resolve that precedes it) when the marker
exists. The marker write site is also updated to use the
client-computed UID for the key so the read and write paths agree
(in production the server-returned UID is the same value — both sides
derive it from the canonical `digest || owner || valid_to` layout —
and a divergence is now surfaced via a Warn).
Tests (24, all green natively + wasm32-wasip2):
* Existing `poll_ready_submits_order_and_persists_submitted_uid` and
`poll_ready_resolves_non_empty_app_data_then_submits` updated to
compute the expected marker key via `compute_uid_hex` instead of
hardcoding `submitted:0xfeedface` (the mock orderbook's stub UID,
which now triggers the divergence Warn so we also assert that).
* New `poll_ready_skips_submit_when_submitted_uid_already_in_store`:
seeds the marker, dispatches a block tick, asserts
`submit_order` (and the preceding appData resolve) are NOT called
and that the expected Info log appears.
Out of scope (deferred): the same idempotency pattern could be
applied to ethflow-watcher's `observed:{uid}` marker (already correct
there — the GET-not-POST design makes this naturally idempotent).
…econnects (COW-1086)
Adds a positive-recovery Info log when the block subscription
resumes after a silence ≥ 60 s, covering the observability gap
identified in the 2026-06-23 Sepolia soak.
## Background
The 2026-06-23 soak surfaced this sequence:
09:05:43 ERROR WS connection error: WebSocket protocol error:
Connection reset without closing handshake
target=alloy_transport_ws::native
(no further WS-related lines for ~1 h)
10:02:24 INFO indexed watch:... ← twap-monitor activity resumes
10:05:24 INFO ethflow observed ... ← ethflow-watcher activity resumes
`docker ps` showed 0 restarts and the container stayed healthy
throughout — alloy's transport layer reconnected internally without
the engine's `reconnecting_block_task` ever observing
`inner.next().await -> None`. So the engine never entered its
"stream ended → backoff → subscription reopened" path, and the
existing `block subscription reopened` Info log (COW-1071) never
fired. The transport-layer ERROR followed by silence is
indistinguishable from a hung engine on a soak dashboard.
## What changes
In `reconnecting_block_task`, on every yielded item compare
`now.duration_since(last_event)` against `BLOCK_GAP_LOG_THRESHOLD`
(60 s, 5× Sepolia block time). When the gap meets or exceeds the
threshold, emit:
INFO chain_id=... gap_s=... kind="block"
"stream gap closed - first event after silence
(likely an alloy-internal transport reconnect)"
The gap-detection logic is factored into a small synchronous helper
`block_stream_gap_to_log(now, last_event, threshold) -> Option<Duration>`
so it can be unit-tested without spinning up an async runtime or a
real provider.
## Why blocks only (not logs)
Block subscriptions have predictable cadence — Sepolia produces a
new block every ~12 s, mainnet every ~12 s. A 60 s silence is
therefore anomalous and worth surfacing. Log subscriptions, by
contrast, are inherently sparse (driven by on-chain user activity),
so the same threshold would fire false positives on quiet windows.
The existing `log subscription reopened` log already handles the
engine-detectable reconnect for log streams.
## Tests
4 new unit tests on the gap-detection helper:
* `block_stream_gap_to_log_returns_none_when_no_prior_event`
* `block_stream_gap_to_log_returns_none_when_under_threshold`
* `block_stream_gap_to_log_returns_some_at_threshold_boundary`
* `block_stream_gap_to_log_returns_some_when_well_over_threshold`
All 90 nexum-engine tests pass (86 existing + 4 new). Clippy strict
clean, fmt clean. Wasm build untouched.
## Out of scope
* End-to-end test of `reconnecting_block_task` against a mock
provider — no existing scaffolding for that path, and the gap
helper covers the decision logic deterministically.
* Suppressing or downgrading the `alloy_transport_ws::native` ERROR
itself — it is a legitimate transport-layer event, just one whose
recovery wasn't previously observable. The new Info line closes
that loop without losing the original signal.
## Live validation
The next time alloy auto-reconnects internally on the soak VM, the
new line will surface as a structured JSON event with
`gap_s=<seconds>` so the soak dashboard can correlate it with the
preceding transport ERROR.
) Squash of PR #68 - 9 markdown files reconciled (5 vapor items rephrased as future direction + capability-gating diagrams aligned to link-time enforcement). Verified: cargo doc --workspace --no-deps clean.
`observe_placement` matches on the `Result<String, HostError>` returned
by `host.cow_api_request(...)`. The M5 conflict resolution (COW-1082
ErrorResp data forwarding) accidentally pasted a wildcard arm that
belongs to `apply_submit_retry`'s `match classify_api_error(...)` (which
matches a `RetryAction` enum). On a `Result`, the wildcard is both
unreachable (`Ok(_)` and `Err(_)` already cover everything) and
references an `err` binding that doesn't exist in its scope:
error[E0425]: cannot find value `err` in this scope
--> modules/ethflow-watcher/src/strategy.rs:205:21
--> modules/ethflow-watcher/src/strategy.rs:205:31
The `Err(err) if err.code == 404` arm + bare `Err(err)` arm already
classify every error case. Drop the spurious `_ =>` arm; bring
`observe_placement` back to fmt/clippy/test green on dev/m5-base.
…terError Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #1 (remaining enums introduced on the M5 multi-chain pass). - `EnvVarError` (engine_config.rs): introduced with the COW-1071 env-var substitution path. Snake_case variant labels feed the boot-time `tracing::error!(error_kind = ...)` call sites in `main.rs`. - `FilterError` (supervisor.rs): introduced with the M5 multi-chain log-filter parsing. Snake_case variant labels feed the `tracing::warn!(error_kind = ...)` log emitted when a `[[subscription]]` address or topic fails to parse. The audit's M3 / M4 derives landed on the milestones that introduced the enums; these two complete the workspace-wide IntoStaticStr pass flagged in audit Major #1 on the milestones that own them.
Audit reference: milestone-rubric-grant-audit-2026-06-25.md, Major #6. The rubric forbids em-dashes in "code, rustdoc, commit messages, PR bodies, or review comments". `.toml` is technically a grey zone but these comments surface verbatim when operators `cat engine.docker.toml` or `engine.example.toml` during deployment onboarding. Mechanical find/replace to ` - ` (ASCII hyphen with spaces). Files touched: - engine.example.toml: 2 em-dashes (lines 20, 38) - engine.docker.toml: 4 em-dashes (lines 4, 5, 6, 31)
…rse (audit JC5) shepherd-backtest's offline replay harness carried its own `AddressParseError` enum (hex-decode + length check). The shape overlaps directly with the `AddressParse` typed error introduced into `shepherd-sdk` by the audit JC5 pass. Extend `shepherd_sdk::address` with a single-address `parse_address` helper alongside the existing `parse_address_list` (the `InvalidAddress` variant covers both call sites via the `index` field). Replay's `fixtures::parse_address` becomes a thin wrapper that calls the SDK and converts the `Address` to the `[u8; 20]` shape the strategy consumes via `LogView::address`. Drops the now-unused `thiserror` dependency from shepherd-backtest; `hex` stays for topic/data decoding.
brunota20
force-pushed
the
feat/composable-cow-m2-integration
branch
from
82a5adc to
c81ed22
Compare
Adds the batched poll path
(`ComposableCoW.batchGetTradeableOrdersWithSignature`) behind a runtime
flag (`module.toml::[config].use_batch_poll`, default `"false"`).
When the flag is on, the per-block poll loop collapses N separate
`eth_call`s into a single batched call:
1. Collect every ungated `watch:` row.
2. Build a single `batchGetTradeableOrdersWithSignatureCall` with
one `BatchOrderRequest` per watch.
3. Issue one `eth_call` against `ComposableCoW`.
4. Decode `BatchOrderResult[]` via
`cowprotocol::decode_batch_order_results`.
5. Dispatch each `BatchOrderOutcome` through the existing submit /
lifecycle paths:
- `Submitted { order, signature }` -> `submit_ready` (unchanged).
- `PollHint(PollOutcome)` -> `cow_poll_to_sdk` ->
`outcome_to_update` -> `apply_watch_update` (same gate state
as the legacy path).
- `ComposableCoWError(_)` -> drop the watch.
- `UnknownRevert(_)` -> leave watch in place + log Warn.
Default `"false"` because today's deployed `ComposableCoW` does not
expose the batch view; flipping the flag against today's contracts
would make every poll-tick `eth_call` revert and every watch retry
next block. Legacy `poll_all_watches` is untouched.
Also bumps `[patch.crates-io] cowprotocol` to a head that lands the
matching bindings (`BatchOrderRequest` / `BatchOrderResult`,
`PollOutcome`, `BatchOrderOutcome`, `ComposableCoWError`,
`decode_batch_order_results`, `registered_topic_filter_by_handler`).
brunota20
changed the title
brunota20
force-pushed
the
feat/composable-cow-m2-integration
branch
from
c81ed22 to
0d1a1c1
Compare
jean-neiverth
force-pushed
the
dev/m5-base
branch
from
9ad747e to
8cb1b43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the batched poll path
(
ComposableCoW.batchGetTradeableOrdersWithSignature) behind a runtimeflag (
module.toml::[config].use_batch_poll, default"false").When the flag is on, the per-block poll loop collapses N separate
eth_calls into a single batched call:watch:row.batchGetTradeableOrdersWithSignatureCallwith oneBatchOrderRequestper watch.eth_callagainstComposableCoW.BatchOrderResult[]viacowprotocol::decode_batch_order_results.BatchOrderOutcomethrough the existing submit /lifecycle code:
Submitted { order, signature }→submit_ready(unchanged).PollHint(PollOutcome)→cow_poll_to_sdk→outcome_to_update→apply_watch_update(same gate state asthe legacy path).
ComposableCoWError(_)→ drop the watch.UnknownRevert(_)→ leave the watch in place + log Warn.Default
"false"because today's deployedComposableCoWdoes notexpose the batch view; flipping the flag against today's contracts
would make every poll-tick
eth_callrevert and every watch retrynext block. Legacy
poll_all_watchesis untouched.What changed
Cargo.toml:[patch.crates-io] cowprotocolrev bumped to pickup the matching bindings (
BatchOrderRequest/BatchOrderResult,PollOutcome,BatchOrderOutcome,ComposableCoWError,decode_batch_order_results,registered_topic_filter_by_handler).modules/twap-monitor/module.toml: new[config]block withuse_batch_poll = "false"(default).modules/twap-monitor/src/lib.rs:Guest::initreadsuse_batch_pollfrom config and writes the runtime flag intolocal-store for
strategy::on_blockto read.modules/twap-monitor/src/strategy.rs: newpoll_all_watches_batched(gated by the runtime flag) +cow_poll_to_sdkbridge + 7 new batched-dispatch tests.Out of scope
ConditionalOrderRegisteredsubscription.Requires extending
crates/nexum-engine/src/manifest/types.rs::SubscriptionKind::Logfrom
event_signature: Option<String>to a 4-slottopics: [Option<String>; 4]. Tracked as a follow-up; the binding-shape test(
conditional_order_registered_decodes_indexed_handler) pins thewire format so the eventual subscription patch is mechanical.
getOrderInfo. The TWAP module has no pre-flightreads to collapse; the helper will land with whoever consumes it
(EthFlow watcher / future strategy modules).
_twap_calldata.pyworkaround. That cleanupis gated on the new contracts being live — the workaround is still
required against today's deployed TWAP.
Rollout
use_batch_poll = "false". No behaviour change in production.use_batch_poll = "true"against the new deployment._twap_calldata.py.Manifest schema bump for indexed-topic subscriptions.
Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace --all-featurestestnet runbook with
use_batch_poll = "true"to confirm thebatched path against a live deployment.
scripts/_twap_calldata.py+ verifye2e-onchain.shagainst the new TWAP.