Off-chain mech requests: second request to the same mech gets 401 until the first settles on-chain (client must advance the wire nonce itself)

[bennyjo]

Summary

The mech's off-chain admission gate (mech v0.35.0+, task_execution/handlers.py) expects each request from a sender to carry the next sequential wire nonce:

wire_nonce == mapNonces[sender] + len(accepted[sender]) + len(settling[sender])

The mech team confirms this is intended, and that a client sending several requests is expected to use x, x+1, x+2 itself. pearl-connect does not: MechAllowances.register_offchain_digest reads mapNonces(safe) and signs with that value for every request, and mech-client 0.21.2 does the same inside send_request (services/marketplace_service.py, still true on mech-client main). Because an accepted nonce stays reserved on the mech until its delivery batch settles on-chain, the second off-chain request from this service to a given mech is rejected with

Offchain request rejected: wire nonce below sender's next expected slot (HTTP 401)

for the whole settlement window, whether the requests are parallel or minutes apart. The only recovery today is legacy_on_chain=true, which costs a marketplace transaction and about 0.13 POL per request.

Evidence (Polygon, 2026-09-03, sender = service safe 0x9657842c…5934, from pearl-connect's activity log)

Mech 21 (0x76a9a2…ab7):

Time (UTC)	Wire nonce sent	Outcome
17:26:20	14	503 eip1271 isValidSignature call timed out (infra, nonce not consumed)
17:26:59	14	accepted, delivered off-chain
17:27:12	14 (new request_id)	401 nonce below expected
17:28:58	on-chain fallback	delivered

Mech 44 (0xe7f818…00b):

Time (UTC)	Wire nonce sent	Outcome
20:54:12	16	503 on-chain mapNonces read failed (infra, nonce not consumed)
20:54:33	16	accepted, delivered off-chain
20:55:06	16 (new request_id)	401 nonce below expected
20:58:20	on-chain fallback	delivered

Suggested approach (open to better ideas)

1. mech-client: it would help if callers could influence the wire nonce. Two shapes come to mind: a nonce / nonce_offset argument on send_request (and the single-request helpers), or per-(sender, mech) in-flight tracking inside the library that adds the count of accepted-but-unsettled requests to mapNonces. The library already does current_nonce + i within one batched call, so the offset form looks like a small extension. Tracked separately in Off-chain send_request always uses on-chain mapNonces as the wire nonce; a sender's second request to a mech gets 401 until settlement mech-client#254.
2. pearl-connect: could track accepted-but-unsettled off-chain requests per (chain, mech) and derive the wire nonce as mapNonces(safe) + in_flight, both in register_offchain_digest (the pre-registered allowance digest) and in the value handed to mech-client, so the two stay identical. The count would be pruned when mapNonces advances past a tracked nonce. On a 401 NONCE_BELOW_EXPECTED it might advance and retry once before falling back to on-chain, keeping the activity-log record of the fallback it writes today.
3. Mech side, optional: would the team consider including expected_next_nonce in the 401 body? A client that lost track (restart, or a second process sharing the safe) could then resync without guessing. Not needed for 1 and 2.

Happy to adjust the split between the two repos if the maintainers see it differently.

Limits of a client-side fix

In-flight tracking only knows about requests this process sent. Two processes sharing one safe, or a restart mid-window, would still collide until settlement. That seems acceptable for Pearl (one connect service per safe), and it is why 3 is listed as the safety net.

Not part of this issue

The two 503s in the tables (eip1271 isValidSignature call timed out, on-chain mapNonces read failed) are mech-side RPC timeouts. The gate handles them correctly by not consuming the nonce, and a retry succeeds.

Related

• Off-chain send_request always uses on-chain mapNonces as the wire nonce; a sender's second request to a mech gets 401 until settlement mech-client#254: the library-side half (caller-controlled wire nonce or in-flight tracking in send_request).