Espresso 3b: TEE batcher (re-hosted) - #459
Conversation
f840eee to
12b46a6
Compare
c78ed9f to
319b3ab
Compare
12b46a6 to
f8480f8
Compare
319b3ab to
3e7dcd9
Compare
9330de1 to
1616378
Compare
6cf6a1f to
eb6ff32
Compare
1616378 to
25a6c63
Compare
eb6ff32 to
ab13906
Compare
| // Sign represents the interface for signing things via eth_sign. | ||
| func (s *SignerClient) Sign(ctx context.Context, address common.Address, data []byte) ([]byte, error) { | ||
| var result hexutil.Bytes | ||
| if err := s.client.CallContext(ctx, &result, "eth_sign", address, data); err != nil { |
There was a problem hiding this comment.
This eth_sign call can't work against op-signer, and I don't think op-signer should be extended to make it work either.
op-signer doesn't serve eth_sign. Its server registers only two namespaces — eth (eth_signTransaction) and opsigner (signBlockPayload, signBlockPayloadV2): https://github.com/ethereum-optimism/infra/blob/main/op-signer/service/service.go#L73-L82. There is no arbitrary-data signing method. And this SignerClient can only talk to op-signer in the first place: NewSignerClient dials with op-signer's mutual-TLS and then handshakes with a health_status ping before returning, so pointing it at a plain geth or another HSM front-end fails at construction. So the call here errors method-not-found at runtime against a real op-signer.
The reason op-signer has no such method is deliberate, and it's why I'd argue against adding one. An HSM-backed signer must never sign raw bytes the caller hands it. If it did, a compromised batcher could pass a 32-byte value that is really the sighash of an L1 transaction spending the funded key, or a block payload for equivocation, and the HSM would sign it. That's why every op-signer method reconstructs the thing being signed server-side from typed arguments and binds a domain tag and chain id into the hash — see BlockPayloadArgs (domain, chainId, payloadBytes) and Message().ToSigningHash(). The client never sends a bare hash. Adding an eth_sign that signs any digest would remove exactly that protection for a key that also signs L1 transactions.
There's a second, backend-independent problem: eth_sign applies the EIP-191 prefix ("\x19Ethereum Signed Message:\n32" || hash), but the verify side recovers over the raw digest (crypto.SigToPub(batchHash, sig) in op-node/rollup/derive/espresso_batch.go):
optimism/op-node/rollup/derive/espresso_batch.go
Lines 105 to 107 in 014f88d
If remote HSM signing of Espresso batches is a requirement, the right shape is a purpose-built op-signer method modeled on signBlockPayload: the client sends typed args (a fixed domain tag, the L2 chain id / namespace, and the batch commitment), op-signer reconstructs the domain-separated digest and signs it with the HSM key, and op-node verifies the same digest. That also resolves the separate domain-separation gap (the batch digest is currently a bare keccak256(rlp(batch)) with no namespace binding). It is a change in the op-signer repo, so it can't land from this PR alone — until it exists, only the local private-key ChainSigner actually works. I'd suggest dropping this eth_sign helper and the clientSigner branch here rather than shipping a path that can't sign or verify.
There was a problem hiding this comment.
Jean is going to look and respond here as he did this work.
There was a problem hiding this comment.
Thanks for the thorough writeup. The analysis is right about op-signer, and the fact that it reads as targeting op-signer at all is a documentation gap on our side, so let me fill in the missing context first.
This Sign call doesn't talk to op-signer. The --signer.endpoint it's deployed against is espresso-kms-signer (https://github.com/EspressoSystems/espresso-kms-signer), a small AWS KMS signing sidecar we built specifically to speak the signer protocol this batcher uses: health_status, eth_signTransaction, and eth_sign (https://github.com/EspressoSystems/espresso-kms-signer/blob/758e4d0/src/rpc.rs#L26-L46). It runs as an ECS sidecar next to op-batcher-tee and has done full batch-posting cycles on our kms test devnets (batches accepted on L1 and on HotShot). On the client side, NewSignerClient isn't actually op-signer-specific: mTLS only kicks in when tlsConfig.Enabled is set (https://github.com/celo-org/optimism/blob/c828d61cd9/op-service/signer/client.go#L31-L65) (plain HTTP otherwise), and the handshake is just a health_status call into a Go string, which the sidecar answers. That genericness is what let us add KMS signing with no batcher code changes.
On EIP-191: agreed that a standard eth_sign would break recovery, but the sidecar's eth_sign is deliberately non-standard; it signs the raw 32-byte digest with no message prefix and returns r||s||v with v ∈ {0,1}, (i.e. go-ethereum's crypto). Sign convention, which makes it semantically identical to the privateKeySigner path in this PR (crypto.Sign(hash, privKey) (https://github.com/celo-org/optimism/blob/c828d61cd9/op-service/crypto/espresso.go#L161)). Both verify the same way under SigToPub. And it's pinned rather than hoped-for: the sidecar's fixture generator (https://github.com/EspressoSystems/espresso-kms-signer/blob/758e4d0/tests/fixtures/gen/main.go#L158-L178) is a Go program that imports op-service/signer itself and records the exact JSON-RPC params bytes geth's RPC client marshals (including the base64 []byte encoding), which CI replays against the production handler; there's also a localstack test that runs the real KMS path end-to-end and asserts the recovered address. That said, you're completely right that calling a method eth_sign while breaking eth_sign semantics is asking for exactly this confusion. I'd be happy to rename it in a follow-up, and we'll add a doc comment on SignerClient.Sign pointing at the sidecar and its semantics either way. (Small note: the espresso_batch.go verify code you linked has since moved into espresso-streamers digest recovery (https://github.com/EspressoSystems/espresso-streamers/blob/1884a718fbf7/op/derivation/espresso_batch.go#L102-L104).)
On "an HSM signer should never sign raw caller-supplied bytes", no pushback on the principle, and I'd rather be precise about what it costs us here. A raw-digest endpoint does mean the sidecar's eth_signTransaction guards (chainId, from, to-allowlist) only protect against a buggy caller, not a malicious one; anyone who can reach the endpoint can get a signature over an arbitrary digest, including an L1 tx sighash. What bounds the damage is that this key was never a batch-eligibility authority: batch acceptance in TEE mode requires an EIP-712 commitment signature from the ephemeral key generated inside the Nitro enclave and verified on-chain via the TEE verifier; the sidecar never touches that key. So a compromised sidecar (or its host) can spend the batcher address's gas funds and inject noise into our own HotShot namespace, but it can't make derivation accept a batch the enclave didn't produce. That's our documented trust model: the sidecar is trusted for availability, not integrity.
You're also right about the missing domain separation, and that one stands on its own: the namespace lives in the transaction envelope outside the signed bytes, so the signature binds neither chain nor namespace, under the local key just as much as the remote one. Fixing it means changing the digest every verifier reconstructs, so it's a coordinated change across the batcher and espresso-streamers with a migration story for payloads already in the stream, and we'll file it as a tracked issue rather than fold it into this PR.
Where I'd push back is on the remedy. A typed, domain-separated signing method modeled on signBlockPayload is the right end state, but it belongs in espresso-kms-signer (op-signer isn't in this deployment), and it should land together with the digest-scheme change since both alter what verifiers recover over. Dropping clientSigner in the meantime wouldn't remove the capability this comment worries about; it would move the key from KMS hardware into batcher memory, which is a strict downgrade for the same attack surface. So my proposal: keep clientSigner/Sign as-is here, add a comment linking the sidecar and its non-standard semantics, and file two linked follow-ups, one for the typed domain-separated method (including the eth_sign rename/retirement) and one for the digest-scheme migration (which I will discuss with the team). Happy to talk through the typed-method design if you have opinions on the shape!
There was a problem hiding this comment.
Hi @jjeangal, I hadn't realised this was talking to the kms signer.
So yes, agreed on the followups 👍
05288e9 to
78e33c9
Compare
|
Hi @lukeiannucci, there are a few conflicts which need resolving before CI will run. |
Bring in op-service/crypto/espresso.go (ChainSigner interface unifying SignTransaction and arbitrary-data signing) and op-service/signer/espresso.go (SignerClient.Sign wrapper around eth_sign). Required by the Espresso batcher to sign batch-authentication payloads with either a remote signer or a local private key, in addition to the existing transaction-signing path. Co-authored-by: OpenCode <noreply@opencode.ai>
Introduce op-node/rollup/derive/espresso_batch.go defining EspressoBatch (a SingularBatch with block number, L1 info deposit transaction, and signer address attached), along with BlockToEspressoBatch and the unmarshaler used by the streamer. Also pulls in the github.com/EspressoSystems/espresso-network/sdks/go dependency, which provides the Espresso transaction and namespace types. Consumed by the Espresso batcher (next commits) to convert L2 blocks into batches submitted to Espresso, and to round-trip those batches back through the streamer. Co-authored-by: OpenCode <noreply@opencode.ai>
Bring in the parts of the espresso/ shared package that are TEE-only: - espresso/cli.go: full CLI flag set for --espresso.enabled, query service URLs, light-client/L1 endpoints, batch-authenticator address, receipt-verification tuning, namespace/origin-height parameters used to construct the espresso-streamer. (The --espresso.fallback-auth-lead-time flag lives in op-batcher/flags/flags.go and was added by the fallback PR.) - espresso/interface.go: EspressoStreamer[B] interface that wraps github.com/EspressoSystems/espresso-streamers/op.BatchStreamer. - espresso/ethclient.go: AdaptL1BlockRefClient adapter (used by cli.go to construct the streamer) and FetchEspressoBatcherAddress helper. Also adds the EspressoSystems/espresso-network/sdks/go and EspressoSystems/espresso-streamers Go module dependencies. The regenerated BatchAuthenticator bindings already live in the fallback PR's espresso/bindings/. Co-authored-by: OpenCode <noreply@opencode.ai>
Bring in op-batcher/enclave/attestation.go: a thin wrapper around the hf/nsm library that obtains an AWS Nitro NSM attestation document over a given public key. Used by the Espresso batcher (next commit) to attach a TEE attestation to its registration with the BatchAuthenticator. Adds the github.com/hf/nsm dependency. Builds on all platforms; NSM device access is only attempted at runtime when invoked from inside a Nitro enclave. Co-authored-by: OpenCode <noreply@opencode.ai>
Add the Espresso TEE batcher write-path on top of the fallback batcher: - op-batcher/batcher/espresso.go: Espresso submission loop (peeks the channel manager, converts each L2 block to an EspressoBatch, submits it to Espresso, waits for inclusion, and then posts the batch txs to L1 with TEE-attested BatchAuthenticator.authenticateBatchInfo calls). - op-batcher/batcher/espresso_service.go: EspressoBatcherConfig, initEspresso (Espresso client / light-client construction, optional TEE attestation gathering), and the initChainSigner hook that wraps the txmgr into a opcrypto.ChainSigner. - op-batcher/batcher/espresso_helpers_test.go and espresso_transaction_submitter_test.go: unit tests for the helpers and the TEE transaction submitter. Extends the existing fallback wiring: - op-batcher/batcher/espresso_driver.go: adds EspressoDriverSetup fields (Client/LightClient/ChainSigner/SequencerAddress/Attestation), batcherL1Adapter, setupEspressoStreamer, startEspressoLoops, resetEspressoStreamer; extends dispatchAuthenticatedSendTx with the TEE branch (always authenticates when Espresso.Enabled). - op-batcher/batcher/espresso_active.go: adds isBatcherActive (queries BatchAuthenticator.activeIsEspresso to gate publishing against this batcher's role). - op-batcher/batcher/driver.go: extends DriverSetup with the Espresso EspressoDriverSetup field; adds espressoSubmitter / espressoStreamer / teeVerifierAddress / degradedLog fields on BatchSubmitter; calls setupEspressoStreamer in NewBatchSubmitter; branches StartBatchSubmitting on Espresso.Enabled to call startEspressoLoops; calls resetEspressoStreamer in clearState. - op-batcher/batcher/service.go: BatcherConfig.Espresso field; EspressoClient / EspressoLightClient / ChainSigner / Attestation runtime fields; initEspresso / initChainSigner / applyEspressoDriverSetup call-outs. - op-batcher/batcher/config.go: thread Espresso espresso.CLIConfig through CLIConfig. - op-batcher/flags/flags.go: register espresso.CLIFlags (TEE-only flags; the --espresso.fallback-auth-lead-time flag added by the fallback PR continues to live in op-batcher/flags/flags.go). Also adds op-service/log/repeat_state.go (RepeatStateLogger) and its test, used by the Espresso submission loop's tick-driven warnings. A safeTestRecorder helper is inlined into the test to avoid pulling in the unrelated debouncer. Adds the github.com/hf/nitrite dependency (transitively required by hf/nsm for attestation document parsing). Co-authored-by: OpenCode <noreply@opencode.ai>
The TEE batcher's Espresso submission path called Txmgr.Send directly for both the authenticateBatchInfo tx and the batch inbox tx, and ran inside authGroup, so it bypassed MaxPendingTransactions, assigned nonces nondeterministically (violating Holocene's in-order L1 inclusion requirement), and never checked whether the auth tx reverted — a reverted authenticateBatchInfo emits no event, so the verifier would silently drop the batch. Submit both txs through the ordered queue.Send path on the publishing-loop goroutine (auth first, batch second) so the auth tx takes the lower nonce and is mined first, and both stay under MaxPendingTransactions. A watcher goroutine (tracked by authGroup) collects both receipts, fails the pair if the auth tx reverted, runs the lookback-window check, and emits a single synthetic receipt. This is the same fix already applied to the fallback path; extract the shared submission + receipt-watching flow into submitAuthenticatedBatch / watchAuthReceipts so both paths reuse it and differ only in how the authenticateBatchInfo calldata is built (TEE-attested signature vs empty signature). Rename fallback_auth.go to espresso_auth.go to reflect the shared scope, and restructure the tests: one suite drives the shared flow directly, plus per-path tests asserting the distinguishing auth calldata (empty sig vs a recoverable EIP-712 signature). Co-authored-by: OpenCode <noreply@opencode.ai>
Port TestBatchRoundtrip from the integration branch and fold it into espresso_batch_test.go. It is the only test covering ToEspressoTransaction and the batcher->derivation serialization path; it asserts the decoded batch matches the original and that the recovered signer is the batcher. Also drop the decodedBlock.ExecutionWitness() comparison in TestEspressoBatchConversion: that method does not exist on the op-geth types.Block pinned in the rebase-18 base, so go vet of the derive_test package failed to build. EspressoBatch/ToBlock carries no execution witness. Co-authored-by: OpenCode <noreply@opencode.ai>
f227870 to
a243125
Compare
| // Sign is a function that provides the ability to sign a transaction | ||
| func (c *Config) SignTransaction(ctx context.Context, address common.Address, tx *types.Transaction) (*types.Transaction, error) { | ||
| return c.ChainSigner.SignTransaction(ctx, address, tx) | ||
| } | ||
|
|
||
| // Sign is a function that provides the ability to sign a hash | ||
| func (c *Config) Sign(ctx context.Context, hash []byte) ([]byte, error) { | ||
| return c.ChainSigner.Sign(ctx, hash) | ||
| } |
There was a problem hiding this comment.
These two methods seem redundant, can't they be inlined in the methods below?
There was a problem hiding this comment.
Yea just double checked, should be fine to inline
57f013d
| batch := l.EspressoStreamer().Peek(ctx) | ||
| if batch == nil { | ||
| return nil | ||
| } | ||
|
|
||
| // Check if we can set the tip if not set | ||
| if tip == (common.Hash{}) && (*batch).Number() == syncStatus.SafeL2.Number+1 { | ||
| l.Log.Info( | ||
| "setting tip to safe l2 hash", | ||
| "batchNr", (*batch).Number(), | ||
| "batchParent", (*batch).Header().ParentHash.Hex(), | ||
| "tip", tip, | ||
| ) | ||
| tip = syncStatus.SafeL2.Hash | ||
| } | ||
|
|
||
| if tip == (common.Hash{}) { | ||
| l.Log.Warn( | ||
| "tip is not set, taking available batch", | ||
| "blockParentHash", (*batch).Header().ParentHash.Hex(), | ||
| "blockHash", (*batch).Header().Hash().Hex(), | ||
| ) | ||
| return batch | ||
| } | ||
|
|
||
| if (*batch).Header().ParentHash != tip { | ||
| l.Log.Warn( | ||
| "head batch fork mismatch, seeking to proper head", | ||
| "batchNr", (*batch).Number(), | ||
| "batchParent", (*batch).Header().ParentHash, | ||
| "tip", tip, | ||
| ) | ||
| l.EspressoStreamer().SetProperHead(tip) | ||
| return nil | ||
| } |
There was a problem hiding this comment.
@jjeangal There can be a bug here, if for some reason the streamer was reset and channel manager was not, the channel manager can serve tip hash for block 100. Then here we call SetProperHead(tip(block 100)) Since the streamer is at position 50, it would discard 50 because its expecting 49s hash, and got 100 instead. Essentially stalling the streamer.
Fix: Check that the channel manager tip with block number and next position streamer matches. If not dont call SetProperHead. Or do the tip tracking in streamer
4 participants
Based on #448
Pulls in the Espresso/TEE batcher .
op-batcher/enclave/attestation.goand modifies the driver to add an Espresso path. Bulk of the changes is inespresso_-prefixed files.This is #447, re-hosted from an in-repo branch (now properly stacked).