feat(codec): zero-copy SBE codec#52
Merged
Merged
Conversation
… Codec paths Add an `sbe` feature (no new dependencies) with SbeMessage, SbeHeader, SbeFrame<T>, and SbeCodec: topics carry owned, header-validated SBE wire frames, so handlers wrap sbe-tool-generated flyweight decoders directly over the received buffer with no intermediate struct. Extend the Codec trait with defaulted encode_bytes/decode_owned so backends can move owned buffers through the codec instead of copying: - NATS: publish and consume are fully zero-copy (Bytes end to end) - InMemory: envelope payload is now Bytes; publish/consume zero-copy - Kafka/RabbitMQ: publish is zero-copy into the client; consume keeps the borrowed-slice path (client-owned buffers), costing one memcpy - SNS/SQS and Redis Streams reject binary SBE frames at publish time via the existing encode_to_string UTF-8 check schemaId/templateId are validated on every frame; wire version and blockLength pass through per SBE extension semantics via frame.header(). bytes is promoted from an optional (kafka/nats) to a required dependency; it was already unconditionally in the tree via tokio-util.
…ck length, real-broker tests Address independent review findings: - SbeCodec::encode_to_string now fails unconditionally with SbeCodecError::StringTransportUnsupported instead of relying on the default UTF-8 check, which would let UTF-8-valid frames ride SNS/SQS and Redis Streams and make failures data-dependent - SbeFrame::new rejects frames shorter than header + acting blockLength (RootBlockTruncated), so flyweight root-block reads cannot run past the buffer - SBE round-trip integration tests against real NATS, Kafka, and RabbitMQ brokers; publish-rejection tests on Redis Streams and SNS/SQS (LocalStack) using a UTF-8-valid frame - CI matrix runs each backend suite with the sbe feature enabled - README and docs: codec reference links and updated rejection semantics
…decode and republish Address review finding: lapin hands over an owned delivery buffer, but the consumer still decoded through the borrowed-slice path, costing SBE frames a copy per delivery. Introduce ReceivedDelivery, created at receipt: the payload moves out of delivery.data into a reference-counted Bytes once. Codec::decode_owned gets a refcount clone, and the hold-queue retry/defer republish paths borrow the same allocation. The concurrent, concurrent-sequenced, and DLQ consume loops all route through it; route_retry/route_defer take the payload explicitly instead of reading delivery.data.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
shove-docs | 1992530 | Commit Preview URL Branch Preview URL |
Jul 07 2026, 03:26 PM |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Simple Binary Encoding (SBE) codec support behind a new
sbecargo feature (zero additional dependencies), preserving SBE's flyweight model end to end: publishing passes the user-encoded frame through without re-encoding, and handlers wrapsbe-tool-generated decoders directly over the received buffer.Design
SbeFrame<T>is the topic message type: an owned, header-validated wire frame backed by reference-countedBytes.Tis a zero-sized type tag implementingSbeMessage(schema id, template id, byte order — the constantssbe-toolgenerates).SbeCodecvalidatesschemaId/templateIdon every decode (typedSbeCodecErrorwith expected/actual ids), and requires the buffer to cover the header plus the actingblockLength. Wireversion/blockLengthpass through per SBE extension semantics viaframe.header(), so cross-version decoding works.Codectrait extension (backward compatible): new defaultedencode_bytes/decode_ownedmethods let backends move owned buffers through the codec. Existing custom codecs compile and behave unchanged.encode_to_stringfails unconditionally (StringTransportUnsupported) — even for frames that happen to be valid UTF-8 — so SNS/SQS and Redis Streams failures are deterministic rather than data-dependent.Per-backend copy behavior
ReceivedDeliveryshares one buffer across decode and retry/DLQ republish)Testing
sbeenabled.Review
Independent codex review loop: 3 rounds, 4 findings fixed (UTF-8-passthrough on string transports, missing root-block length validation, missing real-broker coverage, RabbitMQ consume copy), round 3 returned zero findings.
🤖 Generated with Claude Code