Skip to content

Add to_segment_views for zero-copy serialization - #405

Merged
haata merged 1 commit into
capnproto:masterfrom
BigTailFox:feat/to-segment-views
Jul 3, 2026
Merged

Add to_segment_views for zero-copy serialization#405
haata merged 1 commit into
capnproto:masterfrom
BigTailFox:feat/to-segment-views

Conversation

@BigTailFox

Copy link
Copy Markdown
Contributor

This is a implementation of #402

Summary

This PR adds a new opt-in zero-copy API for segmented message output:

segment_views = msg.to_segment_views()

to_segment_views() returns a sequence of read-only Python buffer exporters that point directly at the Cap'n Proto message builder's output segments. This lets high-throughput callers pass segments to buffer-consuming transports without first copying each segment into a Python bytes object.

The existing to_segments() API is preserved and still returns Python-owned copied bytes objects.

Motivation

to_segments() currently materializes each Cap'n Proto output segment as Python bytes, which is safe and backwards-compatible but forces a copy before the data reaches a transport or IPC layer. Cap'n Proto's segmented layout is naturally suited for scatter-gather and multipart transports, so exposing borrowed read-only segment views gives performance-sensitive users a way to avoid that extra copy.

Changes

  • Adds internal _SegmentViews and _SegmentView Cython types.
  • Implements the Python buffer protocol for each _SegmentView using PyBuffer_FillInfo(..., readonly=1, ...).
  • Adds _DynamicStructBuilder.to_segment_views().
  • Keeps _DynamicStructBuilder.to_segments() behavior unchanged and updates its docstring to clarify that it returns copied bytes.
  • Documents the new API and its lifetime constraints in the Byte Segments quickstart section.
  • Adds tests covering read-only buffer behavior, roundtripping through from_segments(), compatibility with to_segments(), root-only behavior, and lifetime pinning.

Compatibility and Lifetime

to_segments() remains the compatibility API:

segments = msg.to_segments()
assert isinstance(segments[0], bytes)

to_segment_views() is explicitly borrowed:

segment_views = msg.to_segment_views()
view = memoryview(segment_views[0])
assert view.readonly

The returned segment views keep the underlying message builder alive, including custom allocator-backed builders. Callers must not mutate, reset, or reuse the builder while any returned segment view is still in use, because the views point into the builder's arena.

Testing

Ran with the repository .venv:

.venv/bin/python setup.py build_ext --inplace
.venv/bin/python -m pytest test/test_serialization.py test/test_get_data_view.py test/test_py_custom_message_builder.py
.venv/bin/python -m ruff check test/test_serialization.py

Result:

33 passed
All checks passed

Also manually verified that memoryview(msg.to_segment_views()[0]) is read-only and has byte content matching msg.to_segments()[0].

Full-suite note: running the full suite with local socket permissions enabled produced 146 passed, 1 xfailed, 1 failed. The remaining failure was test_bundled_import_hook when running from the source checkout, where the local capnp/ source directory does not contain bundled .capnp package data. The installed .venv package does contain stream.capnp, and from capnp import stream_capnp succeeds when importing from the installed package path.

@haata
haata merged commit 40189f9 into capnproto:master Jul 3, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants