Add buffer-protocol fast path for setting primitive List(T) fields (numpy/memoryview → memcpy)

[htking95]

Summary

Setting a primitive list field (e.g. List(Int16), List(Float32)) from a numpy array or any buffer-protocol object currently has no fast path. The only supported route is field = arr.tolist(), which boxes every element into a PyObject, then _from_list
(capnp/lib/capnp.pyx:1073) iterates in Python and unboxes each one back into a C scalar via _DynamicListBuilder._set. For numerical workloads this dominates serialization time and holds
the GIL for the entire duration.

The capnp wire format for List(Int16) is bit-identical to what numpy.ndarray.tobytes() produces (packed little-endian), so the work pycapnp does today is purely overhead — the bytes that eventually land in the segment are the same bytes the user already had in
memory.

Reproduction

pycapnp 2.2.2, Python 3.11, 8192-element int16 array, measured per-message including to_bytes():

import capnp, numpy as np, time, tempfile

list_schema = """@0xabcdef0123456789;
struct Test { samples @0 :List(Int16); }
"""
data_schema = """@0xabcdef0123456790;
struct Test2 { samples @0 :Data; }
"""
# ... load both schemas ...

arr = np.random.randint(-32768, 32767, size=8192, dtype=np.int16)
N = 200

# Current options for List(Int16):
t0 = time.perf_counter()
for _ in range(N):
    m = list_mod.Test.new_message()
    m.samples = arr.tolist()
    m.to_bytes()
print((time.perf_counter() - t0) * 1000 / N, "ms  List(Int16) tolist()")

t0 = time.perf_counter()
for _ in range(N):
    m = list_mod.Test.new_message()
    lst = m.init("samples", len(arr))
    for i in range(len(arr)):
        lst[i] = int(arr[i])
    m.to_bytes()
print((time.perf_counter() - t0) * 1000 / N, "ms  List(Int16) init+loop")

# Data field, for reference (already has the fast path):
t0 = time.perf_counter()
for _ in range(N):
    m = data_mod.Test2.new_message()
    m.samples = arr.tobytes()
    m.to_bytes()
print((time.perf_counter() - t0) * 1000 / N, "ms  Data tobytes()")

Results

Approach	ms / 8192-sample message	Relative
List(Int16) ← arr.tolist()	0.314	1× (baseline)
List(Int16) ← init() + per-element loop	1.273	4× slower
Data ← arr.tobytes()	0.002	~150× faster

Currently rejected with KjException: Value type mismatch [6 == 8] (capnp type 6 = List, 8 = Data):

• field = arr (numpy array)
• field = memoryview(arr)
• field = arr.tobytes()
• field = array.array('h', ...)

Why it matters

Real-world impact in our codebase (a ultrasound acquisition pipeline running at 60 Hz with ~16 contexts per packet of 8192 samples each):

• tolist() path: ~0.31 ms × 16 × 60 ≈ 300 ms/s of GIL-blocked serialization, enough to starve the asyncio event loop and cause ~89% of downstream WebSocket frames to be coalesced.
• Data field equivalent: ~3 ms/s — effectively free.

We worked around this by changing our schema from List(Int16) to Data and serializing with tobytes(), but this requires coordinated schema bumps across all consumers of the message and a backward-compat decoder for older captures. A pycapnp-level fast path would
let users keep the semantically correct typed-list schema and still get memcpy throughput.

Proposed approach

Add a branch in _setDynamicField (capnp/lib/capnp.pyx:820) before the list/tuple cases that:

1. Checks PyObject_CheckBuffer(value).
2. Looks up the destination field's schema and confirms it's List(T) where T is a fixed-width primitive (Int8/16/32/64, UInt8/16/32/64, Float32/64, Bool).
3. Acquires a Py_buffer and validates:
   • C-contiguous,
   • itemsize matches the capnp element width,
   • format char matches the element type and signedness (with optional explicit endianness check; capnp wire format is always little-endian).
4. Calls thisptr.init(field, n_elements) to allocate the list, then memcpys the buffer into the resulting capnp::List<T>::Builder's underlying storage via the C++ API.
5. Releases the buffer.

There is prior art for the buffer-acquisition pattern in _setMemoryview (line 771), which does the equivalent thing for Data/Text fields. This change generalizes that to typed primitive
lists.

Edge cases / open questions

• List(Bool) — capnp packs bools as a bitfield, not bytes. Should fall through to the slow path or require an explicit packed-bits input.
• Endianness — capnp wire format is little-endian. On little-endian hosts (x86/ARM in practice) the buffer can be memcpy'd directly. On big-endian hosts the fast path should either byte-swap or fall back.
• Format-string strictness — should 'h' (native short) be accepted, or only '<h' (explicit little-endian)? Numpy's default int16 reports as native, so accepting native + checking host endianness seems most ergonomic.
• Reverse direction (reading) — out of scope for this issue, but a _DynamicListReader.__buffer__ / numpy view would be a natural follow-up.

Environment

• pycapnp 2.2.2 (latest as of filing)
• Python 3.11
• Verified the absence of any existing fast path by reading _setDynamicField, _from_list, _setMemoryview, and grepping CHANGELOG.md for numpy / buffer / List( / primitive. The only buffer-protocol support is in from_bytes (whole-message
  deserialization) and Data-field reads, neither of which apply to setting primitive list fields.

---

Happy to put up a PR if the maintainers agree on the approach — wanted to confirm direction (and the List(Bool) / endianness handling) first.

[haata]

Sorry it took so long to review this.

I am ok with adding this as long as it doesn't break any backward compatibility and that there are tests added as well to ensure this. We do build/test for a bunch of the platforms so the endianness should be fine (as long as there are tests).