speed up evt3 and aedat4 decoding - #29
Merged
AdamDHines merged 3 commits intoSep 3, 2026
Merged
Conversation
Both readers spent most of their time on structure rather than on decoding.
Output is unchanged: `examples/decode_check` and `examples/aedat_check` compare
the new path against the old one event for event on real recordings, and the
335 unit tests pass.
Measured on a 2-core x86-64 host, medians of 15 runs, files in the page cache,
on the recordings Faery publishes as decoder test vectors (`evt3.raw`,
1,218,618 events at 1280x720; `davis346.aedat4`, 78,830 events in 236 LZ4
packets):
read_raw 109.4 ms -> 44.8 ms 2.44x
read_aedat4 2.82 ms -> 1.69 ms 1.67x (two cores; scales with more)
slice_time, 33 ms 6.875 ms -> 0.339 ms 20.3x
slice_index(0, n) 69.5 ms -> 49.3 ms 1.41x
EVT3:
* `read_raw` decoded the whole file twice. It called `open_raw_slice`, which
decodes everything to build the checkpoint index, and then `slice_index`
decoded it all again. An eager read needs none of that index, so with a
declared `% geometry` the file is now decoded once; a header without one still
takes the old path, since the index pass is what establishes the extent.
* `slice_index` and `slice_time` read one word at a time through `read_exact` --
4.7 million calls for a 9.4 MB file -- while the index pass already read in
chunks. Over identical bytes with the identical decoder the per-word path was
2.7x slower. Both now decode out of a chunk buffer.
* The four output columns started at capacity 0 and doubled their way up. The
slice methods know the count exactly and reserve it; the eager reader projects
the total from the density of the first chunk, since a vectorised EVT3 word
can emit anywhere from a fraction of an event to twelve.
* Events were bounds-checked twice, once in the closure and again inside `push`.
* Checkpoints sat every 1 MiB, so a short window replayed up to a megabyte of
words it then discarded. The interval is now sized from the file, aiming at
200k checkpoints and clamped to 64 KiB..4 MiB. That is where the 20x on
`slice_time` comes from; a 30 GB recording's index grows from about 2 MB to
about 13 MB.
AEDAT 4:
* Packets are self-contained -- compressed body, flatbuffer, absolute
timestamps -- and the packet table gives offsets and counts up front, so
unlike a stateful EVT3 stream they need not be decoded in sequence.
`slice_index` now decodes them with rayon and concatenates the columns in
packet order. Below eight packets it stays sequential, so a real-time loop
pulling one window pays no thread-pool hand-off.
* `read_body` reopened the file for every packet, and each packet allocated a
fresh buffer that `vec![0u8; size]` zeroed immediately before overwriting it.
One handle and two reused buffers now travel through the packet loop.
Two things that did not help and are deliberately absent: a fat-LTO release
profile changed nothing (and `panic = "abort"` would stop pyo3 turning panics
into Python exceptions), and `lz4_flex`'s frame decoder is slower than the `lz4`
crate already in the tree -- 982 MB/s against 1208 MB/s over the same packets.
The four `examples/` are the harness for all of the above. They are added with
`git add -f`, since `.gitignore` excludes `examples/`.
Merged
`cargo clippy -- -D warnings` fails on them, and it is right to: the optimisation is what killed them. `Compression::decode` allocated a fresh `Vec` per packet. Every caller now goes through `decode_into` with one buffer reused across the file, which is a measurable part of the 1.67x — a few hundred packets of similar size against a few hundred Vecs each growing by doubling. `Aedat4SliceSource::each_event` opened its own `File` and `Buffers` for one packet. Every caller now hands in ones it reuses; dropping the per-packet `File::open` was the other measurable part. Deleted rather than marked `#[allow(dead_code)]`. Both are second implementations of a path that is still live, and an uncalled copy cannot be tested and can only drift from the one that runs. The doc links that pointed at them now point at the versions that survive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vj722xDQa64yshAdG3seYg
Pre-existing and unrelated to the decoder work, but it is the next thing
`cargo clippy -- -D warnings` stops on: the run that reported the AEDAT 4 dead
code never got past `eventcv-core` to reach it, so fixing only that would have
failed CI again on the following line.
The arm's body was one `if self.dragging && self.is_cloud`, which is now the
arm's guard. Behaviour is identical: the only other arm a cursor move can reach
is the final `_ => {}`.
Drop this commit if you would rather clear the viewer's lints separately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vj722xDQa64yshAdG3seYg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.