Control channel: scope buffer page reads to referenced pool entries and attachments#4523
Open
ransomr wants to merge 6 commits into
Open
Control channel: scope buffer page reads to referenced pool entries and attachments#4523ransomr wants to merge 6 commits into
ransomr wants to merge 6 commits into
Conversation
Paged sample-history reads from the realtime buffer (open_sample_history_from / open_sample_history_tail) previously loaded and parsed the sample's entire message pool, call pool, and every attachment body on every page request, making control-channel event polls O(sample) instead of O(page). SampleHistory pools are now position-keyed mappings; page-scoped reads collect the input_refs/call_refs positions and attachment hashes actually referenced by the page's events (including refs inside the loaded pool entries) and fetch only those rows. Full-history reads still load complete pools, and events_data (dense pools for .eval serialization) now guards against misuse on page-scoped histories. Pool-ref resolution accepts position-keyed mappings alongside dense lists. Fixes #80 Co-authored-by: Ransom Richardson <1209015+ransomr@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the contiguity heuristic in SampleHistory.events_data with an explicit page_scoped field set by the construction site, so pages that reference a contiguous pool prefix also raise instead of returning truncated pools. Bind events_data to a local in eval_retry_error_from_history to avoid rebuilding the dense pools twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add POOL_REF_FIELDS in event._pool as the single registry of event fields carrying range-encoded pool refs, with collect_pool_ref_positions() replacing the hardcoded collector in the buffer database. A new test introspects the event models for *_refs fields and fails when one isn't registered, so page-scoped reads can't silently drop pool entries for a future ref field. Also add strict=True to the SampleHistory pool re-keying zips so a length mismatch raises instead of truncating. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-80-20260714-2155 # Conflicts: # CHANGELOG.md
… histories on the SampleBuffer interface Review round 2 feedback: move the export path's pool-ref remapping into event._pool as remap_pool_refs(), walking the POOL_REF_FIELDS registry so a newly registered ref field is remapped without a code change, and note on the abstract open_sample_history_tail/_from docstrings that they yield page-scoped histories (sparse pools, events_data raises). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains:
What is the current behavior? (You can also link to an open issue here)
Fixes meridianlabs-ai#80
When a control-channel events page is served from the realtime sample buffer (
open_sample_history_from→_sample_history), the pagelimitonly bounds the event rows. Each page request also unconditionally loads andjson.loadsthe sample's entire message pool and call pool, and fetches every attachment body for the sample (tens of MB on image-heavy transcripts). Since the read runs synchronously on the eval's own event loop, a read-only monitor pollinginspect ctl eventsimposes repeated blocking pauses on in-flight samples.What is the new behavior?
Fix option 1 from the issue — page-scoped pool/attachment loading:
SampleHistorypools are now position-keyed mappings (dict[int, ...]) rather than dense lists. Full-history reads carry every entry (contiguous keys from 0); page reads carry only referenced positions._sample_history(page_scoped=True)collects theinput_refs/call_refspositional ranges and theattachment://hashes referenced by the page's event JSON (plus refs inside the loaded pool entries, since large message content is itself stored as an attachment), and loads only those rows (WHERE pos IN (...)viaROW_NUMBERrank,WHERE hash IN (...)scoped to the sample;INlists chunked under SQLite's bound-parameter limit).open_sample_history_fromandopen_sample_history_tail) are page-scoped;open_sample_history(full reads: streaming flush, recorder, retry errors) still loads complete pools._expand_refs/resolve_model_event_inputs/resolve_model_event_calls/materialize_pooled_eventsaccept position-keyed mappings alongside dense lists; the buffer transcript history provider resolves against the mappings.SampleHistory.events_data(dense pools for.evalserialization) became a property that raises on a page-scoped (sparse) history rather than silently misaligning refs.Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
No — all changed interfaces are internal (
inspect_ai.log._recorders.buffer.*,inspect_ai.event._pool). Publicresolve_*helpers only widened their accepted parameter types.Other information:
Verification:
pytest tests/log tests/checkpoint tests/_control tests/_eval/test_retry_error_events.py(all passing),ruff check/ruff format, andmypyclean on changed files. New tests intests/log/test_sample_history.pycover page-scoped message/call pool loading, attachment scoping (event-level and pool-entry-level), end-to-end materialization throughBufferTranscriptHistoryProvider.events_from, and theevents_dataguard.The keyset-cursor option (#2 in the issue) is not included; the
ROW_NUMBERCTE still scans the sample's event rows per page, but no longer parses or ships unreferenced pool/attachment content.🤖 Generated with Claude Code