Repository navigation
Foundation: latency capture, run_record capture_date and n, context_size and limits documentation (Refs #312) - #394
Draft
CompleteDotTech wants to merge 3 commits into
Draft
CompleteDotTech wants to merge 3 commits into
CompleteDotTech wants to merge 3 commits into
Conversation
… date and N, count context size (Refs #312) Slice A of #312, measurement honesty. Latency lives in the recorder's sidecar ledger and on CallRecord only; the audit-line format and tests/request_audit.py are unchanged. - live: every HTTP attempt is timed with an injectable clock (default time.perf_counter), read immediately before and after client.system_one. The answering attempt's seconds are on CallRecord.latency_seconds and LiveBackend.last_latency_seconds, each retried attempt carries its own in last_retried, and the sidecar lines gain latency_seconds. A reading that is not a finite non-negative difference is stored as null. It is not in responses.json or the replay key. - live: load_ledger_latency(ledger_path, replay_key) reads a recorded line's latency, date and model, or returns None when it was not measured. - vision: run_record gains capture_date and n, added to a row only when given (a synthetic or scripted row takes n and refuses capture_date). - vision: context_size(state, questions) returns characters and utf8_bytes and no token key; comparison_table prints "not measured" wherever a value is None. - docs: the 32k per-question, 64k per-request and 262k (long-context header) limits with the models-page source and the absence of an upstream token estimator (live.md); latency boundaries and the ledger reader (live.md); accounting rows, context size and the table (vision.md); the seeded workload pattern with a replay-key-stability test template (offline-and-live.md). - tests: injected-clock tests, ledger-line and loader tests, run_record, context_size and table tests, and the workload test. Four existing sidecar equality tests now state the latency field. Deferred to later slices: the guard check for synthetic or scripted notebooks printing "N tokens" (item 4), the audited multi-arm pattern (item 7) and the comparison helper that refuses same-model or moving-alias arms (item 8). Refs #312 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #312
Slice A of #312, measurement honesty. Latency is kept in the recorder's sidecar ledger and on
CallRecordonly; the audit-line format andtests/request_audit.pyare unchanged.What changed, per acceptance item
LiveBackendtimes every HTTP attempt with an injectable clock (defaulttime.perf_counter), read immediately before and afterclient.system_one. Seconds are onCallRecord.latency_seconds,LiveBackend.last_latency_seconds, eachlast_retriedentry and the sidecar lines (recordedline,retriedentries, failed lines). A reading that is not a finite non-negative difference, or a clock that raises, is stored as null. Not inresponses.jsonor the replay key.vision.run_record(..., capture_date=None, n=None): keys added to a row only when given (existing rows are unchanged); a synthetic or scripted row takesnand refusescapture_date.live.load_ledger_latency(ledger_path, replay_key)returnsLedgerLatency(latency_seconds, date, model)orNone("not measured").run_recordtests stay unchangedvision.context_size(state, questions)returns{"characters", "utf8_bytes"}(no key naming tokens);vision.comparison_table(rows)printsnot measuredfor everyNone.docs/vision.mddocs/live.md: 64k per-request / 32k per-question and 262k withTypeSafe-Danger-Zone-Long-Context: true, source https://docs.typesafe.ai/models (read 2026-10-31); no token-counting page or endpoint in https://docs.typesafe.ai/llms.txt, so no estimator.docs/offline-and-live.md:fieldsholds{seed, target_characters, ...},helpers.build_stategenerates the text, with a replay-key-stability test template; the same test is intests/test_multimodal_backends.py.Also documented: the latency boundaries and the ledger reader (
docs/live.md), accounting rows, context size and the comparison table (docs/vision.md).Deferred to later slices
tests/test_recipe_guard.py, which another open pull request owns).Checks run (light checks only; no notebook execution)
ruff checkandruff format --checkon the five changed Python files: pass, with local ruff 0.14.0 (the repository pins 0.17.2, not installed here).python tools/check_hygiene.py: pass (634 files).python tools/render_catalog.py --check: pass.git diff --check: clean.tests/test_live.py,tests/test_vision.py,tests/test_multimodal_backends.pywithtest_importing_and_injecting_works_with_the_sdk_blockeddeselected: 648 passed, 2 skipped. Collection listed first (650 of 651 ids; none mention notebooks or kernels).origin/mainsource (87 failures there across the three modules: the new tests, the four updated sidecar tests, and the environment failure named below) and passes here; 37 hand-made mutants of the new rules and boundaries are each killed by at least one test.Not yet verified
test_importing_and_injecting_works_with_the_sdk_blockedfails on unmodifiedorigin/maintoo on the author's host (the package is not installed, and the test runspython -I); it was not run here.typesafe_sdkis not installed), so what the real SDK does insidesystem_oneis not exercised.Prior work
AuditedBackendaudit lines; untouched here.Generated with Claude Code