Skip to content

Foundation: latency capture, run_record capture_date and n, context_size and limits documentation (Refs #312) - #394

Draft
CompleteDotTech wants to merge 3 commits into
mainfrom
foundation/measurement-honesty-312a
Draft

CompleteDotTech wants to merge 3 commits into
mainfrom
foundation/measurement-honesty-312a

Conversation

@CompleteDotTech

Copy link
Copy Markdown
Contributor

Refs #312

Slice A of #312, measurement honesty. Latency is kept in the recorder's sidecar ledger and on CallRecord only; the audit-line format and tests/request_audit.py are unchanged.

What changed, per acceptance item

Item What Kind
1 LiveBackend times every HTTP attempt with an injectable clock (default time.perf_counter), read immediately before and after client.system_one. Seconds are on CallRecord.latency_seconds, LiveBackend.last_latency_seconds, each last_retried entry and the sidecar lines (recorded line, retried entries, failed lines). A reading that is not a finite non-negative difference, or a clock that raises, is stored as null. Not in responses.json or the replay key. fixed
2 vision.run_record(..., capture_date=None, n=None): keys added to a row only when given (existing rows are unchanged); a synthetic or scripted row takes n and refuses capture_date. live.load_ledger_latency(ledger_path, replay_key) returns LedgerLatency(latency_seconds, date, model) or None ("not measured"). fixed; "keys only when given" is a decision, so the existing run_record tests stay unchanged
3 vision.context_size(state, questions) returns {"characters", "utf8_bytes"} (no key naming tokens); vision.comparison_table(rows) prints not measured for every None. fixed; the counting rule (all strings, keys and question names included, no JSON syntax, images not counted) is a decision, documented in docs/vision.md
5 docs/live.md: 64k per-request / 32k per-question and 262k with TypeSafe-Danger-Zone-Long-Context: true, source https://docs.typesafe.ai/models (read 2026-10-31); no token-counting page or endpoint in https://docs.typesafe.ai/llms.txt, so no estimator. documented limit
6 docs/offline-and-live.md: fields holds {seed, target_characters, ...}, helpers.build_state generates the text, with a replay-key-stability test template; the same test is in tests/test_multimodal_backends.py. documented pattern

Also documented: the latency boundaries and the ledger reader (docs/live.md), accounting rows, context size and the comparison table (docs/vision.md).

Deferred to later slices

Checks run (light checks only; no notebook execution)

  • ruff check and ruff format --check on the five changed Python files: pass, with local ruff 0.14.0 (the repository pins 0.17.2, not installed here).
  • python tools/check_hygiene.py: pass (634 files). python tools/render_catalog.py --check: pass. git diff --check: clean.
  • tests/test_live.py, tests/test_vision.py, tests/test_multimodal_backends.py with test_importing_and_injecting_works_with_the_sdk_blocked deselected: 648 passed, 2 skipped. Collection listed first (650 of 651 ids; none mention notebooks or kernels).
  • Each new behaviour test fails on the unmodified origin/main source (87 failures there across the three modules: the new tests, the four updated sidecar tests, and the environment failure named below) and passes here; 37 hand-made mutants of the new rules and boundaries are each killed by at least one test.
  • The documentation examples were executed (the output blocks are their real output); the test template passes.

Not yet verified

  • test_importing_and_injecting_works_with_the_sdk_blocked fails on unmodified origin/main too on the author's host (the package is not installed, and the test runs python -I); it was not run here.
  • No SDK-gated test ran (typesafe_sdk is not installed), so what the real SDK does inside system_one is not exercised.
  • The full suite, the recipe guard and every test that executes notebooks were not run.
  • The limits come from the docs page as read on 2026-10-31; nothing was measured against the live API.
  • Two independent Opus reviews are required (shared interface: the ledger format).

Prior work

Generated with Claude Code

CompleteDotTech and others added 2 commits October 31, 2026 16:27
… date and N, count context size (Refs #312)

Slice A of #312, measurement honesty. Latency lives in the recorder's sidecar
ledger and on CallRecord only; the audit-line format and tests/request_audit.py
are unchanged.

- live: every HTTP attempt is timed with an injectable clock (default
  time.perf_counter), read immediately before and after client.system_one. The
  answering attempt's seconds are on CallRecord.latency_seconds and
  LiveBackend.last_latency_seconds, each retried attempt carries its own in
  last_retried, and the sidecar lines gain latency_seconds. A reading that is
  not a finite non-negative difference is stored as null. It is not in
  responses.json or the replay key.
- live: load_ledger_latency(ledger_path, replay_key) reads a recorded line's
  latency, date and model, or returns None when it was not measured.
- vision: run_record gains capture_date and n, added to a row only when given
  (a synthetic or scripted row takes n and refuses capture_date).
- vision: context_size(state, questions) returns characters and utf8_bytes
  and no token key; comparison_table prints "not measured" wherever a value is
  None.
- docs: the 32k per-question, 64k per-request and 262k (long-context header)
  limits with the models-page source and the absence of an upstream token
  estimator (live.md); latency boundaries and the ledger reader (live.md);
  accounting rows, context size and the table (vision.md); the seeded workload
  pattern with a replay-key-stability test template (offline-and-live.md).
- tests: injected-clock tests, ledger-line and loader tests, run_record,
  context_size and table tests, and the workload test. Four existing sidecar
  equality tests now state the latency field.

Deferred to later slices: the guard check for synthetic or scripted notebooks
printing "N tokens" (item 4), the audited multi-arm pattern (item 7) and the
comparison helper that refuses same-model or moving-alias arms (item 8).

Refs #312

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Brings in #387, #375 and #392. No conflicts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant