Foundation: latency capture, run_record capture_date and n, context_size and limits documentation (Refs #312)

docs/live.md

@@ -83,6 +83,25 @@ apply when a header is enabled; enabling a flag is not a measurement of accuracy
 latency or cost. See [images](https://docs.typesafe.ai/images),
 [Box](https://docs.typesafe.ai/primitives/box), and [models](https://docs.typesafe.ai/models).
 
+### Context limits
+
+These are the limits the models page states, read on 2026-10-31 at
+<https://docs.typesafe.ai/models>. Its table gives the context length of `jev-1.14.0` and
+`jev-1.13.0` as "64k per-request / 32k per-question". The page defines the per-request context as
+the state plus all questions combined and the per-question context as the state plus any one
+question, and a request has to fit both. It sets no separate limit on the number of questions.
+Sending the header `TypeSafe-Danger-Zone-Long-Context: true`, which `long_context=True` adds,
+raises both limits to 262k tokens on either model. The page adds that long context costs the same
+per token and is slower and less accurate than staying inside the default limits.
+
+The page states the 262k figure in tokens and does not say what a "k" in its table counts, so this
+cookbook converts none of the three limits to characters, and it does not check a request against
+them. [`context_size`](vision.md#context-size-and-the-comparison-table) counts characters and
+UTF-8 bytes, which are not tokens. The one token count the cookbook can show is the
+`usage.input_tokens` of a recorded response, which may be `null`. The documentation index at
+<https://docs.typesafe.ai/llms.txt> (read on 2026-10-31) lists no page or endpoint for counting or
+estimating tokens, and the cookbook has no estimator.
+
 ## What a call does
 
 - Requests are built from the question objects: `state` as given, each question as the raw
@@ -97,8 +116,9 @@ latency or cost. See [images](https://docs.typesafe.ai/images),
 - The `request_id` (the `x-typesafe-request-id` header, when present) is not part of the stored
   response format, and `DecisionResult` rejects unknown fields, so it is kept beside the result:
   `backend.last_request_id`, `backend.request_ids`, `backend.records` (one `CallRecord` per
-  successful call: replay key, model, request id, usage, date, and `images` and `distinct_images`,
-  both 0 unless the call carried image assets) and the recorder's optional sidecar.
+  successful call: replay key, model, request id, usage, date, `images` and `distinct_images`
+  (both 0 unless the call carried image assets) and `latency_seconds`, see [Latency](#latency))
+  and the recorder's optional sidecar.
 
 ## Budget guard
 
@@ -165,6 +185,53 @@ returning an SDK-style response. The backend requires no opt-in or key for it (i
 requests; a client without that parameter that retries inside itself is invisible to the budget,
 so build it with retries off.
 
+## Latency
+
+Each HTTP attempt is timed with the backend's `clock` (default `time.perf_counter`, a monotonic
+clock; pass `clock=` to inject one in a test). The clock is read once immediately before
+`client.system_one(...)` is called and once immediately after that call returns or raises. The
+difference, rounded to a nanosecond, is the attempt's `latency_seconds`.
+
+| What | In the timed window? |
+| --- | --- |
+| Whatever the client does inside the one `system_one` call | Yes |
+| The budget check that counts the attempt | No: it comes first, so `requests_made` already includes the attempt at the first reading |
+| The cookbook's validation of the response | No: it comes after the second reading |
+| The backoff sleep before a retry | No: it falls between one attempt's second reading and the next attempt's first |
+| The recorder's ledger write | No: it happens after `decide` has returned |
+
+The cookbook does not look inside the client, so `latency_seconds` is the wall-clock time of that
+call as seen from the machine that made it. It is not a server-side or model-only time.
+
+Where it is kept:
+
+- `CallRecord.latency_seconds` and `backend.last_latency_seconds` hold the answering attempt's.
+- Each entry of `backend.last_retried` holds the `latency_seconds` of the failed attempt it
+  describes.
+- The recorder's sidecar holds it on a `recorded` line (the answering attempt), on every entry of
+  `retried`, and on a failed line (the attempt that ended the call; null when that call sent
+  nothing, such as a budget stop or a refusal).
+- It is not in the responses file and not in the replay key: recording one request under two
+  different clocks writes byte-identical response files.
+- `backend.ledger()`, the in-memory summary, has no latency field.
+
+A latency is null when the clock gave no usable reading: a difference that is negative or not
+finite, a clock that raises, or readings that cannot be subtracted. A call that took several
+attempts has one latency for its answering attempt; the failed attempts' latencies are in
+`retried`, and the backoff between attempts is in none of them, so their sum is not the call's
+elapsed time.
+
+### Reading a captured latency back
+
+`load_ledger_latency(ledger_path, replay_key)` returns a `LedgerLatency(latency_seconds, date,
+model)` for the `recorded` line stored under the key. It returns `None`, meaning "not measured",
+when the ledger has no `recorded` line under that key or the line has no latency (a ledger
+written before latency was captured has none). It reads only the line under the key itself: not a
+`!failed-N` line, not a `superseded` history entry, and nothing from a replay. It raises
+`FileNotFoundError` for a ledger file that does not exist and `ValueError` for a latency that is
+not a finite, non-negative number, or a line without a text date or model.
+[vision.md](vision.md#honest-accounting) shows the result feeding `run_record`.
+
 ## Recorder
 
 ```python
@@ -233,27 +300,32 @@ report = record(
   there, and a path that is the same file as `responses_path` (including through a different
   spelling or a link). Never commit it as a fixture. It is rewritten atomically after every call.
   - A call that produced a response is `{replay_key: {status: "recorded", request_id, model, date,
-    input_tokens, output_tokens, attempts, retried}}`, written before that response is merged, so
-    a merge conflict still leaves its request id. `request_id` is the id of the answering
-    attempt. `attempts` is every HTTP attempt that call spent, the answering one included (1 when
+    input_tokens, output_tokens, latency_seconds, attempts, retried}}`, written before that
+    response is merged, so a merge conflict still leaves its request id. `request_id` is the id
+    of the answering attempt, and `latency_seconds` is that attempt's [latency](#latency).
+    `attempts` is every HTTP attempt that call spent, the answering one included (1 when
     nothing was retried). `retried` lists the failed attempts it retried past, oldest first, as
-    `{error_type, http_status, request_id}` (`http_status` and `request_id` are null when the
-    attempt got no response, for example a timeout); it is `[]` when nothing was retried.
+    `{error_type, http_status, request_id, latency_seconds}` (`http_status` and `request_id` are
+    null when the attempt got no response, for example a timeout; `latency_seconds` is that
+    attempt's own latency); it is `[]` when nothing was retried.
     `images` and `distinct_images` (the image copies and the distinct assets, counts only, never
     bytes or references) are added only when the call carried image assets; a text-only call's
     line has neither.
   - A call that produced nothing to record is written before the error is re-raised, as
     `{replay_key + "!failed-N": {status, error_type, http_status, request_id, model, date,
-    attempts, retried}}`. `status` is `invalid_response` (the API answered but the result failed
-    validation, or contained the key), `error` (an HTTP or connection error, or any other
+    latency_seconds, attempts, retried}}`. `status` is `invalid_response` (the API answered but
+    the result failed validation, or contained the key), `error` (an HTTP or connection error, or any other
     exception that ended the call, including a request refused before anything was sent),
     `timeout` or `budget_stopped`. `error_type`, `http_status` and `request_id` describe the last
     attempt (the one that ended the call; a timeout has no request id; for `budget_stopped`
-    `error_type` is `BudgetExceeded` and the other two are null). A `decide` call refused before
-    anything was sent (`ImageRequestTooLarge` for an image request over a 32 MiB cap, or
+    `error_type` is `BudgetExceeded` and the other two are null). `latency_seconds` is the
+    [latency](#latency) of that same attempt, and null when the call sent nothing (a budget stop
+    or a refusal, including a budget stop after earlier attempts were retried). A `decide` call
+    refused before anything was sent (`ImageRequestTooLarge` for an image request over a 32 MiB cap, or
     `LiveConfigError` for a request the backend is not set up for, such as image assets without
     `images=True`) is `status: "error"` with `error_type` the exception's class name,
-    `http_status` and `request_id` null, `attempts: 0` and `retried: []`; it is not a paid call.
+    `http_status`, `request_id` and `latency_seconds` null, `attempts: 0` and `retried: []`; it is
+    not a paid call.
     A request that fails validation is not one of these: it raises from `record` before `decide`
     is called and gets no line. `attempts` is the HTTP attempts this call spent: 0 when
     nothing was sent (a budget already spent, or a refusal as above), and otherwise includes any

docs/offline-and-live.md

@@ -315,3 +315,69 @@ A recorded fixture contains the answers and their provenance, never the state
 maintainer with a key. The notebook executor (`tools/execute_notebook.py`) never runs live and removes
 `JEV_COOKBOOK_*` and `TYPESAFE_*` from the kernel it starts, so a committed notebook cannot hold a
 live result by accident ([recipe-template.md](recipe-template.md#executing-a-notebook)).
+
+## Long-context workloads
+
+A long-context study needs inputs of a chosen size, and a text of thousands of characters does not
+belong in `inputs.jsonl`, where a reviewer has to read every line. Keep the recipe for the text in
+the example and generate the text:
+
+- `fields` holds the parameters, for example `{"seed": 7, "target_characters": 2000}`, and
+  whatever else decides the text (the position of an evidence passage, which distractors are
+  included).
+- The recipe's `helpers.build_state(fields)` builds the state from them. The notebook and the
+  recipe's tests call that same function ([fixtures.md](fixtures.md) describes `fields` and
+  `build_state`).
+- The replay key is a hash of the state `build_state` returns, so it follows from `fields` alone.
+  It is stable for a given seed when `build_state` depends on nothing but `fields`: the example
+  below hashes `"<seed>:<index>"` with SHA-256 for each sentence, so no clock, environment
+  variable or interpreter setting enters the text.
+
+The test below pins the replay key for one seed, shows that another seed and another length each
+give a different key, and shows that the row a recipe would commit stays short. In a recipe,
+`build_state` lives in `helpers.py`; replace the pinned key with the one your own `build_state`
+gives for the committed seed, after checking that it is the text you meant.
+
+```python
+import hashlib
+import json
+
+from jev_cookbook import Noul, replay_key
+
+WORDS = ("ledger", "invoice", "shipment", "warranty", "renewal", "audit", "tariff", "carrier")
+QUESTIONS = {"renewal": Noul(instructions="Does the document mention a renewal?")}
+
+
+def build_state(fields):  # in a recipe this lives in helpers.py
+    seed, target = fields["seed"], fields["target_characters"]
+    sentences, size, index = [], 0, 0
+    while size <= target:
+        digest = hashlib.sha256(f"{seed}:{index}".encode()).digest()
+        sentence = " ".join(WORDS[b % len(WORDS)] for b in digest[:8]) + "."
+        sentences.append(sentence)
+        size += len(sentence) + 1
+        index += 1
+    return {"document": " ".join(sentences)[:target]}
+
+
+def test_the_replay_key_is_stable_for_a_seed():
+    fields = {"seed": 7, "target_characters": 2000}
+    state = build_state(fields)
+    key = replay_key(state, QUESTIONS)
+    assert len(state["document"]) == 2000
+    assert build_state(dict(fields)) == state
+    assert key == "e02f7cf95ab3d7f540cfe056b94f9e437a739bc2494a64d86fc694eba4e642c9"
+    assert replay_key(build_state({**fields, "seed": 8}), QUESTIONS) != key
+    assert replay_key(build_state({**fields, "target_characters": 2001}), QUESTIONS) != key
+    row = json.dumps({"id": "e1", "split": "test", "fields": fields, "replay_keys": [key]})
+    assert len(row) < 300 and state["document"][:20] not in row
+```
+
+To report how large a generated text is, use
+[`context_size`](vision.md#context-size-and-the-comparison-table): it returns characters and
+UTF-8 bytes, which are not tokens. The token count of a request is the `usage.input_tokens` of a
+recorded response, and the limits that apply are in [live.md](live.md#context-limits). A character
+count or an offline run does not demonstrate a measured 262k-token inference
+([multimodal-track.md](multimodal-track.md#what-an-evaluation-must-show)). Report latency, input
+tokens and cost only when they were measured and recorded, with the model, the capture date and N,
+as section 2 of [CONTRIBUTING.md](../CONTRIBUTING.md) requires.

docs/vision.md

@@ -726,14 +726,16 @@ fields = {
 ## Honest accounting
 
 `run_record(result, mode=...)` creates one row with mode, returned model,
-input/output tokens, latency in seconds and cost in USD:
+input/output tokens, latency in seconds and cost in USD, and the capture date and N when you
+pass them:
 
 ```python
 from jev_cookbook.vision import run_record
 
 # result is a DecisionResult from the recipe's existing replay or live backend.
-# row = run_record(result, mode="synthetic")
-# row = run_record(result, mode="recorded", measured_latency_seconds=captured_seconds)
+# row = run_record(result, mode="synthetic", n=len(examples))
+# row = run_record(result, mode="recorded", measured_latency_seconds=captured_seconds,
+#                  capture_date="2026-10-31", n=len(examples))
 ```
 
 Synthetic and scripted rows contain `None` for model, tokens, latency and cost,
@@ -745,10 +747,92 @@ Provide `measured_latency_seconds` only for actual inference timing, and
 `measured_cost_usd` only for an actual measured charge. Replay execution time is
 not inference latency. Token-derived price calculations are estimates and do not
 belong in the measured-cost field. The helper neither times a request nor infers a
-price. For a replay of recorded responses, load any actual measurements from the
-capture record; leave missing ones null.
+price. For a replay of recorded responses, read the latency the recorder captured with
+`jev_cookbook.live.load_ledger_latency` ([live](live.md#latency) says what the timing
+covers) and leave a measurement that was not captured null.
+
+`capture_date` is a `YYYY-MM-DD` calendar date string and `n` a positive integer: the number of
+examples the row's numbers cover. Each is a key of the row only when you pass it, so a row built
+without them has the keys it had before. A synthetic or scripted row takes `n` and refuses
+`capture_date`, because nothing was captured. `2026-02-30`, `20261031` and a `datetime.date`
+object are refused as a capture date.
+
+```python
+from jev_cookbook.live import load_ledger_latency
+from jev_cookbook.vision import run_record
+
+# `ledger` is the sidecar that live.record(..., ledger_path=...) wrote, `key` the replay key of
+# the request, and `result` the recorded DecisionResult replayed for that key.
+measured = load_ledger_latency(ledger, key)  # None when no latency was captured for this key
+row = run_record(
+    result,
+    mode="recorded",
+    measured_latency_seconds=None if measured is None else measured.latency_seconds,
+    capture_date=None if measured is None else measured.date,
+    n=1,
+)
+```
+
+For a ledger line that holds 0.8 seconds captured on 2026-10-31 and a result that reported 61
+input tokens, `row` is:
+
+```text
+{'model': 'jev-1.13.0', 'input_tokens': 61, 'output_tokens': 0, 'latency_seconds': 0.8, 'cost_usd': None, 'mode': 'recorded', 'capture_date': '2026-10-31', 'n': 1}
+```
 
 Continue to use `run_header`, the recipe's measured-results disclosure and the
 [offline/live contract](offline-and-live.md). Accounting rows do not make a
 synthetic pipeline check a model benchmark, or turn validation selection into a
 held-out result.
+
+### Context size and the comparison table
+
+`context_size(state, questions)` counts the text of a request on your machine and returns
+`{"characters": ..., "utf8_bytes": ...}`, the sum of `len(s)` and of `len(s.encode("utf-8"))`
+over the strings it covers. They are counts of characters and bytes, not tokens: the result has
+no token key, and the API's own count is the `usage.input_tokens` of a recorded response.
+
+The strings covered are the ones in the state (a string, the items of a list, or the keys and
+string values of an object, at any depth) and, for each question, its name and every string in
+its instructions and criteria, object keys included, so a `Choice`'s option names count.
+Numbers, booleans, JSON punctuation and the question type are not counted. An image reference
+counts as the text it is; the image behind it does not. A string shared by two questions is
+counted once per question. Pass one question to get the state plus that question, the
+per-question context that [live](live.md#context-limits) describes. A string that cannot be
+encoded as UTF-8, such as a lone surrogate, raises `ValueError`.
+
+`comparison_table(rows)` formats rows as aligned text. A row is a mapping with keys from `label`,
+`mode`, `model`, `characters`, `utf8_bytes`, `input_tokens`, `output_tokens`, `latency_seconds`,
+`cost_usd`, `capture_date` and `n`: normally a `run_record` row plus a label and the output of
+`context_size`. A column appears when at least one row has the key. A row without the key shows
+`-`; a key whose value is `None` shows `not measured`, so a synthetic or scripted row, or a
+recorded row whose response reported no `input_tokens`, shows no number it does not have. It
+raises `ValueError` for no rows, for a key outside that list, and for a synthetic or scripted row
+that holds a model, tokens, latency or cost.
+
+```python
+from jev_cookbook.vision import comparison_table, context_size
+
+# Continuing from above: `state` and `questions` are the request, `row` is the recorded row,
+# `synthetic_result` was replayed from a synthetic fixture, and `unreported` is a recorded result
+# whose response reported no usage.
+size = context_size(state, questions)  # {'characters': 123, 'utf8_bytes': 123}
+rows = [
+    {"label": "synthetic check", **run_record(synthetic_result, mode="synthetic", n=1), **size},
+    {"label": "recorded", **row, **size},
+    {"label": "no usage reported", **run_record(unreported, mode="recorded", n=1), **size},
+]
+print(comparison_table(rows))
+```
+
+```text
+label              mode       model         characters  utf8_bytes  input_tokens  output_tokens  latency_seconds  cost_usd      capture_date  n
+-----------------  ---------  ------------  ----------  ----------  ------------  -------------  ---------------  ------------  ------------  -
+synthetic check    synthetic  not measured  123         123         not measured  not measured   not measured     not measured  -             1
+recorded           recorded   jev-1.13.0    123         123         61            0              0.8              not measured  2026-10-31    1
+no usage reported  recorded   jev-1.13.0    123         123         not measured  not measured   not measured     not measured  -             1
+```
+
+A character count or a synthetic row does not demonstrate a measured long-context inference
+([multimodal-track.md](multimodal-track.md#what-an-evaluation-must-show)). `run_record` takes
+`input_tokens` from the recorded response and gives a synthetic or scripted row none.