diff --git a/docs/designs/context_window_usage.md b/docs/designs/context_window_usage.md index 5bacdfa0..98bebc8f 100644 --- a/docs/designs/context_window_usage.md +++ b/docs/designs/context_window_usage.md @@ -1,4 +1,4 @@ -# Design: Context window usage — measurement and UI transfer +# Design: Context window usage — token-count measurement and UI transfer - **Status:** Draft - **Dependencies:** @@ -12,62 +12,54 @@ Long QuickApp sessions accumulate user, assistant, and tool messages plus system Observable symptoms today: 1. **No context meter** — QuickApps does not compute or expose fill percentage to the UI. -2. **Usage collection is presentation-gated** — `prompt_tokens` from the completion stream is parsed only when `SHOW_USAGE_STATISTICS=true`, and rendered as a markdown stage ([`usage_statistics_service.py`](../../src/quickapp/usage_statistics/usage_statistics_service.py)), not as structured state. -3. **No denominator** — `model.get()` is used for pricing only ([`_pricing_service.py`](../../src/quickapp/usage_statistics/_pricing_service.py)); `ModelLimits` (`max_total_tokens`, `max_prompt_tokens`) is ignored. -4. **Provider-specific semantics** — Models apply prompt caching, reasoning tokens, char-based billing, and other optimizations. `prompt_tokens` alone does not mean the same thing across OpenAI, Anthropic, and Google adapters; QuickApps get only `prompt_tokens` and `completion_tokens` ([`parse.py`](../../src/quickapp/common/chat_completion_stream/parse.py)). -5. **Tokenize gap with attachments** — Today `aidial-adapter-openai` implements `/tokenize` with **tiktoken (text only)**. It does not count images, PDFs, or other `custom_content.attachments`. Tokenize refinement is **unreliable** for attachment-heavy orchestrator payloads until adapters add multimodal counting (see [Proposed Core/Adapter requirements](#proposed-coreadapter-requirements)). +1. **No provider-native preflight count** — legacy DIAL `/tokenize` is not equivalent to the provider count APIs that understand tools, images, PDFs, and other multimodal inputs. +1. **No denominator** — `model.get()` is used for pricing only ([`_pricing_service.py`](../../src/quickapp/usage_statistics/_pricing_service.py)); `ModelLimits` (`max_total_tokens`, `max_prompt_tokens`) is ignored. +1. **Provider-specific count APIs** — OpenAI, Anthropic, and Gemini expose different token-count endpoints and response shapes. +1. **Tokenize gap with attachments** — Today `aidial-adapter-openai` implements `/tokenize` with **tiktoken (text only)**. It does not count images, PDFs, tools, or other `custom_content.attachments` the way provider APIs do. -A future **history compaction** feature needs a reliable signal for when to compact. This design defines **measurement**, **UI transfer**, and a **compaction hook** — not compaction logic. +A future **history compaction** feature needs a reliable input-token count for when to compact. This design defines **measurement**, **UI transfer**, and a **compaction hook** — not compaction logic. Post-completion usage and billing information are intentionally out of scope for this feature. ## Design Goals - **G1 — Dual consumer:** One `context_usage` snapshot serves UI notification and a future compaction policy (`compaction_recommended`). - **G2 — Per-app feature gate:** Opt in via `features.context_usage.enabled` on the app manifest (default off). -- **G3 — Internal usage independent of cost UI:** Collect stream usage when context usage is enabled, regardless of `SHOW_USAGE_STATISTICS`. -- **G4 — Two-tier measurement:** Provider usage by default; DIAL **tokenize** when coarse fill ≥ **60%** (fixed constant). +- **G3 — Count-only measurement:** Use a DIAL token-count endpoint before the LLM call. Do not depend on completion stream `usage`, billing stats, or `SHOW_USAGE_STATISTICS`. +- **G4 — Provider-native count contract:** DIAL Core routes one count request shape to the correct adapter; adapters call provider-native count APIs and normalize responses. - **G5 — UI-ready contract:** Structured `custom_content.state.context_usage` — not a markdown stage. -- **G6 — Honest cross-model semantics:** Pass through provider breakdown; tokenize is compaction authority when available; log warning when provider % and tokenize % diverge by > **5** percentage points. +- **G6 — Honest cross-model semantics:** Count the exact logical input that the model receives: messages, system/instructions, tools, multimodal parts, and files when supported by the provider. - **G7 — Compaction out of scope:** Set `compaction_recommended` only; no compaction algorithm in this design. --- ## Use Cases -### UC-1: UI shows context fill after a turn +### UC-1: UI shows context fill for an orchestrator call -**Trigger:** User completes a turn in an app with `features.context_usage.enabled: true`. +**Trigger:** QuickApps is about to call the orchestrator deployment for an app with `features.context_usage.enabled: true`. -**Behavior:** QuickApps enables `stream_options.include_usage`, reads `prompt_tokens` from the last orchestrator iteration, resolves `limit_tokens` from `model.get(deployment.model).limits`, writes `context_usage` to the final assistant message `custom_content.state`. +**Behavior:** QuickApps resolves `limit_tokens` from `model.get(deployment.model).limits`, sends the exact orchestrator input payload to the DIAL count endpoint, and writes `context_usage` to the final assistant message `custom_content.state`. -**Outcome:** UI displays e.g. “68% context used” from `refined.percent` or `percent`. +**Outcome:** UI displays e.g. “68% context used” from `percent`. -### UC-2: Tokenize refinement near limit (text-only) +### UC-2: Multiple orchestrator iterations -**Trigger:** Coarse fill ≥ 60%, deployment advertises `features.tokenize`, and orchestrator payload has **no attachments**. +**Trigger:** One user turn produces several LLM/tool iterations. -**Behavior:** QuickApps POSTs to `/openai/deployments/{id}/tokenize` with the same payload shape as the orchestrator completion. Snapshot includes `refined` block with tokenize-based fill. +**Behavior:** QuickApps counts each orchestrator LLM input payload before sending it and keeps the latest count snapshot. -**Outcome:** UI shows refined %; backend may set `compaction_recommended` when effective fill ≥ 80%. +**Outcome:** The final assistant message contains the context fill for the last orchestrator LLM call in the turn. Counts are not summed across iterations. -### UC-2b: Attachments present — skip tokenize +### UC-3: Attachments and multimodal payloads -**Trigger:** Payload includes `custom_content.attachments` (or equivalent); coarse fill ≥ 60%. +**Trigger:** The orchestrator input contains images, PDFs, DIAL file URLs, `custom_content.attachments`, or tool schemas. -**Behavior:** QuickApps skips tokenize (aidial-adapter-openai tiktoken would under-count). Uses provider `prompt_tokens` only; sets `refine_skipped_reason: "attachments_present"`. +**Behavior:** QuickApps uses the DIAL count endpoint only if the deployment advertises provider-native count support for the same modalities as completions. It must not fall back to text-only tiktoken. -**Outcome:** UI % reflects post-completion provider usage, not a low tiktoken estimate. - -### UC-3: Provider vs tokenize divergence - -**Trigger:** Both provider and tokenize counts exist; `|percent − refined.percent| > 5`. - -**Behavior:** Backend logs a warning with both values. UI shows **refined only** (resolved product decision). - -**Outcome:** Operators can investigate adapter normalization; user sees the more precise number. +**Outcome:** Attachment-heavy sessions receive a provider-aligned input count, or the snapshot records that counting was unavailable rather than showing a false low estimate. ### UC-4: Compaction hook (no compaction yet) -**Trigger:** Effective fill ≥ 80% (refined if present, else provider `percent`). +**Trigger:** Counted fill ≥ 80%. **Behavior:** `compaction_recommended: true` on snapshot. Future compaction reads this flag per [history compaction design](history_compaction_ui_backend.md). @@ -100,7 +92,7 @@ A future **history compaction** feature needs a reliable signal for when to comp ``` - **Owner:** Config layer; read per request from `ApplicationConfig`. -- **Semantics:** When `enabled` is false or omitted, no `include_usage` for context metering, no `context_usage` state write, no tokenize calls. +- **Semantics:** When `enabled` is false or omitted, no token-count calls and no `context_usage` state write. - **Change:** New config model + schema dump (`make dump_app_schema`). --- @@ -118,63 +110,276 @@ A future **history compaction** feature needs a reliable signal for when to comp | `max_prompt_tokens` + `max_completion_tokens` | Use `max_prompt_tokens` for fill % | | Missing or `model.get` fails | `limit_tokens: null` — show token count only, hide % | -- **Change:** `OrchestratorCapabilities` extended with `model_id`, `limit_tokens`, `tokenize_supported` (from `deployment.features.tokenize`). Fix model key: use `Deployment.model`, not `deployment_id`, for `model.get()`. +- **Change:** `OrchestratorCapabilities` extended with `model_id`, `limit_tokens`, and `token_count_supported`. Fix model key: use `Deployment.model`, not `deployment_id`, for `model.get()`. + +--- + +### Concern 3: DIAL token-count API + +QuickApps should depend on one DIAL contract instead of direct OpenAI, Anthropic, or Gemini count APIs. + +```mermaid +sequenceDiagram + participant QuickApps + participant DialCore as DIAL_Core + participant Adapter + participant Provider as Provider_API + + QuickApps->>DialCore: POST count request for deployment + DialCore->>DialCore: Resolve deployment and adapter + DialCore->>Adapter: Forward count payload + alt OpenAI deployment + Adapter->>Provider: POST /v1/responses/input_tokens + else Anthropic deployment + Adapter->>Provider: POST /v1/messages/count_tokens + else Gemini deployment + Adapter->>Provider: models.countTokens + end + Provider-->>Adapter: Native count response + Adapter->>Adapter: Normalize to response.input_tokens shape + Adapter-->>DialCore: Normalized count response + DialCore-->>QuickApps: response.input_tokens shape +``` + +**Responsibilities:** + +- **DIAL Core** — Accept the request, resolve deployment → adapter, return the normalized response or standard DIAL error envelope. +- **Adapter** — Translate DIAL request → provider count API payload; map provider response → OpenAI-shaped output. +- **QuickApps** — Sends one request shape; reads one response shape. No provider-specific count logic. + +#### Endpoint options + +Exact path is for the DIAL Core team to decide. + +| Option | Path | Notes | +|--------|------|-------| +| OpenAI-aligned | `POST /openai/deployments/{deployment_id}/responses/input_tokens` | Mirrors OpenAI URL shape. | +| Provider-neutral | `POST /deployments/{deployment_id}/count_tokens` | Shorter; same response body. | + +**Request body:** Payload that adapters can forward to provider count APIs. Ideally aligned with OpenAI Responses API input (`model`, `input`, `instructions`, `tools`, multimodal parts). DIAL Core may alternatively accept Chat Completions-shaped requests and let each adapter translate. + +**Response body:** Normalized `response.input_tokens` object. + +**Errors:** Standard DIAL error envelope; adapters map provider 4xx/5xx. + +--- + +### Concern 4: Target response contract + +All adapters normalize to this shape. It mirrors OpenAI `response.input_tokens`. + +#### Required fields + +```json +{ + "object": "response.input_tokens", + "input_tokens": 328 +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `object` | string | Always `"response.input_tokens"`. | +| `input_tokens` | integer | Normalized input token count for the full logical request: messages, system/instructions, tools, and multimodal parts. | + +The count endpoint is **input-only**. Do not add `output_tokens`; provider count APIs do not return output counts. + +#### Optional extensions + +```json +{ + "object": "response.input_tokens", + "input_tokens": 328, + "input_tokens_details": { + "cached_tokens": 0 + }, + "provider_details": {} +} +``` + +| Field | When to populate | +|-------|------------------| +| `input_tokens_details.cached_tokens` | Only if the provider count API returns a cache breakdown pre-send. Usually omitted or `0`. | +| `provider_details` | Provider-specific count fields with no OpenAI mapping. Empty or omitted for v1. | + +--- + +### Concern 5: Provider API reference + +#### OpenAI + +| Resource | URL | +|----------|-----| +| Token counting guide | https://developers.openai.com/api/docs/guides/token-counting | +| Count input tokens | `POST https://api.openai.com/v1/responses/input_tokens` | +| Count input tokens API reference | https://developers.openai.com/api/reference/resources/responses/input_tokens | +| Migrate to Responses API | https://platform.openai.com/docs/guides/migrate-to-responses | + +Count response: + +```json +{ + "object": "response.input_tokens", + "input_tokens": 328 +} +``` + +The count endpoint accepts the same input format as `responses.create` (text, messages, images, files, tools, instructions) and returns the input token count the model will receive. + +#### Anthropic + +| Resource | URL | +|----------|-----| +| Messages API reference | https://docs.anthropic.com/en/api/messages | +| Count tokens endpoint | `POST https://api.anthropic.com/v1/messages/count_tokens` | +| Count tokens section | https://docs.anthropic.com/en/api/messages#count-tokens | + +Count response: + +```json +{ "input_tokens": 2095 } +``` + +#### Google Gemini + +| Resource | URL | +|----------|-----| +| Understand and count tokens | https://ai.google.dev/gemini-api/docs/tokens | +| CountTokens API (`models.countTokens`) | https://ai.google.dev/api/tokens#method:-models.counttokens | + +Count response: + +```json +{ "totalTokenCount": 328 } +``` + +SDKs may expose `total_tokens` or `totalTokenCount` depending on language. --- -### Concern 3: Usage collection (decoupled from usage statistics) +### Concern 6: Count mapping tables + +Tables use columns: + +- **DIAL / OpenAI target** — field in the normalized DIAL response +- **Provider source** — field from the provider API +- **Rule** — how the adapter maps it +- **Excessive in provider** — provider has it; OpenAI has no equivalent +- **Missing in provider** — OpenAI has it; provider does not -- **What:** - - `_ChatCompletionConfigBuilder` sets `stream_options.include_usage: true` when `features.context_usage.enabled`. - - Extend `ChunkUsageFootprint` and `parse.py` to capture `total_tokens`, `prompt_tokens_details` (incl. `cached_tokens`), `completion_tokens_details` (incl. `reasoning_tokens`), and a `provider_details` pass-through bag for adapter-specific fields. - - New `ContextUsageService` owns snapshot building; **not** gated on `SHOW_USAGE_STATISTICS`. -- **Owner:** `core/agent` + `common/chat_completion_stream`. -- **Semantics — what to aggregate:** +#### OpenAI adapter — pass-through -| Metric | Rule | -|--------|------| -| **Context fill** | `prompt_tokens` from the **latest** orchestrator LLM call in the request (full payload). **Do not sum** `prompt_tokens` across tool-call iterations. | -| **Completion total** (optional, informational) | Sum `completion_tokens` across orchestrator iterations in the request. | -| **Cross-turn** | Each turn’s snapshot reflects that turn’s last orchestrator `prompt_tokens`. | +Provider API: `POST /v1/responses/input_tokens` -- **Change:** Orchestrator calls `ContextUsageService` after each iteration; persists final snapshot on last assistant message. +| DIAL / OpenAI target | Provider source | Rule | Excessive in provider | Missing in provider | +|----------------------|-----------------|------|-----------------------|---------------------| +| `object` | `object` | Pass through (`"response.input_tokens"`) | — | — | +| `input_tokens` | `input_tokens` | Pass through | — | — | + +No conversion needed. This adapter is the reference implementation. + +**Note on legacy `/tokenize`:** Today `aidial-adapter-openai` uses tiktoken (text-only). The consolidated endpoint should call `/v1/responses/input_tokens` for parity with OpenAI multimodal and tool-aware counting. + +#### Anthropic adapter + +Provider API: `POST /v1/messages/count_tokens` + +| DIAL / OpenAI target | Provider source | Rule | Excessive in provider | Missing in provider | +|----------------------|-----------------|------|-----------------------|---------------------| +| `object` | — | Set `"response.input_tokens"` | — | `object` discriminator | +| `input_tokens` | `input_tokens` | Direct map | — | — | +| `input_tokens_details.cached_tokens` | — | Omit or `0` pre-send | — | No cache breakdown in count API | +| `provider_details` | — | Empty for count | — | — | + +#### Gemini adapter + +Provider API: `count_tokens` / `CountTokens` + +| DIAL / OpenAI target | Provider source | Rule | Excessive in provider | Missing in provider | +|----------------------|-----------------|------|-----------------------|---------------------| +| `object` | — | Set `"response.input_tokens"` | — | `object` discriminator | +| `input_tokens` | `totalTokenCount` / `total_tokens` | Rename to `input_tokens` | `*_token_count` naming suffix | — | +| `input_tokens_details.cached_tokens` | — | Omit pre-send | — | Structured `input_tokens_details` | +| `provider_details` | — | Empty for count | — | — | + +#### Cross-provider summary + +| | OpenAI | Anthropic | Gemini | +|---|--------|-----------|--------| +| **Count API** | `input_tokens` | `count_tokens` | `count_tokens` | +| **Count HTTP** | `POST /v1/responses/input_tokens` | `POST /v1/messages/count_tokens` | `models.countTokens` | +| **Count response field** | `input_tokens` | `input_tokens` | `totalTokenCount` / `total_tokens` | +| **Response object type** | `response.input_tokens` | none | none | +| **Cache in count response** | No | No | No | --- -### Concern 4: Two-tier measurement (provider → tokenize) +### Concern 7: Count normalization rules + +1. **Always emit `object: "response.input_tokens"`** — Anthropic and Gemini count APIs have no object discriminator. +2. **OpenAI:** pass through unchanged. +3. **Anthropic count:** `input_tokens` → `input_tokens` (direct). +4. **Gemini count:** `totalTokenCount` / `total_tokens` → `input_tokens` (rename only). +5. **Do not add `output_tokens`** to the count response — all three providers' count APIs are input-only. + +Pseudocode: + +```python +def to_response_input_tokens_openai(provider: dict) -> dict: + return { + "object": provider["object"], + "input_tokens": provider["input_tokens"], + } + + +def to_response_input_tokens_anthropic(provider: dict) -> dict: + return { + "object": "response.input_tokens", + "input_tokens": provider["input_tokens"], + } + + +def to_response_input_tokens_gemini(provider: dict) -> dict: + count = provider.get("totalTokenCount") or provider.get("total_tokens") + return { + "object": "response.input_tokens", + "input_tokens": count, + } +``` + +--- + +### Concern 8: QuickApps measurement flow - **What:** Fixed constants in code (not per-app config): | Constant | Value | Purpose | |----------|-------|---------| -| `REFINE_THRESHOLD_PERCENT` | 60 | Call tokenize when coarse % ≥ this | -| `DIVERGENCE_WARN_THRESHOLD_PP` | 5 | Log warning when \|provider % − tokenize %\| exceeds this | | `COMPACTION_THRESHOLD_PERCENT` | 80 | Set `compaction_recommended` (hook only) | - **Owner:** `ContextUsageService`. - **Semantics:** - 1. After latest iteration: `percent_coarse = context_fill_tokens / limit_tokens`. - 2. If `percent_coarse >= 60` and `deployment.features.tokenize` and payload has **no orchestrator attachments** in the serialized completion request: POST tokenize (`TokenizeInputRequest` matching completion payload). Populate `refined` with tokenize count and `source: dial_tokenize`, `confidence: high`. - 3. If payload contains attachments (any `custom_content.attachments` or equivalent content parts on messages/tools path): **skip tokenize refine** — tiktoken-based adapters (e.g. **aidial-adapter-openai** today) under-count. Use provider `prompt_tokens` only; `confidence: medium`; optional `refine_skipped_reason: "attachments_present"` on snapshot. - 4. If tokenize unavailable or returns error: keep provider snapshot, `confidence: medium`. - 5. **Compaction hook:** `compaction_recommended` when effective fill ≥ 80% — effective fill = `refined.context_fill_tokens` when refined exists, else provider `context_fill_tokens`. - 6. If both provider and refined exist and \|Δ\| > 5pp: `logger.warning(...)` with deployment id, both counts, both percents. + 1. Before each orchestrator LLM call, build the exact logical input payload for that call. + 2. Send the payload to the DIAL count endpoint for the deployment. + 3. Set `context_fill_tokens = input_tokens`. + 4. Set `percent = context_fill_tokens / limit_tokens * 100` when `limit_tokens` is known. + 5. Set `compaction_recommended` when `percent >= 80`. + 6. Persist the last successful snapshot for the turn on the final assistant message. + 7. If the count endpoint is unavailable or errors, log and emit a snapshot with `source: "unavailable"` only if there is useful structured failure state for the UI; otherwise omit `context_usage`. -- **Principle:** **Post-call provider `prompt_tokens` is the authority when attachments are present.** Tokenize is the compaction decision authority for **text-only** payloads when the adapter implements provider-aligned counting (not text-only tiktoken). +- **Principle:** The DIAL token-count endpoint is the authority. QuickApps must not use stream `usage` or text-only `/tokenize` as the product meter for this feature. ```mermaid sequenceDiagram Orchestrator->>Limits: model.get limits once - Orchestrator->>LLM_stream: completion with include_usage - LLM_stream-->>Orchestrator: usage prompt_tokens details - Orchestrator->>ContextUsageService: update snapshot latest prompt_tokens - ContextUsageService->>ContextUsageService: percent_coarse - alt percent_coarse ge 60 and tokenize supported and no attachments - ContextUsageService->>DIAL_tokenize: tokenize request payload - DIAL_tokenize-->>ContextUsageService: token_count - ContextUsageService->>ContextUsageService: build refined block - else attachments in payload - Note over ContextUsageService: skip tokenize use provider only + Orchestrator->>ContextUsageService: exact LLM input payload + ContextUsageService->>DIAL_count: count request payload + DIAL_count-->>ContextUsageService: response.input_tokens + ContextUsageService->>ContextUsageService: compute percent and compaction flag + alt count succeeded + Orchestrator->>LLM_stream: completion + else count failed + Note over ContextUsageService: log failure; continue without meter end ContextUsageService->>State: context_usage on final assistant message State-->>UI: stream custom_content.state @@ -182,21 +387,33 @@ sequenceDiagram --- -### Concern 5: Provider cache and optimization semantics +### Concern 9: Known count pitfalls -- **Owner:** QuickApps receive only prompt_tokens and completion_tokens. Probably, tokenizers might return more detailed information. [Proposed Core/Adapter requirements](#proposed-coreadapter-requirements). -- **Semantics:** +1. **Gemini tools in count:** Ensure tools in the DIAL request are passed to CountTokens the same way as `generateContent`; `count_tokens` may omit tool schemas if the adapter omits them. +2. **OpenAI legacy `/tokenize`:** Provider count API handles multimodal content and tools; legacy DIAL `/tokenize` (tiktoken) does not. +3. **All providers:** Count endpoint returns input tokens only — never invent `output_tokens` on the count response. +4. **Count vs later completion:** Count APIs estimate the logical input before send; actual provider execution can still differ if adapters mutate payloads between count and completion. The adapter contract must prevent that. +5. **Gemini SDK aliases:** Some SDKs expose `total_tokens`; adapters should accept both `totalTokenCount` and `total_tokens`. +6. **Bedrock-hosted Claude:** Field name aliases and availability of `count_tokens` on Bedrock vs direct Anthropic API need adapter validation. + +--- + +### Concern 10: Relationship to existing DIAL `/tokenize` + +DIAL today exposes `POST /openai/deployments/{deployment_id}/tokenize` (via `aidial_sdk.deployment.tokenize`). The OpenAI adapter implements this with **tiktoken** (text BPE only). + +| | Existing `/tokenize` | Consolidated count endpoint | +|---|---------------------|-----------------------------| +| OpenAI implementation | tiktoken (text only) | `POST /v1/responses/input_tokens` (provider-native) | +| Response shape | `TokenizeInputRequest` / per-chunk `token_count` | `response.input_tokens` | +| Multimodal / tools | Not counted accurately | Counted per provider rules | +| Cross-provider | Each adapter may differ | One normalized response contract | -| Pattern | Context fill numerator | Notes | -|---------|------------------------|-------| -| OpenAI-style | `prompt_tokens` | `cached_tokens` ⊆ prompt; expose separately in `provider_details` | -| Anthropic cache | `input_tokens + cache_read_input_tokens + cache_creation_input_tokens` (adapter-normalized to `prompt_tokens` or `provider_details`) | Uncached `input_tokens` alone **understates** fill | -| Gemini | `prompt_token_count` (adapter-normalized to `prompt_tokens`) | May include `cached_content_token_count`, `thoughts_token_count`; char-priced adapters need tokenize | -| Reasoning models | Prompt fill unchanged | `reasoning_tokens` in completion details only; affects output headroom | +Whether the new endpoint **replaces**, **supplements**, or **coexists** with `/tokenize` is an open question for DIAL Core. --- -### Concern 6: UI transfer contract (primary deliverable) +### Concern 11: UI transfer contract (primary deliverable) - **What:** `custom_content.state.context_usage` on the **final assistant message** of each turn; also merged into top-level `choice.set_state()` via existing orchestrator persistence. - **Owner:** QuickApps writes; UI reads and persists (Track T1 in [history compaction doc](history_compaction_ui_backend.md)). @@ -205,11 +422,11 @@ sequenceDiagram | Field | UI behavior | |-------|-------------| -| Primary % | `refined.percent` if `refined` present, else `percent` | -| Tokens | `refined.context_fill_tokens` if refined, else `context_fill_tokens`; always show `limit_tokens` when known | +| Primary % | `percent` | +| Tokens | Show `context_fill_tokens`; show `limit_tokens` when known | | Thresholds | < 60% neutral; 60–85% warning; > 85% or `compaction_recommended` strong warning | | Unknown limit | Show token count; hide % or “limit unknown” | -| Low confidence | No `refined` and `confidence: medium` → optional “~” prefix | +| Count unavailable | Hide the meter or show a neutral unavailable state, depending on product decision | - **UI must not:** recompute % from message length; drop `context_usage` on reload. @@ -222,26 +439,20 @@ sequenceDiagram "limit_tokens": 128000, "context_fill_tokens": 82000, "percent": 64.1, - "source": "provider", - "confidence": "medium", - "refined": { - "context_fill_tokens": 79100, - "percent": 61.8, - "source": "dial_tokenize", - "confidence": "high" - }, + "source": "dial_token_count", + "confidence": "high", "provider_details": { - "prompt_tokens": 82000, - "completion_tokens": 1200, - "cached_tokens": 45000, - "reasoning_tokens": null, + "object": "response.input_tokens", + "input_tokens": 82000, + "input_tokens_details": { + "cached_tokens": 0 + }, "extra": {} }, "compaction_recommended": false, "model_id": "gpt-4o", "deployment_id": "my-deployment", - "measured_at_iteration": 3, - "completion_tokens_total": 5400 + "measured_at_iteration": 3 } } ``` @@ -250,34 +461,32 @@ sequenceDiagram |-------|------|----------|-------------| | `schema_version` | int | yes | Start at `1` | | `limit_tokens` | int \| null | yes | Denominator from `ModelLimits` | -| `context_fill_tokens` | int | yes | Coarse fill (provider) | +| `context_fill_tokens` | int | yes | `input_tokens` from normalized count response | | `percent` | float \| null | yes | `context_fill_tokens / limit_tokens * 100`; null if no limit | -| `source` | string | yes | `provider` | -| `confidence` | string | yes | `medium` \| `high` | -| `refined` | object \| null | no | Present after tokenize at ≥ 60% | -| `provider_details` | object | no | Raw / normalized usage breakdown | +| `source` | string | yes | `dial_token_count` | +| `confidence` | string | yes | `high` when provider-native count succeeds; `unknown` only for explicit unavailable state | +| `provider_details` | object | no | Raw / normalized count response details | | `compaction_recommended` | bool | yes | Hook for future compaction | | `model_id` | string | yes | `Deployment.model` | | `deployment_id` | string | yes | Orchestrator deployment id | | `measured_at_iteration` | int | yes | Last orchestrator iteration index (1-based) | -| `completion_tokens_total` | int | no | Sum of completion tokens this request | -| `refine_skipped_reason` | string \| null | no | e.g. `"attachments_present"` when tokenize skipped | +| `count_unavailable_reason` | string \| null | no | Optional machine-readable reason when a snapshot is emitted without a count | --- -### Concern 7: `DialTokenizeClient` +### Concern 12: `DialTokenCountClient` -- **What:** HTTP client for `POST /openai/deployments/{deployment_id}/tokenize` using `aidial_sdk.deployment.tokenize` request/response shapes. +- **What:** HTTP client for the selected DIAL count endpoint. - **Owner:** `dial_core_services`. -- **Semantics:** Input = `TokenizeInputRequest` wrapping the serialized chat completion request (messages, tools, model). Output = sum of successful `token_count` values. On error: log, skip `refined`, keep provider snapshot. -- **Change:** New client; `aidial_client` has no tokenize wrapper today. +- **Semantics:** Input = exact logical orchestrator LLM input payload. Output = normalized `response.input_tokens`. On error: log, continue without a product meter for that turn. +- **Change:** New client; `aidial_client` has no wrapper for the proposed count endpoint today. --- ## Secondary Fixes -- **Orchestrator `model_name` for usage:** Record `Deployment.model` in context snapshots (today orchestrator usage uses `deployment_id` — wrong for `model.get()`). -- **Parse layer:** Preserve full usage object instead of two integers — benefits context usage and optionally enriches usage statistics later. +- **Orchestrator `model_id` for limits:** Record `Deployment.model` in context snapshots (deployment id is wrong for `model.get()`). +- **Payload parity:** Ensure the payload passed to the count endpoint is identical in meaning to the payload passed to the completion endpoint. --- @@ -289,6 +498,8 @@ sequenceDiagram - Client-side live typing estimate. - Adapting tool offload byte threshold to model limits ([large_tool_responses.md](large_tool_responses.md)). - Billing / cost display (`SHOW_USAGE_STATISTICS` unchanged). +- Post-completion usage normalization (`prompt_tokens`, `completion_tokens`, `reasoning_tokens`, billing totals). +- DIAL Core or adapter code in this repository. --- @@ -310,9 +521,10 @@ sequenceDiagram | ID | Check | |----|--------| | R1 | After a turn with context usage enabled, final assistant message contains `custom_content.state.context_usage` with `schema_version: 1`. | -| R2 | UI displays `refined.percent` when `refined` is present; otherwise `percent`. | +| R2 | UI displays `percent` when `limit_tokens` is known. | | R3 | Reload + resend: `context_usage` on prior assistant rows is preserved until overwritten by a new turn. | | R4 | App with `context_usage.enabled: false` emits no `context_usage` field. | +| R5 | Attachment-heavy payloads are counted only through the provider-native DIAL count endpoint, never through text-only tiktoken. | --- @@ -326,7 +538,7 @@ None. Feature is off by default (`features.context_usage` omitted). - Optional `features.context_usage` block on app manifest. - Optional `custom_content.state.context_usage` on assistant messages when enabled. -- `stream_options.include_usage` added to orchestrator calls only when feature enabled. +- New DIAL count endpoint consumed only when the feature is enabled. --- @@ -335,22 +547,20 @@ None. Feature is off by default (`features.context_usage` omitted). | Component | Addition / change | |-----------|-------------------| | `config/application.py` | `ContextUsageConfig`, `Features.context_usage` | -| `core/agent/_chat_completion_config_builder.py` | `include_usage` when `context_usage.enabled` | -| `common/chat_completion_stream/models.py`, `parse.py` | Extended usage footprint + `provider_details` | -| `dial_core_services/` | `ModelLimitsService`, `DialTokenizeClient` | -| `core/agent/context_usage_service.py` (new) | Snapshot, refine, compaction hook, divergence warning | +| `dial_core_services/` | `ModelLimitsService`, `DialTokenCountClient` | +| `core/agent/context_usage_service.py` (new) | Count request, snapshot, compaction hook | | `core/agent/orchestrator.py` | Invoke service; persist `context_usage` on assistant state | -| `core/agent/orchestrator_capabilities.py` | `model_id`, `limit_tokens`, `tokenize_supported` | +| `core/agent/orchestrator_capabilities.py` | `model_id`, `limit_tokens`, `token_count_supported` | | UI (separate repo) | Read `state.context_usage`; threshold styling; persist state | -| DIAL Core / adapters | [Proposed Core/Adapter requirements](#proposed-coreadapter-requirements) | +| DIAL Core / adapters | Provider-native count endpoint and normalized `response.input_tokens` contract | --- ## Proposed Core/Adapter requirements -QuickApps implements the mixed measurement strategy in this design, but **correct cross-model behavior depends on DIAL Core and adapters**. Model names below use product families referenced in QuickApps integration tests and fleet config; adapter teams map each deployment to the upstream API version. +QuickApps implements the UI state and compaction hook in this design, but **correct cross-model behavior depends on DIAL Core and adapters**. Model names below use product families referenced in QuickApps integration tests and fleet config; adapter teams map each deployment to the upstream API version. -**QuickApps rule:** below 60% coarse fill, use provider stream usage (cheap). At ≥ 60%, call **tokenize** only for **text-only** payloads (no attachments in the completion request). When attachments are present, **provider `prompt_tokens` from the actual completion is the authority** — do not call tokenize on tiktoken-based adapters. Provider breakdown remains in `provider_details` for tooltips and ops debugging. +**QuickApps rule:** when context usage is enabled, call the consolidated DIAL token-count endpoint for the exact orchestrator input payload. Do not use stream usage or legacy text-only `/tokenize` as a fallback product meter. ### Current adapter limitation (OpenAI) @@ -358,17 +568,18 @@ QuickApps implements the mixed measurement strategy in this design, but **correc - `custom_content.attachments` (DIAL file URLs, images, PDFs), - multimodal `content` parts, -- provider image-tiling / PDF extraction rules. +- provider image-tiling / PDF extraction rules, +- tool-schema overhead the same way provider APIs do. -Until the adapter adds multimodal tokenize (DC-8), QuickApps **must skip** tokenize refine when the orchestrator payload includes attachments and rely on stream `usage.prompt_tokens` after the real completion. +The consolidated count endpoint must call the provider-native count API instead of the legacy tiktoken path. ### How each model family should count context fill -| Family | Coarse meter (stream `prompt_tokens`) | Tokenize authority | Cache / optimization gotcha | -|--------|--------------------------------------|--------------------|-----------------------------| -| **GPT-5.2 / GPT-5.5** | OpenAI `prompt_tokens` = full prompt in window (incl. attachments after completion) | Provider token-count API on full request — **not tiktoken** when attachments present | **Today:** adapter tokenize = tiktoken → text-only. `cached_tokens` ⊆ `prompt_tokens`. | -| **Gemini 3.5 / 3.1 mini & pro** | `prompt_token_count` → normalized to `prompt_tokens` | Gemini `count_tokens` / CountTokens — **not** tiktoken | Implicit cache from 2.5+; report `cached_content_token_count`. Some adapters bill by **chars** — limits must still be in **tokens**. Thinking → `thoughts_token_count` (output). | -| **Claude 4.6 / 4.7 Sonnet, Haiku, Opus** | **Sum:** `input_tokens + cache_read_input_tokens + cache_creation_input_tokens` | Anthropic count API on full request | **`input_tokens` alone understates fill** when prompt caching is on. Cache is explicit (`cache_control`), not automatic like OpenAI. | +| Family | Count authority | Normalized field | Cache / optimization gotcha | +|--------|-----------------|------------------|----------------------------| +| **GPT-5.2 / GPT-5.5** | OpenAI `POST /v1/responses/input_tokens` on full request | `input_tokens` | Legacy adapter tokenize = tiktoken → text-only. Provider count handles multimodal/tools. | +| **Gemini 3.5 / 3.1 mini & pro** | Gemini `count_tokens` / CountTokens | `totalTokenCount` / `total_tokens` → `input_tokens` | Some adapters bill by **chars**; limits must still be in **tokens**. | +| **Claude 4.6 / 4.7 Sonnet, Haiku, Opus** | Anthropic `POST /v1/messages/count_tokens` | `input_tokens` | Direct Anthropic and Bedrock-hosted Claude may differ in availability / aliases. | ### Cross-cutting requirements (all models) @@ -376,75 +587,95 @@ Until the adapter adds multimodal tokenize (DC-8), QuickApps **must skip** token |----|-------------|-------|-----| | DC-1 | `GET model` returns accurate `limits` (`max_prompt_tokens` and/or `max_total_tokens`) per deployment’s underlying model | DIAL Core | Denominator for % | | DC-2 | `limits` use **token** units for context window (not chars), even when `pricing.unit` is `char_without_whitespace` | DIAL Core / adapter | Avoid % on incompatible units | -| DC-3 | `features.tokenize: true` only when tokenize endpoint is implemented and matches provider counting | Adapter | Gate tokenize calls | -| DC-4 | `POST .../tokenize` accepts `TokenizeInputRequest` with full chat payload (messages + tools + system in messages) and returns token count **identical to pre-send logical input** for that deployment | Adapter | Compaction authority | -| DC-5 | Chat completion stream honors `stream_options.include_usage: true` and emits a final chunk with `usage` | Adapter | Coarse meter | -| DC-6 | Normalize usage to OpenAI `CompletionUsage` **minimum** (`prompt_tokens`, `completion_tokens`, `total_tokens`) plus optional `*_details` and `provider_details` bag for non-OpenAI fields | Adapter | Single parse path in QuickApps | -| DC-7 | `tokenizer_model` on `ModelInfo` when a stable tokenizer id exists | DIAL Core | Future local fallback (out of scope for QuickApps v1) | -| DC-8 | Tokenize with `TokenizeInputRequest` counts **the same modalities** as chat completions: resolve DIAL file URLs, images, PDFs, `custom_content.attachments` — via **provider count API**, not text-only tiktoken | Adapter (aidial-adapter-openai et al.) | Attachment-heavy QuickApp threads | -| DC-9 | Optional deployment capability (e.g. `features.tokenize_multimodal` or documented guarantee) so QuickApps knows when tokenize refine is safe with attachments | DIAL Core / adapter | Avoid false precision | +| DC-3 | DIAL Core exposes one token-count endpoint and routes deployment → adapter | DIAL Core | Single client contract | +| DC-4 | Count endpoint accepts the full logical request payload: messages/input, system/instructions, tools, multimodal content, files, and attachments | DIAL Core / adapter | Match model input | +| DC-5 | Adapter calls provider-native count API, not text-only tiktoken, whenever multimodal/tools may be present | Adapter | Accurate count | +| DC-6 | Adapter normalizes count response to `response.input_tokens` | Adapter | Single parse path in QuickApps | +| DC-7 | `features.token_count` or equivalent deployment capability identifies deployments where the count endpoint is supported and provider-aligned | DIAL Core / adapter | Gate count calls | +| DC-8 | Count payload and completion payload have equivalent semantics; adapters must not count one shape and complete with another | Adapter | Avoid false precision | +| DC-9 | Golden fixtures prove fixed payload → normalized count matches provider native API with 0 delta for count | Adapter | Regression guard | ### Per-family adapter must-haves | ID | GPT-5.2 / GPT-5.5 | Gemini 3.5 / 3.1 mini & pro | Claude 4.6 / 4.7 Sonnet, Haiku, Opus | |----|-------------------|----------------------------|--------------------------------------| -| A1 | Pass `prompt_tokens_details.cached_tokens` | Map `prompt_token_count` → `prompt_tokens` | Sum input + cache read + cache creation → `prompt_tokens` | -| A2 | Pass `reasoning_tokens` in completion details | Map `cached_content_token_count`, `thoughts_token_count` | Pass cache fields in `provider_details` | -| A3 | Tokenize = OpenAI **input token count API** on full multimodal request (replace tiktoken-only path) | Tokenize via CountTokens, not tiktoken | Tokenize via Anthropic count API | +| A1 | Count = OpenAI **input token count API** on full multimodal request | Count via CountTokens, not tiktoken | Count via Anthropic count API | +| A2 | Pass through `object` and `input_tokens` | Map `totalTokenCount` / `total_tokens` → `input_tokens` | Add `object: "response.input_tokens"` | +| A3 | Replace or supplement text-only `/tokenize` with provider-native count | Include tools in CountTokens the same way as `generateContent` | Validate direct Anthropic vs Bedrock count availability | | A4 | Correct 128k–1M limits per variant | `input_token_limit` / `output_token_limit` on model metadata | 200k-class limits per tier (Sonnet / Haiku / Opus) | ### OpenAI family — GPT-5.2, GPT-5.5 | Topic | Upstream behavior | Adapter requirement | |-------|-------------------|---------------------| -| Tokenizer | tiktoken / model-specific BPE for **text**; images/PDFs use provider rules | **Today:** tokenize uses tiktoken only — **insufficient for attachments**. **Target:** OpenAI input token count API (or Responses count) on same payload as completions | -| Context fill | `usage.prompt_tokens` = full prompt size in window | Pass through unchanged | -| Prompt caching | `usage.prompt_tokens_details.cached_tokens` | Map to `prompt_tokens_details.cached_tokens`; cached ⊆ prompt | -| Reasoning / thinking | `completion_tokens_details.reasoning_tokens` billed and counted in completion budget | Pass through in `completion_tokens_details`; **exclude from prompt fill** | +| Tokenizer | tiktoken / model-specific BPE for **text**; images/PDFs use provider rules | **Today:** `/tokenize` uses tiktoken only — insufficient for attachments. **Target:** OpenAI input token count API on same payload as completions | +| Context fill | `response.input_tokens.input_tokens` = full input size in window | Pass through unchanged | +| Prompt caching | Count endpoint normally does not expose cache breakdown | Omit `input_tokens_details` unless provider returns it pre-send | | Context window | Model-specific (e.g. 128k–1M depending on variant) | Expose correct `max_prompt_tokens` or `max_total_tokens` on model metadata | -| GPT-5.x note | May use Responses API or Chat Completions depending on deployment | Adapter tokenize + usage must reflect **the same API path** as completions | +| GPT-5.x note | May use Responses API or Chat Completions depending on deployment | Adapter count request must reflect **the same API path** as completions | -**Fill formula (normalized):** `context_fill_tokens = prompt_tokens` +**Fill formula (normalized):** `context_fill_tokens = input_tokens` ### Google Gemini — 3.5 mini/pro, 3.1 mini/pro (and 2.5+ lineage) | Topic | Upstream behavior | Adapter requirement | |-------|-------------------|---------------------| -| Tokenizer | SentencePiece / Gemini tokenizer (not tiktoken) | Tokenize via `count_tokens` / CountTokens API — **do not use tiktoken** | -| Context fill | `usage_metadata.prompt_token_count` | Map to `usage.prompt_tokens` | -| Implicit caching | `cached_content_token_count` in `usage_metadata` (2.5+ / 3.x) | Pass in `provider_details.cached_content_token_count`; fill = full `prompt_token_count` | -| Thinking models | `thoughts_token_count` in output metadata | Map to `completion_tokens_details` or `provider_details.thoughts_token_count` | -| Char-priced adapters | Some deployments bill by `char_without_whitespace` | **Still** expose token `limits` and tokenize in **tokens**; document pricing unit separately | +| Tokenizer | SentencePiece / Gemini tokenizer (not tiktoken) | Count via `count_tokens` / CountTokens API — do not use tiktoken | +| Context fill | `totalTokenCount` from CountTokens | Map to `input_tokens` | +| Implicit caching | Usually not exposed in count response | Omit cache details pre-send | +| Char-priced adapters | Some deployments bill by `char_without_whitespace` | **Still** expose token `limits` and count in **tokens**; document pricing unit separately | | Context window | Per-model `input_token_limit` / `output_token_limit` | Map to `max_prompt_tokens` + `max_completion_tokens` or `max_total_tokens` | -| 3.x caching | Implicit cache min 4096 tokens (per Google docs) | Usage must report cache hits when present | -**Fill formula (normalized):** `context_fill_tokens = prompt_tokens` (adapter-mapped from `prompt_token_count`) +**Fill formula (normalized):** `context_fill_tokens = input_tokens` (adapter-mapped from `totalTokenCount` / `total_tokens`) ### Anthropic — 4.6 / 4.7 Sonnet, Haiku, Opus | Topic | Upstream behavior | Adapter requirement | |-------|-------------------|---------------------| -| Tokenizer | Anthropic tokenizer | Tokenize endpoint uses Anthropic `count_tokens` or equivalent | -| Context fill | **Not** `input_tokens` alone when caching is active | **Normalize:** `prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens` | -| Prompt caching | `cache_read_input_tokens`, `cache_creation_input_tokens` separate from `input_tokens` | Pass all three in `provider_details`; use sum for `prompt_tokens` | -| Extended thinking | May affect output token accounting | Pass thinking-related fields in completion details; not part of prompt fill | +| Tokenizer | Anthropic tokenizer | Count endpoint uses Anthropic `count_tokens` or equivalent | +| Context fill | `input_tokens` from count response | Direct map | +| Prompt caching | Count response normally does not return cache breakdown | Omit cache details pre-send | | Context window | Model-specific (200k+ for Sonnet/Opus class) | Accurate `limits` on model metadata | | Bedrock vs direct API | Field names may differ | Adapter normalizes to DC-6 shape | -**Fill formula (normalized):** `context_fill_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens` +**Fill formula (normalized):** `context_fill_tokens = input_tokens` ### Summary matrix -| Model family | Coarse meter (`prompt_tokens`) | Tokenize at ≥ 60% | Known pitfall | -|--------------|-------------------------------|-------------------|---------------| -| GPT-5.2 / GPT-5.5 | Provider `prompt_tokens` (post-completion) | Tokenize only if text-only; else provider only | **Tiktoken tokenize ignores attachments**; reasoning tokens in completion details | -| Gemini 3.5 / 3.1 mini/pro | Adapter-mapped `prompt_token_count` | CountTokens / tokenize | Char pricing ≠ token limits; implicit cache | -| Claude 4.6 / 4.7 Sonnet/Haiku/Opus | **Sum** of input + cache read + cache creation | Anthropic count API | `input_tokens` alone understates fill | +| Model family | Count meter | Known pitfall | +|--------------|-------------|---------------| +| GPT-5.2 / GPT-5.5 | OpenAI `response.input_tokens.input_tokens` | Text-only `/tokenize` ignores attachments and tools | +| Gemini 3.5 / 3.1 mini/pro | Adapter-mapped `totalTokenCount` / `total_tokens` | Char pricing ≠ token limits; tools must be included in CountTokens | +| Claude 4.6 / 4.7 Sonnet/Haiku/Opus | Anthropic `input_tokens` | Bedrock/direct API differences need validation | ### Adapter acceptance tests (suggested) -1. Golden fixture: fixed messages + tools → tokenize count equals provider `prompt_tokens` within **1%** (or document intentional delta). -2. Cached second turn: cache fields populated; normalized `prompt_tokens` reflects **full** window occupancy. -3. Stream final chunk includes `usage` when `include_usage: true`. -4. `model.limits` matches vendor-documented context window for each listed deployment. -5. **Attachment fixture:** messages + image/PDF attachment → completion `prompt_tokens` ≈ multimodal tokenize count (within 1%); tiktoken-only count must **not** be used for product meter. +1. Golden fixture: fixed messages + tools → normalized count matches provider native count API with **0 delta**. +2. `model.limits` matches vendor-documented context window for each listed deployment. +3. Attachment fixture: messages + image/PDF attachment → consolidated endpoint uses provider-native count; tiktoken-only count must **not** be used for product meter. +4. Tool fixture: messages + tool definitions → count includes tool-schema overhead when the provider count API supports it. +5. Provider aliases: Gemini `totalTokenCount` and SDK `total_tokens` both normalize to `input_tokens`. + +--- + +## Open Questions + +1. **DIAL endpoint path** — OpenAI-aligned (`/responses/input_tokens`) vs provider-neutral (`/count_tokens`)? +2. **Request body shape** — Accept OpenAI Responses API input directly, or Chat Completions-shaped requests with per-adapter translation? +3. **`/tokenize` migration** — Deprecate tiktoken-based `/tokenize` for OpenAI deployments, or keep both? +4. **`provider_details` on v1** — Include the optional bag on the count response, or defer? +5. **Gemini tools in count** — How to guarantee CountTokens receives the same tool definitions as `generateContent`? +6. **Bedrock-hosted Claude** — Field name aliases and availability of `count_tokens` on Bedrock vs direct Anthropic API. + +--- + +## Suggested Next Steps + +| Team | Action | +|------|--------| +| **DIAL Core** | Choose endpoint path and request contract; define routing deployment → adapter. | +| **aidial-adapter-openai** | Implement pass-through to `POST /v1/responses/input_tokens`; document `/tokenize` relationship. | +| **aidial-adapter-anthropic** | Implement `count_tokens` → `response.input_tokens` mapping. | +| **aidial-adapter-gemini** | Implement CountTokens → `response.input_tokens`; validate tool/multimodal parity. | +| **QuickApps** | Add `DialTokenCountClient`, `ContextUsageService`, feature gate, model limit resolution, and `custom_content.state.context_usage`. | +| **All** | Golden fixtures: fixed payload → normalized count matches provider native API. | diff --git a/docs/designs/history_compaction_investigation.md b/docs/designs/history_compaction_investigation.md new file mode 100644 index 00000000..2caee93b --- /dev/null +++ b/docs/designs/history_compaction_investigation.md @@ -0,0 +1,533 @@ +# History Compaction Investigation + +This note investigates how long conversation history is stored and compacted in coding agents, provider APIs, and agent +libraries, then maps those approaches to QuickApps. It is a companion to +[`history_compaction_strategy.md`](history_compaction_strategy.md), not a formal design document. + +## Executive Summary + +QuickApps' main constraint is that the frontend currently sends the full visible `messages` array to the backend on every +new user turn. The backend then restores assistant `custom_content.state.tool_execution_history` into assistant/tool +message pairs and sends an expanded effective history to the orchestrator model. + +The most practical first step is backend-owned effective-history compaction: + +1. The UI keeps sending the full visible transcript. +1. QuickApps stores a compacted checkpoint on the next assistant message under `custom_content.state`. +1. On later requests, the backend uses the latest valid checkpoint to ignore older messages for model input. +1. The model receives protected setup context, a synthetic compacted-history message, recent raw episodes, and the new + user message. + +This solves the model context-window problem without requiring a UI contract rewrite. UI-side transcript canonicalization +is still valuable, but it should be treated as a later performance and payload-size optimization. + +No reviewed library is a clean drop-in for QuickApps. The best path is to copy proven reducer patterns from +LangChain/LangGraph and Semantic Kernel while keeping QuickApps-specific episode grouping, attachment handling, +`custom_content.state` checkpointing, and DIAL Chat Completions compatibility. + +## Current UI to QuickApp Communication + +Today's communication model is stateless from the backend's perspective: + +1. The UI sends a full `messages` array for each turn. +1. `_MessagesSetup.extract_tool_calls` expands `custom_content.state.tool_execution_history` from assistant rows into + assistant/tool message pairs. +1. Request-level message transformers run. +1. The orchestrator invokes the model and executes tools in a loop. +1. `Orchestrator._persisting_state` writes new `tool_execution_history` to the final assistant state via + `choice.set_state(...)`. +1. The UI persists the assistant row and sends it back on the next turn. + +This means the browser-visible transcript and backend-effective transcript are currently close to the same logical +history after restoration. History compaction introduces a useful distinction: + +- **Visible transcript:** what the user sees and expects in the browser. +- **Effective model history:** what QuickApps sends to the orchestrator model after restoration, checkpoint selection, + compaction, and recent-suffix selection. + +Keeping those separate is the key to shipping useful compaction without forcing the UI to stop sending full history. + +## Provider and Product Approaches + +### Claude Code and Claude API + +Claude Code and the Claude API use server-side or product-level compaction when conversations approach the context +window. Older conversation content is summarized into a structured compaction block. Later requests can continue from +that block instead of replaying every prior message. + +Important traits: + +- Compaction is lossy. +- The compacted representation is provider-native. +- Claude Code can manually compact with `/compact` and can automatically compact under context pressure. +- Durable instructions are safer on disk, for example in root `CLAUDE.md`, because pure chat history can be summarized + away. +- The user-visible transcript may exist separately from the model's active compacted context. + +Applicability to QuickApps: + +- Strong conceptual fit: checkpoint plus recent suffix is the same core idea. +- Not directly portable unless QuickApps is using Anthropic APIs with their native `context_management` surface. +- The durable-instructions lesson applies: app configuration, system prompt, tool definitions, and required setup context + should be regenerated or protected, not summarized as ordinary chat. + +### Cursor + +Cursor automatically summarizes long chats near the context limit and exposes manual summarization through `/summarize`. +Public details are less precise than Claude/OpenAI documentation, but the observed pattern is conventional prompt-based +conversation summarization. + +Important traits: + +- Summary becomes the seed for continued work. +- The process is lossy and can omit old rules, exact errors, or nuanced decisions. +- Cursor keeps the product chat experience separate from the model's compacted working context. + +Applicability to QuickApps: + +- Useful as product precedent: users can tolerate backend-effective compaction while the visible chat remains intact. +- Also a warning: generic prose summaries are not enough for tool-heavy agents. QuickApps needs structured tool outcomes, + attachments, artifacts, decisions, and open questions. + +### OpenAI Responses and Codex + +OpenAI Responses supports native compaction in two ways: + +- Server-side auto-compaction using `context_management` with a compact threshold. +- Explicit stateless compaction using `/responses/compact`. + +The compacted output includes an opaque `type=compaction` item with encrypted content. That item must be round-tripped in +future Responses requests. It is not meant to be human-readable. + +Important traits: + +- Strongest provider-native support among the reviewed APIs. +- Designed for Responses API input/output items, not classic Chat Completions `messages`. +- Can preserve hidden reasoning state in provider-specific opaque form. +- The latest compaction item becomes the anchor; prior items can be omitted in stateless item-array chaining. + +Applicability to QuickApps: + +- Not a direct fit for the current DIAL Chat Completions path. +- Not portable across DIAL deployments, Anthropic, Gemini, or arbitrary model adapters. +- Useful as an optional future provider capability for OpenAI Responses deployments if DIAL exposes that surface. +- Very useful as a pattern: "opaque or structured checkpoint plus suffix" beats replaying full history. + +### Common Agent Pattern + +Most practical agents converge on this model: + +1. Persist the full transcript for audit, UI, and recovery. +1. Build a smaller model-facing context per turn. +1. Protect system/setup context. +1. Keep the newest turns raw. +1. Summarize or structure older turns. +1. Treat tool-call groups as atomic units. + +This matches the direction in `history_compaction_strategy.md`. + +## Library Overview and QuickApps Applicability + +### OpenAI Agents SDK + +What it provides: + +- `OpenAIResponsesCompactionSession` wraps a session backend and calls `responses.compact`. +- Supports automatic or forced compaction. +- Can compact based on a decision hook. +- Rewrites the underlying session with the compacted Responses item list. + +Covered strategy areas: + +- Checkpointing and latest-compaction-anchor behavior. +- Provider-native opaque compaction. +- Session storage rewrite after compaction. + +Applicability to QuickApps: + +- **Direct reuse:** Low for current QuickApps, because it targets OpenAI Responses items. +- **Adaptable ideas:** High. The session wrapper pattern is useful: an outer compaction layer can decorate an underlying + history store and replace the effective history while preserving raw session data elsewhere. +- **Risk:** Pulling in the SDK would introduce a provider-specific model/runtime layer that conflicts with QuickApps' + DIAL multi-provider abstraction. + +### Pydantic AI + +What it provides: + +- `OpenAICompaction` for OpenAI Responses compaction. +- `AnthropicCompaction` for Anthropic context management. +- Provider-specific `CompactionPart` handling. +- Capability routing that chooses provider-native behavior where available. + +Covered strategy areas: + +- Provider capability abstraction. +- Native compaction blocks/items. +- Token-threshold-driven server-side compaction. + +Applicability to QuickApps: + +- **Direct reuse:** Low. Pydantic AI is an agent framework and model abstraction, while QuickApps already has its own DI, + config, DIAL client, orchestrator, tools, and message pipeline. +- **Adaptable ideas:** Medium to high. The "capability" framing is useful if QuickApps later supports provider-native + compaction for selected deployments while falling back to generic backend summaries for others. +- **Risk:** Provider-native compaction output may not be compatible with DIAL Chat Completions messages or the UI + `custom_content.state` contract. + +### LangChain and LangGraph + +What they provide: + +- Token-aware trimming utilities such as `trim_messages`. +- Running-summary memory patterns. +- Graph state with full checkpoints plus reduced LLM input. +- Summarization nodes that maintain a separate summary and keep recent messages raw. + +Covered strategy areas: + +- Sliding-window recent suffix. +- Token-budget trimming. +- Running summary plus recent raw tail. +- Full state persistence separated from model-facing state. + +Applicability to QuickApps: + +- **Direct reuse:** Low to medium. Individual utilities may be reusable only after adapting message types, but bringing in + LangChain/LangGraph as a runtime would be excessive. +- **Adaptable ideas:** Very high. QuickApps should copy the pattern: full history remains available, while each model call + receives a managed subset. The summary-plus-recent-suffix shape maps directly to `HistoryCompactionService`. +- **Gap:** Generic LangChain messages do not know about QuickApps `custom_content.attachments`, + `custom_content.state.tool_execution_history`, DIAL files, artifacts, or tool-result propagation rules. + +### Semantic Kernel + +What it provides: + +- Chat history reducers for truncation, token count, and summarization. +- A reducer abstraction that can sit in front of chat completion. +- Options to include function-call/function-result content in summaries. + +Covered strategy areas: + +- Reducer abstraction. +- Message-count and token-count thresholds. +- Summary message insertion. + +Applicability to QuickApps: + +- **Direct reuse:** Low. It is another framework with different message and tool abstractions. +- **Adaptable ideas:** High. A QuickApps-native `HistoryCompactionService` can act like a reducer over restored messages. +- **Important caution:** Tool-call pair preservation is a known sharp edge in reducer-style implementations. QuickApps + must make assistant tool-call messages plus matching tool results an atomic episode boundary. + +### LlamaIndex + +What it provides: + +- Token-limited short-term memory. +- Optional long-term memory blocks. +- Fact extraction and vector-memory blocks. +- Chat history flushing from short-term to long-term storage. + +Covered strategy areas: + +- Recent raw suffix. +- Long-term fact extraction. +- Retrieval-backed memory. + +Applicability to QuickApps: + +- **Direct reuse:** Low for MVP history compaction. +- **Adaptable ideas:** Medium for future memory features. Fact extraction or vector recall may help cross-session memory + later, but they do not solve the immediate "UI sends full transcript and model context overflows" problem. +- **Risk:** Retrieval memory is not deterministic enough to replace an explicit compaction checkpoint for ongoing tasks. + +## Strategy Coverage from `history_compaction_strategy.md` + +The existing strategy has several QuickApps-specific concerns. Library coverage is partial: + +| Strategy concern | Library support | QuickApps-specific work still needed | +|------------------|-----------------|--------------------------------------| +| Protected prefix | Common in trimmers; preserve system message | Decide which setup, skill, context-enrichment, and synthetic messages are protected or regenerated | +| Episode grouping | Some frameworks warn about tool pairs | Build a QuickApps `HistoryEpisodeBuilder` that understands assistant/tool pairs, attachments, stages, and checkpoints | +| Content-aware reducers | Generic summary memories exist | Implement reducers for tool outcomes, attachments, audio/transcripts, stage noise, and artifacts | +| Attachment/artifact registry | Mostly not covered | Store DIAL file URLs, generated artifacts, observed facts, and "referenced but not read" status | +| Compacted checkpoint state | Provider-native for OpenAI/Anthropic; summary state in frameworks | Define `custom_content.state.history_compaction` schema and validation | +| Recursive shrink | Usually not turnkey | Token-count compacted payload and retry shrink with ordered pruning | +| UI round-trip | Framework sessions handle their own stores | Fit the checkpoint into the existing assistant-message state the UI already preserves | + +The conclusion is that libraries can inform the implementation but should not own it. + +## Approaches Without UI Changes + +### Option A: Backend-Only Effective-History Compaction + +How it works: + +- UI keeps sending full visible history. +- Backend finds the latest valid `history_compaction` checkpoint on an assistant message. +- Backend ignores messages before that anchor for model input. +- Backend injects a synthetic compacted-history assistant message and keeps recent raw episodes. +- Backend writes an updated checkpoint to the next assistant state. + +Benefits: + +- Minimal UI work. +- Solves the model context-window problem. +- Keeps browser transcript behavior unchanged. +- Fits current `custom_content.state` persistence. +- Allows QuickApps-specific structured compaction. + +Effort: + +- Medium to high backend effort. +- Requires token-count preflight, episode grouping, reducers, checkpoint schema, and tests. + +Limitations: + +- Browser still uploads full history. +- Backend still receives and parses full JSON. +- Old assistant rows can still carry large `tool_execution_history` until backend chooses to ignore them. +- UI cannot show a user-facing "this range was compacted" state unless we add UI work later. + +Library fit: + +- Best served by custom QuickApps code borrowing patterns from LangChain/LangGraph and Semantic Kernel. + +### Option B: Backend Trim-Only Safety Valve + +How it works: + +- When context is too large, backend drops old messages or old tool bodies without summarizing. +- It preserves system/setup messages and recent episodes. + +Benefits: + +- Lower implementation effort than full summarization. +- Can prevent some over-limit calls quickly. + +Effort: + +- Low to medium. + +Limitations: + +- Loses important decisions and tool outcomes. +- Weak fit for long task continuity. +- Dangerous unless episode boundaries are exact. +- Does not satisfy attachment/artifact preservation goals. + +Library fit: + +- Similar to `trim_messages` or chat history truncation reducers. +- Still needs QuickApps message adaptation and tool-pair safety. + +### Option C: Provider-Native Compaction Where Available + +How it works: + +- If a deployment supports native compaction, QuickApps enables it in provider-specific request settings or calls a + provider-specific compact endpoint. +- Generic backend compaction remains the fallback. + +Benefits: + +- High quality for providers with first-class compaction. +- Can preserve provider-specific hidden reasoning state. +- Less custom summarization logic for supported providers. + +Effort: + +- Medium to high because DIAL must expose the provider capability and compaction items in a compatible way. + +Limitations: + +- Not portable across all DIAL deployments. +- Opaque compaction items do not map cleanly to Chat Completions `messages`. +- Harder to render, inspect, validate, or migrate. + +Library fit: + +- OpenAI Agents SDK and Pydantic AI are useful references, but direct use is unlikely. + +### Option D: Long-Term Memory or Retrieval as a Complement + +How it works: + +- Old history is flushed into facts, vector memory, or a session store. +- Relevant memories are retrieved on later turns. + +Benefits: + +- Useful across long-lived sessions. +- Can preserve reusable user preferences, facts, or workflow lessons. + +Effort: + +- High if done well. + +Limitations: + +- Does not replace per-turn compaction. +- Retrieval can miss important context. +- Harder to prove correctness for ongoing tool-heavy tasks. + +Library fit: + +- LlamaIndex is the strongest inspiration here. +- Better suited for a later memory feature than MVP compaction. + +## Suggested UI Changes + +UI changes are best treated as phases. The first backend-only implementation should define the checkpoint shape so later +UI improvements can build on it. + +### Less Invasive: Preserve and Surface Checkpoint State + +What changes: + +- UI keeps sending full history. +- UI guarantees unknown `custom_content.state` keys are persisted and resent. +- UI may display a small "history compacted" marker based on assistant state. + +Benefits: + +- Minimal contract change. +- Makes backend-only compaction reliable across reloads. +- Gives users some visibility without changing payload semantics. + +Effort: + +- Low to medium, depending on current serializer fidelity. + +Library impact: + +- Does not make provider libraries easier to use, but it preserves the backend's custom checkpoint. + +### Moderately Invasive: Backend-Returned Compacted Effective Transcript + +What changes: + +- Backend returns a canonical compacted effective transcript or patch. +- UI stores both the visible transcript and a backend-sendable transcript. +- Future requests send the compacted effective transcript instead of the full visible transcript. + +Benefits: + +- Reduces upload size and backend parsing cost. +- Avoids repeatedly resending huge old `tool_execution_history`. +- Makes backend behavior more explicit and testable with before/after golden files. + +Effort: + +- Medium to high. + +Risks: + +- UI must avoid confusing "what user sees" with "what backend receives". +- Debugging needs tools to inspect both tracks. +- Message IDs and anchor validation become important. + +Library impact: + +- Makes OpenAI-style stateless compaction patterns easier to imitate because the compacted transcript can become the + canonical next request payload. +- Still does not make OpenAI Responses compaction directly portable unless the model path also changes to Responses + items. + +### More Invasive: Dual Transcript Model + +What changes: + +- UI explicitly stores: + - visible transcript for rendering; + - model transcript for backend requests. +- Backend can send replacement model transcript chunks after compaction. +- UI may keep visible old messages while sending only checkpoint plus suffix. + +Benefits: + +- Best balance of UX and backend efficiency. +- Preserves user-visible history while reducing model and transport context. +- Enables clean UI markers for compacted ranges. + +Effort: + +- High. + +Risks: + +- Requires careful reconciliation when users branch, edit, retry, or resume conversations. +- Requires strong message identity and migration rules. + +Library impact: + +- Makes reducer/session-library concepts easier to apply because the UI has an explicit model-history store. +- Provider-native opaque compaction still needs provider-specific transport support. + +### Most Invasive: Server-Owned Session History + +What changes: + +- UI sends a `session_id` plus new user input instead of the full message array. +- Backend owns the transcript, compaction, checkpointing, and replay. + +Benefits: + +- Eliminates repeated full-history upload. +- Allows robust backend-managed compaction and storage. +- Aligns with framework session models like OpenAI Agents SDK sessions or LangGraph checkpointers. + +Effort: + +- Very high. + +Risks: + +- Major API and product change. +- Requires session lifecycle, storage, retention, auth, replay, branching, and export semantics. +- Harder for stateless deployments. + +Library impact: + +- This is the path where session libraries become most applicable. +- It is likely too large for the first QuickApps history compaction milestone. + +## Recommended Direction + +Recommended path: + +1. **Track 1: Backend-only effective-history compaction.** + - Keep the UI contract unchanged. + - Store a structured checkpoint in assistant `custom_content.state.history_compaction`. + - Reconstruct effective history from checkpoint plus recent suffix. + - Preserve tool-call episodes, attachments, artifacts, decisions, and open questions. + +1. **Track 2: UI checkpoint fidelity and visibility.** + - Confirm the UI preserves unknown state keys. + - Optionally show compacted-history markers. + +1. **Track 3: Optional UI canonicalization.** + - Add a backend-returned compacted effective transcript or patch. + - Let UI send the compacted model transcript while rendering full visible transcript. + +1. **Track 4: Provider-native compaction and long-term memory.** + - Add optional provider-native compaction when DIAL exposes compatible capabilities. + - Consider LlamaIndex-like fact/vector memory only after MVP compaction is stable. + +The investigation points toward a QuickApps-native implementation. Libraries provide useful vocabulary and algorithms, +but QuickApps' cross-provider DIAL surface, `aidial_sdk` message shape, `custom_content.state`, restored tool history, +and attachment/artifact semantics require a custom reducer and checkpoint service. + +## Follow-Up Decisions + +- **Feature placement:** Move history compaction under `features.history_compaction`. Enabling history compaction should also + enable the backend context-usage measurement path, since compaction depends on provider-native token-count preflight. + This fits the expected direction that `features.context_usage.enabled` may become enabled by default soon. +- **Checkpoint location:** `custom_content.state` is the practical place to round-trip compaction checkpoints in the + current protocol, but it is a workaround rather than a clean long-term session-history model. +- **UI visibility:** Whether the UI should expose a visible "history compacted" marker needs discussion with Core, UI, + and adapter teams. +- **Provider-native compaction:** A provider-neutral DIAL compaction capability may be possible, but it pulls in adapter + team work and increases solution complexity. Treat it as a later optimization, not the MVP dependency. diff --git a/docs/designs/history_compaction_strategy.md b/docs/designs/history_compaction_strategy.md new file mode 100644 index 00000000..13b938fd --- /dev/null +++ b/docs/designs/history_compaction_strategy.md @@ -0,0 +1,504 @@ +# Design: History compaction strategy + +- **Status:** Draft +- **Dependencies:** + - [Context window usage](context_window_usage.md) - provider-native token-count preflight, model limits, and `context_usage` state. + - [History compaction UI/backend communication](history_compaction_ui_backend.md) - `custom_content.state` round-trip contract and optional UI trimming tracks. + +## Problem Statement + +Long QuickApp conversations can exceed the orchestrator model context window because the UI sends the full `messages` array on every turn. The backend then restores assistant `custom_content.state.tool_execution_history` into assistant/tool message pairs and sends that expanded history to the LLM. + +Today this creates several failure modes: + +1. **Tool history expands hidden state into prompt tokens** - a single assistant row can carry a large tool-call history in `custom_content.state`; after restoration it can dominate the context window. +1. **No effective-history shortening** - even if the backend can detect that the request is close to the context limit, it has no strategy for replacing old history with a smaller representation. +1. **State can only be persisted on the next assistant message** - compaction performed for the current request must also be written to the next assistant response so the UI can round-trip the checkpoint on the following turn. +1. **Attachments and artifacts can be lost by naive summarization** - compacting prose alone risks dropping attachment names, file URLs, generated artifact references, and whether a document was actually inspected. +1. **A summary can still be too large** - compaction needs its own token budget and retry path instead of assuming that any summary will fit. + +This design defines the backend compaction strategy, compacted state shape, and message-selection rules. It builds on the token-count hook from `context_window_usage.md` and the UI persistence contract from `history_compaction_ui_backend.md`. + +## Design Goals + +- **G1 - Prevent over-limit upstream calls:** When token-count preflight exceeds the compaction threshold, build a smaller effective history before calling the orchestrator model. +- **G2 - Preserve conversation semantics:** Retain user goals, constraints, decisions, unresolved tasks, tool outcomes, and artifact references while dropping low-value bulk. +- **G3 - Preserve tool integrity:** Never create invalid chat history with orphan tool messages or assistant tool calls without matching tool results. +- **G4 - Preserve attachment identity:** Keep structured attachment and artifact references even when their surrounding messages are summarized. +- **G5 - Bound compacted state size:** Token-count the compacted effective payload and recursively shrink the compacted state if it still exceeds budget. +- **G6 - Persist the checkpoint:** Write compaction metadata to the next assistant message `custom_content.state` so the next request can anchor on it. +- **G7 - Keep configuration small:** Use one backend compaction pipeline with internal content-aware reducers, not many user-facing strategy knobs. + +--- + +## Use Cases + +### UC-1: Long plain-text thread reaches the threshold + +**Trigger:** Token-count preflight for the orchestrator request exceeds the configured compaction threshold. + +**Behavior:** QuickApp keeps the protected prefix and recent suffix raw, summarizes older plain-text turns into compacted state, rebuilds the effective history, then token-counts the rebuilt payload. + +**Outcome:** The current LLM call succeeds with a shorter prompt. The final assistant message includes the compaction checkpoint in `custom_content.state` for the next turn. + +### UC-2: Old assistant row contains large restored tool history + +**Trigger:** Request setup expands `state.tool_execution_history` from an old assistant message, making the prompt too large. + +**Behavior:** QuickApp groups assistant/tool messages into tool episodes and summarizes old episodes as tool outcomes, preserving durable IDs, URLs, artifact names, errors, and any useful final result. + +**Outcome:** The LLM sees the outcome of old tool work without receiving all raw tool bodies. + +### UC-3: User uploaded attachments + +**Trigger:** Compactable history contains messages with `custom_content.attachments` or tool results that reference DIAL files. + +**Behavior:** QuickApp records attachment metadata in a structured registry and summarizes only observed or extracted facts. If an attachment was uploaded but never read, the state says it was referenced but not analyzed. + +**Outcome:** Attachment names and references survive compaction without inventing document contents. + +### UC-4: Audio or transcript-heavy messages are old enough to compact + +**Trigger:** Compactable history contains audio-derived content, transcripts, or speech-to-text tool results. + +**Behavior:** QuickApp reduces the transcript into concise text while preserving speaker intent, key facts, decisions, and any referenced attachments or artifacts. + +**Outcome:** The LLM receives the useful content from the audio without carrying the full transcript in the prompt. + +### UC-5: Compacted history is still too large + +**Trigger:** The rebuilt effective payload still exceeds the target after the first compaction attempt. + +**Behavior:** QuickApp runs recursive shrink steps: compress the summary, prune low-value sections, reduce attachment/tool details to metadata, and retest with token count. + +**Outcome:** The effective payload fits the budget or QuickApp returns a controlled error instead of sending an over-limit LLM request. + +--- + +## Proposed Design + +### Concern 1: Runtime placement and payload parity + +- **What:** Add a history-compaction step that shares the exact message-preparation path used for orchestrator token counting and completion. +- **Owner:** QuickApp backend, in the agent/orchestrator request path. +- **Semantics:** + 1. Request setup restores `custom_content.state.tool_execution_history` into assistant/tool message pairs. + 1. Request-level message transformers run. + 1. If a valid `history_compaction` checkpoint exists, QuickApp reconstructs the baseline effective history from that checkpoint plus the later raw suffix; otherwise it counts the restored transcript. + 1. QuickApp builds the same logical orchestrator input that completion will use, including pre-invocation transformers currently applied in `_ChatCompletionConfigBuilder._prepare_messages`. + 1. The raw effective payload is sent to the token-count endpoint. + 1. If context fill is below threshold, no compaction runs. + 1. If context fill is above threshold, QuickApp builds a compacted message list, runs it through the same pre-invocation preparation path, and token-counts that prepared payload again. + 1. The prepared compacted payload that was counted is passed to completion without another divergent message transformation step. + 1. The compaction checkpoint is written to the next assistant message state. +- **Change:** The model no longer always receives the fully restored UI transcript. It receives an effective transcript derived from the UI transcript plus the latest valid compaction checkpoint, and the counted payload stays identical in meaning to the completion payload. + +Implementation should avoid a second, hidden message-preparation path. Either: + +1. Refactor `_ChatCompletionConfigBuilder` so token counting, compaction, and `AssistantInvoker` all consume one shared prepared-payload builder. +1. Or split message preparation from payload assembly so compaction can replace the `messages` field and then reuse the same prepared payload for both token count and completion. + +```mermaid +flowchart TD + requestMessages[Request messages] + restoredHistory[Restore tool history] + requestTransformers[Run request transformers] + checkpointCheck[Apply latest compaction checkpoint if present] + prepareRaw[Prepare completion payload] + countRaw[Token count prepared payload] + thresholdCheck{Over threshold} + compact[Build compacted effective history] + prepareCompacted[Prepare compacted payload] + countCompacted[Token count compacted payload] + llmCall[Call orchestrator LLM] + assistantState[Write checkpoint to assistant state] + + requestMessages --> restoredHistory + restoredHistory --> requestTransformers + requestTransformers --> checkpointCheck + checkpointCheck --> prepareRaw + prepareRaw --> countRaw + countRaw --> thresholdCheck + thresholdCheck -->|"No"| llmCall + thresholdCheck -->|"Yes"| compact + compact --> prepareCompacted + prepareCompacted --> countCompacted + countCompacted --> llmCall + llmCall --> assistantState +``` + +### Concern 2: History zones + +- **What:** Split the conversation into protected prefix, compactable body, and recent suffix. +- **Owner:** `HistoryCompactionService`. +- **Semantics:** + - **Protected prefix:** System/setup content that must remain raw or be regenerated by backend. This includes system messages and required assistant/tool pairs used for skill loading or internal context enrichment when they are needed for app behavior. + - **Compactable body:** Older user/assistant/tool episodes that can be replaced by a compacted summary and structured references. + - **Recent suffix:** Last N turns or episodes kept raw to preserve local coherence for the next answer. +- **Change:** Compaction operates on episode groups, not arbitrary individual messages. + +Protected messages can still contribute to a separate "active context summary" if they are expensive enrichment outputs, but the MVP should avoid dropping required setup messages. If a setup/enrichment message is deterministic, prefer regeneration or raw retention over summarizing it as user conversation. + +### Concern 3: Episode grouping + +- **What:** Normalize messages into compaction episodes before selecting what to compact. +- **Owner:** `HistoryEpisodeBuilder`. +- **Semantics:** Episodes are the smallest safe compaction unit: + - Plain user message plus assistant answer. + - Assistant tool-call message plus matching tool result messages and the assistant follow-up. + - Attachment upload plus related list/read/get-content tool activity. + - Short clarification sequences grouped into one requirements episode. + - Existing compaction anchor plus later messages. +- **Change:** Tool messages are never compacted without their owning assistant tool call. This preserves OpenAI-style tool integrity. + +### Concern 4: Content-aware reducers + +- **What:** Use one compaction pipeline with internal reducers selected by episode shape. +- **Owner:** `HistoryCompactionService`. +- **Semantics:** Reducers produce a common compacted representation. They are internal implementation details, not separate public strategies. + +Recommended reducers: + +| Reducer | Input shape | Output | +|---------|-------------|--------| +| `PlainConversationReducer` | Text-only user/assistant turns | Goals, constraints, decisions, open questions, final answers | +| `ToolEpisodeReducer` | Assistant/tool message groups | Tool intent, relevant arguments, outcome, errors, durable IDs/URLs | +| `AttachmentEpisodeReducer` | Messages or tool results with attachments/files | Attachment registry entries and observed facts only | +| `AudioTranscriptReducer` | Audio messages, transcripts, speech-to-text tool results | Concise text summary with speaker intent and key facts | +| `StageNoiseReducer` | Reasoning/progress/stage-heavy content | User-visible conclusion or nothing if purely diagnostic | +| `ArtifactReferenceReducer` | Generated files, charts, exports, links | Artifact name, URL, purpose, status | + +- **Change:** `history_compaction` enables the pipeline; reducer selection is automatic based on episode shape. + +### Concern 5: Attachment and artifact registry + +- **What:** Store attachment and artifact metadata separately from prose summary. +- **Owner:** `HistoryCompactionService`. +- **Semantics:** Every attachment or generated artifact found in compacted history gets a small structured entry. The prose summary may reference these entries but does not need to carry all metadata inline. +- **Change:** Compaction no longer risks losing names or URLs when old messages are omitted from effective history. + +Illustrative entry: + +```json +{ + "name": "contract.pdf", + "url": "files/abc/contract.pdf", + "source": "user", + "content_type": "application/pdf", + "status": "referenced", + "observed_facts": [], + "first_seen_at": { + "message_index": 12 + } +} +``` + +Attachment statuses: + +| Status | Meaning | +|--------|---------| +| `referenced` | Attachment appeared in history but was not read or analyzed by the backend/model. | +| `read` | Attachment content was loaded or exposed through a tool result. | +| `summarized` | Relevant extracted content is represented in compacted summary. | +| `generated` | Artifact was created by QuickApp or a tool and sent to the user. | + +### Concern 6: Compacted state schema + +- **What:** Store compaction checkpoint under `custom_content.state.history_compaction`. +- **Owner:** QuickApp writes; UI preserves and resends. +- **Semantics:** The latest valid assistant message with `history_compaction` is the anchor. On the next request, messages before that anchor can be ignored for effective LLM input and replaced by the compacted state. +- **Change:** Replaces illustrative `history_compacted` and `history_summary` fields from the UI/backend draft with a versioned object. + +Anchor semantics: + +- The primary anchor rule is "latest valid assistant message containing `custom_content.state.history_compaction`". +- If the UI/backend message contract provides stable message ids, store the compacted-through id and use it for validation. +- `message_index` is diagnostic only; it can help tests and logs, but it must not be the sole source of truth because UI-side trimming can renumber messages. + +Draft schema: + +```json +{ + "history_compaction": { + "schema_version": 1, + "compacted_through": { + "message_id": "msg_018", + "message_index": 18, + "anchor_role": "assistant" + }, + "summary": "User is building a QuickApps history compaction design. Key decisions: ...", + "preserved_facts": [ + "Compaction runs before the current LLM call when token count exceeds threshold.", + "The compacted checkpoint is written to the next assistant message state." + ], + "open_questions": [], + "attachments": [ + { + "name": "contract.pdf", + "url": "files/abc/contract.pdf", + "source": "user", + "content_type": "application/pdf", + "status": "referenced", + "observed_facts": [] + } + ], + "artifacts": [], + "tool_outcomes": [ + { + "tool_name": "internal_attachments_get_content", + "outcome": "Loaded selected pages from contract.pdf", + "status": "success" + } + ], + "token_budget": { + "target_tokens": 8192, + "estimated_tokens": 2100 + } + } +} +``` + +### Concern 7: Compaction feature configuration + +- **What:** Add `HistoryCompactionConfig` under `features.history_compaction`, with an optional summarizer deployment. +- **Owner:** Config layer and compaction service. +- **Semantics:** History compaction is a feature-level capability because it depends on context-usage measurement rather than on a specific orchestrator deployment parameter. When `features.history_compaction.enabled` is `true`, QuickApp also enables the context-usage measurement path required for token-count preflight, even if `features.context_usage` is omitted. If `summarizer_deployment` is set, QuickApp uses it for summarization. If omitted, QuickApp falls back to the main orchestrator deployment. +- **Change:** Moves compaction config under `features`, keeps summarization configurable without requiring a separate application-level tool or deployment type, and makes the dependency on `context_usage` explicit. + +Config fields: + +| Field | Default | Validation | Omitted behavior | +|-------|---------|------------|------------------| +| `enabled` | `false` | boolean | No compaction, no summarizer calls, and no `history_compaction` state is written. | +| `trigger_percent` | `85` | `1 <= value <= 100` | Compaction starts when token-counted input reaches 85% of the prompt limit. | +| `target_percent` | `70` | `1 <= value < trigger_percent` | Recursive shrink targets 70% after compaction. | +| `keep_recent_turns` | `3` | integer `>= 1` | Keep the latest three user-facing turns or equivalent tool episodes raw. | +| `summary_budget_tokens` | derived from `target_percent`, capped at `8192` | integer `>= 512` when set | Use a bounded summary budget computed from the model limit. | +| `summarizer_deployment` | `null` | same deployment config shape as `orchestrator.deployment` | Use the main orchestrator deployment as the summarizer. | + +`features.history_compaction` should be a stable disabled-by-default config block, not preview-gated. The field is inert when omitted or when `enabled` is `false`. When `enabled` is `true`, config resolution treats `features.context_usage` as enabled for the backend measurement path because compaction cannot safely decide when or how much to shrink without provider-native token-count preflight. This remains compatible with making `features.context_usage.enabled` default to `true` in a nearby release. + +Example: + +```json +{ + "features": { + "history_compaction": { + "enabled": true, + "trigger_percent": 85, + "target_percent": 70, + "keep_recent_turns": 3, + "summary_budget_tokens": 8192, + "summarizer_deployment": { + "deployment_id": "gpt-5.2-mini" + } + } + }, + "orchestrator": { + "deployment": { + "deployment_id": "gpt-5.2" + } + } +} +``` + +### Concern 8: Recursive shrink and failure behavior + +- **What:** Treat compaction as successful only if the rebuilt effective payload fits the target budget. +- **Owner:** `HistoryCompactionService` and `ContextUsageService`. +- **Semantics:** + 1. Build first compacted state. + 1. Rebuild effective messages. + 1. Token-count the effective payload. + 1. If it still exceeds target, shrink the compacted state with stricter instructions. + 1. Drop low-value sections in order: stage details, verbose tool bodies, repeated errors, non-recent observed facts, attachment observed facts while keeping attachment identity. + 1. Retry up to a small fixed limit. + 1. If still too large, return a controlled context-limit error. +- **Change:** QuickApp avoids sending an upstream LLM request that is known to be too large. + +### Concern 9: Effective history reconstruction + +- **What:** Convert compacted state plus raw suffix into valid model messages. +- **Owner:** `HistoryCompactionService`. +- **Semantics:** The compacted state is injected as a synthetic assistant message after the protected prefix and before the recent suffix. The message content uses a deterministic envelope so golden tests can compare the effective history. +- **Change:** The backend can ignore UI messages before the anchor while still preserving their meaning in a compacted replacement. + +MVP synthetic message envelope: + +```text +Previous conversation was compacted by QuickApp. + +Summary: +{summary} + +Preserved facts: +- {fact} + +Open questions: +- {question} + +Attachments and artifacts: +- {name} ({status}): {url_or_description} + +Tool outcomes: +- {tool_name}: {status}; {outcome} +``` + +Provider-specific role or envelope changes are future compatibility work and must keep the same compacted state schema. + +Illustrative effective order: + +1. System/setup messages. +1. Protected skill/context-enrichment messages that must remain raw. +1. Synthetic compacted history message derived from `history_compaction`. +1. Recent raw user/assistant/tool episodes. +1. New user message. + +--- + +## Secondary Fixes + +### Logging and privacy + +Compacted summaries may contain sensitive user data. Logs should report sizes, token counts, and reducer names, not full summaries or attachment contents. + +### Metrics + +Add counters for compaction triggered, compaction succeeded, recursive shrink count, final percent, controlled failures, and reducer categories used. This will show whether content-aware reducers are worth their complexity. + +--- + +## Out of Scope + +- **UI-side deletion of old messages:** Deferred to Track 2 in `history_compaction_ui_backend.md`. Backend effective-history compaction works even when UI keeps sending the full transcript. +- **Server-owned session history:** Deferred to a larger session model. This design stays compatible with stateless UI-resends-full-history behavior. +- **Vector/RAG memory over chat history:** Useful later, but too large for MVP and requires retrieval correctness work. +- **Provider-specific summarization prompts:** MVP should use one conservative summarization prompt with structured output. +- **Exact attachment content extraction:** Compaction records attachment identity and observed facts. It does not introduce new document parsing beyond existing tools. + +--- + +## Configuration / Usage Examples + +### Minimal enabled config + +```json +{ + "features": { + "history_compaction": { + "enabled": true + } + }, + "orchestrator": { + "deployment": { + "deployment_id": "gpt-5.2" + } + } +} +``` + +### Summarizer fallback + +If `features.history_compaction.summarizer_deployment` is omitted, QuickApp uses `orchestrator.deployment` for compaction. This keeps existing apps easy to enable while allowing production apps to choose a cheaper or faster summarizer. + +### Context usage dependency + +Enabling `features.history_compaction.enabled` also enables the backend context-usage measurement path needed for compaction decisions. Apps may still configure `features.context_usage` directly when they need UI-visible context-meter behavior, but compaction does not require the app author to set both feature blocks. + +### Acceptance checks + +| ID | Check | +|----|-------| +| R1 | A request above `trigger_percent` compacts before the current LLM call, not only after the turn. | +| R2 | Final assistant state contains `history_compaction.schema_version: 1` after compaction succeeds. | +| R3 | Next request with the checkpoint can omit messages before the anchor from effective LLM input. | +| R4 | Tool messages are compacted only as complete assistant/tool episodes. | +| R5 | Attachments from compacted history appear in `history_compaction.attachments` with name, source, and status. | +| R6 | Audio/transcript-heavy compacted history is represented as concise text, not raw transcript bulk. | +| R7 | If compacted payload still exceeds budget after retries, QuickApp returns a controlled error and does not call the orchestrator LLM. | + +--- + +## Migration + +### Breaking changes + +None. The feature is disabled by default and relies on optional config plus optional `custom_content.state.history_compaction`. + +### Non-breaking changes + +- Adds optional `features.history_compaction` config. +- Adds optional `custom_content.state.history_compaction` on assistant messages. +- Reuses the existing UI requirement to preserve unknown `custom_content.state` fields. +- Keeps the UI full-history request contract unchanged for MVP. +- Exposes one public compaction mode in v1; content-aware reducers remain internal policy. + +## Summary of Changes + +| Area | Addition or change | +|------|--------------------| +| Config | `features.history_compaction.enabled`, `trigger_percent`, `target_percent`, `keep_recent_turns`, `summary_budget_tokens`, optional `summarizer_deployment`; enabling it also enables backend `context_usage` measurement | +| Agent request path | Compaction step that reuses the same prepared-payload path for token counting and orchestrator LLM invocation | +| State | Versioned `custom_content.state.history_compaction` checkpoint | +| Compaction service | Episode grouping, content-aware reducers, recursive shrink, effective-history reconstruction | +| Attachments | Structured attachment/artifact registry preserved outside prose summary | +| Tests | Token-threshold trigger, tool-episode integrity, attachment preservation, recursive shrink, next-turn checkpoint reuse | + +--- + +## Review Notes — Round 1 + +- **Reviewer:** Claude (quickapps-design-review skill) +- **Date:** 2026-06-24 + +### Verdict + +Blocking issues must be addressed. The design is well structured and covers the hard semantic cases, especially tool episodes and attachment identity, but it needs a more exact runtime placement and config contract before implementation can proceed safely. + +### Blocking issues + +1. **Proposed Design / Concern 1: Runtime placement** — The doc says compaction runs after "normal message transformers" and after the "chat-completion payload is built for token counting", but the current LLM payload is also changed later by pre-invocation transformers in `_ChatCompletionConfigBuilder._prepare_messages` (`src/quickapp/core/agent/_chat_completion_config_builder.py:78`) before `chat.completions.create` (`src/quickapp/core/agent/assistant_invoker.py:34`). If compaction/counting is inserted only between `_RequestContextSetup.setup_messages` and `AssistantInvoker.invoke`, it can measure or compact a payload that is not exactly what the model receives, contradicting the linked context-usage design's "exact logical input" requirement. + **Suggestion:** Specify the concrete integration point: either move compaction/token-counting behind the same payload-preparation path used by `_ChatCompletionConfigBuilder`, or refactor that builder so compaction receives the post-transform effective payload and then passes the same messages into completion. + +2. **Proposed Design / Concern 7: Compaction model configuration** — The example introduces `enabled`, `trigger_percent`, `target_percent`, `keep_recent_turns`, `summary_budget_tokens`, and `summarizer_deployment`, but only `summarizer_deployment` has omitted-field semantics. The design goals promise "Keep configuration small", while the README rubric requires explicit defaults and non-obvious behavior. Without defaults, `None` handling, validation bounds, and preview-gating decision, implementers cannot know whether this lands as a stable public schema or a preview field. + **Suggestion:** Add a compact config table covering each field's default, validation range, omitted behavior, and whether `orchestrator.history_compaction` is gated by `ENABLE_PREVIEW_FEATURES` / `PreviewField` or is a stable disabled-by-default feature. + +### Suggestions + +1. **Proposed Design / Concern 6: Compacted state schema** — The schema uses `compacted_through.message_index` in the main contract and only later notes in Secondary Fixes that indexes are fragile. Because the UI/backend dependency already calls out optional trimming tracks, an index-like anchor in the primary schema could become stale as soon as Track 2 exists. + **Suggestion:** Move the "prefer stable message id / latest valid checkpoint" rule into Concern 6 and make `message_index` explicitly illustrative or diagnostic rather than the primary anchor contract. + +2. **Proposed Design / Concern 9: Effective history reconstruction** — The design says the synthetic compacted message can be "assistant or system-adjacent" and that the exact role "should match provider compatibility". That is a reasonable implementation concern, but it leaves the persisted state schema and golden tests without a deterministic expected message shape. + **Suggestion:** Pick the MVP role and content envelope in the design, then list provider-specific deviations as future compatibility work if needed. + +### Nits + +1. **Proposed Design / Concern 4: Content-aware reducers** — "QuickApp does not expose multiple compaction strategies in configuration for v1" is useful compatibility information, but the skill rubric asks to keep non-change prose out of design-body sections where possible. + **Suggestion:** Move this sentence to Migration / Non-breaking changes or shorten it into the config concern's default behavior. + +--- + +## Review Notes — Round 2 + +- **Reviewer:** Claude (quickapps-design-review skill) +- **Date:** 2026-06-24 + +### Verdict + +Ready for approval pending minor suggestions. The Round 1 blockers are resolved: payload parity is now explicit, config defaults and gating are specified, anchor semantics moved into the schema concern, and the synthetic message envelope is deterministic enough for golden tests. + +### Suggestions + +1. **Proposed Design / Concern 7: Compaction model configuration** — The doc now says enabling compaction requires `features.context_usage.enabled: true`, but it does not state what happens when `orchestrator.history_compaction.enabled: true` is configured without that dependency, or when the token-count endpoint is unavailable. This matters because the linked context-usage design allows count failures to continue without a meter, while this design's G1 and UC-5 promise to avoid known over-limit calls and return controlled errors. + **Suggestion:** Add one sentence or config-table row that defines this failure mode: schema/config validation error, initialization warning with compaction disabled, or controlled runtime error before orchestrator completion. + +2. **Proposed Design / Concern 1: Runtime placement and payload parity** — The runtime flow covers first-time compaction, and Concern 6 says the latest valid `history_compaction` message is the anchor. The flow does not explicitly show whether an existing checkpoint is applied before the "raw effective payload" token count on the next turn. + **Suggestion:** Add a short step after request transformers: "If a valid checkpoint exists, reconstruct baseline effective history from it before counting; if no checkpoint exists, count the restored transcript." That makes the cross-turn behavior line up with R3 and the UI/backend contract. + +### Changes since previous round + +1. **Resolved:** Round 1 blocking issue on runtime placement. Concern 1 now names `_ChatCompletionConfigBuilder._prepare_messages`, requires shared preparation for token count and completion, and updates the flow diagram. +2. **Resolved:** Round 1 blocking issue on config defaults and gating. Concern 7 now includes defaults, validation ranges, omitted behavior, and the stable disabled-by-default decision. +3. **Resolved:** Round 1 suggestion on anchor fragility. Concern 6 now makes latest-valid-checkpoint the primary rule and treats `message_index` as diagnostic only. +4. **Resolved:** Round 1 suggestion on effective history reconstruction. Concern 9 now chooses a synthetic assistant message and deterministic envelope for MVP. +5. **Resolved:** Round 1 nit on non-change prose. The reducer strategy note was moved into positive behavior in Concern 4 and compatibility wording in Migration.