Skip to content

docs: draft the invariants of each component (keep-alive and the sweep first) - #820

Draft
amiddavid wants to merge 11 commits into
mainfrom
feat/component-invariants
Draft

amiddavid wants to merge 11 commits into
mainfrom
feat/component-invariants

Conversation

@amiddavid

Copy link
Copy Markdown
Collaborator

Summary

A draft for inline review: docs/invariants.md. It defines, for each component, the rules that must hold on real traffic, and how to check each one from metadata. A monitor will later run these checks every day or week.

This PR is for comments. It is not ready to merge.

What is in it

  • The component list, grouped, with where each one's purpose is written.
  • Keep-alive (K1 to K6) and the sweep (S1 to S5): proposed invariants, with baselines from a copy of the laptop dashboard database.
  • 4 suspected breaches:
    • B1: two counters give 28.7% and 6.7% for the sweep's false-drop rate.
    • B2: requests labelled hit that re-wrote the prefix.
    • B3: toolfilter reports acted = 0 but mutated = 3204.
    • B4: cachesplit never acted.
  • 5 gaps. None of them needs stored content.
  • A draft monitor spec.

What to review

  • Confirm, change or drop each invariant.
  • Give the thresholds. Every number in the document is a proposal.
  • Decide on each suspected breach: open an issue now, or keep it in the document only.

Data dependence

None. The document and its queries read metadata columns only. No code change.

Open questions

  • The components with no laptop baseline are marked as gaps, as you decided.
  • The production part (cg2, tenants, real thresholds, who gets the report) goes into the internal repo, later.

…-alive and the sweep

A first draft for review. It lists the components, proposes invariants
for keep-alive and the sweep with baselines from a copy of the laptop
dashboard database, and lists the suspected breaches and the gaps.
Metadata only. No code change.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
Add T1 to T4 for the option B fallback: a disable by reason, with the
Claude Code version, and the one cache rewrite it costs. The signal
names come with the #898 PR, so they are gap G6 until then.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
Fill T1 to T4 with the metric, the /stats field and the log line from
PR #900 (open, not reviewed). Add how each reason counts, the disable
scope, today's known-good hash, and the log join T4 needs.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
From sweep-mcp's brief and a check of main: purpose, the signals that
exist, 9 invariants with the laptop baseline since the switch to
ultrahigh, and gaps G7 (no cg_* metric) and G8 (a missed tool, a purge).

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
Rewrite K1 to K6 for the per-wait ping plan (#867) with a v0.6.6
baseline, add K7 (no sweep question in a tool wait) and K8 (no ping
when done), close G1 with wake_cause. Add breaches B5 to B7. Update T4
and the #900 signals. Add the expand-rate section E1 to E3 and gap G9
(no per-component expand count).

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
Fill E1 to E3 with fleet totals from a read-only scan of the cg2
dashboard database (2026-11-04, 2 days, 13,338 real requests), add E4
(expands per 1,000 requests), and schedule them weekly. Per-tenant
numbers stay out of the repo.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
The break is real, at the end of system + tools, on xhigh too, and not
caused by keep-alive. Two candidate causes; sweep-mcp checks the
deferral one.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
…fix)

The ping and the request keep the cold clock under different keys on
ultrahigh, so the sweep cuts a prefix a ping just kept warm. S6 counts
sweep applies within the cache lifetime of a ping read, daily.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
cache-strategies traced each of the 5: two restarts dropped the planned
pings, a subagent system prompt rewrite, a new subagent thread, and a
ping plan that ended 3 s early. B2 gets 3 confirmed mislabels. B8: a
restart drops the planned pings. B9: a 20k read is the shared head.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>
The disable metric now has 2 reasons, unknown_description and break.
T4 counts break only. Releases up to v0.6.7 still send the old label;
the monitor ignores it.

Assisted-By: Claude Opus 5.5
Signed-off-by: DAVID AMID <DAVIDA@il.ibm.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New/ToDo

Development

Successfully merging this pull request may close these issues.

1 participant