feat(oncall-agent): read-only Slack-triggered cluster triage agent - #293
Draft
manan164 wants to merge 48 commits into
Draft
feat(oncall-agent): read-only Slack-triggered cluster triage agent#293manan164 wants to merge 48 commits into
manan164 wants to merge 48 commits into
Conversation
Scaffolds an Agentspan agent that triages Orkes SaaS health-check alerts. It listens on the Slack alert channel, reads the failing health_check execution by id, and runs READ-ONLY agent-handler commands against the ah5r-prod Conductor API to investigate, then replies in-thread with a root-cause hypothesis. Advisory/dry-run only — no remediating actions. - sql_guard: deterministic SELECT-only guard (not LLM-trusted) for the run_sql_select tool; rejects DML/DDL, multi-statement, comment-smuggling. - conductor_client: dispatch read-only agent-handler workflows + poll; reads org/cluster/cloudEnvironmentTag off the failing execution so only the executionId is threaded into tools. - tools: 9 read-only investigation tools mapped to agent-handler commands. - agent: Claude triage loop with a component->investigation runbook. - slack_app: Socket Mode listener -> triage -> threaded reply. - tests: sql_guard + alert parsing, deterministic (no LLM), validated by proving each fails before passing per CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ention Follows the pattern in sdk/python/examples/91_slack_autofix_agent.py (per PR #135): poll the alert channel with conversations.history + reply via chat.postMessage using a bot token only — no slack_bolt / Socket Mode, no app-level token. Run-once or --loop, dedup via a local state file. Slack I/O lives in a deterministic poller; the triage agent stays pure (investigates a given execution id). Adds test_poller.py covering alert-only triage, cross-poll dedup, and failure reporting (fakes, no network/LLM; validated fail-then-pass per CLAUDE.md). Config: drop SLACK_APP_TOKEN; add SLACK_ALERT_CHANNEL (required), ONCALL_POLL_INTERVAL, ONCALL_STATE_FILE. requirements: drop slack-bolt, add requests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- test_tools_readonly: source-level invariant that no mutating/privileged agent-handler command is wired into tools.py, and SQL goes through the SELECT guard. Validated by adding DELETE_POD and confirming failure. - scripts/smoke_dispatch.py: LLM-free live check against ah5r-prod — reads the failing execution's cluster context, dispatches read-only commands (get_pods_data, get_cluster_metrics, SELECT 1), asserts COMPLETED + output shape. The L1 verification step; run with the Conductor app key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r domain Two bugs found during the first live run against ah5r-prod (viz-stage): 1. cloudEnvironmentTag is NOT in the health_check workflow input (it's produced by prepare_agent_handler's output), but sql_conductor reads it from workflow.input — derive it as c<orgId[:5]>-<clusterName> when absent. 2. Dispatched commands set no task_to_domain, so customer-cluster tasks (e.g. collect_metrics) sat in the default queue and the in-cluster agent never polled them -> TIMED_OUT. Mirror the control plane: wildcard "*" -> the cluster domain (orgId#-#clusterName), with orchestration tasks pinned to NO_DOMAIN. Switched dispatch to StartWorkflowRequest to carry task_to_domain. Verified live: GET_PODS_DATA and PULL_LOGS now COMPLETE end-to-end with real viz-stage data. Adds deterministic regression tests (fake client) for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated end-to-end against ah5r-prod (viz-stage Redis-critical alert): the agent now autonomously reads the health-check data and pulls server + worker logs to reach a root-cause hypothesis. Fixes found during the live run: - runtime_compat: on macOS, run conductor tool-workers as THREADS, not forked processes. Forked children segfault in getaddrinfo (Network.framework is not fork-safe) and 'spawn' can't pickle the worker's thread lock. agentspan ships a thread shim but gates it to Windows; reuse it on macOS. No-op on Linux (where fork is safe — how this runs in prod). Wired into triage + slack paths. - get_incident_details: surface parse_conductor_cluster_data (redis.usage, decider_queue_size = running workflows, indexer_queue_size, heap, cpu, postgres) so the agent reads the queue numbers from the health-check JSON instead of deriving them via SQL. - runbook: treat queue/usage as the symptom; find the cause in CONDUCTOR SERVER and WORKER pod logs (+ pod events). Explicitly forbid ad-hoc SQL on large tables like `workflow` (decider_queue_size already is the running-workflow count). run_sql_select is a last resort, not the primary tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…check alerts - agent.py: add a compact alert-type playbook covering all 17 HealthIssue types (resource saturation, component-down, pod, networking, self-describing). Preserves the existing Redis/CPU/heap guidance; routes the model symptom -> evidence -> cite. - tools.py: add read-only get_ingress_info (GET_INGRESS_INFO) for DNS / domain resolution / reachability alerts (empty ingress address = LB not provisioned). - test_tools_dispatch.py: assert each tool dispatches its expected agent-handler command (incl. get_ingress_info) via a fake dispatcher. Validated fail-then-pass. - test_tools_readonly.py: fix command names to match the AgentHandlerCommand enum (ROLLOUT_RESTART, KUBECTL_UNRESTRICTED, ...) so the guard actually catches them; add heavy/disruptive commands (DOWNLOAD_HEAP_DUMP, etc.) to the deny list.
…verage Verified against the source workflow/worker (not assumed): - get_incident_details ref "issues" matches health_check.json taskReferenceName, and the issues task output carries the per-issue severity+description text the playbook matches on (HealthCheckIssuesWorker) — the 17-type playbook is actually reachable. - alert.py: the real Slack text is markdown (*`[CRITICAL]`* … _Org's_ env *`cluster`*) + emoji + appended execution URL. The old _CLUSTER_RE choked on the italic/bold markers and returned org/cluster=None. Strip *,` decoration and tolerate italic _ so org/cluster parse; execution id + severity were already fine. Added a test built from the exact worker+notify format; validated fail-then-pass. - test_playbook_coverage.py: deterministic guard that all 17 HealthIssue types stay in the playbook; validated it fails when a type is dropped.
…ai.agents The main merge into this branch (cd689ae) renamed the Python SDK package from `agentspan` to `conductor` (dist `conductor-agent-sdk`), import root `conductor.ai.agents`. oncall-agent still imported `agentspan.agents`, so the module could not import its SDK at all on the post-merge branch. Swap the five SDK imports (Agent, tool, AgentRuntime, worker_manager shim) to the new namespace. Left the local `agentspan_server_url` config field + AGENTSPAN_SERVER_URL env var as-is — they're our own naming, not SDK symbols, and the env var is documented. Full suite passes (37) against the new namespace.
…spatcher Runs the actual conductor.ai.agents AgentRuntime + agent reasoning + tool-calling against the local server, with the Conductor dispatcher replaced by canned fixtures so it never touches ah5r-prod. Validation is deterministic (CLAUDE.md): asserts every dispatched command is read-only and the runtime returned output — does NOT LLM-judge the hypothesis. Verified live: agent reads the incident first, then dispatches only read-only commands (GET_CLUSTER_METRICS, GET_PODS_DATA, GET_POD_EVENTS, PULL_LOGS x2), follows the Redis->decider-queue->logs playbook, and emits the Issue/Findings/Root-cause/Next-step summary. Proves the SDK migration runs end-to-end and the read-only guard holds at runtime.
…DOMAIN_REACHABILITY playbook Two-step logic for the 'domain X is down' (reachability/502) alert: first rule out Conductor itself (server pod crashloop/OOM via pods/events/logs); if the server is healthy, treat the 502 as the network/ingress layer. Encodes the ingress-nginx stale-endpoint failure mode (pod restart -> new IP -> a controller replica keeps routing to the dead IP, inconsistent across replicas), with graceful degradation if the ingress namespace isn't visible to the read-only tools. Fixes the prior blind spot where a provisioned-LB reachability alert was reflexively called 'external'. No coverage-test change: 'domain X is down' anchor preserved.
…CHRONIC The agent triaged every alert as a fresh incident, missing that a chronic flapper (this one fired on ~30% of recent one-staging health-checks) is a standing capacity problem, not a page. Adds get_alert_recurrence: one search filtered by the unique clusterId UUID (NOT the fuzzy cluster name), classified locally by a pure, retention-aware summarize_recurrence(). Instructions now run the recurrence check early and lead the summary with NEW/RECURRING + the retention caveat (true onset may predate the search window -> Slack/Prometheus). Tests: pure classifier, validity proven by mutation; ISO-8601 start_time parsing locked (caught live).
The server returns agent output as {result, finishReason, context,
rejectionReason}; the Slack reply path concatenated that dict into the
message header and crashed (TypeError) at the last step of the live
e2e run. Extract the text via runtime_compat.summary_text (accepts both
the dict shape and the older plain-string output) in both the poller
and the CLI triage path. Repro test: DictOutputRuntime in test_poller,
failed with the exact live TypeError before the fix.
Live e2e showed 'I have all the evidence needed. Let me compile...' leaking into the Slack post; the final message is posted verbatim.
The instruction-only fix didn't hold — the second live run still leaked 'I have all the data I need...' into the Slack reply. Enforce the output contract in summary_text(): drop everything before the mandated *Issue*: header. Test extended with the real leaked-preamble shape; failed before.
…r reported NEW Live miss (2026-07-22): agent passed signature 'Pod Failed' (pod id correctly stripped per instructions) but the reason text is 'Pod orkes-agent-…-shldc Failed' — substring match found nothing, so a 100/100-runs chronic flapper was reported first-seen/NEW. Match every significant signature word as a token in any position; drop numeric tokens on both sides (percentages vary per firing). Live re-check now yields RECURRING/CHRONIC matched=100/100.
The digest channel posts one message per (cluster, alert-type) incident and edits it in place with an occurrence counter — polling it gives flapper dedup for free. The headline text carries no execution URL; the original alert is quoted inside a section block. Flatten text + block mrkdwn (message_text) before parsing, so both raw-channel and digest messages parse with the same parser, and the occurrence count reaches the triage prompt. Fixture is the live At-Bay HEAP_HIGH digest.
…scorable report Production gate: human-scored hypotheses over real alerts. eval_select dedupes the flapper-dominated stream to unique incidents (cluster + number-stripped token set, same normalization as recurrence matching; poll-timeout noise excluded). scripts/eval_batch.py replays each through the agent (one shared runtime) and writes a markdown report with a Useful/Partly/Wrong score line per incident. Reports are generated output and are not committed.
…ifests Two-container pod: agentspan/server:latest sidecar (ANTHROPIC_API_KEY from secret at boot — the known cold-start requirement) + poller image (python:3.11-slim, SDK installed from sdk/python). Entrypoint waits for the sidecar's /health so the SDK never tries to auto-install a CLI in the container. Digest channel is the default source; Recreate strategy keeps the dedup state single-writer (PVC); kill switch = scale to 0. Image builds and imports verified (11 tools registered).
… not -Xmx bumps Team guidance from Manan: raising the heap ceiling by default masks leaks. The playbook now mandates heap dump -> dominant retainers (MAT) -> map to recently deployed changes; rolling restart only as short-term relief; limit increase only after the dump justifies it.
…P_DUMP Team decision: for heap alerts the agent should not tell the engineer to capture a dump — it dispatches ah5r-prod's download_heap_dump on the single highest-heap pod (once per incident; jmap is stop-the-world, so the tool and playbook both forbid multi-pod dumps and non-memory use) and reports the stored dump paths for MAT + recent-changes analysis. DOWNLOAD_HEAP_DUMP moved off the banned list with the rationale recorded in the guard test. Dispatch contract covered by deterministic tests.
SLACK_ALERT_CHANNEL now accepts a comma-separated list; dedup state is kept per channel (ts values are channel-scoped in Slack), the legacy single-channel state shape migrates onto the first configured channel, and a failing channel cannot starve the others. k8s manifest watches both the raw alert channel and the aggregator digest channel.
…or CPU triage download_thread_dump(execution_id, pod) dispatches ah5r-prod's download_thread_dump workflow (jstack — cheap, near-zero pause) and returns the stored dump paths. Playbook CPU EVIDENCE RULE: when a CPU alert's cause isn't visible in logs, dump the hottest pod's threads and include the paths in the summary. DOWNLOAD_THREAD_DUMP moved off the banned list alongside DOWNLOAD_HEAP_DUMP with rationale in the guard test; DOWNLOAD_ALL_POD_LOGS stays banned.
…e it If the draft next step tells the engineer to check/count/verify something the agent's own read-only tools can answer, the agent must do it and move the answer into Findings; the final next step may contain only actions it cannot take (remediation, offline analysis, business decisions). When a check is impossible read-only, say exactly why instead of delegating it.
…window Live: the raw channel fired the same shailesh-test-gcp TIMED_OUT alert 4x in <1h (fresh execution id each firing) and each got a full LLM triage + thread posts. alert_signature() = sorted word tokens with URLs stripped and number/hex tokens dropped — stable across firings of one incident, distinct across clusters/types. Within ONCALL_SIGNATURE_COOLDOWN (default 3600s) a repeated signature is marked processed but not re-triaged and posts nothing; suppression is cross-channel, so an incident seen in both raw and digest channels is triaged once.
…fail playbook run_kubectl_read dispatches KUBECTL_UNRESTRICTED behind a deterministic allowlist guard (kubectl_guard: get/describe/logs/top/events/explain/ auth can-i/rollout history|status; shell metacharacters rejected) — same philosophy as sql_guard, validated live on orkes-wvuf-prod (namespaced reads work; the agent SA is namespace-scoped so -A is Forbidden, documented). Playbook: TIMED_OUT/lost-telemetry alerts now fast-fail to AGENT_DOWN after one hung probe — every tool executes through the in-cluster agent, so when the agent is down there is no read-only path (kubectl_unrestricted included: RunKubectlWorker runs in orkes-saas-agent) and a human with kubectl/cloud-API access is required.
…ture Live: one-staging fired the same CPU-100% alert twice in 10 min naming a different conductor pod each time (…-65pkn vs …-pv6m5) — the 5-char k8s pod suffix survived the hex-only filter, so the cooldown saw two incidents. Drop every mixed digit+letter token (identifiers by nature: uuid/hex fragments, ReplicaSet hashes, pod suffixes); subsumes the old hexish rule.
…t wording Live (orkes-prod): the same CPU condition fired as 'following issue' naming pod …-q9h9d, then as 'following 2 issues' adding pod …-pvxjg — an all-letter k8s suffix the digit+letter rule misses, plus plural boilerplate. k8s suffixes are vowel-free by design, so drop short vowel-less tokens; normalize issues->issue. Repro test from the real message pair failed before.
…g triage once Live: the raw channel's TIMED_OUT message and the digest channel's aggregated message referenced the same execution but tokenize differently, so signature suppression missed the pair and the same execution was triaged twice. An execution id, once triaged, is never triaged again (state['executions'], newest-500 cap); signature cooldown still handles fresh executions of a flapping incident.
Live failure (2026-07-23, twice): a transient network blip left
conductor-python's shared httpx client with a dead socket ('Bad file
descriptor') and a poisoned auth token; every retry reused the broken
client, tools failed indefinitely, and the sequential poll loop wedged
for ~4h. ConductorDispatcher now routes every client call through
_call(): on failure it rebuilds the workflow client (fresh pool + fresh
token exchange) and retries once. Injectable client_factory for tests.
…runs Live: the arm was set in memory before a multi-minute triage but only saved after it; an exception escaping mid-triage (failed Slack post) is absorbed by the per-channel guard and silently discarded the arm — the state file stayed 30+ min stale while duplicates of orkes-prod and zweorksksp001 re-triaged. Save immediately after arming; the post-triage save still records processed/last_ts. Crash-mid-triage now trades a duplicate triage for a possible dangling 'starting' marker (visible, rare, cheaper).
…on, no queue-number-only attributions Manan's review: CPU alerts were converging on 'decider backlog' with empty ERROR greps as the only log step — sweeper churn logs at INFO, so the grep found nothing and the agent inferred causation from the queue size. Now mandatory on CPU alerts: unfiltered 300-line tail of the hottest pod with dominant-pattern-by-volume cited as evidence, INFO-marker greps (sweeper/decider/timed-out/S3-abort/broken-pipe), then thread dump; queue-number-only attribution is forbidden — the summary must cite log volume or state the logs did not confirm.
Manan's review: 'fired 11 of the last 100 health-checks (11%)' counts against ALL runs — most of which pass — and buries the ratio on-call actually needs: of the checks that FAILED, how many were this alert? RecurrenceReport now carries failing_count and fraction_of_failing, and the RECURRING summary reads 'N of the M failing checks (X% of failures)' alongside the window count.
…ing them Manan's review: every repeat firing of a known incident burned a full LLM triage. Now the first firing runs the full investigation and its *Likely root cause* is remembered per signature (state['incidents']); repeat firings within ONCALL_FULL_TRIAGE_INTERVAL (default 6h) get a deterministic in-thread update built from memory — prior diagnosis, firing count, span — with zero LLM tokens. A full re-investigation runs after the interval WITH the prior diagnosis in the prompt (verify + report delta, not rediscover). Signature changes (severity escalation, new issue set) bypass memory by design. Chronic clusters drop from ~24 to ~4 LLM triages/day.
…nt, one memory Live: endpoint-dev's memory was seeded from the raw-channel form, then the digest form of the same incident signed differently (headline + occurrence tokens) and bought a duplicate full triage. signable_text() extracts the blockquoted original alert from digest messages (raw messages pass through), so raw and digest forms share a signature — cross-channel dedup and incident memory now cover both.
…ent top-level Live 2026-07-27: twilio-non-prod-us paged 12 times (24 agent pods Failed) and both triages posted an invented root cause — "the cluster has been deregistered from the Orkes control plane" — because get_context saw clusterId AND clusterName as null and the model reasoned from the nulls. Twilio's health_check schedulers pass only organizationId top-level and put the cluster under agentHandlerRequest.clusterName; Ocean's pass clusterName/clusterId top-level, which is why this never surfaced. clusterName is load-bearing: dispatch builds the agent routing domain from it, so it became "<org>#-#None", prepare_agent_handler FAILED, and every read-only tool returned nothing. Verified against the live execution — with the fallback, prepare_agent_handler completes and get_pods_data / kubectl reads return real cluster data. clusterId stays unresolved for these clusters (it appears nowhere in the execution, all 28 tasks scanned), so get_alert_recurrence still reports no_cluster_id and GET_CLUSTER_METRICS times out for them — the incident's own clusterData covers the metrics. Fixing that belongs in the scheduler definitions.
…26-07-30 outage - never diagnose from a single execution (compare prior runs; invariant signature = persistent, not transient/GC) - Conductor-has-failed + healthy pods -> probe serving path (:5000 UI front-end vs :8080 API), not the JVM - capture evidence (events, kill -3 stacks, live ingress logs) before recommending restarts; restarts destroyed evidence and fixed nothing - repeated identical failure after a restart = retry-storm re-wedge: recommend load-shed before any further restart - always surface FailedScheduling / zero-headroom when pods churn
…pped the loop runtime.run() blocks, so a triage that never returns takes the whole poll loop with it. Seen live: iteration 3's pull_pod_logs and get_cluster_metrics hit responseTimeout: 3600, went TIMED_OUT -> back to SCHEDULED, and nothing polled them again (polldata lastPollTime froze at Sat 10:30). The DO_WHILE never closed, so runtime.run() never returned, so the loop sat on one incident from Sat 01:25 to Mon 12:48 — 59h, zero alerts triaged, process up the whole time. Terminating the workflow by hand released it instantly, which is the proof the loop was parked in that one call. The SDK's own `timeout` is not a wall-clock bound — _poll_status_until_ complete does `elapsed += 1` per iteration while each iteration also makes a network call, so under the connection churn in the log it drifts arbitrarily far behind real time. Its 30000s default should have fired Sat ~09:45 and didn't. So the authoritative deadline has to live here: run the agent on a daemon thread, join with a wall-clock timeout, and abandon it if it overruns. The bound is passed to runtime.run() as well so the orphan thread eventually exits on its own. An abandoned triage posts an ⌛ in-thread and the loop moves on. The dedup arm is already persisted before triage starts, so the incident is not immediately re-triaged. Tests: HangingRuntime blocks in run(); assert the loop returns on the deadline rather than on release, that the next incident still triages, and that the bound reaches the runtime. Validated by reverting the guard — all three fail and the suite goes 2.2s -> 60s.
runtime.run() returns normally when the execution ends FAILED or TERMINATED —
only .status says otherwise. Nothing checked it, so summary_text() fell
through to str(output) and the loop posted the raw result dict into the
thread and stored it in incident memory as that incident's root cause.
Found while clearing the 59h wedge: terminating the stuck execution by hand
made the loop post
{'result': None, 'finishReason': 'TOOL_CALLS', 'context': {}, ...}
to the westpac-mesh-prod thread. Two older entries with the same shape were
already sitting in the live state file, so this has been happening quietly
for a while. A poisoned entry outlives the incident: it is replayed verbatim
as the "PRIOR DIAGNOSIS" block on every later firing of that signature, so
the agent re-investigates against garbage.
Treat non-COMPLETED as the failure it is — the thread gets ":warning: Triage
failed: agent execution TERMINATED: <reason>" and incident memory is left
untouched.
Three multi-hour AuditBoard outages, same cause, re-derived from scratch each
time: auditboard-prod 2026-07-30 (8h29m), auditboard-postprd 2026-08-03 (8h20m)
and 2026-08-05. Root cause is orkes-conductor's own PR #3796 / CCOR-13223: a
decider VIRTUAL thread borrows a Redis connection, enters a synchronized monitor
inside commons-pool2 GenericObjectPool.create(), and on JDK 21 pins its carrier
while holding the pool's ReentrantLock. It is never rescheduled, the lock is
never released, and every thread needing Redis queues forever.
It took ~8h per incident because the evidence actively misleads:
- CPU sits at ~0.1% on every pod, which reads as "no load, cluster is fine".
Parked threads burn nothing; low CPU is CONFIRMATORY.
- all pods Running/Ready with 0 restarts (there is no readiness probe, so
Ready is a latch set at boot and never re-evaluated).
- jstack shows ~230 threads waiting on the pool lock and NO owner anywhere.
Not a paradox: AQS guarantees an owner, but jstack does not attribute locks
held by virtual threads. The owner is invisible in every dump format, which
is why years of thread dumps produced nothing.
- the health_check's own conductor task is TIMED_OUT with "Task poll timed
out" while the agent half succeeds — the check is starved, not failing.
- pod logs stop dead ~90s before the first failed check and never resume.
The playbook now carries the fingerprint, the thread-dump confirmation (incl.
why the lock looks ownerless), the version gate (#2943 introduced it 2025-09-04,
#3796 fixed it, first release v5.5.0-rc1 — 5.2.x/5.3.x/5.4.x are exposed with no
feature flag), and the three wrong turns: do not call it GC/heap/load, do not
blame Kubernetes or ingress (`curl localhost:8080/health` from inside a wedged
pod disproves that in one line — it never leaves the netns), and restart is a
remedy while the fix is the upgrade. Capture the dump BEFORE restarting.
Tests: deterministic substring guards, no LLM, matching the existing
test_playbook_coverage.py style. Validated by deleting the block — all 5 fail.
…counts Every CPU/heap alert came back as some variant of "sweeper churn over N stale RUNNING workflows" — ~20 of them across 8 clusters in 8 days. On the AuditBoard clusters that answer was simply wrong: the JVM was not churning, it was WEDGED on a lock, and the backlog was a symptom. Two causes, both ours: 1. The agent could not see a single stack frame. download_thread_dump uploads to S3 and returns only `paths`; its docstring calls it "the go-to evidence... the dump names the busy threads", but the model gets back a filename. So it was structurally forced to reason from queue COUNTS. 2. The playbook told it what to find: "for CPU look for a hot loop / tight retry / sweeper churn", and "CPU saturation usually logs at INFO ... sweeper churn". It went looking for sweeper churn and found sweeper churn, every time. Confirmation bias encoded in the prompt. thread_summary.py turns a dump into facts the model can reason over: thread states, what the RUNNABLE threads are actually executing (only RUNNABLE threads can burn CPU), lock pileups, and whether a contended lock has NO owner. It emits a deterministic `verdict` the model is told not to contradict. Run against the real 2026-08-03 dump it says, with no prior knowledge: WEDGE, not load: 223 threads parked on one lock (0x00000006ca7c0c30) and ZERO application threads are RUNNABLE. CPU is near-zero because parked threads consume none. No thread owns that lock in this dump — jstack does not attribute locks held by VIRTUAL threads, so the owner is unreportable. Subtlety worth keeping: sun.nio.ch.EPoll.wait is RUNNABLE but parked in the kernel. Counting it as app work flipped the verdict to "CPU is being spent in app code" on a JVM doing nothing at all — the exact misread. Pinned by a test. get_thread_summary() feeds it, pulling only the RUNNABLE slice and lock addresses via targeted greps so it fits the agent-handler's ~8KB result cap instead of hauling 2.4MB. Playbook now: threads FIRST on every cpu/heap alert; "sweeper churn" and "decider backlog" are BANNED without thread evidence; no evidence -> say "cause not established" and stop. And when the sweeper genuinely IS hot, don't stop at a total — bucket via the workflow_running metric or GROUP BY workflow_name, sample 2-3 real workflow ids from the sweeper log lines, and establish WHY they never terminate. Whether the sweeps are necessary is the actionable finding. Tests: 5 parser tests on synthetic dumps + 4 playbook guards. Both mutation- validated (first attempt at the playbook mutation was a no-op — anchors were inverted so nothing was removed; redone by line range, 3 fail as expected).
…ve pod
Validated end-to-end for the first time and it returned zeros. Three real bugs,
none of which unit tests could have caught:
1. `.replace("_", " ")` — a hack to inject spaces into grep patterns, since the
remote side splits the command on whitespace — also rewrote the dump FILENAME:
/tmp/oncall_threads.txt -> "/tmp/oncall threads.txt". It wrote one file and
grepped another. Fixed by using single-token grep patterns and a hyphenated path.
2. Grepping for RUNNABLE found nothing: `Thread.dump_to_file -format=plain` carries
NO `Thread.State:` line, and no lock addresses either — just `#id "name"` headers
and frames. jstack has both but only writes to stdout, which the handler truncates
at ~8KB, and `sh -c` redirection does not survive this channel (verified), so
jstack cannot reach a file.
3. A loop variable named `top` shadowed the `top: int = 6` parameter later passed to
most_common(top). Latent: the jstack-format fixtures never enter that branch.
Since the dump is ~700KB against an ~8KB result cap, per-thread retrieval is
impossible here. The tool now reports COUNTERS — total/parked/active plus a grep -c
per diagnostic frame (redis pool borrow, lock acquire, sweeper, decider, jdbc,
socket read). That is enough to separate "wedged" from "genuinely busy", which is
the question a CPU alert asks. Live check on a healthy pod: 521 threads, 473 parked,
48 active, sweeper=1 — and it correctly declines to blame a backlog.
grep -c exits 1 on zero matches and the handler returns empty for a non-zero exit;
that is a legitimate 0, not a failure (total_threads proves the plumbing works).
summarize_thread_dump() is retained for offline analysis of a full dump pulled off a
pod — it is what produced the auditboard RCA — and now parses both jstack and plain
formats, inferring state from the top frame when the format omits it.
Known gap: lock OWNERSHIP still needs `jcmd <pid> Thread.print -l` run by hand; no
format that fits this channel carries it.
…sweeps
"N RUNNING workflows are saturating CPU" was never actionable, because a backlog
count cannot say whether the sweeper is doing real work or spinning. The codebase
answers it exactly.
OrkesWorkflowSweeper.sweep() drains the decider queue in only two places — workflow
null, or status terminal. Everything else stays queued and is swept again forever.
And the wasted case is already instrumented: after decide() the sweeper compares the
task list before/after, and when nothing changed it logs
"Going to repair the task {id} / {ref}, with status {st}, workflow = {wf},
timeout = {n}, now-wait = {n}"
and increments `queue_message_repushed` (tagged taskType, namespace). So every one of
those lines is a sweep that advanced nothing and merely re-pushed a queue message —
and the line names the task, its status, the owning workflow and how long it waited.
analyze_sweeper_waste() counts those against "Running sweeper for workflow" from ONE
grepped window (separate greps return different slices, which would make the ratio
meaningless) and reports waste_ratio, the dominant stuck task ref, workflows re-swept
within the window, and the longest-waiting tasks. Verdict distinguishes three cases:
no sweeper activity at all (do NOT blame the backlog), lock-acquire failures with no
repairs (threads contending — the wedge signature, check thread state), and genuine
waste (name the stuck task ref; fix the workflows, not the CPU limit).
Verified against a live cluster: regex matches the real WARN line, e.g. a `wait_basic`
task SCHEDULED on workflow adc5cc00-fdb8-11ef — an old workflow whose WAIT task never
advances, which is precisely the class this is meant to surface. Current recent window
on that cluster is clean (82 sweeps, 0 repairs), so the tool carries a caveat to widen
the window before concluding the work is genuine.
Note SQL cannot substitute here: workflow_archive is EMPTY on that cluster (RUNNING
state is Redis-only) and the in-cluster Conductor API returns 401 from inside the pod,
so the sweeper's own log line is the only cheap source of per-workflow truth.
Tests: 6 cases on synthetic logs, mutation-validated by disabling repair detection —
4 fail as expected. 143 total green.
…uote a bare total
"16,555 RUNNING workflows are saturating CPU" is not actionable. "Two definitions
are 93% of the backlog and both start on 17 March" is. get_running_backlog_buckets
turns the first into the second, and the two live clusters show why it matters —
same alert, opposite responses:
auditboard-postprd 1,651 RUNNING | oldest 2024-12-17 | 93.3% FAILED
-> rot: 20-month-old workflows, no dominant definition, cluster is unhealthy
one-staging 16,500 RUNNING | oldest 2026-03-16 | 0.66% FAILED
-> leak: COM_Account_Creation_Request (49%) + COM_Notification_Email_Workflow
(44%) = 93%, both starting mid-March. Everything else on that cluster is fine.
Shape of the data, learned against both:
- the parent workflow_archive returns NOTHING; it does not route. Query shards.
- fleets vary wildly: 2 shards vs 301 holding 150.7M rows.
- sharding is by HASH, not time — every one-staging shard held 510K rows within
0.3% of the others, so there is no "latest" shard and one shard is a sound 1/N
sample. Small fleets are read exactly; large ones sampled and flagged ESTIMATED
so a scaled figure is never presented as measured.
- json_data is EMPTY for RUNNING rows (live state is Redis-only until terminal),
so this names WHICH definitions and how old, not which task is stuck. Pair with
analyze_sweeper_waste for the task-level reason.
Bug found by running it against both clusters rather than one: listing all 301
shards exceeds the agent-handler's ~8KB result cap and comes back EMPTY, so
one-staging silently reported nothing. Shard discovery is now a LIMIT 4 listing
plus a separate count(*) for the true fleet size — without that split a 301-shard
fleet scales by 4 and under-reports 75x. Pinned by a regression test.
Failure rate is surfaced separately: 93.3% FAILED on auditboard is its own finding
and must not be buried under the backlog story.
7 tests on the real numbers from both clusters. 150 total green.
… being told Both multi-hour outages this month were fixed upstream WEEKS before they happened. Nobody connected the running image tag to the merged PR, so each was re-diagnosed from scratch over ~8 hours: auditboard-postprd 5.2.97 JDK-21 carrier pin fixed by #3796, 2026-07-14 one-staging 5.4.3 WAIT/HUMAN re-sweep fixed by #3754 + #3775, 2026-07 check_known_issues() closes that loop. It reads the conductor image tag off the deployment and matches it against a table of known issues plus the symptoms the agent has already measured — no human recognition required. The signals are ones the code itself emits: `queue_message_repushed` is tagged with taskType, so WAIT/HUMAN/EVENT being re-pushed at a high waste_ratio on a pre-5.5.0 cluster IS the #3754/#3775 fingerprint; decider-executor-vthread names plus GenericObjectPool.borrowObject frames are the #3796 one. Verified live: 5.4.3 + WAIT + waste 0.95 -> wait-human-resweep-cadence, cites #3775. 5.2.97 + vthread names + pool frames -> jdk21-carrier-pin, cites #3796. Correct issue for each cluster, unaided. Version parsing fails CLOSED. `fix-archive-feature-async-reconcile-latest` is really deployed in this fleet; parsing that as a version would tell a vulnerable cluster it is safe, so a non-semver tag reports UNKNOWN and matches nothing. Mutation-validated by making the parser fail open — the guard test fails as expected. Being on an old version is NOT a diagnosis: exposure without matching symptoms returns "do not assume it is one of these". That distinction is the difference between a version check and a real triage. Also fixed a cosmetic leak of the raw tuple "(5, 5, 0)" into the Slack-facing verdict. 8 new tests on real fleet versions; 159 green.
…opped polling Twice the process stayed alive while individual SDK worker threads silently died. Observed 2026-08-10: get_incident_details last poll 08 Aug 18:01 (2.1 days ago) get_alert_recurrence last poll 08 Aug 18:21 (2.1 days ago) pull_pod_logs last poll 10 Aug 21:12 (0 min ago) <- same process, fine Every triage then ran the LLM, forked those two tools onto queues nobody polled, and the JOIN sat until the 1800s deadline killed it. Task-level timing on a dead run: LLM_CHAT_COMPLETE 0.1 min COMPLETED get_incident_details - CANCELED (never started) get_alert_recurrence - CANCELED (never started) JOIN 30.8 min CANCELED DO_WHILE 30.9 min CANCELED 26+ consecutive dead triages over 2.1 days. Not slow — never executed. A guard inside the tool cannot help, because the tool never runs. So this checks polldata BEFORE starting: if a required worker is stale or absent, skip the triage, post exactly why and how stale, and move on. An instant accurate finding instead of a silent 30-minute timeout. Fails OPEN by design: a transient probe error reports age 0, never "dead". Blocking triage on a flaky HTTP call would be a worse outage than the one this guards against. "no poller registered" and "registered then died" stay distinguishable — different faults, and both differ from healthy. This is the blast radius, NOT the root cause. Why the worker threads die is still unknown and lives in conductor-python's task runner; a restart still revives them. Same failure took down the loop for 59h on 08-03 under different task types, where I noted the stall was never root-caused. It came back. Also corrects two wrong diagnoses of mine from this week, recorded so they are not repeated: the timeouts were NOT caused by the CPU tools I added (they predate them, and the rollback build timed out too), and NOT by unhealthy target clusters (that correlation was just which clusters alert most often). Task-level timing settled it. Tests: 7 unit on real observed ages + 2 end-to-end asserting the LLM is not invoked when workers are dead. Mutation-validated by disabling the preflight.
…old start The guard I added in 9659ed5 blocked ALL triages on a freshly restarted process and posted this to Slack for every alert: Triage skipped — on-call agent tool workers are not polling. get_incident_details, get_alert_recurrence last polled 23 min ago (stale beyond 120s). Wrong, and self-defeating. Workers are spawned LAZILY by runtime.run() -> _prepare_workers, so on a new process they have never polled and polldata still shows the PREVIOUS process's timestamps. The guard read that as death and blocked the very call that would have started the workers. It turned a healthy agent into a dead one. Fix: report dead only on a healthy -> stale TRANSITION observed within this process. A task type never yet seen polling is "warming_up", not dead, and the triage proceeds. That still catches the real fault (workers polled, then died — 2.1 days, 26+ dead triages) while leaving cold start alone. Tests pin the exact Slack case: 23 min stale on a fresh process must NOT block; the same 23 min AFTER a healthy reading must block. Mutation-validated by removing the exemption. 171 green. My own guard caused a worse outage than the bug it guards against. Worth remembering that a preflight which can block the happy path needs the happy path tested first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(oncall-agent): read-only Slack-triggered cluster triage agent
What this is
An Agentspan agent that triages Orkes SaaS cluster health-check alerts. It polls the Slack
alert channel (Web API, no Socket Mode), parses the
executionIdfrom the failinghealth_checkexecution URL, runs read-only agent-handler commands against the ah5r-prodConductor API to investigate, and replies in-thread with a root-cause hypothesis.
It is strictly read-only and advisory — it never takes a remediating action. SQL is gated by
a deterministic SELECT-only guard (
sql_guard.py), not by trusting the model. The agent readsorganizationId/clusterName/cloudEnvironmentTagoff the failing execution, so the LLMonly threads the
executionIdinto each tool — it cannot target the wrong cluster and no secretspass through tool args.
Done ✅
Slack ingestion — Web API poller (
conversations.history+chat.postMessage), bot tokenonly, run-once /
--loop, state-file dedup. Matchessdk/python/examples/91_slack_autofix_agent.py.Alert parsing (
alert.py) — extracts execution id + severity + org/cluster from the alert text.Conductor dispatch (
conductor_client.py) — starts agent-handler command workflows onah5r-prod via conductor-python (app key/secret), polls to completion, derives + caches cluster context.
Read-only tool set (
tools.py) —get_incident_details,get_cluster_metrics,get_infrastructure_metrics,get_pods_data,get_deployments_info,get_pod_events,get_top_output,pull_pod_logs,get_ingress_info,run_sql_select(SELECT-guarded).SQL safety guard (
sql_guard.py) — SELECT/WITH/EXPLAIN/SHOW only; rejects every mutation,multi-statement, and comment-smuggling case before it reaches the DB.
Per-alert-type triage playbook for all 17 health-check alert types (
agent.py) — symptom →evidence-to-gather → what-to-cite, matched off the
issuestext. Keeps the strong Redis /decider-queue / CPU / heap guidance and extends to component-down, pod, networking, and
self-describing alerts (table below).
Tests (deterministic, no LLM in the assertion path, per
CLAUDE.md) —test_sql_guard.py,test_alert.py,test_poller.py,test_conductor_client.py,test_tools_readonly.py(read-only safety guard, pinned to the real
AgentHandlerCommandenum names),test_tools_dispatch.py(each tool dispatches its expected agent-handler command). 45 passing.Local run path (
python -m oncall_agent.main [triage <execId>]),.env.example, README.Live end-to-end run (2026-07-22) — the FULL production path: poller read a real MAJOR
alert from a Slack channel (
collective-staging,Pod orkes-agent-deployment-* Failed,exec
d552305e-85ce-11f1), parsed it, ran the triage agent with real read-only agent-handlerdispatches to ah5r-prod (~80s, recurrence check + pods + events + logs + metrics), and posted
a correct root-cause hypothesis in-thread (stale Failed pod superseded by a 2026-07-20
rollout; cluster already self-healed). The first run caught a real bug at the last step —
the runtime returns
result.outputas a dict ({result, finishReason, ...}) and the Slackpost crashed concatenating it; fixed via
runtime_compat.summary_text()with a repro testthat failed with the exact live TypeError first. An earlier CLI-path run (2026-07-03,
triage <execId>) had validated the investigation loop.Digest-channel support (2026-07-22) — polls the alert-aggregator channel (one message per
(cluster, alert-type) incident, edited in place with an occurrence counter → flapper dedup for
free). Block-text flattening (
alert.message_text); validated live (At-Bay HEAP_HIGH triagedin-thread). Token-based recurrence matching fixed the flapper-reported-as-NEW miss.
Eval batch tooling (
scripts/eval_batch.py+eval_select.py) — dedupes the raw stream tounique incidents and replays them into a human-scorable markdown report. First run: 6/6 unique
48h incidents triaged clean (2× pod-failed, 3× CPU, 1× heap), ~90s each.
Containerization (
Dockerfile,deploy/k8s.yaml) — two-container pod (agentspan serversidecar + poller), digest channel default, single-writer state on PVC, kill switch = scale to 0.
Image build + in-container import smoke verified.
To do 🚧
kubectl apply.HITL approval gate.
The 17 health-check alert types — all now have a playbook
Source of truth:
HealthIssueenum inorkes-saas/.../worker/HealthCheckIssuesWorker.java. "Approach" = howagent.pytriages it.REDIS_CRITICAL_USAGEREDIS_HIGH_USAGECONDUCTOR_HIGH_HEAP_USAGECONDUCTOR_HIGH_CPU_USAGECONDUCTOR_ERROR_LOGS_COUNT_EXCEEDED_THRESHOLDCONDUCTOR_WARN_LOGS_COUNT_EXCEEDED_THRESHOLDCONDUCTOR_HEALTHY(failed)WORKERS_HEALTHY(failed)PROMETHEUS_NOT_RUNNINGPOD_NOT_RUNNINGPOD_RESTARTEDDNS_HEALTHY(failed)get_ingress_info— no address ⇒ LB unprovisioned; else external → infraDOMAIN_RESOLUTIONget_ingress_info→ resolve vs escalate to infraDOMAIN_REACHABILITYget_ingress_info→ reachable vs escalate to infraAUTH_STALEDOMAIN_CERTIFICATE_WILL_EXPIRERESPONSE_TIMEOn testing the playbook: the playbook is prompt text — by
CLAUDE.mdrule 1 it can't beLLM-judged in unit tests and isn't deterministically assertable, so it has no unit test by design.
What is tested deterministically: the read-only safety guard and the per-tool dispatch contract
(
get_ingress_info→GET_INGRESS_INFO, mutating commands stay unreachable). Playbook quality isvalidated in the eval step above.
Testing
Out of scope (v1)
Triage + read-only investigation only. No remediation (restart/scale/rollback).