Skip to content

feat(sight): agent crash full-stack observability#661

Open
jfeng18 wants to merge 1 commit into
alibaba:mainfrom
jfeng18:feat/lineage-tree-and-agent-mode
Open

feat(sight): agent crash full-stack observability#661
jfeng18 wants to merge 1 commit into
alibaba:mainfrom
jfeng18:feat/lineage-tree-and-agent-mode

Conversation

@jfeng18

@jfeng18 jfeng18 commented May 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Agent 进程层级全栈可观测:血缘树 + 精准 crash 检测 + blast radius 量化 + token 按层级归因 + CLI crash report + token --by-type 聚合查询。

agent 崩了,系统知道崩没崩、为什么崩、影响多大、崩的时候在干什么、哪个子进程在烧钱。

Strategic positioning

This PR is the AgentSight foundation for the OS-middle-layer recoverability and task-level observability substrate: process supervision, exit-reason capture, Agent/SubAgent/Tool lineage, crash blast-radius reporting, and token attribution by process type.

It deliberately does not implement the scheduling/control-plane pieces from the broader Agent Infra roadmap: no cgroup QoS control, no LSM enforcement, no transparent MaaS cache proxy, and no closed-loop task recovery orchestration. Those can build on this observability substrate later.

Problem

  1. crash 检测靠猜:用 pending LLM calls 判断 crash,SIGSEGV 在两次调用间 → 漏判,正常退出 SSE 没收完 → 误判
  2. 子进程完全不可见:SubAgent/Tool 崩溃静默丢弃(scanner 只认 root agent)
  3. token 无层级归因:不知道哪个子 agent 在烧钱,无法按 Agent/SubAgent/Tool 分层分析
  4. 启动时已有 agent 不在 lineage tree:scanner 初始扫描发现的 agent 没有 lineage 节点

Solution

改动 效果
BPF 采集 procmon.h 加 exit_code,BPF_CORE_READ task_struct crash/normal 从内核事实判定
血缘树 lineage/mod.rs:pid→ppid 树 + Agent/SubAgent/Tool 分类 + root_agent_ancestor 进程层级结构化
crash 判定 classify_exit() 纯函数替代 pending-calls 猜测 假阴性/假阳性修复
blast radius ProcessType → total_session_loss/partial/recoverable 量化崩溃影响,severity 按类型分级
SubAgent/Tool crash Exit handler 扩展:lineage 分类的子进程 signal crash 也检测 子进程不再静默
token 归因 LLMCall/PendingCallInfo 加 process_type,SQLite+Logtail 全链路 按层级分析消耗
token --by-type GenAISqliteStore 聚合查询 + CLI agentsight token --by-type 一行看哪层在烧钱
启动补种 构造函数末尾 insert_and_classify existing_agents 已有 agent 也在 lineage tree
CLI interruption get 结构化 crash report(信号/进程树/blast radius/process type) 一眼看懂崩溃现场
bug fix pid_agent_name_cache PID 复用清理 + proctrace exit_code TODO 修复 数据准确性
race fix proctrace Exit 不再删 lineage node,defer 到 procmon Exit handler 防止 crash 事件因事件排序竞态静默丢失
CI fix background.rs tmp_dir 用 CARGO_MANIFEST_DIR 替代 temp_dir 修复 llvm-cov 下 SQLITE_READONLY

Crash detection truth table

场景 exit_code 旧判定 新判定
SIGSEGV 无 pending 11 漏判 crash(Critical, blast=total_session_loss)
SIGKILL (OOM) 9 crash(仅有pending) crash(Critical)+oom
SIGTERM 优雅停止 15 crash(若有pending) normal(graceful)
exit(0)+pending SSE 0 误判(crash) no crash event (pending lifecycle remains separate)
Tool SIGSEGV 11 静默丢弃 crash(Medium, blast=recoverable)
SubAgent SIGSEGV 11 静默丢弃 crash(High, blast=partial)
子进程 exit(1) 跟随父死 256 N/A 不产出(非 signal,正常清理)

CLI examples

$ agentsight interruption get <id>

  --- Crash Report ---
  Cause:        SIGSEGV (signal 11, coredump: true)
  OOM Killed:   no
  Process Type: agent
  Blast Radius: total_session_loss
  Children at Exit: [5678]

  Process Tree at Crash:
    `-- cosh [1234] (agent)
        `-- node [5678] (tool)

$ agentsight token --by-type --hours 24

Token Usage by Process Type (last 24h)

TYPE           CALLS        INPUT       OUTPUT        TOTAL
------------------------------------------------------------
agent              15       12,345        8,901       21,246
sub_agent           3        2,100        1,500        3,600
tool                1          200          100          300
------------------------------------------------------------
TOTAL                                                25,146

Verification

Check Status
cargo fmt/clippy/test (1031 tests) pass
check-arch-boundaries.py pass
BPF verifier load (kernel 6.6.102+) E2E pass
SIGSEGV root agent → crash + blast_radius=total_session_loss E2E pass
SIGTERM → no crash event E2E pass
子进程 exit(1) → no crash (噪音修复) E2E pass
process_type in genai_events SQLite E2E pass
CLI crash report 含 Process Type + Blast Radius E2E pass
token_usage_by_process_type 排除 pending 行 区分性测试
Adversarial workflow review (2 rounds, 14 agents) 0 critical/high

What is NOT changed

  • Not a cgroup/QoS scheduler: no cpu.weight, cpu.max, burst control, or dynamic resource enforcement
  • Not an LSM/sandbox policy engine: no AppArmor/SELinux/eBPF-LSM allow/deny control
  • Not a transparent MaaS cache proxy: no socket redirection or LLM response cache interception
  • Not closed-loop recovery orchestration: crash reports/checkpoint context only; resume/retry policy remains out of scope
  • health/checker.rs:serve 进程没 BPF exit_code,保持 pending-calls 逻辑(已知限制)
  • procmon pid namespace:保留 host tgid,容器 PID namespace 场景标注限制

@jfeng18 jfeng18 requested a review from chengshuyi as a code owner May 28, 2026 15:25
@github-actions github-actions Bot added the component:sight src/agentsight/ label May 28, 2026
@CLAassistant

CLAassistant commented May 28, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch from 6ccd232 to e4c79a5 Compare May 28, 2026 15:32
@jfeng18 jfeng18 changed the title feat(sight): add blood lineage tree and AGENT_MODE=1 detection feat(sight): blood lineage tree + idle-burst-idle scheduling May 28, 2026
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch from 0f14dfc to f50ad4d Compare May 29, 2026 04:00
@jfeng18 jfeng18 changed the title feat(sight): blood lineage tree + idle-burst-idle scheduling feat(sight): blood lineage tree with AGENT_MODE detection May 29, 2026
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch from f50ad4d to 3c04313 Compare June 2, 2026 06:16
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch 7 times, most recently from 6e3f294 to 4556654 Compare June 10, 2026 03:06
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch 5 times, most recently from f051d52 to 9123d4c Compare June 26, 2026 16:44
@jfeng18 jfeng18 changed the title feat(sight): blood lineage tree with AGENT_MODE detection feat(sight): process lineage tree + three consumers Jun 26, 2026
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch from d41952f to 3653d22 Compare June 27, 2026 12:45
@jfeng18 jfeng18 changed the title feat(sight): process lineage tree + three consumers feat(sight): agent crash full-stack observability Jun 27, 2026
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch 4 times, most recently from 6d992b1 to 3bf059d Compare June 27, 2026 14:14
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch 6 times, most recently from c94e018 to e1649de Compare June 30, 2026 07:32
BPF exit_code capture + classify_exit replaces pending-calls heuristic.
Lineage tree, enriched crash detail, CLI crash report, pid cache fix.
@jfeng18 jfeng18 force-pushed the feat/lineage-tree-and-agent-mode branch from e1649de to b54578c Compare June 30, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:sight src/agentsight/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants