Skip to content

failure-attribution.md

Latest commit

 

History

History
61 lines (46 loc) · 2.22 KB

failure-attribution.md

File metadata and controls

61 lines (46 loc) · 2.22 KB
Raw

Deterministic Failure Attribution

Why Rules Own the Classification

FlowLens must produce the same category for the same redacted evidence, work without a model key, remain unit-testable, and avoid prompt-injection influence. Therefore the rule engine owns primary_cause. The optional LLM explainer may rewrite the diagnosis into natural language, but it cannot change the category, confidence, evidence, or suggested action.

Priority

Rules are evaluated in this order:

  1. CHECKPOINT_ROLLBACK or USER_INTERRUPTED
  2. GUARDRAIL_DENIED or LOOP_DETECTED
  3. MCP_AUTH_ERROR or MCP_SESSION_ERROR
  4. MEMORY_ERROR
  5. SUBAGENT_TIMEOUT or SUBAGENT_ERROR
  6. TOOL_ERROR or MCP_TOOL_ERROR
  7. MODEL_ERROR
  8. UNKNOWN_RUNTIME_ERROR
  9. NONE

The first match is primary. Later matches remain visible as contributing_causes. A successful Run is classified NONE even if Metrics retain recovered intermediate errors.

Output Contract

primary_cause, contributing_causes, phase, confidence,
root_cause, evidence, suggested_action

Evidence references event seq, event_type, summary, span_id, and parent_span_id. This lets the dashboard open the exact redacted Timeline item. FlowLens does not infer causality from the final exception alone: a Tool error may be downstream of an MCP authentication failure, while a run error is often only the terminal symptom.

Multi-cause Example

If a run contains a guardrail denial, a Subagent timeout, and a terminal model error, GUARDRAIL_DENIED is primary. Subagent and model failures are contributing causes with their own evidence. This preserves the full failure surface without making the primary result ambiguous.

Iterating Rules

Every new or changed rule requires:

  • a synthetic or runtime-derived privacy-safe event fixture;
  • a test for the intended primary cause;
  • a simultaneous-cause test proving priority and contributing causes;
  • an explicit confidence and remediation message;
  • a Replay update when the category is part of the public demo taxonomy.

Unknown evidence is not silently forced into a known category. A failed run with no specific match returns UNKNOWN_RUNTIME_ERROR at low confidence so future rules can be added from observed, redacted fixtures.