Skip to content

test: add cross-framework governed failure conformance - #189

Merged
brunovicco merged 4 commits into
mainfrom
test/phase-54-cross-framework-executor-failure-conformance
Sep 6, 2026
Merged

brunovicco merged 4 commits into
mainfrom
test/phase-54-cross-framework-executor-failure-conformance

Conversation

@brunovicco

@brunovicco brunovicco commented Sep 6, 2026 •

Copy link
Copy Markdown
Owner

Phase 54 — cross-framework governed executor failure conformance

This phase extends the existing framework conformance contract with one post-executor failure scenario and required no production-code change.

Concrete gap

The previous conformance suite covered successful execution and pre-executor security states, but not the Phase 49 GovernedActionExecutionError / ActionExecutionFailureEvidence path. Phase 53 demonstrated that a framework could satisfy the existing conformance suite while still replacing application-owned failure provenance.

Design

  • adds one deterministic failing executor inside the existing cross-framework conformance test;
  • executes the same exact-allow action directly through GovernedActionRuntime and through LangGraph, CrewAI, LlamaIndex and Agno;
  • requires GovernedActionExecutionError from every framework;
  • requires framework ActionExecutionFailureEvidence to equal the application baseline exactly;
  • requires exactly one executor invocation;
  • requires exact action/context/authorization plus execution_attempted=true, failure_reason=executor_error, and external_side_effect_state=unknown;
  • requires raw executor text to stay out of structured evidence and governed error text while remaining only as the local Python exception cause.

Result

All four framework adapters already conform after the Phase 53 Agno fix. No LangGraph, CrewAI, LlamaIndex, Agno, application-runtime, authorization, approval, or policy code had to change.

The resulting contract is:

authorized executor failure -> exactly one attempt + baseline-equivalent application-owned failure evidence across frameworks

and:

framework conformance includes failure provenance, not only successful/pre-executor states

CI evidence

Functional CI #486 passed on b18528ea6dbf98f813cdb583134afbc33002ce6a, proving the new failure scenario across the direct application baseline and all four frameworks before documentation.

Final clean CI #489 passed on d6a22ba4dd6a425a7014e3e370470b93db6d5524 after documentation and removal of the temporary docs workflow.

Final quality evidence:

  • Ruff lint: all checks passed;
  • Ruff format: 298 files already formatted;
  • Pyright: 0 errors, 0 warnings, 0 informations;
  • pytest: 544/544 passed;
  • coverage: 95.49%;
  • Bandit: no issues identified;
  • pip-audit: no known vulnerabilities, with the repository's existing 5 explicit ignores;
  • MCP v2 compatibility + STDIO: green;
  • governed MCP v2 compatibility + STDIO: green;
  • authenticated governed MCP v2 compatibility + STDIO: green;
  • OpenTelemetry analysis observation check: green.

Documentation

  • CHANGELOG.md records the cross-framework governed-failure conformance evidence;
  • docs/security/GOVERNED_AGENT_ACTIONS.md adds authorized executor failure to the cross-framework matrix and documents baseline-equivalent ActionExecutionFailureEvidence, exactly one attempt, raw-error exclusion, and external_side_effect_state=unknown.

Final scope

The final diff contains exactly three files:

  • tests/integration/test_governed_action_framework_conformance.py;
  • CHANGELOG.md;
  • docs/security/GOVERNED_AGENT_ACTIONS.md.

No production code, framework adapter, application runtime, dependency, workflow helper, historical artifact, or published release remains changed. This phase adds no retry, idempotency, rollback, compensation, two-phase commit, or distributed transaction semantics.

Closes #188

@brunovicco
brunovicco marked this pull request as ready for review September 6, 2026 04:56
@brunovicco
brunovicco merged commit 00db8d2 into main Sep 6, 2026
1 check passed
@brunovicco
brunovicco deleted the test/phase-54-cross-framework-executor-failure-conformance branch September 6, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 54: add cross-framework governed executor failure conformance

1 participant