Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/dependency-audits/2026-09-03-external-evaluator-handoff.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
title: External Evaluator Handoff Example Dependencies
last_reviewed: 2026-09-03
owner: agt-maintainers
---

# External Evaluator Handoff Example Dependencies

## Which Dependencies Changed And Why

The new `examples/external-evaluator-handoff/requirements.txt` declares two
bounded dependencies for the standalone example:

- `agent-governance-toolkit-core>=5.0.0,<6.0` supplies the existing
`DecisionBOM` model and the public `sha256_jcs` digest helper. The v5 floor is
required so the example does not teach direct use of a raw cryptographic
primitive outside the SDK boundary.
- `pytest>=8.0.0,<10.0` is used only to run the example's local regression
tests. It is not imported by the runnable example.

These dependencies are isolated to the example and do not change any AGT
package or repository-wide runtime dependency.

## Security Advisory Relevance

This change is not a security-advisory remediation and does not add a new
cryptographic implementation. Content digests are delegated to AGT's existing
public SDK helper. `pytest` is test-only, and the repository's dependency
review and vulnerability checks remain authoritative for the resolved graph.

## Breaking Change Risk Assessment

**Risk: low and example-local.** The example requires AGT core v5 because the
public digest helper is part of that supported surface. Users pinned to AGT
core v4 cannot run this example without upgrading, but no existing package,
API, policy, or runtime behavior is changed. The upper bounds keep resolution
within the currently supported major versions.

## Validation And Rollback

Validation covers the example test suite, formatting and lint checks, strict
JSON output, documentation links, and the repository dependency and
unauthorized-crypto gates. Rollback consists of removing the standalone
example and this audit record; no production data or migration is involved.
105 changes: 105 additions & 0 deletions examples/external-evaluator-handoff/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# External Evaluator Handoff

<!-- cspell:ignore SAEE -->

> Status: experimental, community-driven example

This example converts one or more AGT Decision BOMs into a deterministic,
strict-JSON request for a downstream evaluator. It demonstrates an
interface-first boundary: AGT remains the runtime governance and observation
source, while an external system may perform post-execution fitness,
adaptation, stability, or other longitudinal evaluation.

The example is deliberately offline. It does not call an external service,
change an AGT policy decision, authorize an action, mutate an audit record, or
turn an evaluation result into a governance decision.

## Why this boundary exists

Runtime governance and post-execution evaluation answer different questions:

- AGT answers whether an action was allowed, which policy applied, and what
governance signals were observed.
- A downstream evaluator may compare multiple observed decisions over time and
produce a reviewable assessment.
- That assessment is input to a separate review process. It is not permission
and does not override AGT.

The handoff builds on AGT's existing
[`DecisionBOM`](../../agent-governance-python/agent-mesh/src/agentmesh/governance/decision_bom.py)
instead of introducing a second audit or policy model.

## Prerequisites

- Python 3.11+
- No API keys or network service required at runtime

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r examples/external-evaluator-handoff/requirements.txt
```

## Run

From the repository root:

```bash
python examples/external-evaluator-handoff/external_evaluator_handoff.py
```

The script prints one synthetic request. Its shape contains:

- a content-derived `request_id`;
- source Decision BOM observations;
- only explicitly allowlisted extra fields;
- fixed read-only and zero-authority declarations.

Example boundary:

```json
{
"authority_boundary": {
"evaluation_result_is_governance_decision": false,
"execution_authorized": false,
"policy_decision_overridden": false,
"read_only": true,
"source_records_mutated": false
}
}
```

## Test

```bash
PYTHONPATH=agent-governance-python/agent-mesh/src \
pytest -q examples/external-evaluator-handoff/test_external_evaluator_handoff.py
```

The tests cover deterministic output, exact field allowlisting, source
immutability, timezone rejection, empty-input rejection, strict JSON values,
and the permanent authority boundary.

## Data and security notes

- Decision BOM fields can contain policy, context, or trace data. The exporter
therefore includes no optional fields unless their exact names are
allowlisted by the caller.
- Allowlisted values are normalized into detached strict-JSON copies. Mutating
a constructed request therefore does not mutate the source Decision BOM.
- The sample uses synthetic identifiers and values. Review tenant, privacy,
retention, and cross-border requirements before exporting real records.
- `source_completeness` describes Decision BOM reconstruction coverage. It does
not prove that an event was correct, authorized, or complete in the real
world.
- A content hash generated through AGT's public `sha256_jcs` SDK helper
identifies the canonical payload bytes before `request_id` is added, so the
identifier itself is excluded from its hash input. The hash is not a
signature, attestation, or proof of truth.

## Prior art and interoperability intent

The interface boundary was informed by SAEE's Evolution Intelligence Layer:
`https://github.com/joy7758/SAEE`. No SAEE source code, engine implementation,
or runtime dependency is included. The request is framework-neutral so other
external evaluators can consume the same observation boundary.
168 changes: 168 additions & 0 deletions examples/external-evaluator-handoff/external_evaluator_handoff.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
"""Export AGT Decision BOMs to a read-only external-evaluation request.

This example deliberately stops at the interoperability boundary. It does not
call an evaluator, mutate AGT records, or turn an evaluation result into a
governance decision.
"""

from __future__ import annotations

import json
from collections.abc import Collection, Sequence
from datetime import datetime, timezone
from typing import Any

from agentmesh.governance.approval_protocol import sha256_jcs
from agentmesh.governance.decision_bom import BOMField, BOMFieldCategory, DecisionBOM


# cspell:ignore utcoffset
SCHEMA_VERSION = "0.1"


def _utc_timestamp(value: datetime) -> str:
"""Return an RFC 3339 UTC timestamp, rejecting timezone-free values."""
if value.tzinfo is None or value.utcoffset() is None:
raise ValueError("timestamps must include a timezone")
return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")


def _export_field(field: BOMField) -> dict[str, Any]:
"""Export one explicitly allowlisted BOM field with fail-closed JSON checks."""
try:
serialized_value = json.dumps(
field.value,
allow_nan=False,
ensure_ascii=False,
separators=(",", ":"),
sort_keys=True,
)
exported_value = json.loads(serialized_value)
except (TypeError, ValueError) as exc:
raise ValueError(f"field {field.name!r} is not strict-JSON serializable") from exc

return {
"name": field.name,
"category": field.category.value,
"value": exported_value,
"source": field.source,
"confidence": field.confidence,
"inferred": field.inferred,
}


def build_external_evaluation_request(
decisions: Sequence[DecisionBOM],
*,
generated_at: datetime,
allowed_field_names: Collection[str] = (),
) -> dict[str, Any]:
"""Build a deterministic, offline request for a downstream evaluator.

Args:
decisions: Reconstructed AGT decisions to expose as observations.
generated_at: Time at which this handoff request was created. Callers
must pass a timezone-aware value so replays are unambiguous.
allowed_field_names: Exact Decision BOM field names permitted to cross
the boundary. The default is empty to avoid exporting arbitrary
policy, context, or trace data by accident.

Returns:
A strict-JSON-compatible dictionary. Its authority boundary is
intentionally fixed: the downstream evaluator receives observations
but cannot authorize actions, override policy, or mutate source records.

Raises:
ValueError: If no decisions are provided, timestamps are timezone-free,
or an allowlisted value is not strict-JSON serializable.
"""
if not decisions:
raise ValueError("at least one Decision BOM is required")

generated_at_utc = _utc_timestamp(generated_at)
allowlist = frozenset(allowed_field_names)
observations: list[dict[str, Any]] = []

for decision in decisions:
fields = [_export_field(field) for field in decision.fields if field.name in allowlist]
observations.append(
{
"decision_id": decision.decision_id,
"observed_at": _utc_timestamp(decision.timestamp),
"agent_id": decision.agent_id,
"action_requested": decision.action_requested,
"governance_outcome": decision.outcome,
"source_completeness": decision.completeness_score,
"sources_queried": list(decision.sources_queried),
"fields": fields,
}
)

request: dict[str, Any] = {
"schema_version": SCHEMA_VERSION,
"purpose": "post_execution_external_evaluation",
"generated_at": generated_at_utc,
"source": {
"system": "agent-governance-toolkit",
"representation": "decision_bom",
"decision_count": len(observations),
},
"observations": observations,
"authority_boundary": {
"read_only": True,
"source_records_mutated": False,
"execution_authorized": False,
"policy_decision_overridden": False,
"evaluation_result_is_governance_decision": False,
},
}

digest = sha256_jcs(request)
request["request_id"] = f"eval_{digest.removeprefix('sha256:')}"
return request


def _sample_decisions(now: datetime) -> list[DecisionBOM]:
"""Create synthetic Decision BOMs for the runnable example."""
return [
DecisionBOM(
decision_id="decision-001",
timestamp=now,
agent_id="did:mesh:synthetic-agent",
action_requested="read_inventory",
outcome="allow",
fields=[
BOMField(
name="latency_ms",
category=BOMFieldCategory.OUTCOME,
value=42,
source="synthetic_trace",
),
BOMField(
name="internal_policy_context",
category=BOMFieldCategory.POLICY,
value={"rule": "allow-read"},
source="synthetic_policy",
),
],
sources_queried=["audit", "policy", "trace"],
completeness_score=0.8,
)
]


def main() -> None:
"""Print one synthetic, offline evaluation handoff request."""
now = datetime.now(timezone.utc)
request = build_external_evaluation_request(
_sample_decisions(now),
generated_at=now,
allowed_field_names={"latency_ms"},
)
print(json.dumps(request, indent=2, sort_keys=True))


if __name__ == "__main__":
main()
2 changes: 2 additions & 0 deletions examples/external-evaluator-handoff/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
agent-governance-toolkit-core>=5.0.0,<6.0
pytest>=8.0.0,<10.0
Loading
Loading