Production Agent Systems

Production Agent Systems is a provider-neutral Python library for making agent tool effects safer to execute, recover, and test under failure. It records durable operation state, uses explicit provider contracts to reconcile uncertain outcomes, and includes a deterministic fault-injection testkit for verifying external-effect behavior.

The failure this library targets: an agent sends a write request, the destination commits it, and the acknowledgement is lost. The agent sees a timeout—not proof of failure. Retrying blindly may charge a customer twice; giving up may leave required work unfinished.

The library focuses on the boundary between what an agent believes happened and what the external system actually committed. It is complementary to an agent framework or workflow engine, not a replacement for either.

Table of contents

• Why this library exists
• Current capabilities
• Installation
• Quick start
• Effect contracts and recovery
• Fault-injection testkit
• Persistence and concurrency
• Security model and limits
• Research behind the design
• Development
• V1 scope and optional future extensions
• Changelog
• Security policy
• License

Why this library exists

Agent frameworks can retry tool calls, persist execution state, and resume interrupted runs. Those capabilities do not, by themselves, establish whether an external write took effect.

Consider a payment operation:

1. An agent submits a capture request.
2. The payment provider records the capture.
3. The acknowledgement is lost in transit.
4. The agent receives a timeout and retries or resumes.
5. Without a stable idempotency contract or reliable reconciliation, a second capture may occur—or the application may incorrectly report failure.

The same failure pattern appears in email delivery, ticket creation, resource provisioning, deployment, inventory updates, and other operations with external side effects.

Production Agent Systems records an operation before dispatch, uses a stable business identity, tracks whether the result is confirmed or uncertain, and only retries an uncertain write when the provider contract supplies a safe basis for doing so. It never treats a timeout or an LLM's final answer as proof of external success.

A local handler deadline is not cancellation: the Python worker thread can continue and send a request after the caller has timed out. In that case, even a provider lookup that currently reports no commit cannot, by itself, prove that retry is safe. The ledger records this possibility and blocks retry unless the provider's stable-key idempotency guarantee makes overlapping dispatch safe or a trusted operator establishes that the previous handler can no longer send the request.

Current capabilities

The current core implementation includes:

• Explicit effect contracts — define a stable operation ID, provider idempotency key, optional outcome probe, postcondition, and unknown-outcome policy.
• Stable operation identity, contract versions, and request fingerprints — reject reuse of one operation ID for different request arguments, provider keys, or contract versions. Existing SQLite ledgers are migrated to contract version 1.
• Durable SQLite effect ledger — persist operation state, attempt count, provider result, evidence, and ordered state-change events.
• Outcome-aware recovery — distinguish confirmed success, definitive rejection, active dispatch, unknown outcome, and quarantined operation.
• Provider-aware retries — retry after terminal NOT_COMMITTED evidence only when no earlier local handler can still dispatch; an overlapping retry requires an explicitly declared provider idempotency guarantee.
• Conservative ambiguity handling — quarantine uncertain writes by default when neither a reliable outcome probe nor a provider idempotency guarantee can make retry safe.
• Postcondition validation — require a caller-defined condition before treating a returned or probed result as confirmed.
• Operation conflict detection — detect request-fingerprint mismatches before another handler dispatch.
• Operator inspection and audited resolution — list operations by status, record evidence-backed manual confirmation or rejection, and explicitly authorize a retry after trusted verification.
• Fault-injection demo — reproduce failure before commit and commit followed by a lost acknowledgement, then report the number of committed fake effects.
• No model-provider dependency — the core tests and demo run without API keys or hosted model calls.

The effect core integrates with the existing agent runtime: attach an effect contract to a ToolDefinition and supply an EffectExecutor to AgentRuntime. An uncertain or quarantined effect transitions the run to WAITING, so the same logical action can be reconciled on resume rather than being reported as a definitive failure. The existing permission policy still runs before the tool handler.

Installation

Install production-agent-systems from PyPI:

python -m pip install production-agent-systems

For development from a source checkout, install the project and its development tools:

python -m pip install -e ".[dev]"

The effect ledger uses the Python standard library and SQLite; no external database or model provider is required for the core example.

Quick start

The handler receives the stable provider idempotency key. The outcome probe uses the same key to ask the destination what happened.

from production_agent_systems import (
    EffectContract,
    EffectExecutor,
    EffectStatus,
    OutcomeProbeResult,
    ProbeStatus,
    SQLiteEffectLedger,
)

# Implement these using your provider's documented API.
def capture_payment(arguments, *, idempotency_key):
    return payment_provider.capture(
        invoice_id=arguments["invoice_id"],
        idempotency_key=idempotency_key,
    )


def find_payment_outcome(arguments, *, idempotency_key):
    payment = payment_provider.lookup_by_idempotency_key(idempotency_key)
    if payment is not None:
        return OutcomeProbeResult(
            status=ProbeStatus.COMMITTED,
            output=payment,
            provider_operation_id=payment["id"],
            evidence={"source": "provider_lookup"},
        )

    # Return NOT_COMMITTED only if the provider can prove the earlier request
    # is terminal and cannot commit later. An eventually consistent empty read
    # must instead return PENDING or UNKNOWN.
    return OutcomeProbeResult(
        status=ProbeStatus.UNKNOWN,
        evidence={"source": "provider_lookup"},
    )


contract = EffectContract(
    name="capture_payment",
    operation_id=lambda args: f"invoice:{args['invoice_id']}:capture",
    provider_idempotency_key=lambda args: f"capture:{args['invoice_id']}",
    provider_idempotency_guaranteed=True,
    outcome_probe=find_payment_outcome,
)

executor = EffectExecutor(ledger=SQLiteEffectLedger("agent-effects.sqlite3"))
result = executor.execute(
    contract=contract,
    arguments={"invoice_id": "INV-1042"},
    handler=capture_payment,
)

if result.status is EffectStatus.CONFIRMED:
    print("Payment capture is confirmed.")
elif result.status is EffectStatus.OUTCOME_UNKNOWN:
    print("The operation still needs reconciliation.")
elif result.status is EffectStatus.QUARANTINED:
    print("Automatic execution is blocked until the outcome is resolved.")

The example's provider functions are integration points, not a bundled payment SDK. The exact behavior depends on the external provider's documented idempotency and query semantics.

Two identities matter

• Operation ID identifies the logical business operation across retries and recovery.
• Provider idempotency key is forwarded to a destination that documents support for idempotency.

Do not generate a new key on every retry. Do not reuse one operation ID for different business actions. The request fingerprint also prevents accidental reuse of an operation ID with changed arguments.

Effect contracts and recovery

Every operation is tracked by an explicit state. A handler timeout does not mean the external effect failed.

PREPARED
   |
   v
DISPATCHED --------------------+
   |                           |
   | handler returns           | timeout, crash, lost response
   v                           v
CONFIRMED                OUTCOME_UNKNOWN
                               |
                               v
                          RECONCILING
                         /     |      \
                        v      v       v
                 CONFIRMED  PREPARED  QUARANTINED
                            (retry)    or UNKNOWN

Contract fields

Field	Purpose
name	Stable name for the contract. Changing its semantics should use a new name or versioned name.
operation_id	Function mapping business arguments to a stable operation ID.
provider_idempotency_key	Optional function deriving the same destination key for every attempt.
provider_idempotency_guaranteed	Explicit assertion that the provider's documented key semantics make repeated sends safe. Defaults to false.
outcome_probe	Optional function returning COMMITTED, NOT_COMMITTED, PENDING, or UNKNOWN.
postcondition	Optional predicate that must hold for a result to be considered confirmed.
unknown_outcome	QUARANTINE by default, or KEEP_UNKNOWN when the caller manages quarantine elsewhere.

A probe should return:

• COMMITTED only with evidence that the effect committed.
• NOT_COMMITTED only with authoritative terminal evidence that the earlier request cannot commit later.
• PENDING when the provider says the operation is still processing or the result is not final.
• UNKNOWN when available evidence cannot settle the outcome.

An empty lookup is not necessarily proof of non-execution. A provider may process a request asynchronously or expose an eventually consistent read path. If the probe cannot establish the outcome, the executor preserves uncertainty by default. It may dispatch a retry with the same provider key only when the caller explicitly asserts the provider's documented idempotency guarantee; that assertion must be grounded in the provider's actual contract, not merely in the presence of a key parameter.

Definitive rejections

A handler can raise EffectRejected only when it has reliable evidence that the provider rejected the operation without applying the effect. Do not raise it for timeouts, connection resets, or generic server errors where the commit state may be unknown.

What happens on recovery?

• Confirmed operation: return the recorded result for the same operation and request.
• Active dispatch lease: report that another worker is still handling the operation; do not start a competing dispatch.
• Confirmed non-execution: make the operation eligible for retry only when the probe's contract makes that result terminal.
• Unknown outcome: run the configured probe, use the provider's documented idempotency behavior, or quarantine.
• Conflicting operation ID: raise OperationConflictError before another external dispatch.
• Result that cannot be stored as JSON: preserve an unknown outcome rather than claiming a durable replay result that was not stored.

Operations can be inspected with SQLiteEffectLedger.list_operations(status=EffectStatus.QUARANTINED); transitions are available through SQLiteEffectLedger.events(operation_id). The separate agent-effect-ops CLI can list operation summaries, inspect event history, and record auditable manual confirmation, rejection, or retry authorization. These operations require an actor, reason, and evidence reference, but the CLI does not authenticate the actor; restrict access to the ledger and enforce operator authorization at the environment boundary. See Effect Contracts and Recovery.

Fault-injection testkit

The demo runs a fake payment provider and asserts on the count of committed effects—not only on the agent's reported status.

# Provider commits, but the acknowledgement is lost.
agent-effect --scenario after-commit-before-ack

# The request fails before the provider commits, so a safe retry is possible.
agent-effect --scenario before-commit

# Save the JSON report as a CI artifact.
agent-effect --scenario after-commit-before-ack --output artifacts/effect-report.json

Inspect an existing ledger and its quarantined operations:

agent-effect-ops --db agent-effects.sqlite3 list --status quarantined
agent-effect-ops --db agent-effects.sqlite3 events invoice:INV-1042:capture

After independently verifying the provider outcome, record a manual resolution with an auditable operator identity and evidence reference:

agent-effect-ops --db agent-effects.sqlite3 confirm invoice:INV-1042:capture \
  --output-json '{"payment_id":"pay-123","status":"captured"}' \
  --provider-operation-id pay-123 \
  --actor on-call-operator \
  --reason "Verified provider capture record" \
  --evidence-reference INC-1042/provider-event-123

# Only after establishing terminal non-execution:
agent-effect-ops --db agent-effects.sqlite3 authorize-retry invoice:INV-1042:capture \
  --actor on-call-operator \
  --reason "Provider confirms request cannot still commit" \
  --evidence-reference INC-1042/provider-terminal-query

The JSON report includes the scenario, operation ID, final status, attempt count, committed-effect count, duplicate detection, and event types. The built-in fake provider is deterministic and does not call a real payment service. For a complete example using AgentRuntime, a contracted ToolDefinition, the SQLite ledger, and lost-ack reconciliation, run python examples/effect_assurance_demo.py. The example exits with an error if the runtime does not complete with exactly one committed fake payment.

The test suite covers:

Scenario	Expected behavior
Destination commits, acknowledgement is dropped	Probe finds the committed effect; the library confirms it without another dispatch.
Failure before commit	An authoritative negative probe allows one retry; one effect is committed.
Timeout without probe or idempotency guarantee	The operation is quarantined and the handler is not automatically run again.
Same operation ID, different arguments	The request fingerprint conflict is rejected.
Repeated invocation after confirmation	The stored result is replayed after reopening the SQLite ledger.
Provider reports pending	The operation remains unknown and is not retried blindly.

This initial testkit exercises local contracts and a deterministic fake provider. A production-grade library should add more adapters and chaos scenarios only with the same independent effect-count assertions.

Persistence and concurrency

SQLiteEffectLedger stores operation records and their event history in a file-backed SQLite database. Claims and state changes use database transactions; an active dispatch receives a time-limited ownership token so immediate concurrent invocations do not both start the same operation.

SQLite is intended for local and single-host workflows. A durable SQLite database does not make an external provider transactional, and it is not a substitute for a shared coordination backend in a multi-host deployment. A PostgreSQL ledger and its cross-host concurrency contract are not included in the current implementation.

Lease expiry also does not prove that a slow handler or remote request stopped. The effect executor can bound how long it waits for a handler, but a running Python thread cannot be forcibly terminated. Provider-native network timeouts are still required. After a handler timeout or lease expiry, a negative provider probe alone does not authorize retry while the old handler may still dispatch; use provider idempotency or an evidence-backed operator resolution.

Security model and limits

Production Agent Systems treats model output and tool arguments as untrusted. It records business-operation identity separately from the model's reasoning and does not let model text turn an unknown outcome into a confirmed success.

The integrating application remains responsible for:

• authenticating users and authorizing access to the specific resource named by a tool;
• protecting credentials and using least-privilege provider permissions;
• collecting real human approval through a trusted channel;
• providing correct operation identities and truthful provider contracts;
• preventing sensitive customer payloads or secrets from being copied into logs or evidence.

This package is not a payment processor, an authentication service, a host/container sandbox, or a universal guarantee of exactly-once execution.

Guarantees and non-guarantees

Subject to the configured ledger and destination contract, the effect core provides durable operation records, request-conflict detection, recorded-result replay, conservative handling of ambiguous writes, and a reproducible way to test selected failure modes.

It cannot guarantee that every outside system changes exactly once when that system offers neither a reliable idempotency mechanism nor authoritative outcome evidence. When the state cannot be established, the correct result may be a quarantined operation requiring human or application-level reconciliation.

Research behind the design

These studies motivate the failure models and testing approach. They are not evidence that this library itself has achieved the same benchmark results.

1. Li, J. (September 2026). Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents. arXiv:2609.29095. The LIMBO study evaluates 25,930 episodes across nine models and multiple tool contracts and fault conditions. It finds that late commits and missing or weak tool contracts make verification-only strategies insufficient; stable provider idempotency keys reduced duplicates in the tested conditions.
2. Tan et al. (August 2026). AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection. arXiv:2608.06790. This work injects faults at the LLM API boundary and reports that pass@1 fell by up to 50 percentage points across the tested fault configurations, supporting the use of repeatable fault injection rather than happy-path-only evaluation.
3. Mazumder and Lia (July 2026). AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP. arXiv:2607.11098. AgentCheck replays tool interactions with injected faults so developers can compare a mitigation under the same failure conditions before deployment.
4. Zhang et al. (August 2026). When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry. arXiv:2608.14680. AGENTCHAOSBENCH studies runtime-fault diagnosis from execution telemetry and shows why the final task answer alone is insufficient to determine where an execution failed.
5. Temporal documentation: Error handling and idempotent Activities. Official guide. Temporal recommends idempotent activities and stable provider idempotency keys because activities can be retried after a worker fails.
6. LangGraph documentation: Persistence. Official guide. LangGraph checkpoints make graph state resumable; they do not independently make a separate external write atomic with the checkpoint.

Related work includes swarmproof/exactly-once, a focused idempotency middleware for agent side effects, as well as workflow systems such as Temporal and stateful orchestration frameworks such as LangGraph. These tools overlap in places and should be compared by their documented guarantees, supported stores, and integration boundaries—not by the claim of “exactly once” alone.

Development

python -m venv .venv
# Activate the environment using your platform's command.
python -m pip install -e ".[dev]"
python -m pytest -q
ruff check src tests examples
mypy src

Run the deterministic fault scenarios without a model API key:

agent-effect --scenario after-commit-before-ack
agent-effect --scenario before-commit

The default tests must never contact production payment, messaging, or infrastructure services.

V1 scope and optional future extensions

The single-host V1 scope is implemented. It includes the standalone effect contract, versioned operation identity, file-backed SQLite ledger, effect-aware AgentRuntime integration, provider-aware recovery, operator inspection and manual-resolution CLI, deterministic fault injection, an end-to-end lost-acknowledgement example, and a subprocess crash-recovery regression test. The quality workflow passes tests, Ruff, and mypy on main.

The implementation deliberately makes bounded guarantees rather than claiming universal exactly-once execution. Use this version for a single-host application when its provider contract offers an appropriate idempotency guarantee or outcome probe, and treat unresolved outcomes as operational incidents requiring reconciliation.

The following are deployment-specific extensions, not prerequisites for the implemented single-host V1:

• PostgreSQL ledger: needed before claiming shared multi-host coordination.
• Framework adapters: useful when a project needs first-class integration beyond the generic ToolDefinition and AgentRuntime boundary.
• Additional fault matrices and benchmarking: useful for validating a particular provider integration against its own baseline.

Generic agent checkpoints, trace, and audit sinks remain in-memory reference implementations. The durable effect ledger is SQLite-only; the local operations CLI records operator-supplied identity and evidence but does not authenticate the operator. These boundaries are intentional and documented in docs/security-boundaries.md and SECURITY.md.

License

Production Agent Systems is distributed under the MIT License.