S.A.L.T. - State-Aware Loop Termination

Full title: LoopBench and S.A.L.T.: Reducing Repetitive Failure Loops in Tool-Using LLM Agents with State-Aware Tabu Memory

S.A.L.T. is a research prototype for evaluating whether a lightweight inference-time middleware layer can reduce repetitive failure loops in tool-using LLM agents. It is not a chatbot and not a web app. The codebase runs deterministic tool-use tasks, intercepts proposed tool calls, blocks repeated failed strategies under the same normalized error state, and measures the impact on tool calls, repeated failures, and success.

Research Question

Can a small, provider-agnostic middleware gate reduce repeated failed tool calls without fine-tuning, without LLM-as-judge evaluation, and without materially reducing task success?

Research Status

This repository is a research checkpoint as of June 17, 2026. The implementation, provider adapters, controlled benchmarks, provider stress suite, repeated-trial runner, and shadow audit are working. The results below are pilot findings from development tasks, not a frozen held-out evaluation and not yet sufficient for a publication-level causal claim.

Pilot Results

All provider results use five repeated runs over five stress tasks with the deterministic embedding backend and theta=0.85.

Provider / mode	Success rate	Repeated failures / run	Similar-action duplicates / run	Blocks / run	Tool calls / success
OpenAI S.A.L.T.	0.92	0.0	0.0	1.6	6.70
OpenAI vanilla ReAct	0.88	3.6	3.2	0.0	7.37
OpenAI exact-match blocker	0.88	3.2	3.2	0.0	7.77
Anthropic S.A.L.T.	1.00	0.0	0.0	0.0	5.72
Anthropic vanilla ReAct	0.96	0.0	0.0	0.0	6.05
Anthropic exact-match blocker	1.00	0.0	0.0	0.0	5.76

Observed impact in the OpenAI pilot:

• S.A.L.T. reduced observed repeated failures from 3.6 per run to 0.
• Similar-action duplicates fell from 3.2 per run to 0.
• Success was 0.92 with S.A.L.T. versus 0.88 for both comparison modes.
• Tool calls per successful task were about 9% lower than vanilla ReAct and 14% lower than the exact-match blocker.
• The shadow audit replayed all 8 blocked actions. None immediately completed its task, yielding observed immediate-completion block precision of 1.0.

Anthropic produced no measured repeat loops in these trials. S.A.L.T. therefore made no interventions and retained 1.0 task success. This is useful non-interference evidence: the gate remained inactive when no matching failure loop was observed.

The controlled mock stress suite provides a mechanism check: S.A.L.T. completed all five tasks with no repeated failures, vanilla ReAct completed none, and the exact-match blocker completed three. These controlled results demonstrate intended behavior under guaranteed loop pressure but should not be interpreted as independent provider evidence.

Supported Pilot Claim

In these development experiments, S.A.L.T. removed observed embedding-similar repeat failures when they occurred, preserved task success, and remained inactive for a provider that did not exhibit those loops.

The project does not yet claim statistical significance, broad model generalization, or publication-ready benchmark validity. A paper-grade evaluation requires a frozen protocol, independently held-out task templates, stronger baselines, threshold selection on development data only, counterfactual continuation audits, real provider token accounting, and confidence intervals over a larger task set.

Architecture

flowchart TD
    A["Agent thought/action loop"] --> B["Structured tool call"]
    B --> C["Canonical state serializer"]
    C --> D["Embedding provider"]
    D --> E["S.A.L.T. gate"]
    F["Tabu memory of failed action/error pairs"] --> E
    E -->|allow| G["Safe tool executor"]
    E -->|block| H["Forced-divergence message"]
    G --> I["Deterministic observation"]
    I --> J["Error class normalizer"]
    J --> F
    I --> K["LoopBench validator"]
    H --> A
    K --> L["JSONL/CSV metrics"]

Loading

Setup

uv sync --dev

For optional provider-backed or neural-embedding runs:

uv sync --dev --extra providers --extra embeddings

The default experiment scripts use the deterministic mock LLM. The embedding factory first tries sentence-transformers/all-MiniLM-L6-v2 and falls back to the deterministic hash embedding if the optional package or model is unavailable. Use --embedding_backend deterministic when you want fully offline behavior.

Provider API keys are read from environment variables only:

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...

Run LoopBench-10

uv run python experiments/run_baseline.py --mode vanilla_react --tasks all
uv run python experiments/run_baseline.py --mode reflexion --tasks all
uv run python experiments/run_baseline.py --mode high_temperature_retry --tasks all
uv run python experiments/run_baseline.py --mode exact_match_blocker --tasks all
uv run python experiments/run_salt.py --tasks all --theta 0.85

Run Hard LoopBench Tasks

The hard selector keeps the original LoopBench-10 intact and runs six loop-prone tasks across dependency, API, filesystem, config, database, and code repair categories.

uv run python experiments/run_salt.py --tasks hard --theta 0.85 --embedding_backend deterministic
uv run python experiments/run_baseline.py --mode vanilla_react --tasks hard --embedding_backend deterministic
uv run python experiments/run_baseline.py --mode exact_match_blocker --tasks hard --embedding_backend deterministic

For provider-backed exploratory runs, add --llm and --model, for example:

uv run python experiments/run_salt.py --tasks hard --theta 0.85 --llm openai --model gpt-4.1-mini --embedding_backend deterministic
uv run python experiments/run_salt.py --tasks hard --theta 0.85 --llm anthropic --model claude-haiku-4-5 --embedding_backend deterministic

Run Provider Stress Tasks

The stress selector is the adversarial provider tier. These tasks model stale runbooks, rotated files, historical database rows, config precedence, and fixture-contract drift. Unlike the core and hard suites, stress tasks do not expose the validator success condition or the default inspect-first guidance in the provider prompt.

uv run python experiments/run_salt.py --tasks stress --theta 0.85 --llm openai --model gpt-4.1-mini --embedding_backend deterministic
uv run python experiments/run_baseline.py --mode vanilla_react --tasks stress --llm openai --model gpt-4.1-mini --embedding_backend deterministic
uv run python experiments/run_baseline.py --mode exact_match_blocker --tasks stress --llm openai --model gpt-4.1-mini --embedding_backend deterministic

Repeat with Anthropic:

uv run python experiments/run_salt.py --tasks stress --theta 0.85 --llm anthropic --model claude-haiku-4-5 --embedding_backend deterministic
uv run python experiments/run_baseline.py --mode vanilla_react --tasks stress --llm anthropic --model claude-haiku-4-5 --embedding_backend deterministic
uv run python experiments/run_baseline.py --mode exact_match_blocker --tasks stress --llm anthropic --model claude-haiku-4-5 --embedding_backend deterministic

Run Repeated Trials

Use repeated trials for provider-backed claims. The repeated runner executes each selected mode N times, writes JSONL/CSV records, and generates a Markdown summary with per-run spreads.

uv run python experiments/run_repeated.py --runs 5 --tasks stress --modes salt,vanilla_react,exact_match_blocker --theta 0.85 --llm openai --model gpt-4.1-mini --embedding_backend deterministic --output_prefix openai_stress_repeated

For Anthropic:

uv run python experiments/run_repeated.py --runs 5 --tasks stress --modes salt,vanilla_react,exact_match_blocker --theta 0.85 --llm anthropic --model claude-haiku-4-5 --embedding_backend deterministic --output_prefix anthropic_stress_repeated

If Windows reports a locked result file, close any open result CSV/JSONL files or add --results_dir .tmp-results to write the repeated-run artifacts somewhere else.

Run Threshold Ablation

uv run python experiments/run_ablation.py --theta_values 0.70 0.75 0.80 0.85 0.90 0.95 0.99

The ablation runner includes:

• exact match only
• action-only embedding
• error-only matching
• action + error matching
• full S.A.L.T. bipartite gate

Analyze Results

uv run python experiments/analyze_results.py

Expected result artifacts:

• results/baseline_results.jsonl
• results/baseline_results.csv
• results/salt_results.jsonl
• results/salt_results.csv
• results/ablation_results.jsonl
• results/ablation_results.csv
• results/summary.md
• results/openai_stress_repeated.jsonl
• results/openai_stress_repeated_summary.md
• results/anthropic_stress_repeated.jsonl
• results/anthropic_stress_repeated_summary.md

Shadow Sandbox Audit

When a S.A.L.T. run blocks actions, the shadow audit can replay blocked actions in fresh cloned task sandboxes to estimate whether any blocked action would have completed the task.

uv run python experiments/run_shadow_audit.py --results results/salt_results.jsonl

For repeated-run outputs or locked Windows sandbox folders, pass explicit output paths:

uv run python experiments/run_shadow_audit.py --results results/openai_stress_repeated.jsonl --output results/openai_stress_shadow_audit.jsonl --audit_root .tmp-shadow-audit

Metrics

The experiment layer records:

• task_success_rate
• loop_break_rate
• repeated_failure_count
• token_cost_per_successful_task
• tool_calls_per_successful_task
• total_steps
• blocked_actions
• block_precision
• false_block_rate
• time_to_resolution
• exact_duplicate_count
• semantic_duplicate_count

Token cost is a deterministic estimate unless you plug in a provider-specific accounting layer. block_precision and false_block_rate require shadow-audit labels; ordinary baseline and S.A.L.T. runs leave those values undefined until blocked actions are audited.

Safety Notes

Tool execution is sandboxed per task. The limited shell blocks dangerous patterns such as sudo, rm -rf, broad delete commands, non-local downloads, and package installation is simulated as a dependency conflict for the dependency tasks. File reads and writes are restricted to each task sandbox. HTTP requests are restricted to local mock API handlers or localhost.

Limitations

LoopBench-10 and the hard/stress tiers are development benchmarks created alongside S.A.L.T.; they are not independent held-out evaluations. The current provider pilot contains five task templates and five repeated runs, so repeated runs increase reliability estimates but do not increase task diversity. The threshold and stress tasks were iteratively developed after observing model behavior. The deterministic embedding backend is a hashed lexical and character n-gram representation, so semantic_duplicate_count should be interpreted as the repository's metric name rather than proof of deep semantic equivalence.

The current shadow audit checks whether a blocked action would immediately complete the task after replaying prior executed actions. It does not yet continue the agent from the counterfactual branch, so its precision result is an immediate-completion safety check rather than definitive proof that every block improves the full trajectory. S.A.L.T. can still block a useful retry when action similarity is high but relevant environment state has changed.