🧠 The Agency Circuit

A Neuro-Symbolic Architecture for Mitigating Policy Collapse in Reinforcement Learning

An interpretable meta-controller that detects when an RL agent has "given up" — and acts to restore its sense of agency.

Paper · Architecture · Quick Start · Results · Citation

---

📖 Overview

Deep Reinforcement Learning agents can achieve human-like performance, yet they remain vulnerable to policy collapse — a catastrophic failure where, after a streak of negative outcomes, the agent defaults to a state of paralyzing inaction. This failure mode is strikingly analogous to the psychological phenomenon of learned helplessness.

The Agency Circuit is a neuro-symbolic meta-controller, inspired by the antagonistic mPFC–DRN neural circuit that governs resilience to uncontrollable stress in mammals. It augments any standard RL agent with the ability to:

• 🩺 Learn a continuous, sub-symbolic metric of helplessness from experience (the neural subsystem).
• 🔎 Reason symbolically about when that metric signals imminent collapse (interpretable Boolean predicates).
• ⚡ Intervene with a targeted micro-action to re-assert agency and break the agent out of its passive state.

In a deceptive trap gridworld, the Agency Circuit (AC-DQN) escapes traps where standard DQN and curiosity-driven DQN+ICM baselines collapse — and it does so with an interpretable, inspectable internal state.

📄 Shahid, M. (2026). The Agency Circuit: A Neuro-Symbolic Solution for Mitigating Policy Collapse in Reinforcement Learning. In Proc. of AAMAS 2026, Paphos, Cyprus. IFAAMAS. doi:10.65109/TRFJ2704

---

✨ Key Idea: Modeling the Brain's Resilience Circuit

Biology	Agency Circuit Component	Role
Dorsal Raphe Nucleus (DRN) — the brain's "helplessness switch"	DRN_Net (GRU)	Learns a continuous helplessness signal hₜ from interaction history
Medial Prefrontal Cortex (mPFC) — top-down "executive" inhibition	mPFC_Net (feed-forward)	Maintains a control value cₜ that suppresses helplessness
mPFC → DRN inhibitory projection	Effective helplessness h_eff,t = hₜ × (1 − c_{t−1})	The antagonism between surrendering and persevering
Active coping / restored sense of control	Control Exertion Module (CEM)	Fires a micro-action when collapse is imminent

---

🏗️ Architecture

The Deceptive Trap: a 20×20 gridworld with a punishing central trap zone that induces policy collapse in standard agents.

The Agency Circuit is composed of three interconnected modules:

1. Neural State Assessor
   • DRN_Net — a GRU trained self-supervised to predict the (clipped) discounted negative return over a horizon N, yielding raw helplessness hₜ ∈ [0, 1].
   • mPFC_Net — a feed-forward network producing a control value cₜ ∈ [0, 1] that decays deterministically (factor δ) unless the CEM fires.
2. Symbolic Reasoner — evaluates crisp, auditable predicates:
   • IsHelpless ⟸ h_eff,t > θ_h
   • InTrapZone ⟸ sₜ ∈ S_trap
   • NotRecentlyIntervened ⟸ cooldown elapsed
3. Control Exertion Module (CEM) — when IsHelpless ∧ InTrapZone ∧ ¬RecentlyIntervened, it overrides the base policy with a micro-action to restore agency.

The Agency Circuit Loop (per timestep)

1. Assess          hₜ      ← σ(DRN_Net(history))
2. Inhibit         h_eff,t ← hₜ × (1 − c_{t−1})
3. Reason          fire?   ← IsHelpless ∧ InTrapZone ∧ ¬RecentlyIntervened
4. Act             aₜ      ← micro-action   if fire
                           ← π_RL(sₜ)        otherwise
5. Update control  cₜ      ← mPFC_Net(...)  if fire   (assert agency)
                           ← δ · c_{t−1}    otherwise (decay agency)
6. Train           base RL agent + self-supervised DRN_Net & mPFC_Net

---

📂 Project Structure

agency_circuit/
├── config.py                 # All hyperparameters (paper Table 1)
├── environment.py            # Deceptive Trap + Sticky / Teleporter variants
├── neural_state_assessor.py  # DRN_Net (GRU) and mPFC_Net (control)
├── agency_circuit.py         # The Agency Circuit meta-controller (AC-DQN)
├── baselines.py              # DQN and DQN+ICM baseline agents
├── experiments.py            # Experiment runners + plotting
├── main.py                   # CLI entry point
├── requirements.txt          # Python dependencies
├── setup_env.sh / .bat       # One-shot environment setup
└── results/figures/          # Pre-generated paper figures

---

🚀 Quick Start

1. Install

# Clone
git clone https://github.com/mahnoor-shahid/AgencyCircuit.git
cd AgencyCircuit

# Create & activate a virtual environment, then install deps
python -m venv env
source env/bin/activate        # Windows: env\Scripts\activate
pip install -r requirements.txt

Or use the one-shot setup script:

bash setup_env.sh              # Windows: setup_env.bat

2. Run

# Reproduce all experiments
python main.py --all

# Or run a single experiment
python main.py --experiment 1   # Performance vs. baselines
python main.py --experiment 2   # Internal state dynamics
python main.py --experiment 3   # Helplessness-threshold sensitivity
python main.py --experiment 4   # Zero-shot generalization to unseen traps

# Quick demos
python main.py --demo-env       # Explore the gridworld
python main.py --demo-agents    # Inspect agent internals

Results and plots are written to results/.

---

🔬 Experiments

#	Experiment	What it tests	Key metric
1	Performance in a Trap	Can AC-DQN escape where baselines collapse?	Success Rate, Escape Latency
2	Internal State Dynamics	Does the model work via the hypothesized mechanism?	hₜ, cₜ, h_eff,t time series
3	Hyperparameter Sensitivity	How robust is performance to the threshold θ_h?	Final success rate vs. θ_h
4	Generalization to Unseen Traps	Is resilience a reusable skill?	Zero-shot success on Sticky & Teleporter traps

---

📊 Results

Performance & resilience in the Deceptive Trap — AC-DQN solves the task while DQN and DQN+ICM collapse.

Internal dynamics — helplessness rises in the trap, the CEM fires, the control value spikes, and effective helplessness is suppressed, letting the base policy escape.

Zero-shot generalization — AC-DQN transfers its "enter, struggle, escape" skill to unseen Sticky and Teleporter traps; baselines fail everywhere.

📌 All quantitative results in the paper are averaged over 10 independent runs. Re-running the experiments regenerates the figures above in results/.

---

⚙️ Key Hyperparameters

Defaults follow Table 1 of the paper (config.py):

Parameter	Symbol	Default	Description
Helplessness threshold	θ_h	0.8	Tolerance for failure before intervening
History sequence length	k	50	Transitions fed to the DRN GRU
Control decay	δ	0.995	How quickly a sense of control fades
Prediction horizon	N	20	Foresight of the helplessness estimate
CEM cooldown	N_cooldown	20	Min. steps between interventions
Learning rate	α	1e-4	Adam optimizer
Discount factor	γ	0.99	—
Replay buffer	|B|	50,000	—
DQN hidden layers	—	[256, 256]	Shared by all agents for a fair comparison

---

🌍 The Deceptive Trap Environment

• 20×20 gridworld, state = flattened 400-dim one-hot of the agent's (x, y) cell.
• Actions: {up, down, left, right}, deterministic transitions (walls block movement).
• Rewards: goal +10, wall collision −1, step cost −0.01, and a central 5×5 trap zone with a punishing −0.1 per action.
• Variants (for zero-shot generalization):
  • Sticky Trap — actions inside the trap fail with 90% probability.
  • Teleporter Trap — any action inside the trap resets the agent to the start.

---

📝 Citation

If you use this code or build on the Agency Circuit, please cite the paper:

@inproceedings{shahid2026agency,
  title     = {The Agency Circuit: A Neuro-Symbolic Solution for Mitigating
               Policy Collapse in Reinforcement Learning},
  author    = {Shahid, Mahnoor},
  booktitle = {Proceedings of the 25th International Conference on Autonomous
               Agents and Multiagent Systems (AAMAS 2026)},
  year      = {2026},
  address   = {Paphos, Cyprus},
  publisher = {International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)},
  doi       = {10.65109/TRFJ2704},
  url       = {https://doi.org/10.65109/TRFJ2704}
}

This repository also ships a CITATION.cff file, so you can grab a formatted citation directly from the "Cite this repository" button on GitHub.

---

👤 Author

Mahnoor Shahid — Universität Duisburg-Essen, Essen, Germany ✉️ mahnoor.shahid@uni-due.de

📄 License

The code in this repository is released under the MIT License. The paper is © 2026 IFAAMAS, licensed under CC BY 4.0.