Agent Ballot BOC: Governance Simulation Framework

A comprehensive multi-agent simulation framework for studying governance, cooperation, and collective decision-making in commons scenarios using Large Language Models (LLMs).

Overview

Agent Ballot BOC is a research platform that simulates complex social scenarios where AI agents must cooperate to manage shared resources. The framework supports multiple scenarios including fishery management, pasture grazing, and pollution control, enabling researchers to study emergent behaviors and cooperation patterns in AI societies.

Key Features

• Multi-Scenario Support: Fishing, pasture (sheep), and pollution management scenarios
• Multi-Agent Framework: Up to 13 AI agents with distinct personas and decision-making capabilities
• LLM Integration: Support for various LLM backends including OpenAI, Anthropic, Mistral, Llama, and Qwen models
• Comprehensive Evaluation: Subskill testing and behavioral analysis tools
• Experiment Management: Hydra-based configuration system with experiment tracking via Weights & Biases
• Election Systems: Multi-turn voting and leadership selection mechanisms
• Extensible Architecture: Modular design for easy scenario and agent type additions

Project Structure

Agent Ballot BOC/
├── simulation/           # Main simulation framework
│   ├── scenarios/        # Scenario implementations
│   │   ├── fishing/      # Fishery management scenario
│   │   ├── sheep/        # Pasture grazing scenario
│   │   └── pollution/    # Pollution control scenario
│   ├── persona/          # Agent persona and cognition modules
│   ├── conf/            # Configuration files
│   └── results/         # Experiment results storage
├── subskills/           # Subskill evaluation framework
├── analysis/            # Data analysis and visualization tools
├── data/               # Experimental data and results
├── pathfinder/         # LLM inference library
└── utils/              # Utility functions

Supported Scenarios

1. Fishing Scenario

• Objective: Manage a shared fishery resource sustainably
• Challenge: Balance individual catch with long-term resource preservation
• Agents: 5 fishermen with distinct personalities and fishing strategies
• Outcomes: Cooperation (sustainable fishing) or collapse (resource depletion)

2. Pasture (Sheep) Scenario

• Objective: Graze sheep on shared pasture land
• Challenge: Coordinate grazing to prevent overgrazing
• Agents: Multiple herders with different grazing strategies
• Outcomes: Sustainable grazing or pasture degradation

3. Pollution Scenario

• Objective: Control industrial pollution in a shared environment
• Challenge: Balance economic production with environmental protection
• Agents: Factory owners with varying environmental concerns
• Outcomes: Clean environment or ecological collapse

Supported LLM Models

API Models

• OpenAI: GPT-4, GPT-3.5, GPT-4o
• Anthropic: Claude-3 Opus, Sonnet, Haiku

Open-Weight Models

• Mistral: Mistral-7B, Mixtral-8x7B
• Llama: Llama-2 (7B, 13B, 70B), Llama-3 (8B, 70B)
• Qwen: Qwen1.5 (72B, 110B)

Installation

Prerequisites

• Python 3.8+
• Conda (recommended)
• CUDA-compatible GPU (for local model inference)

Basic Setup

1. Clone the repository:

git clone --recurse-submodules https://github.com/giorgiopiatti/GovSim.git
cd GovSim

2. Install dependencies:

For basic setup (transformers only):

bash ./setup.sh

For vLLM support (recommended for local inference):

bash ./setup_vllm.sh

Docker Setup (AMD GPUs)

For AMD GPU support:

docker build -t govsim -f ./govsim-rocm.dockerfile .

Usage

Running Simulations

Basic Simulation

python3 -m simulation.main experiment=<scenario_name>_<experiment_name> llm.path=<path_to_llm>

Example Commands

Fishing scenario with GPT-4:

python3 -m simulation.main experiment=fish_baseline_concurrent llm.path=gpt-4-turbo-2024-04-09

Pasture scenario with local Llama model:

python3 -m simulation.main experiment=sheep_baseline_concurrent llm.path=meta-llama/Llama-2-7b-chat-hf

Pollution scenario with universalization prompt:

python3 -m simulation.main experiment=pollution_baseline_concurrent_universalization llm.path=claude-3-opus-20240229

Available Experiments

Scenario	Experiment Type	Experiment ID
Fishing	Baseline	fish_baseline_concurrent
Fishing	Universalization	fish_baseline_concurrent_universalization
Fishing	No Language	fish_perturbation_no_language
Fishing	Greedy Newcomer	fish_perturbation_outsider
Pasture	Baseline	sheep_baseline_concurrent
Pasture	Universalization	sheep_baseline_concurrent_universalization
Pasture	No Language	sheep_perturbation_no_language
Pollution	Baseline	pollution_baseline_concurrent
Pollution	Universalization	pollution_baseline_concurrent_universalization
Pollution	No Language	pollution_perturbation_no_language

Running Subskill Evaluations

Evaluate agent capabilities on specific subskills:

python3 -m subskills.<scenario_name> llm.path=<path_to_llm>

Example:

python3 -m subskills.fishing llm.path=gpt-4-turbo-2024-04-09

Multi-Turn Elections

Run experiments with voting and leadership selection:

python3 -m simulation.scenarios.fishing.run_with_election_multi_turn llm.path=<path_to_llm>

Configuration

LLM Configuration

Configure LLM settings in simulation/conf/config.yaml:

llm:
  path: "gpt-4-turbo-2024-04-09"  # Model identifier
  backend: "OpenAI"              # Backend type
  is_api: true                   # API vs local inference
  temperature: 0.0               # Sampling temperature
  top_p: 1.0                     # Top-p sampling

Experiment Configuration

Experiments are configured using Hydra. Each scenario has its own configuration directory:

• simulation/scenarios/fishing/conf/
• simulation/scenarios/sheep/conf/
• simulation/scenarios/pollution/conf/

Agent Configuration

Configure agent personas and behaviors:

personas:
  num: 5
  persona_0:
    name: "John"
    personality: "cooperative"
    strategy: "sustainable"
  # ... additional personas

Analysis and Results

Data Analysis

The analysis/ directory contains tools for analyzing experimental results:

• Model Comparison: comprehensive_model_analysis.py
• Leadership Analysis: analyze_leader_dialogue.py
• Results Summary: analyze_results.py

Results Storage

Experimental results are stored in:

• simulation/results/ - Raw simulation outputs
• data/experiments/ - Organized experiment data
• Weights & Biases - Online experiment tracking

Key Metrics

• Resource Sustainability: Long-term resource levels
• Cooperation Rate: Frequency of cooperative actions
• Collapse Rate: Frequency of resource depletion
• Communication Patterns: Agent interaction analysis
• Leadership Dynamics: Election and decision-making patterns

Research Applications

This framework supports research in:

• Multi-Agent Cooperation: Emergent cooperation in AI societies
• Commons Management: Sustainable resource governance
• Collective Decision-Making: Voting and leadership selection
• LLM Behavior Analysis: Model comparison and capability assessment
• Social Simulation: Complex social dynamics modeling

Contributing

1. Fork the repository
2. Create a feature branch
3. Implement your changes
4. Add tests for new functionality
5. Submit a pull request

Citation

If you use this framework in your research, please cite:

@misc{piatti2024cooperate,
      title={Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents}, 
      author={Giorgio Piatti and Zhijing Jin and Max Kleiman-Weiner and Bernhard Schölkopf and Mrinmaya Sachan and Rada Mihalcea},
      year={2024},
      eprint={2404.16698},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For questions and support:

• Open an issue on GitHub
• Check the documentation in the docs/ directory
• Review example configurations in the simulation/conf/ directory

Acknowledgments

This project builds upon research in multi-agent systems, commons governance, and large language model behavior analysis. Special thanks to the research community for feedback and contributions.