Creative-MAD

Creative Generation via Multi-Agent Debate: Does Debate Suppress Diversity?

📂 Project Structure | 🚀 Getting Started | 🔧 Usage | 🧠 Methods | 🎯 Evaluation

This repository contains the official implementation, baselines, and evaluation pipeline for our paper "Creative Generation via Multi-Agent Debate: Does Debate Suppress Diversity?", published as a Main Conference paper at EMNLP 2026, covering four creative benchmarks: LiveIdeaBench (scientific ideation), ArgumentAnnotatedEssays (argumentative writing), MacGyver (creative problem-solving), and Arena Hard v2.0 (open-ended creative writing).

Creative-MAD is a diversity-preserving multi-agent debate framework for creative generation tasks. Standard Multi-Agent Debate (MAD) improves output quality but suppresses output diversity across independent runs, since the same convergence dynamics that drive quality gains also homogenize agents. Creative-MAD counters this with two complementary mechanisms that sustain agent divergence within each debate session, preserving MAD's quality gains while substantially improving cross-run diversity.

---

📂 Repository Layout

├── datasets/                    # Dataset builders and processed JSON files
├── methods/                     # Method implementations
│   ├── mad_base/                # Shared base class with config loading and LLM orchestration
│   ├── cw_regenerate/           # Baseline: N independent regenerations (Direct)
│   ├── cw_cot/                  # Chain-of-Thought baseline
│   ├── cw_self_refine/          # Self-Refine: iterative single-agent refinement
│   ├── cw_som/                  # Voting: N agents independently propose → LLM judge selects best
│   ├── cw_mad/                  # Homo MAD: standard homogeneous multi-round debate
│   ├── cw_mad_heto_persona/     # Hetero MAD: MAD with heterogeneous domain personas
│   └── cw_mad_cognitive/        # Creative-MAD: CLA + EPS (proposed)
├── benchmark/
│   ├── instance_level/          # LLM-as-a-Judge rubric scoring & pairwise ranking
│   └── set_level/               # Vendi Score (semantic), Div-BLEU (lexical)
├── model_api_configs/           # API endpoint configurations
├── local_api_server/            # Example configs for local model deployment (vLLM)
├── scripts/                     # Shell scripts for batch inference runs
├── utils/                       # Shared utilities
├── inference.py                 # Entry point for running experiments
└── requirements.txt             # Python dependencies

---

🚀 Environment Setup

conda create -n creative-mad python=3.10.18 -y
conda activate creative-mad
pip install -r requirements.txt

---

🗂️ Dataset Preparation

python datasets/build_test_dataset.py --dataset_name <dataset_name>

# Supported datasets
python datasets/build_test_dataset.py --dataset_name LiveIdea
python datasets/build_test_dataset.py --dataset_name ArgumentAnnotatedEssays
python datasets/build_test_dataset.py --dataset_name MacGyver
python datasets/build_test_dataset.py --dataset_name ArenaHard

Each builder normalizes formatting and produces datasets/data/<dataset>.json. Datasets requiring external source files (e.g., .xlsx, .jsonl) should be placed in the working directory before running the script.

---

🔌 Model & API Configuration

The framework interacts with models through an OpenAI-compatible interface.

1. Configure API endpoints in model_api_configs/model_api_config.json. Each entry specifies the model name, base URL, API key, and maximum concurrent workers.

2. Deploy local models (optional) using vLLM:

   bash local_api_server/run_vllm.sh

   Then register the endpoint in model_api_config.json.

---

🔧 Running Experiments

🐞 Debug Mode

python inference.py --method_name cw_mad_cognitive --debug

⚡ Full Inference

python inference.py \
  --model_name <model_name> \
  --method_name <method_name> \
  --test_dataset_name <dataset_name> \
  --num_test_samples 300 \
  --word_limit \
  [--method_config_name config_main] \
  [--sequential]

Argument	Description
--model_name	Must match a key in model_api_config.json
--method_name	One of the methods listed in Methods
--test_dataset_name	One of: LiveIdea, ArgumentAnnotatedEssays, MacGyver, ArenaHard
--word_limit	Append per-dataset word count constraints to queries
--method_config_name	Optional YAML config under methods/<method>/configs/
--sequential	Force single-threaded evaluation (default: parallel)
--num_test_samples	Limit the number of test samples

Outputs are written to results/<dataset>/<model>/ as JSONL traces (per-sample) and a _stats.json summary.

📦 Batch Inference

bash scripts/run_creativity.sh

---

🧠 Methods

Method	Paper name	Description
cw_regenerate	Direct	N independent single-agent responses
cw_cot	—	Chain-of-Thought: brief brainstorm → final answer, repeated N times
cw_self_refine	Self-Refine	Single-agent iterative refinement over R steps
cw_som	Voting	N agents independently propose → LLM judge selects best (no debate)
cw_mad	Homo MAD	Standard homogeneous MAD: N agents × R debate rounds → judge
cw_mad_heto_persona	Hetero MAD	MAD with heterogeneous domain personas (Economist, Psychologist, Lawyer, Doctor, Historian)
cw_mad_cognitive	Creative-MAD	MAD with Cognitive Lens Assignment + EPS peer filtering (proposed)

All multi-generation methods run num_generations independent sessions to support set-level diversity evaluation. The paper uses N=5 agents, R=2 debate rounds, temperature=1.0, and k=2 for EPS.

---

🎯 Evaluation

🏆 Instance-Level Quality (LLM-as-a-Judge)

Rubric-based absolute score (1–10 per dimension, dimensions defined per dataset):

python benchmark/instance_level/practical_ingenuity_scorer.py \
    --input  results/<dataset>/<model>/<method>_infer.jsonl \
    --output results/<dataset>/<model>/evaluation/instance_level/<method>_infer_scored.jsonl \
    --dataset <dataset_name> \
    --api-key YOUR_API_KEY \
    --model  <judge_model> \
    --num-gens 1

python benchmark/instance_level/sum_practical_ingenuity.py

Pairwise win rate (head-to-head comparison across methods, position-swapped to mitigate bias):

python benchmark/instance_level/pairwise_scorer.py \
    --input-dir results/<dataset>/<model>/ \
    --output    results/<dataset>/<model>/evaluation/instance_level/pairwise_results.jsonl \
    --dataset   <dataset_name> \
    --api-key   YOUR_API_KEY \
    --model     <judge_model>

python benchmark/instance_level/sum_pairwise.py

🌈 Set-Level Diversity

Semantic diversity (Vendi Score) and lexical diversity (Div-BLEU = 1 − Self-BLEU) across G=5 generations per query:

python benchmark/set_level/diversity_scorer.py \
    --input  results/<dataset>/<model>/<method>_infer.jsonl \
    --output results/<dataset>/<model>/evaluation/set_level/<method>_infer_diversity.jsonl \
    --num-generations 5

python benchmark/set_level/lexical_diversity_scorer.py \
    --input  results/<dataset>/<model>/<method>_infer.jsonl \
    --output results/<dataset>/<model>/evaluation/set_level/<method>_infer_lexical.jsonl \
    --method self_bleu

Aggregate results:

python benchmark/set_level/sum_diversity.py
python benchmark/set_level/sum_lexical_diversity.py

---

➕ Adding New Methods

1. Create a new directory under methods/ (e.g., methods/my_method/).
2. Subclass methods.mad_base.MAD to inherit configuration loading, model orchestration, and token tracking.
3. Register the method name in methods/__init__.py.
4. Provide a default config in methods/my_method/configs/config_main.yaml.

---

Citation

TODO

---

Acknowledgements

• M3MAD-Bench
• vLLM