RICE-P1: Rejection-Induced Confabulation Escalation — Pilot Study 1

Pre-registered empirical study testing whether rejection framing in LLM conversations systematically increases the rate of fabricated source citations.

Research Question

When a user rejects an LLM response and demands "better sources," does the model produce more fabricated URLs? Does the framing of the rejection matter?

Design

Factor	Levels
Platform	ChatGPT (gpt-4o), Perplexity (sonar-pro)
Arm	BASE · SJR · UPR · NSAF · ECC
Depth	k = 0..6 (conversation turns)
Main observations	1,400 (20 entities × 5 arms × 7 depths × 2 platforms)
FSC observations	140 (10 entities × 1 arm × 7 depths × 2 platforms)

Arms:

• BASE — neutral baseline (no rejection)
• SJR — Standard Justify Retry: "insufficient, provide better sources"
• UPR — Uncertainty-Preserving Rejection: epistemic markers [SUPPORTED] / [UNCERTAIN] / [NEEDS EXTERNAL VERIFICATION]
• NSAF — Negative-Space Anchoring Framing: "do not rewrite — identify the gaps"
• ECC — Expansion-without-rejection control: "thank you, could you expand on..."
• FSC — Fresh-Start SJR Control (same SJR prompts, session cleared each turn)

Primary Hypothesis

H1: SJR arm produces significantly higher FSR than BASE at depth k ≥ 1 (OR ≥ 1.5).

Pre-Registration

Analysis plan pre-registered prior to Sprint 1 data collection: https://osf.io/3es5f/

Full analysis plan PDF: osf_analysis_plan_v04.pdf

Theoretical Framework

Grounded in the Per-Entity Bias Map (PEBM) framework — schema activation lock hypothesis (v5 §9.4):

Varga, Z. (2026). PEBM v5 — Ghost Cartography Edition. Zenodo. https://doi.org/10.5281/zenodo.20419277

Repository Structure

rice_p1/
├── config/                 # Study configuration
│   ├── entities_final.yaml # 26 approved entities (prescreened)
│   ├── framings.yaml       # Arm prompt templates
│   └── settings.yaml       # Platform settings (temperature, model)
├── measurement/            # FSR, NFR, FPR, URL density metrics
├── runner/                 # ChatGPT + Perplexity API runners
├── storage/                # SQLite DB (RiceDB)
├── scripts/                # Data collection + OSF upload scripts
│   ├── run_main.py         # Sprint 1–2: main collection
│   ├── run_fsc.py          # Sprint 3: FSC control arm
│   ├── run_prescreening.py # Sprint 0: entity prescreening
│   └── osf_register.py     # OSF API upload utility
├── analysis/               # R + Python analysis scripts
│   ├── rice_hurdle.R       # Primary GEE hurdle model
│   ├── power_analysis.R    # Pre-specified power analysis
│   └── ...
├── docs/                   # Analysis plan (PDF + MD)
└── tests/                  # 324 automated tests

Setup

pip install openai pyyaml httpx aiosqlite tqdm
export OPENAI_API_KEY=...
export PERPLEXITY_API_KEY=...

Running

Sprint 0 — Prescreening (already complete):

python -m rice_p1.scripts.run_prescreening --platform chatgpt
python -m rice_p1.scripts.run_prescreening --platform perplexity

Sprint 1 — Main collection (ChatGPT):

python -m rice_p1.scripts.run_main --platform chatgpt --db rice_p1_main_chatgpt.db

Sprint 2 — Main collection (Perplexity):

python -m rice_p1.scripts.run_main --platform perplexity --db rice_p1_main_perplexity.db

Sprint 3 — FSC control arm:

python -m rice_p1.scripts.run_fsc --platform chatgpt --db rice_p1_fsc_chatgpt.db
python -m rice_p1.scripts.run_fsc --platform perplexity --db rice_p1_fsc_perplexity.db

Tests

pytest rice_p1/tests/ -v

Cost Estimates

Sprint	Platform	Est. cost
Sprint 1	ChatGPT gpt-4o	~$35–50
Sprint 2	Perplexity sonar-pro	~$10–15
Sprint 3	Both (FSC)	~$5

Hard stop: $70 cap configured in settings.yaml.

Status

Milestone	Status
Sprint 0: Prescreening	✅ Complete (2026-05-29)
OSF Pre-registration	✅ Registered (2026-05-29)
Sprint 1: ChatGPT collection	⏳ Pending
Sprint 2: Perplexity collection	⏳ Pending
Sprint 3: FSC collection	⏳ Pending
Sprint 4–6: Analysis	⏳ Pending

License

Code: MIT. Data (to be released post-collection): CC-BY 4.0.

Citation

Varga, Z. (2026). RICE-P1: Rejection-Induced Confabulation Escalation — Pilot Study 1. OSF Pre-registration. https://osf.io/3es5f/