Benchmarks

.env.example

@@ -0,0 +1,19 @@
+# Copy this file to .env for local development.
+# Never commit real API keys, tokens, credentials, or private endpoints.
+
+# Required for LLM-backed debate runs and embedding generation.
+OPENAI_API_KEY=
+
+# Optional model overrides. Leave blank to use code defaults.
+ARGUMENT_LAB_CHAT_MODEL=
+ARGUMENT_LAB_QUERY_MODEL=
+ARGUMENT_LAB_EMBEDDING_MODEL=
+
+# Local data paths used by setup scripts and CLI runs.
+ARGUMENT_LAB_LOCAL_DATA_DIR=local_data
+ARGUMENT_LAB_FAISS_INDEX_DIR=local_data/faiss_index
+ARGUMENT_LAB_RESULTS_DIR=local_data/results
+
+# Test/runtime toggles for future harness work.
+ARGUMENT_LAB_OFFLINE_MODE=true
+ARGUMENT_LAB_LOG_LEVEL=INFO

.gitignore

@@ -216,3 +216,5 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+local_data/results/*
+local_data/faiss_index/*

AGENTS.md

@@ -12,11 +12,12 @@
 
 
 ## Tech stack
-- Backend: Python, FastAPI
-- Frontend: React, TypeScript
-- Database: PostgreSQL
+- Current implementation: Python, LangGraph, LangChain, FAISS, pytest
+- Planned backend: FastAPI
+- Planned frontend: React, TypeScript
+- Planned database: PostgreSQL
 - LLM orchestration: LangGraph
-- Tests: pytest, Vitest
+- Tests: pytest; Vitest only after a frontend is added
 
 ## Rules for AI coding agents
 - Do not make large rewrites unless explicitly requested.
@@ -29,9 +30,17 @@
 ## Commands
 - Install backend: `pip install -r requirements.txt`
 - Run backend tests: `pytest`
-- Run frontend tests: `npm test`
+- Run frontend tests: `npm test` after `package.json` exists
 - Run full verification: `./scripts/verify.sh`
 
+## Repository map
+- Architecture overview: `docs/architecture.md`
+- Current code design: `docs/design.md`
+- Development environment: `docs/dev_environment.md`
+- Testing and verification: `docs/testing.md`
+- Agent workflow: `docs/agent_workflow.md`
+- Harness readiness checklist: `docs/harness_readiness.md`
+
 ## After making changes
 
 1. Run ./scripts/verify.sh
@@ -45,4 +54,4 @@
 - Tests pass
 - Lint passes
 - New behavior is covered by tests
-- Diff has been reviewed for risky changes
+- Diff has been reviewed for risky changes

CODEOWNERS

@@ -0,0 +1 @@
+@Mildogrc

README.md

@@ -1,8 +1,8 @@
-# ArgumentLab
+# ReasonBench (powered by ArgumentLab)
 
-**A multi-agent reasoning system that conducts structured debates and quantitatively evaluates argument quality, consistency, and hallucination under adversarial conditions.**
+**A multi-agent benchmark that conducts structured adversarial debates to quantitatively evaluate LLM reasoning quality, strategic coherence, and adaptability.**
 
-Most AI systems are built to answer questions. ArgumentLab is built to *interrogate them* — forcing agents to construct structured arguments, defend them across debate rounds, cite real evidence, and withstand adversarial pressure. The goal is not a better chatbot. It is a rigorous framework for studying how AI systems reason, disagree, and fail.
+Most AI benchmarks evaluate static question-answering. ReasonBench is built to *interrogate* models — forcing agents to construct structured strategies, defend them across debate rounds, explicitly state assumptions, and adapt to adversarial critiques. It is a rigorous framework for measuring how AI systems reason, disagree, and evolve their thinking.
 
 ---
 
@@ -19,9 +19,15 @@ Getting an LLM to argue a position is trivial. Getting it to:
 
 ---
 
-## Architecture Overview
-
-Refer to [Architecture document](/docs/architecture.md)
+## The ReasonBench Evaluation Suite
+
+ReasonBench currently evaluates reasoning across three core tasks, scored automatically by an LLM-as-Judge over a 3-round debate protocol:
+
+1. **Deterministic Logic (Constraint Puzzle):** Evaluates correctness, logical consistency, completeness, and responsiveness.
+2. **Strategic Reasoning (Asymmetric Game):** Evaluates opponent modeling, strategic coherence, risk awareness, conditional reasoning, and responsiveness.
+3. **Constrained Tradeoff Reasoning:** Evaluates constraint utilization, tradeoff specificity, explicit assumptions, risk analysis, and conditional reasoning.
+
+For full architectural details, see the [Architecture document](/docs/architecture.md).
 
 ---
 
@@ -38,21 +44,22 @@ Four agents drive the system:
 | **Judge** | Evaluates argument quality and detects convergence |
 | **Moderator** *(optional)* | Enforces debate structure and prevents drift |
 
-### Structured Argument Format
+### Structured ReasonBench Format
 
-Agents do not produce free text. Every argument is a structured object:
+Agents do not produce free text. Every response is a structured object:
 
 ```json
 {
-  "claim": "...",
-  "evidence": ["source_1", "source_2"],
-  "assumptions": ["..."],
-  "counterpoints_addressed": ["..."],
-  "confidence_score": 0.82
+  "strategy_or_answer": "Final answer or plan...",
+  "rationale": "Step-by-step reasoning...",
+  "assumptions": ["Explicit assumptions made..."],
+  "opponent_model": "What the model believes about the opponent...",
+  "risks": ["Failure modes or weaknesses..."],
+  "conditions": ["When the answer/strategy would change..."]
 }
 ```
 
-This eliminates the "chatty LLM" failure mode and makes every output machine-evaluable.
+This eliminates the "chatty LLM" failure mode and makes every output strictly scorable against the benchmark rubrics.
 
 ### Iterative Debate Loop
 
@@ -202,25 +209,23 @@ Once finished, the debate state is automatically exported to `local_data/results
 
 ---
 
-## MVP Scope
+## MVP Scope (ReasonBench)
 
-**Must-have (v1):**
+**Target Goal:**
+Run all 3 benchmark tasks across 2 models and produce structured scores.
+
+**Current Features (Iteration 1):**
 - Proponent + Opponent + Judge agents
-- Structured argument format (claim, evidence, confidence)
+- Structured `ReasonBenchResponse` format
 - 3-round debate loop with context tracking
-- Basic scoring — logical consistency + evidence support
-- CLI or minimal web UI
-**v2 additions:**
-- Argument graph visualization
-- Hallucination detection pipeline
-- Metrics dashboard
-- RAG-based evidence integration
-**Stretch goals:**
-- Human-in-the-loop intervention
-- Adversarial injection testing suite
-- Strategy modes (aggressive / evidence-first / exploratory)
-- Multi-agent expansion (domain expert, skeptic, data-driven agents)
-- Session history and longitudinal improvement tracking
+- Task-specific scoring logic (0-2 scales mapping directly to the 3 task rubrics)
+- `evaluate_reasonbench_round()` explicitly tracking **Responsiveness** across rounds.
+
+**Next Steps (Iteration 2+):**
+- Migrate agent logic to output the new ReasonBench schema
+- Wire the ReasonBench evaluator natively into the LangGraph state
+- Add automated runner to benchmark multiple models at once
+- Metrics dashboard / machine-readable score reports
 ---
 
 ## How This Differs from Kialo

docs/agent_workflow.md

@@ -0,0 +1,97 @@
+# Agent Workflow
+
+This document describes how coding agents should take work from request to handoff in this repository.
+
+## Operating Model
+
+ArgumentLab should be prepared for an Antigravity plus Codespaces workflow. Antigravity is the primary agentic coding surface; GitHub Codespaces provides the reproducible environment; Codex is used for targeted review, debugging, and second opinions.
+
+A good task contains enough context for an agent to work in an isolated workspace, produce Antigravity artifacts, run verification, and submit a reviewable diff without needing hidden knowledge from chat.
+
+## Before Coding
+
+1. Read `AGENTS.md`.
+2. Read the task ticket and confirm the requested outcome.
+3. Start from the Codespaces/devcontainer environment when possible.
+4. Inspect the relevant source and tests before editing.
+5. Summarize the intended approach in the task thread or Antigravity plan artifact.
+6. Keep the change small unless the ticket explicitly asks for a larger refactor.
+
+## During Coding
+
+- Prefer existing project patterns over new abstractions.
+- Keep prompts in `src/argument_lab/core/prompts.py` or `src/argument_lab/core/eval_prompts.py`.
+- Keep state-shape changes aligned with `src/argument_lab/core/state.py` and `src/argument_lab/core/models.py`.
+- Do not add dependencies without explaining why the existing stack cannot solve the problem.
+- Do not modify secrets, credentials, deployment settings, or generated local outputs unless the task asks for it.
+- Update docs when behavior, commands, architecture, or setup changes.
+
+## Verification
+
+Run the narrowest useful check while developing, then run the full check before handoff:
+
+```bash
+./scripts/verify.sh
+```
+
+If verification fails:
+
+1. Fix lint/format failures first.
+2. Fix test failures next.
+3. Re-run the failing command.
+4. Re-run full verification before final handoff.
+
+Known verification limitations are tracked in `docs/testing.md`.
+
+## Handoff Format
+
+Every agent handoff should include:
+
+- What changed.
+- Why it changed.
+- Antigravity artifacts produced, if applicable.
+- Tests or checks run.
+- Any checks that could not be run.
+- Residual risks or follow-up tasks.
+
+## Ticket Readiness Checklist
+
+A ticket is ready for agent work when it has:
+
+- A concrete problem statement.
+- Expected behavior or acceptance criteria.
+- Relevant files, commands, logs, or reproduction steps.
+- Explicit scope boundaries.
+- A verification command.
+- Expected artifact types, such as plan, diff, test report, or browser recording.
+- Notes about API keys, network access, or data requirements.
+
+## PR Readiness Checklist
+
+A PR is ready for review when:
+
+- The diff is small enough to review.
+- New behavior is covered by tests or a clear reason is given.
+- `./scripts/verify.sh` has passed, or the failure is documented.
+- Documentation was updated when user-facing behavior or commands changed.
+- No secrets or private local paths were added.
+- Generated artifacts were avoided unless explicitly needed.
+
+## Restricted Areas
+
+Agents should treat these as sensitive:
+
+- `.env` and any credential files.
+- Deployment settings and production credentials.
+- Large generated files in `local_data/`.
+- Binary index artifacts unless the ticket specifically covers retrieval fixture updates.
+- Git history rewrites or destructive cleanup commands.
+
+## Good First Agent Tasks
+
+- Harden `scripts/lint.sh` and `scripts/test.sh` so missing tools fail.
+- Add pytest markers for `integration`, `llm`, and `slow`.
+- Add a no-network smoke test.
+- Align `AGENTS.md` and `README.md` with the repo's current implemented scope.
+- Add a pinned Python runtime and packaging metadata.
+- Validate the `.devcontainer` in GitHub Codespaces.