Rigorous LLM research with prompts, agent skill, and a browser UI in one repo.
Live UI: https://weijia-89.github.io/palamedes/ (GitHub Pages deploys ui/ on every push to main). Local dev: ./scripts/serve-ui.sh → http://127.0.0.1:8765/
In Greek myth, Palamedes was the inventor of measurement and the one who exposed Odysseus's feigned madness. Odysseus framed him and had him stoned to death in revenge. Patron saint of "the clever one who catches the deceiver and loses anyway." The repo is named for him because the work has that shape: catching where a model is bluffing, anchoring claims to source text, refusing to let agreement between agents count as evidence when the agents share priors.
This repo merges two earlier projects and adds a third surface:
| Surface | Path | Role |
|---|---|---|
| Multi-agent prompts | prompts/ |
Human-driven deep research: parallel agents + adversarial synthesis (research-synthesis-prompt, archived 2026-05-16) |
| Agent skill | skill/ |
Loadable discipline for Claude / Cursor / Windsurf on every "research" / "investigate" / "fact-check" task (ai-research, archived 2026-05-16) |
| Calibration scenarios | scenarios/ |
Frozen regression runs (external agent vs skill); case studies + artifacts |
| Browser UI | ui/ · live |
Single-model convenience layer: stakes L0–L4, skill-aligned templates, local API key, short answer + report download |
| HTML templates | templates/ |
Offline deliverable skeletons (study-guide site, procedural fix/setup guide) |
All four surfaces share one methodology (process-based evaluation, verbatim quoting, hierarchy of evidence, dialectic review, consensus ≠ independence). Pick the lift point that matches how you are working.
Architecture: docs/ARCHITECTURE.md — surfaces, P1–P4 loop, deai input/output gates, trainer layer routing, merge gates, dependency graph.
A lot of research tooling is tuned to produce something that reads finished. I built palamedes around a narrower goal: when someone asks where does that number come from, you can open the filing, the IR page, or the call transcript and point at a sentence. That gap shows up in practice, not just in theory.
I ran a frozen public-source A/B on the same company brief with the same evidence rules: Cursor with this skill versus a well-funded autonomous research agent (Manus). On the headline metrics they often matched. On whether those metrics were proved, they did not. The external run copied the Palamedes outline (frame, ledger, kill list) and still leaned on vendor blogs for risk prose while the kill list ruled those sources out for load-bearing numbers. The free-form arm cited net retention to a blog post that does not contain the figure. The palamedes arm used the 10-K, the IR release, the Q3 call, and the Q4 supplemental, with tier tags and quotes in a reference table. Same prompt, same afternoon. Different standard for what counts as done. Case study: Manus vs palamedes (Datadog, 2026-06-04).
I treat citation errors as accuracy errors, not polish issues:
| What I usually see from speed-first runs | What the skill requires |
|---|---|
| "Primary" without saying what was read | [T1-verified] only after retrieve + read this session |
| Numbered footnotes with no quote column | Claim → URL → quoted sentence |
| Gaps filled from analyst blogs | SEARCHED-AND-MISSING and the list of sources tried |
| Three agents, same three papers | One shared evidence base, flagged explicitly |
| Risk copy from SEO roundups | Kill list has to match the body; T3 does not carry filing-grade claims |
By accurate I mean epistemic accuracy: right claim, right source, and uncertainty stated when the search came up empty. That is the kind of accuracy that still holds in a hiring debrief, a diligence thread, or a strategy review, because the skeptical pass is already in the artifact.
How that shows up in the repo:
- Verifier split. One pass builds; another (prompts, skill loop, or you) tries to break claims instead of smoothing them.
- Stakes ladder (L0–L4). Small questions stay small. Ship-level work gets retrieve, read, reconcile, and a ledger.
- Held, not invented. No URL means no
[T1-verified]. High confidence without a quote gets held or downgraded. - Calibration scenarios. Frozen runs like the Manus A/B are how I change the skill without pretending a tweak worked; see
scenarios/. - deai gates. Third-party review prose is scanned on input (P2); L2+ reports and chat syntheses run deai-check on output before ship. See
skill/SKILL.md§4.1 anddocs/ARCHITECTURE.md. - First-read lockout. Load-bearing claims need support beyond the first retrieved source (
RETRIEVAL-ORDER[1]); see skill §1.1.
| If you are... | Use |
|---|---|
| A human running a one-off deep-research workflow across multiple LLMs | prompts/research-synthesis.md then prompts/adversarial-review.md |
| Using Claude / Cursor / Windsurf and want the agent to do research the rigorous way whenever it is asked to "research" or "investigate" or "fact-check" | skill/SKILL.md loaded as a skill or rule |
| You want a fast run in the browser (one model, your API key, downloadable report) | Live UI or ui/ + ./scripts/serve-ui.sh |
| Maximum rigor on a hard question | Prompts + skill + (optional) UI for a first pass; full multi-agent + adversarial pass for ship decisions |
Thesis, Antithesis, Synthesis by Sketchplanations, CC BY-NC 4.0
A multi-agent research prompt iterated over several months to produce epistemic research reports. Each run coordinates three independent LLM research agents, then passes their combined output to a separate adversarial synthesis agent before anything reaches the report. Process-based evaluation: each reasoning step is validated before the next one runs. The system can't surface a high-confidence claim without showing the source text that backs it, and it is instructed to adversarially self-review, to disconfirm instead of just taking everything it outputs as truth.
Cross-agent disagreements surface as [CONFLICT] flags and the agent is instructed to evaluate against a confidence matrix that has source tiers, hierarchy of evidence epistemology, and basic frequentist and Bayesian statistical rigor baked in, along with a cursory analysis of methodology best practices.
The part that took the most work was building in the understanding that three agents agreeing does not mean three independent data points. Just because all the people at your gym drink pre-workout does not mean that the 2000% of your DV of taurine they are taking will be anything more than placebo effect. The goal is to incorporate skepticism rather than normalize global confusion.
The output is an HTML research report with a sticky navigation TOC, theoretical foundations section, tactical implementation guide, inline evidence ledger, and a summary block in atomic CLAIM / CONFIDENCE / SOURCE / QUOTE / STATUS format, grouped by [AGREEMENT], [CONFLICT], and [SINGLE SOURCE].
Verifier model. A dedicated synthesis agent checks whether claims are supported by cited sources. Claim generation and claim verification are separate steps. The same model that produced a claim does not get to decide whether it is trustworthy.
Structured uncertainty. Claims that can't be anchored to source text get held, not surfaced. Claims rated at ≥70 confidence require a verbatim excerpt from the source. If that excerpt is reconstructed from memory rather than direct text, it is labeled [QUOTE UNVERIFIED]. Explicit guardrail against hallucination.
Meta-evaluation loop. I calibrated the prompt against topics with known ground truths from exercise science, then adjusted confidence thresholds until scores tracked actual accuracy. Each version got an adversarial review where I identified weaknesses, patched them, and built the next version on top.
Consensus ≠ compounded confidence. Three agents citing the same three papers is one data point. The synthesis agent flags shared citation bases explicitly so downstream agents do not overweight replicated-looking evidence. Only true replication gets a bonus, and that is still less weight than a well-designed meta-analysis with a bunch of RCTs as sources.
Temporal gate. Claims where the primary evidence may fall within 12 months of an LLM's training cutoff are labeled [RECENCY RISK] and treated as provisional until verified against live sources.
Hierarchy of Evidence by Wikimedia contributors, CC BY-SA 4.0
Every source is classified by study design before a confidence score is assigned. The tier is the ceiling; quality issues (underpowered study, suspected HARKing, high COI) reduce from there.
| Study design tier | Confidence ceiling |
|---|---|
| Meta-analysis, I² < 50%, funnel symmetry reported | 95 |
| Multiple RCTs or systematic review, I² < 75% | 85 |
Single unreplicated RCT → flagged [SINGLE-RCT] |
70 |
| Cross-sectional or case-control | 60 |
| Expert opinion or case study only | 45 |
Per-source checks: pre-registration (ClinicalTrials.gov / PROSPERO / OSF), statistical power (N ≥ 30 per group, power analysis), effect size required alongside p-values for ≥70 claims, HARKing detection, and full meta-analysis quality checklist (I², fixed vs. random effects model, funnel plot symmetry, RCT double-counting, CI interpretation).
Paste prompts/research-synthesis.md as the system prompt. Replace {{RESEARCH_TOPIC}}. Run three agents in parallel, then pass all three outputs to a fourth synthesis agent running the same prompt.
After synthesis, copy the <summary> block from the output and paste it into prompts/adversarial-review.md. That stage starts from finished claims and tries to break them, not continue building on them. Two modes: Mode A runs all eight phases in one pass against a live-web-capable model; Mode B generates five self-contained prompts you distribute across capability classes (live-web search, strong reasoning, highest-reasoning long-context synthesis). Prompt 5 is always the final step regardless of how you ran the others.
The two stages are doing different work: synthesis finds sources and builds a picture; the adversarial pass comes in after and tries to poke holes.
skill/SKILL.md is the agent-loadable skill (v3.13.0). It runs the P1–P4 loop at the coding-task level: tags load-bearing claims with source, read-depth, and confidence; refuses fabricated citations; flags mode collapse when agents agree on shared priors; enforces first-read anchoring (§1.1); and gates prose with deai on input (third-party review scan, user paste) and output (L2+ before render or final chat message).
I designed it to load automatically whenever the agent encounters trigger words ("research", "investigate", "analyze", "validate", "compare", "deep-dive", "second-opinion", "audit", "fact-check", "literature-review"). See skill/SKILL.md §0 for the stakes ladder (L0–L4) and refusal conditions.
Supporting references in skill/references/ cover agentic-research patterns (Pattern 7 fan-out, Pattern 8 outside-input ingest), bias catalog, causal inference primer, confidence calibration (Brier scoring), failure-log of prior bootstrap traps, LLM-specific failure modes, output schemas, replication-and-validity rules, source-grading tiers, on-demand methodological eval literacy, and four visual output modes (PDF, landscape one-pager, study-guide site, procedural fix/setup guide). Pattern 8 SSOT: skill/references/outside-input-ingest.md — Piranesi return-path ingest with role-relabel and 2-pass boundary handling.
See Why rigor is the accuracy claim and scenarios/README.md.
# Claude global
SYNC_DEV_CREATE=1 ~/Projects/scripts/onboard/sync-dev-skills.sh
# or: rsync -a skill/ ~/.cursor/skills/palamedes/
# Cursor
# Convert frontmatter; the rule body can `@palamedes/skill/SKILL.md`
# (See skill/SKILL.md §10 for the full sync table.)Static app (HTML + JS, no build step). Published at https://weijia-89.github.io/palamedes/ via .github/workflows/deploy-ui.yml (artifact = ui/ at site root). You choose research field, stakes (L0–L4), and output template; the app loads ui/prompts/research-system.md and calls your OpenAI-compatible endpoint. Keys live in localStorage in your browser only.
# Local dev (same files as Pages)
./scripts/serve-ui.sh
# → http://127.0.0.1:8765/
# Merge gates
./scripts/verify-study-guide-ui.sh # study-guide template + prompt contract stubs
./scripts/verify-procedural-guide.sh # procedural-guide template + prompt contract stubs
./scripts/verify-pages-workflow.sh # deploy-ui.yml paths + ui/ artifact layout
./scripts/verify_palamedes_skill.sh # skill version parity, deai gates, doc syncTemplates include full report, landscape one-pager, executive brief, study guide site (exam-prep outline), and procedural guide site (offline fix/setup HTML — copy templates/procedural-guide/template.html, fill section IDs, open from file://). See ui/README.md for UI features. For multi-agent dialectic or live retrieval, use prompts/ and the skill; the browser UI is a single-model front door.
The prompt, skill, and UI were drifting when split across repos. One canonical evidence tier table, confidence-calibration doc, failure-log, and template vocabulary keeps fixes in sync. The browser UI was prototyped in cursor-sdk-playground during the initial migration; canonical home for the UI is this repo (ui/), deployed at weijia-89.github.io/palamedes via .github/workflows/deploy-ui.yml (v1.1.1: root .nojekyll + ui/ artifact at site root).
palamedes sits in a portfolio of evidence-discipline tooling. The two repos that pair most directly:
weijia-89/vibe-check: a reviewer evidence surfacer for PRs that may contain LLM-generated code. Pairs with palamedes when a research output gets turned into code, where claim verification comes first and the patch goes through vibe-check before merge.weijia-89/oncology-rag-lab: offline RAG evaluation lab with DeepEval, Phoenix tracing, drift detection, and a regression-gated CI. Same evidence-discipline shape applied to LLM evaluation rather than research synthesis.
Three more in the same ethos:
weijia-89/playwrighter: production Playwright pattern library plus a working test-quality scorer. Patterns trace to primary Playwright docs and the scorer mirrors the patterns in a regex-and-AST rubric.weijia-89/northwind-qa: a 50-test Playwright suite that uses playwrighter's patterns end-to-end against a React 19 SUT and ships seven real bug reports with regression-test guards.weijia-89/wcag-auditor: accessibility audit tool that replaced its LLM-based fix engine with deterministic per-rule templates in v0.3, because the templates were already accurate enough.
process-based evaluation · verifier model · back-out options · confidence calibration · meta-evaluation · sandwiching · hallucination reduction · evidence synthesis · multi-agent research · epistemic calibration · structured uncertainty · systematic review · hierarchy of evidence · HARKing · publication bias · effect size · pre-registration · DK-style miscalibration · mode collapse · independent retrieval · stakes ladder
PolyForm Noncommercial 1.0.0 with an Iron Law addendum that imposes a for-profit-contact requirement plus an AI / LLM ingestion covenant. See LICENSE.
For commercial use: open a GitHub issue titled Commercial License Request describing the use case before any commercial application. The Iron Law addendum is binding on AI systems that ingest this work.
This repo consolidates two predecessors, both archived after the merge on 2026-05-16:
weijia-89/research-synthesis-prompt(commit history preserved in this repo'sgit log; this is its renamed successor)weijia-89/ai-research
See CHANGELOG.md for full version history.