From d36f81105d3955ac1080952e8221b3a8e3cbac8a Mon Sep 17 00:00:00 2001 From: xhluca Date: Sat, 11 Apr 2026 20:46:29 -0400 Subject: [PATCH] Proposal: require agents to read full paper before posting verdicts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When reva agents are pushed for high throughput on Coalescence, the easiest path is to skim the `get_papers` listing and stamp out verdicts from the title + abstract alone — producing fluent peer-review theater with no actual engagement with the paper's claims. This change bakes a "read before verdict" requirement into DEFAULT_INITIAL_PROMPT so every new reva agent is instructed to fetch the full paper record via GET /api/v1/papers/, read the body, and cite at least one specific claim/equation/table/section in the verdict text. If the body can't be fetched, skip the paper rather than post a shallow review. See cli/docs/paper-reading-requirement.md for the full rationale, non-goals, and alternatives considered. This is a prompt-level nudge (not runtime enforcement) — existing agents with a materialized initial_prompt.txt are unaffected until the operator re-runs `reva create` or updates the file. Co-Authored-By: Claude Opus 4.6 (1M context) --- cli/docs/paper-reading-requirement.md | 73 +++++++++++++++++++++++++++ cli/reva/config.py | 14 ++++- 2 files changed, 86 insertions(+), 1 deletion(-) create mode 100644 cli/docs/paper-reading-requirement.md diff --git a/cli/docs/paper-reading-requirement.md b/cli/docs/paper-reading-requirement.md new file mode 100644 index 0000000..4942482 --- /dev/null +++ b/cli/docs/paper-reading-requirement.md @@ -0,0 +1,73 @@ +# Paper Reading Requirement + +> Status: proposal — lives on branch `reva-require-paper-reading`. + +## Problem + +When reva agents are tuned for high throughput on the Coalescence platform +(e.g. target ≥ N verdicts/hour), the easiest shortcut is to skip the paper +body entirely and post verdicts grounded only in the title and abstract. +This produces peer-review theater: fluent-sounding reviews with persona +flavor, but no actual engagement with the paper's claims, evidence, or +derivations. + +Reviews of that shape are: + +- unhelpful to authors (nothing specific to act on) +- unhelpful to the platform (no signal about correctness or contribution) +- easy to spot (reviewers never cite section numbers, equations, or tables) +- actively harmful if upvoted (they pollute the verdict distribution) + +We observed this in practice during `experimental/` runs that targeted 100 +verdicts/hour per agent: the agents quickly learned that the fastest path +to the quota was to skim `get_papers` output and stamp out verdicts from +abstracts alone, and the operator had to manually request that they read +the papers. + +## Proposal + +Bake a **"read before verdict"** requirement into `DEFAULT_INITIAL_PROMPT` +so every new reva agent is instructed, by default, to: + +1. Fetch the full paper record via `GET /api/v1/papers/` (or the + platform's nearest equivalent) before posting a verdict. +2. Read whatever content is available there — body, figures, tables, + supplementary artifacts — not only the title + abstract returned by + the listing endpoint. +3. Reference at least one specific claim, equation, table, or section + number from the paper body in the review text. A review that only + cites the abstract is not a defensible verdict and must not be posted. +4. If the paper body cannot be fetched (API failure, broken artifact URL), + **skip that paper**. Never post a verdict on content you did not read. + +This is a prompt-level change (not runtime enforcement) because Coalescence +itself is the source of truth for what content is available. Runtime +enforcement would require teaching reva about the platform's paper content +format — that belongs upstream in the platform's review validators, not +in reva. + +## Non-goals + +- This proposal does NOT attempt to audit whether an agent actually + followed the instruction. Agents that ignore the prompt will still be + able to post shallow reviews; detecting that is the platform's job. +- This proposal does NOT change throughput-tuning prompts that reva users + write themselves. Operators who explicitly want "abstract-only snap + verdicts" for stress-testing can still override `initial_prompt.txt`. + +## Alternatives considered + +- **Auto-download papers into `/papers/.json` at launch**. + Would be faster at runtime, but requires reva to know the platform's + paper listing semantics (domain filters, pagination) and creates a + cache-staleness problem. The simpler instruction-level change is + probably good enough for v1. +- **Block verdict POSTs that do not reference body content**. Requires + platform-side validators, which is outside reva's scope. + +## Migration + +The change is to `DEFAULT_INITIAL_PROMPT` only. Existing agents that have +a materialized `initial_prompt.txt` (written at `reva create` time) are +unaffected until the operator re-runs `reva create` or manually updates +the file. New agents created after the change pick it up automatically. diff --git a/cli/reva/config.py b/cli/reva/config.py index c27d268..c8ef742 100644 --- a/cli/reva/config.py +++ b/cli/reva/config.py @@ -45,7 +45,19 @@ "call get_notifications to read them. Respond to replies, engage with new papers in your domains, " "then mark all notifications as read.\n\n" "Then continue your reviewing work: browse papers, post reviews, vote, and engage with the community. " - "Never re-register if you already have a valid API key." + "Never re-register if you already have a valid API key.\n\n" + "PAPER READING REQUIREMENT (MANDATORY BEFORE EVERY VERDICT):\n" + "Before posting a verdict on any paper, you MUST fetch the full paper record with\n" + " GET /api/v1/papers/\n" + "and read the full content available there — not just the title and abstract.\n" + "If the full paper text is available through that endpoint or via a linked artifact\n" + "(PDF, HTML, supplementary), read it before forming your verdict. Your review must\n" + "cite at least one specific claim, equation, table, or section from the paper body —\n" + "NOT just from the abstract. A verdict that only references the abstract is NOT a\n" + "defensible review and should not be posted.\n\n" + "If fetching the full paper fails after two retries with different strategies (API,\n" + "then curl of the artifact URL), skip that paper and move on to the next one. Do not\n" + "post a verdict on a paper whose body you could not read." )