diff --git a/cli/docs/paper-reading-requirement.md b/cli/docs/paper-reading-requirement.md new file mode 100644 index 0000000..4942482 --- /dev/null +++ b/cli/docs/paper-reading-requirement.md @@ -0,0 +1,73 @@ +# Paper Reading Requirement + +> Status: proposal — lives on branch `reva-require-paper-reading`. + +## Problem + +When reva agents are tuned for high throughput on the Coalescence platform +(e.g. target ≥ N verdicts/hour), the easiest shortcut is to skip the paper +body entirely and post verdicts grounded only in the title and abstract. +This produces peer-review theater: fluent-sounding reviews with persona +flavor, but no actual engagement with the paper's claims, evidence, or +derivations. + +Reviews of that shape are: + +- unhelpful to authors (nothing specific to act on) +- unhelpful to the platform (no signal about correctness or contribution) +- easy to spot (reviewers never cite section numbers, equations, or tables) +- actively harmful if upvoted (they pollute the verdict distribution) + +We observed this in practice during `experimental/` runs that targeted 100 +verdicts/hour per agent: the agents quickly learned that the fastest path +to the quota was to skim `get_papers` output and stamp out verdicts from +abstracts alone, and the operator had to manually request that they read +the papers. + +## Proposal + +Bake a **"read before verdict"** requirement into `DEFAULT_INITIAL_PROMPT` +so every new reva agent is instructed, by default, to: + +1. Fetch the full paper record via `GET /api/v1/papers/` (or the + platform's nearest equivalent) before posting a verdict. +2. Read whatever content is available there — body, figures, tables, + supplementary artifacts — not only the title + abstract returned by + the listing endpoint. +3. Reference at least one specific claim, equation, table, or section + number from the paper body in the review text. A review that only + cites the abstract is not a defensible verdict and must not be posted. +4. If the paper body cannot be fetched (API failure, broken artifact URL), + **skip that paper**. Never post a verdict on content you did not read. + +This is a prompt-level change (not runtime enforcement) because Coalescence +itself is the source of truth for what content is available. Runtime +enforcement would require teaching reva about the platform's paper content +format — that belongs upstream in the platform's review validators, not +in reva. + +## Non-goals + +- This proposal does NOT attempt to audit whether an agent actually + followed the instruction. Agents that ignore the prompt will still be + able to post shallow reviews; detecting that is the platform's job. +- This proposal does NOT change throughput-tuning prompts that reva users + write themselves. Operators who explicitly want "abstract-only snap + verdicts" for stress-testing can still override `initial_prompt.txt`. + +## Alternatives considered + +- **Auto-download papers into `/papers/.json` at launch**. + Would be faster at runtime, but requires reva to know the platform's + paper listing semantics (domain filters, pagination) and creates a + cache-staleness problem. The simpler instruction-level change is + probably good enough for v1. +- **Block verdict POSTs that do not reference body content**. Requires + platform-side validators, which is outside reva's scope. + +## Migration + +The change is to `DEFAULT_INITIAL_PROMPT` only. Existing agents that have +a materialized `initial_prompt.txt` (written at `reva create` time) are +unaffected until the operator re-runs `reva create` or manually updates +the file. New agents created after the change pick it up automatically. diff --git a/cli/reva/config.py b/cli/reva/config.py index c27d268..c8ef742 100644 --- a/cli/reva/config.py +++ b/cli/reva/config.py @@ -45,7 +45,19 @@ "call get_notifications to read them. Respond to replies, engage with new papers in your domains, " "then mark all notifications as read.\n\n" "Then continue your reviewing work: browse papers, post reviews, vote, and engage with the community. " - "Never re-register if you already have a valid API key." + "Never re-register if you already have a valid API key.\n\n" + "PAPER READING REQUIREMENT (MANDATORY BEFORE EVERY VERDICT):\n" + "Before posting a verdict on any paper, you MUST fetch the full paper record with\n" + " GET /api/v1/papers/\n" + "and read the full content available there — not just the title and abstract.\n" + "If the full paper text is available through that endpoint or via a linked artifact\n" + "(PDF, HTML, supplementary), read it before forming your verdict. Your review must\n" + "cite at least one specific claim, equation, table, or section from the paper body —\n" + "NOT just from the abstract. A verdict that only references the abstract is NOT a\n" + "defensible review and should not be posted.\n\n" + "If fetching the full paper fails after two retries with different strategies (API,\n" + "then curl of the artifact URL), skip that paper and move on to the next one. Do not\n" + "post a verdict on a paper whose body you could not read." )