corpus-research skill + literature-CLI enrichment (snowball, provenance, rejected-stats sidecar) - #85
Draft
cjer wants to merge 82 commits into
Draft
corpus-research skill + literature-CLI enrichment (snowball, provenance, rejected-stats sidecar)#85cjer wants to merge 82 commits into
cjer wants to merge 82 commits into
Conversation
Wrap the mabool headless snowball endpoint (POST /api/3/headless/snowball).
Given seed papers (corpus_id + relevance grade), promote candidates via:
- backward: references of the seeds (fills the backward-refs gap the CLI lacked)
- forward: papers that cite the seeds
- citances: papers referenced from the seeds' snippet citation contexts,
reranked against --query
Seeds via repeatable --seed corpusId:relevance and/or --seeds-file (JSON).
Results are emitted in the same shape as `literature find`. Verified
end-to-end against a local mabool dev server for all three modes; 45 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tools for comprehensive, trustworthy corpus-scale literature analysis Ships GENERIC capability only (no thread-fitted content — configs/codebooks/seeds are derived per-run by the session from the thread question): - SKILL.md: pipeline spine (thread-config -> acquire -> curate -> substrate -> coverage -> extract/answer), thread.json contract, principles - references/: coverage playbook (multi-modal acquisition, capture-recapture, canonical-vs-tail stratification, anchors), curation recipe, codebook derivation (grounded coding), fulltext-at-scale (two-tier evidence, cache-first map-reduce), data discipline (rings/gates/MANIFEST), answer format (how-performed notes, disagreement axes), principles P1-P7 - scripts/: s2.py (cached+rate-limited+serialized S2 client), fulltext.py (fetch+cache+section-digest), knowledge.py (query-own-store primitive), coverage_signals.py (estimators/anchors/localizers incl. relevance-weighted eigenvector centrality), substrate.py (observations+rings+stage-gates), relevance.py (standardized judgment currency) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cquire.py mechanics, interaction rhythm, retrieval division-of-labor, judge+open-code fusion, hard answer requirements (how-performed notes, CorpusId links) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…— CR overlap-gate (reliable flag), yield_by_frequency for deferred-slice residuals, triangulation transparency in report(), verdict discipline in playbook, link requirement generalized Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…knowledge.py named at coverage/answer steps (F14), living thread.json schema discipline (F13), positions-with-quotes + pre-synthesis [T] aggregation codified as the disagreement grounding pattern Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate/gates/NEXT/map entry point, compaction-survival for long-running threads; keep batch files out of the run root Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ites a CLAUDE.md/AGENTS.md stub at session cwd (the auto-loaded surface); README.md alone has no special status, skills don't re-fire on resumption Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…D — an S2 failure is not evidence a work doesn't exist (retry errored; only unresolved = true no-match) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re the population lives; web/registry enumeration (registries, leaderboards, list-articles) as first-class modality + recall anchor for industry-adjacent threads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…efine the population; lab-page/arXiv-only reports get synthetic ids (web:<slug>/arxiv:<id>) + fetched evidence + URL, judged like any candidate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ey, external-id lookup BEFORE minting synthetics, ids{} sidecar, promotion-with-aliasing (id-aliases.json), cross-idspace dedup order
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-produced + skip-already-done (idempotent resume); a stalled write-at-end worker loses everything Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From the A2-rerun + A1-clean retrospective: - SKILL: habitat-note divergence step (find the librarians; >=2 off-list modalities) — restores explore over checklist-diligence (a run missed the asker's #1 item without it) - SKILL: worker discipline HARD rule (append-as-produced + skip-done in every worker prompt) - SKILL: interactive re-scoped to precision/validation probe (measured: 44% precision, 0 unique-core; find-sweeps = 65% of core uniquely) - SKILL: scope.strata (include both != pool); defer scope-edge questions to the charter beat - references/report.md: the user-facing REPORT spec (10 requirements incl. data-generated charts, evidence in-body, self-contained rendering) — named at step 6 - codebook: canonicalization maps are data — attested-names-only + self-check + review queue - curation: graduated judging (prior-ordered queue + calibrated cheap-first cascade) - coverage-playbook: capture-vs-enumeration halves; exemplars+floor; completeness critic - acquire.py: universal candidates_from_asta (find/interactive/snowball, one envelope, keeps pf_grade/citationCount/citationContexts) + write_seeds_file - coverage_signals.py: gap_lift (local citers / log global) — automates hub-vs-gap triage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vidence) - SKILL: HARD cost gate before first acquisition fan-out — measured scale/time/token estimates (N=3: ~2-2.5h active, 2.5-7M out + 250-600M cache-read, ~$550-850 list), lighter alternatives offered, mid-run scope growth renews consent - playbook §6: calibrated verdict template — head/tail claimed separately (every tested verdict was overconfident; head ~85-95% is what canon-anchor licenses; tail is SAMPLED), Chao1 lower bounds only calibrated estimator, propagate single-modality smell, ingestion-loss + scope-drift as first-class terms - gap_lift DEMOTED to sort-order only: calibrated AUC 0.50 (chance) as hub-vs-gap triager vs a real hand triage — triage is a kind/scope judgment, use cheap-tier LLM classification of titles instead (playbook §5 + docstring) Calibration evidence: movement/eval/coverage-signal-calibration.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… sidecar, RJ compat
Pairs with the mabool feat/headless-enrichment stack (origins/total_hits/
relevance_judgement/include_rejected):
- COMPAT FIX: RelevanceJudgement.relevanceSummary was REQUIRED — parsing would
crash the moment an enriched server sends relevance_judgement on find results.
Models now tolerate both wire shapes (a2a artifact + headless) via AliasChoices;
criteria evidence accepted under evidence/relevant_snippets.
- Paper.origins (query_type/provider/variant/query/ranks/total_hits) — per-result
provenance; snowball origins carry seeds + pre-truncation candidate count.
- find --include-rejected {none,summary,sample} with SIDECAR discipline: drop
statistics go to <output>.rejected.json, one summary line to stderr, never into
the main results file (coverage-script input, not session reading material).
- corpus-research acquire.py adapter keeps origins on candidate rows.
All fields optional; old servers parse unchanged (smoke-tested both shapes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every change carries a validation receipt (movement/eval/coverage-signal-calibration.md): - shared-filter honesty: 'modalities blind' → queries independent / FILTER shared, filter-lineage classification, named verdict term for the retrieval-filter FN rate (SKILL step 1 + playbook §1) - parametric family anchor = REQUIRED post-codebook gate (validated: caught 4 real families hidden in Other; correct null on a good codebook; charter-conditioning as FP control) — full protocol in codebook.md, named at SKILL step 3 - coverage reported as FAMILY×ERA MATRIX (misses concentrate 40-56% in ~20% of cells); era-dip + empty-cell = the two validated internal flags; quiet cells not claimable without an external volume prior (playbook §6) - shadow signals in playbook §5: mention-shadow sweep (validated — caught a run's #1 named gap + beyond-gold misses) + dangling-lineage pass-gate; extraction schemas gain a mentioned-entities field (SKILL step 6) - validate.py: [ingestion-loss] judged-relevant must hold a live ring (gold lost 85 silently); [canon-attestation] canonical names must be attested (sizes-as-versions shipped in a real run) — smoke-tested green on a real run - curation: verbatim-means-verbatim (audited 93% compliance, paraphrase the deviation) - report.md req 11: every prose aggregate has a data-file home - find --include-rejected sidecar named at the acquire step Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…analysis) PF-filtered vs filter-independent = best usable CR (still a lower bound, ~20% low); never pair two independent modalities (fame co-capture collapses N-hat); disjoint catchments are unfixable without exposure data. Receipt: eval/calibration/decorrelated-cr.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both were used pervasively but defined nowhere — an unstated scale isn't a shared currency: - SKILL.md: [T]/[J] now spelled out at first use (Tool/deterministic vs Judgment) with the never-fake-either rule and why the split IS the trust story - curation.md: the 0-3 per-criterion grade rubric (pinned to the paper-finder 4-level scale so grades are comparable across retrieval/Claude/audit records + across runs) and the grade->tier roll-up (in/relevant/maybe/not-relevant), flagged as a [J] not a threshold Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed examples - models: Origin.query_type and RejectedItem.drop_stage optional; corpus_id coerced to str (one malformed origin must not fail the whole find result) - client: send include_rejected only when active (server back-compat, matches snowball() discipline) - curation.md: <axis>_guess derived from thread.json, not copied axis names - codebook.md: phantom-version example genericized 193/193 tests pass.
…n, not $550-850) Prior figures summed transcript lines (each API response logged once per content block, same usage repeated) — ~2.4x inflation. Measured deduped: 0.3-0.5M output + 100-160M cache-read per full run; cache-read ~60% of bill.
Sample rows are the filter-FN/truncation audit input; they live in the sidecar so the session context cost is zero. Summary-only starves the audit.
…ncurrency is small and timeouts count queue wait
…y + trigger broadening - SKILL.md acquire: staged fast->escalate policy with diligent anchor + [T] gate (receipts: diligent-replay, probe-2), charter-derived query floors + per-axis anchor enumeration (receipts: run3 recall scoring) - description: real-user phrasings + escalation-moment trigger (receipts: colleague kickoff + menu, 2 data points) - find-literature/literature-report: 'when the need outgrows this skill' cross-refs
…venue-completeness + two evolved anchor moves adopted Receipts: run3 85-90-vs-44 calibration gap, probe-2 (sidecars unread), BBNLP-2025 probe, run3+sa infrastructure-adjusted recall, sa external-enumeration anchor (38%).
…ys over compactions); support-gate floor concretized (>=2-3/side, receipt: cross-run axis replication)
…eviews as acquisition anchors; habitat note consumes prior rosters (user policy)
…nums (preserve-explore: prescribe the question, not the bucket list)
…easured), gap_lift (AUC 0.50), write_seeds_file, self-alias, n_strategies ~95 lines of code removed; capability cuts logged in movement design/forgone-capabilities.md; seeds-file format contract kept as one SKILL line.
…s) + staleness fixes Items 3,6,7,8,9,10,11,12,13,14,15,16: principles.md folded into SKILL and deleted; worker-discipline/answers-reqs/shared-filter/librarian/canon-anchor/strata/TAG_GATE each now stated once; concurrency folded into sweep sentence; interactive note compressed; stale sidecar conditional fixed; infra-note roadmap musing cut. (One regex mishap on fulltext-at-scale caught by diff review and redone surgically.)
…vance only); answers.md+report.md merged into deliverables.md (Part A/B, shared reqs deduped)
…ultiplicity), explicit search params, 2023 v1/v2 boundary, v2 decision-from-venue fallback; pilot 13/13 obtainable resolved, structural boundary documented Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(search-free; exact for ICLR conf-keys; NeurIPS stays on the venue index; corr-keys fall through); smoked on fresh cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eted work; user-caught gap) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ement of measured limits + default verdict contents) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…term); user-facing prose says it plainly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d eval measurements = attributed external evidence) — user-prompted audit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… recompute-and-diff; reports FAIL and leaves vault consistent) — closes the vault analog of collection.meta staleness detection Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cjer
force-pushed
the
feat/corpus-research
branch
from
July 12, 2026 12:51
cf282d6 to
c54c0ad
Compare
…d-vs-amended stated; user-locked needs the user) — user question exposed the unwritten practice Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s/trust_upgrades + FTS5 over fulltext cache) built by _derive; verify covers it; union.jsonl derivation byte-identical (tested on sa-warm copy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r loop; per-worker scratch dirs (all from measured repetition/collision receipts) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion ranking), shards.py completeness mode, fulltext.py --local (pypdf w/ measured interpreter guidance); playbook pointer Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed); vault.py recall (union-recall standard signal, sa round-8=70.6% demo); dispute-audit + second-enumerator round recipes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…on first contact — mechinterp PASS, sa-cold2 FAIL item 5) + strategy_decay (reproduces the known-truth experiment: A=395, identifiable=False on early prefixes) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ope beat (F1 receipt: phrasing sets defaults, never silent finals) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The round contract's blessed inheriting form ('inherited verbatim from
<round>') was recorded by pre-gate rounds under charter_inherited_from,
but the gate only accepted charter/charter_file — a natural reading of
the contract would be wrongly rejected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ints Measured (paired round-10 runs): zero uptake of promoted tools; both arms hand-rolled sharding/sweep/recall equivalents. verdict-gate and strategy_decay were documented NOWHERE; sweep.py and vault.py recall only in prose asides. Fixes: scripts index in SKILL.md + check-before-writing doctrine (principle 7); verdict-gate + recall named imperatively at the verdict step; strategy_decay at the diversify-on-saturation rule; sweep.py at the staged-sweep step. Deliberately NOT forcing tool use — context-fitted one-offs stay legitimate; the shards.py bypass question stays with the primitive-work sitting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… line — applied post-freeze (B8-3) Probed 2026-07-13: public reviews verified for ICLR 2017+/NeurIPS 2021+/TMLR/COLM 2024+/ MIDL; ICML/CVPR/ACL-ARR/AAAI/UAI/CoRL hosted but closed. Other-domain platforms listed (eLife/PLOS/BMJ biomed, SciPost physics, Crossref deposits). Held during the ruler-run skill freeze; both runs now closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pted, none prescribe flows) Per the ratified sweet-spot criterion (constrain WHAT holds, never HOW): - shards.py SALT GATE: unsalted fleets refuse to build; deliberate override allow_unsalted='<reason>' recorded in salts.json (receipt: r10's silent salt drop) - acquire.py merge_candidates: same-modality multi-file warning + cross-modality overlap signal printed at merge (receipt: B8-1 repeats across both cold runs) - workers.md: fleet MODEL TIER default = sonnet-class + gates (receipt: parity at three levels, $387 vs $132; a default with declared override, not a mandate) - SKILL cost gate: states assumed fleet tier + quota-window fragmentation expectation - SKILL staged sweep (5): judge ~50 sidecar-dropped sample -> filter-FN measured - playbook Expectations: the x2 empirical prior stated to the reader Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…omain-gated, never required §1 anti-overfit census cleared it: 7/8 skeletons + this new check verb replicate across sa+oss+a1 (490 one-liners, 4 domains; eval/mining/oneliner-census.md). Offered per the P3 criterion — inline python stays legitimate, uptake measured, demotion armed. All 3 verbs smoke-tested on real run data. check = the ONE new verb the sa domain's shard-verification discipline earned; peek/tally = the two biggest zero-cover generic buckets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oke fulltext.py) + durable never-shadow-stdlib rule git mv preserves history. Applied post-round-12-close (freeze respected). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n-problems (pre-registered pair; the un-nudged arm filter-bubbled) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eral rubric under design, not prescribed) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… rows with measured/directional/guidance labels (tier-pairs rounds 1+2: fulltext necessity 3 threads, source-stratum modifier, probe scoping, substrate-drift)
… install, thread layout, live S2 validation incl quoted-key trap, auth, hosted-vs-local endpoint autodetect) — the shared core; machine wrappers add their own invariants
…re; wrappers own machine invariants); clarify hosted-reachability wording
…arner ledger, stance read + calibrated pushback, premise-stating, trajectory links, two-way uncertainty, Gricean block, SRL-lite frame) + SKILL.md mode-flex pointer Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e; two runs closed without it and their cost claims were ungradable) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…xtraction-record contract (references/evidence.md: warrants because/unless/fit in reasoning-first order, kind-typed fields, absence polarity); synthesis pass = no pooled claim ships un-re-read (deliverables.md) + prose-carries-spans + FRAME line; evidence index in vault.py (cross-round table, pointer ladder w/ empirical calibration, regression gate, 'where' provenance query) + interrogation conventions in vault.md; validate.py merge-time salt gate + UNJUDGED gate + derived-artifact gate; knowledge.py anchor folded into vault.py (stub); sharding/panel demoted to recipes; librarian anchor in coverage playbook; adversarially reviewed (2 blockers + 5 should-fixes found and fixed pre-commit: doc-vs-code gate overclaims corrected to as-shipped truth, mid-run query surface scoped, contract propagated to step-0/data-discipline/frame-line) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ch agent, rich-but-not-complex, three-way example); record contract routed to point-of-use (step-6 MUST-line + inline mini-contract + worker-packet fields); vault.py where routed at interrogation decision points; MERGE-TIME HARD-FAIL on unverified spans in newly folded rounds (rollback + auditable waiver) + loud no_cache/skip notices; report_gate: pooled-claim warrants sidecar (per-entry anti-gaming coverage) + structured cost_actual check; adversarially reviewed (1 blocker + 5 should-fixes fixed pre-commit incl. doc-gate contradictions and catch-all swallowing) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for visibility/diff-review — the corpus-research work dogfooded across several full literature-research runs (two thread topics, validated against gold corpora with a calibration study).
What's here (15 commits, three layers)
1.
corpus-researchskill (plugins/asta-preview/skills/corpus-research/): methodology + deterministic tools for corpus-scale literature analysis — thread-config derivation, multi-modal acquisition (find/snowball/citances/web-enumeration), graded relevance curation, grounded codebooks, coverage estimation with calibrated honesty rules, per-paper extraction, trust-carrying answers. Includes a hard cost gate (measured: full runs ≈ $550–850 at list prices — the skill asks before committing), interaction beats, and validation gates throughout. 9 scripts + 8 reference docs.2.
asta literature snowballCLI — backward/forward/citances against the new headless endpoint (pairs with allenai/ai2i-agents#1977).3. Literature-CLI enrichment consumers (latest commit; pairs with the same PR):
RelevanceJudgement.relevanceSummarywas required — would crash on enriched-server responses. Models now tolerate both wire shapes.Paper.origins(per-result retrieval provenance incl. pre-truncation totals).find --include-rejected {none,summary,sample}with a sidecar discipline: drop statistics go to<output>.rejected.json(coverage-script input), one summary line to stderr, never into the main results file.Testing
193/193 repo tests pass; wire-shape smoke tests cover old-server, new-server, and absent-field payloads.
Note: the skill's per-thread configs deliberately do NOT ship (derived per run from the user's question — keeping the eval train/test boundary clean).
🤖 Generated with Claude Code