An artificial research institution — built so that it can be proven wrong.
nullius in verba · take nobody's word for it
A multi-agent system that carries a research question through hypothesis, preregistration, experiment design, sandboxed execution, statistical analysis, adversarial challenge, independent replication and peer review — into a claim whose every number traces back to a hashed artifact.
And, more importantly, a way to find out whether any of that helps.
Nullius is evaluated on research questions generated from structural causal models we author. We know the true effect of every intervention — including the ones that are exactly zero — which features are causal and which are spurious, and where a defect was planted. So "did the institution reach the right conclusion, and was it appropriately confident?" is measured against ground truth rather than judged by a language model reading another language model's prose.
The project's own claim is falsifiable:
Institutional structure — preregistration, adversarial challenge, independent replication, and evidence-typed memory — improves the accuracy and calibration of autonomous empirical research relative to an unstructured agent, at a measurable cost.
That may turn out to be false. The benchmark is designed to be able to say so.
It has said so. See FINDINGS.md — generated from the committed protocols and results, and regenerated by CI on every push so it cannot drift from them. The short version, across six registered protocols:
- Two of the four adjudicated predictions were refuted, including the headline one the design document staked out in advance: B4 will capture most of the gain over B3.
- A published result was retracted by a later protocol, in the same document that reported it.
- Six flaws were found in the protocols by running them, not by reviewing them — a model-dependent baseline, an adjudication rule that compared two point estimates, calibration scored on a quantity the rubric does not measure, abstention scored as a correct answer, a prediction whose registered test measured something else, and a single custody draw that could not support the contrasts being measured.
- One mechanism clearly works. Spending experimental seeds where they would change an answer — cheap, and the only arm to beat a do-nothing baseline with an interval excluding zero.
- Calibration improves monotonically down the ladder across every protocol version. The accuracy ordering did not survive; that one did.
Every number in that file is read from a results file whose stored summary re-scores from its own per-item rows. All results to date were produced under a mock provider, which the document says on its own face.
Not a chatbot, not an AutoGPT descendant, not a paper generator, not a RAG application, not a wrapper around an existing agent framework. There is no conversation between agents. There is a ledger.
Norms are invariants, not instructions. A model asked not to rewrite its hypothesis after seeing results will rewrite its hypothesis after seeing results. So preregistration is a content hash written before dispatch and checked by a foreign key; the test split lives only inside a custodian process; a CHECK constraint makes it impossible for an agent-authored number about the holdout to enter the database at all.
No number passes through a language model. Every statistic is computed by library code. Reports are template-rendered from the database, with prose confined to slots the renderer rejects numerals in.
Agents do not converse. Every action is typed state view → validated artifact → append-only event. Replay, provenance, audit and cost control all fall out of that one choice.
Refutation is a success. refuted and inconclusive are terminal states reported with the same prominence as institutional. Nearly half the question bank has a true effect of exactly zero, so a system that always finds something scores badly.
Confidence is computed, never asserted. It is a function of replication count, effect size over interval width, open critical objections, preregistration status, and holdout queries consumed.
nullius station build draws the institution as the facility it is. It opens on the Control
Room — the biggest room, in the middle, joined by corridor to every other one — which says what
the project is, who is in it, what it has run and what does not exist yet, and hands you a button
into any of the other fourteen. Beyond it: one room per department,
laid out from db/enums.py rather than beside it, so every role is stationed somewhere and every
state of the research machine is owned by exactly one room. It is drawn as a cutaway: each room is
a chamber you look into from the side, with a ceiling, a back wall and a floor its furniture and
its people stand on, and between the chambers is the plant that runs the place — a service deck, a
trunk down the sealed wing, and a hall of tanks and trusses underneath. None of that carries a
number, because none of it records anything. The recorded passes move between the rooms, and the
actors walk between the stations of their own rooms and work at them — in the rooms the arm on
display engages. In the rooms it does not, they stand at their posts. The building runs too: a car
in each shaft, packets in the corridors, screens that change, and a plate on every room's wall
saying what that room is doing — WORKING, NO DATA, LOCKED — in the same words the dossier
uses. At rest there is no other writing on it: hover a room to find out which it is,
click it to open what that department is and what it actually did, on a sheet set in that
department's own face, hue and layout — each dossier opens on a brief
written for a reader who has never heard of a preregistration, with the exact wording one click
away, and every figure naming the artifact it was read out of — and press labels to put the
callouts, the captions and the roster back on. Drag to pan,
scroll to zoom.
The two rooms with no corridor into them are the two you genuinely cannot walk into — the Vault, where the Custodian holds the evaluation split in its own process, and the Oracle, which holds the ground truth the institution may never read. Every way out of the pipeline is drawn the same width as the way in, and every one of them currently reads zero, which is a fact about the code and not about the drawing (ADR-0008).
It builds from committed artifacts alone; point it at a ladder's ledger for the per-agent detail.
nullius station build --out site/station.htmlEarly. Under active construction against a public plan — see BUILD_PLAN.md for milestones and their acceptance criteria, and the badge above for what currently passes.
Nothing here claims to work until its acceptance test is green in CI.
BUILD_PLAN.md |
Milestones, acceptance criteria, environment constraints |
docs/01-critique.md |
Analysis of the originating spec: ambiguities, 19 failure modes, where LLMs may and may not be trusted, prior art |
docs/02-architecture.md |
Agent protocol, research state machine, experiment lifecycle, knowledge representation, research economy |
docs/03-data-model.md |
Schema and the invariants expressed as constraints |
docs/04-evaluation.md |
Question bank, metrics, the B0–B7 baseline ladder |
docs/05-security.md |
Threat model and sandbox design |
docs/06-roadmap.md |
Staged roadmap, complexity estimates, the first experiment |
docs/adr/ |
Decision records, including every deviation from the above |
The research economy landed in M9 with a negative finding, which is the kind this project is built to be able to report. Over the question bank, greedy expected-information-gain allocation shows no measurable advantage over a random allocator on correct claims per dollar, and its numbers are identical to those of a control policy that ignores information entirely and ranks on cost alone.
Two things make that honest rather than merely disappointing. Under a mock provider every role forecasts the same thing about every question, so the information term is constant and cannot have contributed — which is why nullius economy sweep re-runs the comparison with forecasts of dialled-up quality, and greedy-EIG still never separates. And the bank's items all cost within a fraction of a percent of each other, so the comparison has little power: the honest reading is no detectable effect at this power, not no effect.
An earlier version of this paragraph claimed greedy-EIG won. That result sat one hundredth of a unit inside its confidence interval and did not survive a reproducibility fix to the kernel's seed derivation. The project caught it with its own machinery.
nullius economy comparegit clone https://github.com/martex-dev/nullius
cd nullius
uv sync
uv run nullius doctorNo database server, no container runtime and no API key are required to build or test the project. See ADR-0001 and ADR-0005 for why.
The default sandbox (SubprocessSandbox) uses AST validation, an import allowlist, a Python audit hook denying sockets and subprocesses, and hard resource limits. It is not a security boundary against a determined adversary. It is defence in depth against accidental and emergent misbehaviour, which is the actual threat while the MVP executes only our own compiled code (ADR-0004).
Code generation (M12) is gated in code on DockerSandbox being active. Do not run untrusted code under the default backend. See docs/05-security.md.
Apache-2.0. See LICENSE.