A persistent, bounded Go program that investigates a software repository the way a careful scientist would: form competing, falsifiable hypotheses, commit predictions before running an experiment, run a real typed experiment, score the prediction against what actually happened, and durably record the whole chain in SQLite — immutably, with full provenance. The database is the product; the CLI and the model calls are disposable scaffolding around it.
Built by inspecting an April-2023 Auto-GPT/ChaosGPT fork and salvaging
almost none of it. See docs/ARCHAEOLOGY.md for what was there and why.
A fresh prompt to a frontier model can guess whether a test suite passes.
It cannot tell you that, across the last 50 investigations, its stated
70%-confidence hypotheses about "flaky vs. deterministically broken" test
failures were actually right about 40% of the time — because that fact
only exists if someone recorded every prediction, immutably, before the
outcome was known, and scored it honestly afterward. That's what this
project keeps. See docs/ARCHITECTURE.md and docs/EPISTEMIC_MODEL.md
for the full design, and docs/DECISIONS.md for milestone-1's acceptance
criteria and the self-critique behind the scope cuts.
Important scope note: this is a scorecard, and — as of milestone 2 —
a narrow corrector: once there's enough history (10+ scored predictions
across closed investigations) showing the system has been meaningfully
overconfident, the next investigation automatically runs under a
stricter strategy (more competing hypotheses), and exactly why is
recorded alongside it (brier calibration shows the evidence at any
time). It is still narrow — one knob (hypothesis count), reacting to one
signal (aggregate overconfidence), never reacting to underconfidence or
adapting per investigation kind. See docs/ARCHITECTURE.md § Scorecard,
and now a (narrow) corrector for the precise distinction (a common and
reasonable misreading of the project's goal) and docs/DECISIONS.md's
Milestone 2 section for what's deliberately still out of scope.
go build -o brier ./cmd/brier
# Run one full investigation cycle against a repository.
./brier run --db brier.db --repo /path/to/some/go/module \
--objective "does the go test suite pass?" --auto-approve 1
# Inspect the entire recorded chain: observations, hypotheses, predictions,
# the experiment, its result, every score, confidence changes, and findings.
./brier show --db brier.db --id 1
# List all investigations recorded in this database.
./brier list --db brier.db
# See the calibration evidence driving strategy selection right now.
./brier calibration --db brier.dbOmit --auto-approve to be prompted interactively before each experiment
executes — the authorization boundary described in
docs/ARCHITECTURE.md § Safety and control.
v1 uses a deterministic, offline mock model provider by default (no API
key needed, and what every test in this repo runs against); see
internal/llm for the provider interface a real one would implement.
go test ./...Tests directly exercise the invariants in docs/EPISTEMIC_MODEL.md —
e.g. that a raw SQL UPDATE against a scored prediction is rejected by
the database itself, not merely absent from the Go API.
See docs/ARCHITECTURE.md § Package layout.