Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions crates/bench-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

`iris-bench`, the command line tool.

Everything this repository can do from a terminal is here and nowhere else, so the tool is the only way anybody runs anything and there is no second path with different defaults on it. `check` says what the machine is and which eligibility gates it passes. `noise`, `overhead` and `resident` are the measurements about the harness itself. `corpus` fetches or generates a pinned corpus into the store. `clickbench` runs the workload against one system, compares the records afterwards, and calibrates them against the public leaderboard.

`run`, `reproduce` and `report` are named and do nothing yet. They are the milestones after this one, and they are in the command list rather than absent from it so that the shape of the finished tool is visible from the help text.

The parts that would be tempting to default are command line arguments instead. Thread count, memory budget, the seed, whether the page cache was dropped, and whether a machine that failed its gates was measured anyway are all in the record, because a reader of a number should not have to know which version of this tool produced it to know what it means.

It refuses to produce a publishable number on a machine that failed its gates, and it says which gate failed. That is not configurable, only overridable in a way that marks the result.

Part of [iris-bench](https://github.com/tamnd/iris-bench). Licensed under Apache-2.0.
5 changes: 3 additions & 2 deletions crates/bench-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -182,11 +182,12 @@ enum Command {
Report,
}

/// The two halves of `clickbench`.
/// The three parts of `clickbench`.
///
/// Running and comparing are separate because the systems are measured one at a time, often on
/// different days, and a comparison that could only happen inside a run would be a comparison that
/// never happened.
/// never happened. Calibrating is separate from both because it reads every record of a run
/// together, and a single record cannot tell a misconfiguration apart from a slower machine.
#[derive(Debug, Subcommand)]
enum ClickbenchCommand {
/// Measure one system.
Expand Down
10 changes: 5 additions & 5 deletions crates/bench-corpus/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# bench-corpus

Corpus manifests, fetching, generation and content addressed storage.
Corpus manifests, fetching, generating, verifying, and the content addressed store.

Every corpus is pinned by a BLAKE3 digest, including the ones that are
Every corpus is pinned by a BLAKE3 digest, including the ones that are generated rather than downloaded. The digest is checked in the same pass that writes the file, so bytes that are not what the manifest promised never land in the store, and a mismatch is a hard failure that prints both digests with no override anywhere. A corpus that quietly changed is worse than one that is missing, because the missing one stops a run and the changed one does not.

generated rather than downloaded. A digest mismatch on fetch is a hard
The asserted row and column counts are checked after the digest, which is what catches a download that stopped early and still parses as a valid file.

failure that prints both digests, because a corpus that quietly changed is
A generated corpus is pinned to the platforms it has actually been produced on, and generating anywhere else is refused with a reason rather than attempted. The generator's own version is checked before it produces a byte, because a different generator writes different bytes and a digest mismatch on its own does not say which of those two things went wrong.

worse than one that is missing.
`docs/CORPORA.md` is the manifest format and what each pinned corpus is.

Part of [iris-bench](https://github.com/tamnd/iris-bench). Licensed under Apache-2.0.
10 changes: 3 additions & 7 deletions crates/bench-report/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
# bench-report

Rendering.
Rendering, including the rules that stop a misleading table being drawn.

The rules that stop a misleading table being drawn are code in this crate
Those rules are code in this crate rather than editorial policy, because a policy is something a person can be in a hurry about. A comparison whose confidence interval crosses one renders as no measurable difference rather than as a winner. A geometric mean never appears without the per query table it came from on the same page. Failed and unsupported queries are shown rather than dropped, since a mean over the queries a system could answer is a different number from a mean over the queries it was asked.

and not editorial policy. A comparison whose interval crosses one renders as

no measurable difference. A geometric mean never appears without its per

query table on the same page. Failed queries are shown rather than dropped.
A result taken on a machine that failed its eligibility gates, or under a configuration this repository had to guess at, renders with that said next to it and not in a footnote. The marker travels with the number wherever the number goes.

Part of [iris-bench](https://github.com/tamnd/iris-bench). Licensed under Apache-2.0.
10 changes: 2 additions & 8 deletions crates/bench-store/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,8 @@

Append only result storage and the claim ledger.

Raw iterations are stored, never summaries. Aggregation happens at read
Raw iterations are stored, never summaries. A store that kept only medians would be a store nobody could re-analyse with a different statistic later, and re-running is expensive enough that the choice of statistic has to stay open long after the machine has moved on to something else.

time, because storing only medians would make it impossible to re-analyse

with a different statistic later and re-running is expensive.

Nothing is deleted. Corrections are new rows that reference the run they

correct.
Nothing is deleted and nothing is edited. A correction is a new row that names the run it corrects, so a result that turned out to be wrong is still readable next to the thing that replaced it. That is what makes the published losses rule mean something: a number that can be quietly removed is a number nobody has to stand behind.

Part of [iris-bench](https://github.com/tamnd/iris-bench). Licensed under Apache-2.0.
Loading