Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions CONTRACTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ Each contract is specified once, in its owner repo. Consumers reference that spe

| Contract | Owner | Consumers | Status |
|---|---|---|---|
| Catalog contract: per-implementation layout, metadata schema, pinned Dockerfile, canonical runner, correctness check, result markers | benchmark-tool | catalog-algorithms, catalog-ssg, catalog-web-frameworks | to specify ([benchmark-tool#29](https://github.com/BenchCommons/benchmark-tool/issues/29)) |
| Measurement output: perfratio JSON schema, versioned | perfratio | benchmark-tool | to specify ([perfratio#240](https://github.com/BenchCommons/perfratio/issues/240)) |
| Benchmark inputs: deterministic samples, checksums, versioned releases | dataset-tool | dataset-names, dataset-commons, catalog-algorithms, benchmark-tool | to specify ([dataset-tool#1](https://github.com/BenchCommons/dataset-tool/issues/1)) |
| Published results: `docs/results/<date>/…`, `results.json` schema | benchmark-tool | website | exists for SSGs (`docs/results/README.md`); to generalize ([website#5](https://github.com/BenchCommons/website/issues/5)) |
| Generator index: `generators.json` | catalog-ssg | website, benchmark-tool | exists (`schema/generators.schema.json`) |
| Catalog contract: per-implementation layout, metadata schema, pinned Dockerfile, canonical runner, correctness check, result markers | tools-benchmark | catalog-algorithms, catalog-ssg, catalog-web-frameworks | to specify ([tools-benchmark#29](https://github.com/BenchCommons/tools-benchmark/issues/29)) |
| Measurement output: perfratio JSON schema, versioned | perfratio | tools-benchmark | to specify ([perfratio#240](https://github.com/BenchCommons/perfratio/issues/240)) |
| Benchmark inputs: deterministic samples, checksums, versioned releases | tools-dataset | dataset-names, dataset-commons, catalog-algorithms, tools-benchmark | to specify ([tools-dataset#1](https://github.com/BenchCommons/tools-dataset/issues/1)) |
| Published results: `docs/results/<date>/…`, `results.json` schema | tools-benchmark | website | exists for SSGs (`docs/results/README.md`); to generalize ([website#5](https://github.com/BenchCommons/website/issues/5)) |
| Generator index: `generators.json` | catalog-ssg | website, tools-benchmark | exists (`schema/generators.schema.json`) |
6 changes: 3 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ Initiatives in priority order. Each one is decomposed into specs inside the repo
| # | Initiative | Tracking | Depends on |
|---|---|---|---|
| 1 | Org organization: naming, dot repos, community files, topics | this repo, [.github#1](https://github.com/BenchCommons/.github/issues/1) | — |
| 2 | Catalog contract and multi-catalog toolset | [benchmark-tool#29](https://github.com/BenchCommons/benchmark-tool/issues/29) | 1 |
| 3 | perfratio as the measurement engine | [benchmark-tool#30](https://github.com/BenchCommons/benchmark-tool/issues/30), [perfratio#240](https://github.com/BenchCommons/perfratio/issues/240) | 2 |
| 4 | Benchmark inputs: the `dataset-*` contract and datasets | [dataset-tool#1](https://github.com/BenchCommons/dataset-tool/issues/1), [dataset-names#24](https://github.com/BenchCommons/dataset-names/issues/24), [dataset-commons#1](https://github.com/BenchCommons/dataset-commons/issues/1) | 1 |
| 2 | Catalog contract and multi-catalog toolset | [tools-benchmark#29](https://github.com/BenchCommons/tools-benchmark/issues/29) | 1 |
| 3 | perfratio as the measurement engine | [tools-benchmark#30](https://github.com/BenchCommons/tools-benchmark/issues/30), [perfratio#240](https://github.com/BenchCommons/perfratio/issues/240) | 2 |
| 4 | Benchmark inputs: the `dataset-*` contract and datasets | [tools-dataset#1](https://github.com/BenchCommons/tools-dataset/issues/1), [dataset-names#24](https://github.com/BenchCommons/dataset-names/issues/24), [dataset-commons#1](https://github.com/BenchCommons/dataset-commons/issues/1) | 1 |
| 5 | catalog-algorithms: classic algorithms across languages | [catalog-algorithms#11](https://github.com/BenchCommons/catalog-algorithms/issues/11) | 2, 3, 4 |
| 6 | Continue TechEmpower Framework Benchmarks as catalog-web-frameworks | [.github#2](https://github.com/BenchCommons/.github/issues/2), [catalog-web-frameworks#1](https://github.com/BenchCommons/catalog-web-frameworks/issues/1) | 2 |
| 7 | Results website with several benchmark families | [website#5](https://github.com/BenchCommons/website/issues/5) | 2, 5 |
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0001-repository-naming.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# 0001 — Repository naming by role

## Status
Accepted (2026-10-12).
Accepted (2026-10-12). The naming of tools is superseded by [ADR 0005](0005-tools-prefix.md).

## Context
The org grew from one benchmark (static site generators) to several families. Names like `ssg-frameworks` did not say what role a repo plays, and new repos (algorithms, web frameworks, datasets) needed a convention.
Expand Down
4 changes: 2 additions & 2 deletions docs/adr/0003-default-branch-and-licenses.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
Proposed. Awaiting the maintainer's decision.

## Context
Default branches are mixed: `master` in benchmark-tool, catalog-algorithms, catalog-ssg, dataset-names and perfratio; `main` in the rest.
Default branches are mixed: `master` in tools-benchmark, catalog-algorithms, catalog-ssg, dataset-names and perfratio; `main` in the rest.

Licenses are also mixed: MIT (website, catalog-algorithms), Apache-2.0 (perfratio, dataset-names), Unlicense (catalog-ssg), and TechEmpower's BSD-3 (benchmark-tool; GitHub reports NOASSERTION). Other repos have none: .github, perfratio-website, catalog-web-frameworks, dataset-commons and dataset-tool.
Licenses are also mixed: MIT (website, catalog-algorithms), Apache-2.0 (perfratio, dataset-names), Unlicense (catalog-ssg), and TechEmpower's BSD-3 (tools-benchmark; GitHub reports NOASSERTION). Other repos have none: .github, perfratio-website, catalog-web-frameworks, dataset-commons and tools-dataset.

## Options
- Branch: standardize on `main`. GitHub retargets open PRs and redirects the old name, but local clones, CI filters and docs must be updated. The other option is to keep each repo as it is.
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0004-continue-techempower-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Proposed. The license review is done (BSD-3-Clause, see below); awaiting the maintainer's decision to start the import.

## Context
TechEmpower officially sunset the project on 2026-03-24 ([FrameworkBenchmarks#10932](https://github.com/TechEmpower/FrameworkBenchmarks/issues/10932)) and archived the repo. FrameworkBenchmarks, TFBVerifier, TFBDatabases, TFBToolset and tfb-status are BSD-3-Clause (© TechEmpower, Inc.). benchmark-tool is already derived from its toolset, and `catalog-web-frameworks` exists for web frameworks.
TechEmpower officially sunset the project on 2026-03-24 ([FrameworkBenchmarks#10932](https://github.com/TechEmpower/FrameworkBenchmarks/issues/10932)) and archived the repo. FrameworkBenchmarks, TFBVerifier, TFBDatabases, TFBToolset and tfb-status are BSD-3-Clause (© TechEmpower, Inc.). tools-benchmark is already derived from its toolset, and `catalog-web-frameworks` exists for web frameworks.

## Decision (proposed)
- Import the framework implementations, with their history, directly from `TechEmpower/FrameworkBenchmarks` into `catalog-web-frameworks`, and adapt them to the catalog contract. The old `BenchCommons/FrameworkBenchmarks` fork was removed; the archived upstream is the source.
Expand Down
19 changes: 19 additions & 0 deletions docs/adr/0005-tools-prefix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# 0005 — Tools use the `tools-*` prefix

## Status
Accepted (2026-10-12). Supersedes the tools rule in [ADR 0001](0001-repository-naming.md).

## Context
ADR 0001 gave tools plain names (`benchmark-tool`, `dataset-tool`), while catalogs and datasets have role prefixes. A shared library for the tools also needed a home.

## Decision
- Tools use the `tools-*` prefix, like `catalog-*` and `dataset-*`:
- `benchmark-tool` was renamed to `tools-benchmark`;
- `dataset-tool` was renamed to `tools-dataset`;
- `tools-commons` is new and holds the shared libraries and utilities of the tools, such as contract checkers and common CLI helpers.
- `perfratio` keeps its product name, because it is a standalone tool with its own website and distribution.

## Consequences
- The prefix gives the role of every repo: `catalog-*`, `dataset-*` and `tools-*`. Sites and dot repos keep their names.
- GitHub redirects the old URLs, but submodules, CI checkouts and docs are updated to the new names.
- Code shared by several tools moves to `tools-commons` instead of being copied between them.
Loading