diff --git a/CONTRACTS.md b/CONTRACTS.md index aaf39cd..b352790 100644 --- a/CONTRACTS.md +++ b/CONTRACTS.md @@ -4,8 +4,8 @@ Each contract is specified once, in its owner repo. Consumers reference that spe | Contract | Owner | Consumers | Status | |---|---|---|---| -| Catalog contract: per-implementation layout, metadata schema, pinned Dockerfile, canonical runner, correctness check, result markers | benchmark-tool | catalog-algorithms, catalog-ssg, catalog-web-frameworks | to specify ([benchmark-tool#29](https://github.com/BenchCommons/benchmark-tool/issues/29)) | -| Measurement output: perfratio JSON schema, versioned | perfratio | benchmark-tool | to specify ([perfratio#240](https://github.com/BenchCommons/perfratio/issues/240)) | -| Benchmark inputs: deterministic samples, checksums, versioned releases | dataset-tool | dataset-names, dataset-commons, catalog-algorithms, benchmark-tool | to specify ([dataset-tool#1](https://github.com/BenchCommons/dataset-tool/issues/1)) | -| Published results: `docs/results//…`, `results.json` schema | benchmark-tool | website | exists for SSGs (`docs/results/README.md`); to generalize ([website#5](https://github.com/BenchCommons/website/issues/5)) | -| Generator index: `generators.json` | catalog-ssg | website, benchmark-tool | exists (`schema/generators.schema.json`) | +| Catalog contract: per-implementation layout, metadata schema, pinned Dockerfile, canonical runner, correctness check, result markers | tools-benchmark | catalog-algorithms, catalog-ssg, catalog-web-frameworks | to specify ([tools-benchmark#29](https://github.com/BenchCommons/tools-benchmark/issues/29)) | +| Measurement output: perfratio JSON schema, versioned | perfratio | tools-benchmark | to specify ([perfratio#240](https://github.com/BenchCommons/perfratio/issues/240)) | +| Benchmark inputs: deterministic samples, checksums, versioned releases | tools-dataset | dataset-names, dataset-commons, catalog-algorithms, tools-benchmark | to specify ([tools-dataset#1](https://github.com/BenchCommons/tools-dataset/issues/1)) | +| Published results: `docs/results//…`, `results.json` schema | tools-benchmark | website | exists for SSGs (`docs/results/README.md`); to generalize ([website#5](https://github.com/BenchCommons/website/issues/5)) | +| Generator index: `generators.json` | catalog-ssg | website, tools-benchmark | exists (`schema/generators.schema.json`) | diff --git a/ROADMAP.md b/ROADMAP.md index d5e3017..257ae6c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,9 +5,9 @@ Initiatives in priority order. Each one is decomposed into specs inside the repo | # | Initiative | Tracking | Depends on | |---|---|---|---| | 1 | Org organization: naming, dot repos, community files, topics | this repo, [.github#1](https://github.com/BenchCommons/.github/issues/1) | — | -| 2 | Catalog contract and multi-catalog toolset | [benchmark-tool#29](https://github.com/BenchCommons/benchmark-tool/issues/29) | 1 | -| 3 | perfratio as the measurement engine | [benchmark-tool#30](https://github.com/BenchCommons/benchmark-tool/issues/30), [perfratio#240](https://github.com/BenchCommons/perfratio/issues/240) | 2 | -| 4 | Benchmark inputs: the `dataset-*` contract and datasets | [dataset-tool#1](https://github.com/BenchCommons/dataset-tool/issues/1), [dataset-names#24](https://github.com/BenchCommons/dataset-names/issues/24), [dataset-commons#1](https://github.com/BenchCommons/dataset-commons/issues/1) | 1 | +| 2 | Catalog contract and multi-catalog toolset | [tools-benchmark#29](https://github.com/BenchCommons/tools-benchmark/issues/29) | 1 | +| 3 | perfratio as the measurement engine | [tools-benchmark#30](https://github.com/BenchCommons/tools-benchmark/issues/30), [perfratio#240](https://github.com/BenchCommons/perfratio/issues/240) | 2 | +| 4 | Benchmark inputs: the `dataset-*` contract and datasets | [tools-dataset#1](https://github.com/BenchCommons/tools-dataset/issues/1), [dataset-names#24](https://github.com/BenchCommons/dataset-names/issues/24), [dataset-commons#1](https://github.com/BenchCommons/dataset-commons/issues/1) | 1 | | 5 | catalog-algorithms: classic algorithms across languages | [catalog-algorithms#11](https://github.com/BenchCommons/catalog-algorithms/issues/11) | 2, 3, 4 | | 6 | Continue TechEmpower Framework Benchmarks as catalog-web-frameworks | [.github#2](https://github.com/BenchCommons/.github/issues/2), [catalog-web-frameworks#1](https://github.com/BenchCommons/catalog-web-frameworks/issues/1) | 2 | | 7 | Results website with several benchmark families | [website#5](https://github.com/BenchCommons/website/issues/5) | 2, 5 | diff --git a/docs/adr/0001-repository-naming.md b/docs/adr/0001-repository-naming.md index c30d736..e79f892 100644 --- a/docs/adr/0001-repository-naming.md +++ b/docs/adr/0001-repository-naming.md @@ -1,7 +1,7 @@ # 0001 — Repository naming by role ## Status -Accepted (2026-10-12). +Accepted (2026-10-12). The naming of tools is superseded by [ADR 0005](0005-tools-prefix.md). ## Context The org grew from one benchmark (static site generators) to several families. Names like `ssg-frameworks` did not say what role a repo plays, and new repos (algorithms, web frameworks, datasets) needed a convention. diff --git a/docs/adr/0003-default-branch-and-licenses.md b/docs/adr/0003-default-branch-and-licenses.md index f8d3819..7694ce9 100644 --- a/docs/adr/0003-default-branch-and-licenses.md +++ b/docs/adr/0003-default-branch-and-licenses.md @@ -4,9 +4,9 @@ Proposed. Awaiting the maintainer's decision. ## Context -Default branches are mixed: `master` in benchmark-tool, catalog-algorithms, catalog-ssg, dataset-names and perfratio; `main` in the rest. +Default branches are mixed: `master` in tools-benchmark, catalog-algorithms, catalog-ssg, dataset-names and perfratio; `main` in the rest. -Licenses are also mixed: MIT (website, catalog-algorithms), Apache-2.0 (perfratio, dataset-names), Unlicense (catalog-ssg), and TechEmpower's BSD-3 (benchmark-tool; GitHub reports NOASSERTION). Other repos have none: .github, perfratio-website, catalog-web-frameworks, dataset-commons and dataset-tool. +Licenses are also mixed: MIT (website, catalog-algorithms), Apache-2.0 (perfratio, dataset-names), Unlicense (catalog-ssg), and TechEmpower's BSD-3 (tools-benchmark; GitHub reports NOASSERTION). Other repos have none: .github, perfratio-website, catalog-web-frameworks, dataset-commons and tools-dataset. ## Options - Branch: standardize on `main`. GitHub retargets open PRs and redirects the old name, but local clones, CI filters and docs must be updated. The other option is to keep each repo as it is. diff --git a/docs/adr/0004-continue-techempower-benchmarks.md b/docs/adr/0004-continue-techempower-benchmarks.md index 214c809..a0c18ac 100644 --- a/docs/adr/0004-continue-techempower-benchmarks.md +++ b/docs/adr/0004-continue-techempower-benchmarks.md @@ -4,7 +4,7 @@ Proposed. The license review is done (BSD-3-Clause, see below); awaiting the maintainer's decision to start the import. ## Context -TechEmpower officially sunset the project on 2026-03-24 ([FrameworkBenchmarks#10932](https://github.com/TechEmpower/FrameworkBenchmarks/issues/10932)) and archived the repo. FrameworkBenchmarks, TFBVerifier, TFBDatabases, TFBToolset and tfb-status are BSD-3-Clause (© TechEmpower, Inc.). benchmark-tool is already derived from its toolset, and `catalog-web-frameworks` exists for web frameworks. +TechEmpower officially sunset the project on 2026-03-24 ([FrameworkBenchmarks#10932](https://github.com/TechEmpower/FrameworkBenchmarks/issues/10932)) and archived the repo. FrameworkBenchmarks, TFBVerifier, TFBDatabases, TFBToolset and tfb-status are BSD-3-Clause (© TechEmpower, Inc.). tools-benchmark is already derived from its toolset, and `catalog-web-frameworks` exists for web frameworks. ## Decision (proposed) - Import the framework implementations, with their history, directly from `TechEmpower/FrameworkBenchmarks` into `catalog-web-frameworks`, and adapt them to the catalog contract. The old `BenchCommons/FrameworkBenchmarks` fork was removed; the archived upstream is the source. diff --git a/docs/adr/0005-tools-prefix.md b/docs/adr/0005-tools-prefix.md new file mode 100644 index 0000000..799d8b8 --- /dev/null +++ b/docs/adr/0005-tools-prefix.md @@ -0,0 +1,19 @@ +# 0005 — Tools use the `tools-*` prefix + +## Status +Accepted (2026-10-12). Supersedes the tools rule in [ADR 0001](0001-repository-naming.md). + +## Context +ADR 0001 gave tools plain names (`benchmark-tool`, `dataset-tool`), while catalogs and datasets have role prefixes. A shared library for the tools also needed a home. + +## Decision +- Tools use the `tools-*` prefix, like `catalog-*` and `dataset-*`: + - `benchmark-tool` was renamed to `tools-benchmark`; + - `dataset-tool` was renamed to `tools-dataset`; + - `tools-commons` is new and holds the shared libraries and utilities of the tools, such as contract checkers and common CLI helpers. +- `perfratio` keeps its product name, because it is a standalone tool with its own website and distribution. + +## Consequences +- The prefix gives the role of every repo: `catalog-*`, `dataset-*` and `tools-*`. Sites and dot repos keep their names. +- GitHub redirects the old URLs, but submodules, CI checkouts and docs are updated to the new names. +- Code shared by several tools moves to `tools-commons` instead of being copied between them.