Support TOML config file by VisruthSK · Pull Request #16 · VisruthSK/foil

VisruthSK · 2026-08-04T09:41:44Z

Closes #2.
Closes #7.
Closes #20.

They previously bypassed the config-to-clap merging entirely, requiring a separate Extras type and hand-rolled TOML matching just for these two fields. Making them ordinary Cli fields lets the same generic default merging handle them, so they can be set at the top level or via CLI flags too, not just inside a benchmark's table.

--benchmark now selects zero or more named benchmarks (repeatable); with none named, every benchmark in the configuration's [benchmarks] table runs, each into its own --output-dir subdirectory, and a one-line-per-benchmark summary prints and writes to report_short.txt. A single selected benchmark, named or not, keeps today's flat layout and full report only.

The main workflow is a b3.toml with named benchmarks and a plain b3 invocation; --benchmark a b c narrows that to a subset without needing --benchmark a --benchmark b --benchmark c.

Command::envs accepts borrowed key/value pairs, so there is no need to clone every entry each time run_in spawns the baseline or candidate.

main() picked suite_output_dir from the first selected benchmark's own resolved output_dir, so a benchmark that overrode output-dir could pull the suite-level summary into its own directory instead of the shared one the README documents. Resolve it from the top-level configuration (or the benchmark's own directory when no top-level default exists) instead of whichever benchmark happened to run first.

… option default_values silently keeps only the first entry when an option's action is Set, so baseline = ["a", "b"] resolved to "a" without any error, unlike the existing empty-list check. Compare against the argument's action and reject anything but a single value for options that cannot repeat.

The multi-benchmark example above the Output section names its benchmarks parse and render; the sample summary output used unrelated names instead.

The top-level Configuration example set command in TOML right next to the [benchmarks] example doing the same thing, with no reason given to prefer one over the other. Command now stays a command-line concern for the single, unnamed case; [benchmarks] is the only TOML path shown, since each entry needs its own command to be selectable. Reordered benchmark keys and switched the example commands to cargo so flag order and example style match across the Usage, top-level Configuration, and benchmarks blocks.

Avoid an eager allocation in read_config's fallback path, and take output_dir by reference in compare since it never needs ownership.

…cing it A benchmark previously had to redefine every variable from the top-level [env] table to add or change just one, since config.extend replaced the env key wholesale like any other option. Merge the two tables key by key instead, letting the benchmark override only what it sets.

…d value The suite-level output directory was read straight from the config file's top-level output-dir key, ignoring an --output-dir argument. With a [benchmarks] table in play, every individual benchmark honored the argument while report_short.txt tried to write into the never-created configured directory instead, crashing after both benchmarks had already run. Capture --output-dir from the recon pass and give it the same precedence over the config file that every other option already has.