diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 4a040430..c9720462 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -167,6 +167,18 @@ the glue that lets `parsers` write into any builder's target without importing c Scans fixture trees and compares against `testdata/integration/golden/*.json`. Tests for enhancements, malformed input, the petstore, aliased schemas, go123-specific forms, and cross-feature coverage. +### `internal/benchmarks/` — what a scan costs + +Measures two go-swagger-generated projects that ship with the repo as one archive +(`testdata/corpus.tgz`, unpacked on demand by the `corpus` package): `dockerctl`, a light client over +a reasonably large API, and `kubeapi`, a heavy server over a very large one. Both vendor their +dependencies, so a measurement needs a clone and nothing else. Everything is gated behind +`CODESCAN_BENCH=1` or run by hand — none of it runs in CI, where a benchmark would measure the +runner. `loader-benchmark/` is the half that measures *released* versions too: it talks only to the +public `Run` API, so `run.sh` compiles the same source against each release tag and against the +working tree. `README.md` carries the results — the loader comparison and the history since v0.33.3 — +and is what `internal/scanner/README.md#loader` cites for its figures. + ### `testdata/` - `testdata/goparsing/...` — historic corpus: classification, petstore, go118/go119/go123 variants, invalid inputs. diff --git a/.gitignore b/.gitignore index 4374ae89..508f9030 100644 --- a/.gitignore +++ b/.gitignore @@ -40,6 +40,9 @@ internal/exportdata/exportdata.zip /genspec-wasi /loader-benchmark -# loader-benchmark scratch: generated baseline module, built probes, raw measurements -hack/loader-benchmark/.work/ -hack/loader-benchmark/results/ +# benchmark corpora: the archive is committed, the tree it unpacks to is not +internal/benchmarks/testdata/corpus/ + +# loader-benchmark scratch: generated version modules, built probes, raw measurements +internal/benchmarks/loader-benchmark/.work/ +internal/benchmarks/loader-benchmark/results/ diff --git a/docs/doc-site/maintainers/_index.md b/docs/doc-site/maintainers/_index.md index 9f3643f0..143e0c6e 100644 --- a/docs/doc-site/maintainers/_index.md +++ b/docs/doc-site/maintainers/_index.md @@ -42,3 +42,6 @@ annotation to both its tutorial and its entry here. - **[The commands]({{% relref "commands" %}})** — how the three CLI tools are put together: why each lives where it does, where their shared flag surface is declared, and what keeps it whole. +- **[Performance]({{% relref "performance" %}})** — what a scan costs: what the + grammar and the loader each changed since the code left go-swagger, and how the + loader options compare warm and cold. diff --git a/docs/doc-site/maintainers/performance.md b/docs/doc-site/maintainers/performance.md new file mode 100644 index 00000000..f9fccd29 --- /dev/null +++ b/docs/doc-site/maintainers/performance.md @@ -0,0 +1,156 @@ +--- +title: "Performance" +weight: 60 +description: | + What a scan costs, and what six months of work did to it. + + The two independent gains — the annotation parser and the package loader — measured on the same generated server, emitting the same document. +--- + +codescan was extracted from [go-swagger](https://github.com/go-swagger/go-swagger) +and has been reworked twice since, in ways that turned out to matter for what a +scan costs. This page is the short version of the evidence: three points in time, +then the loader options available today, measured on one corpus. + +The full method, both corpora and the raw tables live in +[`internal/benchmarks`](https://github.com/go-openapi/codescan/tree/master/internal/benchmarks). + +## The corpus + +Everything below scans **kubeapi**: an API *server stub* that go-swagger generates +from the kubernetes API manifest — not kubernetes itself, and not code that ever +runs. It is here because of its shape: 2352 Go files, ~344k lines, and an +annotated surface of 222 model definitions and **260 route/operation blocks**, +which is as much work as an annotation parser is ever asked to do. + +Every figure on this page comes from a scan that emitted the **same 222 +definitions and 260 paths**. A configuration that got faster by scanning less +would not be an improvement, so the emitted document is checked alongside the +cost. + +## Three points in time + +{{% notice style="note" title="Versions" %}} +**v0.33.3** is roughly the state the code was extracted from go-swagger in. +**v0.35.1** is the first debugged release of the grammar-based annotation parser. +**current** is master in its default configuration. +{{% /notice %}} + +```mermaid +xychart-beta + title "Time for one scan of kubeapi (warm build cache, seconds)" + x-axis ["v0.33.3", "v0.35.1", "current"] + y-axis "seconds" 0 --> 8 + bar [7.087, 1.758, 0.970] +``` + +```mermaid +xychart-beta + title "Memory: total allocated (bars) and peak RSS (line), MB" + x-axis ["v0.33.3", "v0.35.1", "current"] + y-axis "MB" 0 --> 4800 + bar [4555, 1217, 447] + line [1154, 807, 306] +``` + +End to end: **7.3× faster and 10× less allocated**, for the identical document. + +The two charts do not tell the same story, and the difference is the interesting +part. The step from v0.33.3 to v0.35.1 removes 3.3 GB of *allocation* but only +346 MB of *peak* — the old regexp-based parser allocated and discarded, so the +high-water mark stayed near the floor that the loaded package graph sets. The +step to current lowers that floor instead. + +### What each step changed + +**v0.33.3 → v0.35.1 — the annotation parser.** The +[grammar]({{% relref "grammar" %}}) replaced a regexp engine. That was a +correctness, diagnostics and completeness project, not a performance one, and the +speed is a byproduct of dropping Go's `regexp` from the hot path. It is worth +knowing how large the byproduct is on route-heavy code: scanning only this +corpus's models — nothing to emit — the two versions allocate the same to within +0.5 MB, while scanning the routes goes from 4476 MB to 1216 MB. The whole gain is +in reading annotations, and it appears where route and operation bodies do. + +**v0.35.1 → current — the package loader.** Resolving and type-checking the +package graph is the bulk of a scan, and it is now possible to ask for a loader +that reads less of it. That is the next section. + +## How each loader option fares + +Three ways to get the package graph, all producing the same document. What they +disagree about is what they read and what they hold. + +Note the axes: the two charts below are the *same* three configurations, timed on +a warm build cache and on an empty one. Only the scale differs — and it differs +by an order of magnitude. + +```mermaid +xychart-beta + title "Time for one scan, WARM build cache (seconds)" + x-axis ["source deps", "pure-Go", "compiled deps"] + y-axis "seconds" 0 --> 2.5 + bar [1.506, 1.331, 0.970] +``` + +```mermaid +xychart-beta + title "Time for one scan, COLD build cache (seconds)" + x-axis ["source deps", "pure-Go", "compiled deps"] + y-axis "seconds" 0 --> 15 + bar [2.208, 1.359, 14.511] +``` + +```mermaid +xychart-beta + title "Memory: total allocated (bars) and peak RSS (line), MB" + x-axis ["source deps", "pure-Go", "compiled deps"] + y-axis "MB" 0 --> 1300 + bar [1215, 643, 447] + line [751, 412, 306] +``` + +| configuration | option | warm | cold | allocated | peak RSS | build cache it writes | +|---|---|---|---|---|---|---| +| source dependencies | `SkipCompiledDependencies` | 1.506 s | 2.208 s | 1215 MB | 751 MB | 7.7 MB | +| pure-Go loader | `ToolchainFreeLoader` | 1.331 s | **1.359 s** | 643 MB | 412 MB | **4 KB** | +| compiled dependencies | *the default* | **0.970 s** | 14.511 s | **447 MB** | **306 MB** | 231 MB | + +**No configuration wins both cache states**, which is the whole reason there is a +choice to make: + +- **Compiled dependencies** take dependency types from the compiler instead of + reading their source — 19 packages read from source rather than 296. Fastest and + smallest by a wide margin on a warm cache. On a cold one it must *compile* the + closure before it can read it, so it is 6× slower than reading source and writes + 231 MB of build cache. It is the default; `SkipCompiledDependencies` opts out, + and a CI job regenerating a spec from a clean checkout is what that is for. +- **The pure-Go loader** never invokes the go command, so there is no metadata to + populate and nothing to compile: it writes 4 KB of build cache and **its cold + time equals its warm time**. It is the only choice whose cost is predictable, + and it holds about 45% less memory than the standard loader. + +Which to reach for, and what each one gives up, is on the +[Options reference]({{% relref "options-reference" %}}). + +## Reading these numbers + +They are indicative, not a promise. Measured on one machine (Ryzen 7 5800X, +31 GB, go1.26.5, Linux), three warm rounds and a single cold one — cold being a +single-shot state by definition. + +Two cautions carry over to any corpus of your own: + +- **The two gains scale with different things.** The loader acts on the + *dependency closure*, so its win is roughly a fixed amount per project; the + parser acts on the *annotated surface*, so its win is large on a route-heavy + server and small on a client carrying annotations only on its models. On a + smaller tree the loader's share of the total looks much bigger than it does + here. +- **Wall clock needs an idle machine.** Under load these timings move by 3× while + the allocation and peak-RSS figures reproduce to four digits. Distrust a timing + difference under 10%; nothing concluded above rests on one. + +To measure your own tree, the harness takes it as an extra corpus alongside the +two it ships with — see its +[README](https://github.com/go-openapi/codescan/tree/master/internal/benchmarks). diff --git a/docs/doc-site/usage/options-reference.md b/docs/doc-site/usage/options-reference.md index bba476fd..a30259fc 100644 --- a/docs/doc-site/usage/options-reference.md +++ b/docs/doc-site/usage/options-reference.md @@ -104,10 +104,9 @@ it does not cover falls back to source and then to synthesis. {{% notice style="note" %}} The percentages are indicative, not a promise: the balance moves with the size of the tree being scanned, and on a small one the pure-Go loader is *slower* warm than -the standard loader. Measure your own corpus with -[`hack/loader-benchmark`](https://github.com/go-openapi/codescan/tree/master/hack/loader-benchmark), -which compares a released version against a working tree and carries the tables and -the method. +the standard loader. The figures, the corpora they were taken on and the method are in +[`internal/benchmarks`](https://github.com/go-openapi/codescan/tree/master/internal/benchmarks), +whose harness also takes an extra corpus of your own to measure alongside them. {{% /notice %}} | Option | Type | Default | Flag | Section | Effect | diff --git a/hack/loader-benchmark/README.md b/hack/loader-benchmark/README.md deleted file mode 100644 index fcc514f4..00000000 --- a/hack/loader-benchmark/README.md +++ /dev/null @@ -1,205 +0,0 @@ -# loader-benchmark - -Measures what a codescan scan costs, comparing a **released baseline against the working tree** over -external corpora, warm and cold. - -It exists because the benchmarks under `internal/integration/` cannot answer *"how much did we -improve?"*. Those live inside the module, so they do not exist in any earlier release; they can -measure this version against itself and nothing else. This harness talks only to the public -`codescan.Run` API — the one surface every version has — so the same source compiles against a -release tag and against the working tree. - -Not run in CI, by design. On shared runners a benchmark measures the runner. This is an operator -tool, run deliberately, roughly once per minor release. - -**Looking for which loader to use, rather than for the evidence?** The primer is -`internal/scanner/README.md#loader`; this document is where its figures come from. - -## Running it - -Corpora are external, large, and never vendored here — the point of measuring them is that they are -real trees. Name each one through the environment; anything unset is skipped by name rather than -silently defaulting to some other checkout. - -```sh -export CODESCAN_BENCH_DOCKERCTL_DIR=/path/to/generated/client -export CODESCAN_BENCH_KUBEAPI_DIR=/path/to/generated/server -export CODESCAN_BENCH_KUBEAPI_VENDORED_DIR=/path/to/the/same/server/vendored - -./run.sh # warm matrix -CODESCAN_BENCH_COLD=1 ./run.sh # add the cold pass (slow — it compiles dependency closures) -``` - -| variable | meaning | default | -|---|---|---| -| `CODESCAN_BENCH_BASELINE` | release the working tree is compared against | `v0.36.3` | -| `CODESCAN_BENCH_ROUNDS` | measured rounds per warm cell | `3` | -| `CODESCAN_BENCH_COLD` | `1` adds the cold-cache pass | off | -| `CODESCAN_BENCH_PATTERN` | package pattern handed to the scan | `./...` | - -Raw measurements accumulate in `results/` as JSON lines, and the table is a pure function of them: - -```sh -go run . -summarize results/warm.jsonl,results/cold.jsonl -``` - -Both `results/` and `.work/` (the generated baseline module and built probes) are gitignored. - -## How it measures, and why that way - -**One scan per process.** Peak RSS is a process-wide high-water mark, so a second scan in the same -process would report the first one's peak plus its own garbage. - -**A discarded warm-up per configuration, then alternating rounds.** Measuring config-by-config lets -cache and thermal drift land entirely on whichever ran last; alternating spreads it evenly. - -**Shape is recorded with cost.** Every row carries its definition and path counts, because a -configuration that got faster by *scanning less* is not an improvement. In the results below every -configuration emits the identical document for its corpus. - -**The cold pass gives each cell a private empty `GOCACHE`** and leaves `GOMODCACHE` alone, so it -isolates the build cache and nothing else. The operator's own cache is never touched. - -**`retained` is not the loader's peak.** It is measured after `Run` returns, when the package graph is -garbage and only the spec is live — a couple of MB in every configuration. Use peak RSS and total -allocation for the memory story. - -### Two traps worth knowing - -**The baseline build must be built with `GOWORK=off`.** The workspace at the repository root would -substitute the local tree for the released library and quietly measure the working tree twice, with a -label claiming otherwise. `run.sh` handles this; anything reimplementing it must too. - -**Options younger than the baseline sit behind a build tag.** `options_current.go` sets them; -`options_baseline.go` (tag `baseline`) accepts the same flags and *exits* rather than ignoring them. -A baseline binary that silently accepted `-compiled-deps` would emit a default-configuration -measurement under a label saying otherwise — an error invisible in the output. - -## Results - -Measured 2026-08-08, go1.26.5, baseline **v0.36.3**, on one machine (Ryzen 7, 31 GB). v0.36.3 has -exactly one way to load and scan, which is what makes it a baseline. - -**Rows are labelled by configuration, not by default, and they predate v0.36.4 flipping the default.** -So `current` is the source-loading scan, which is now what `SkipCompiledDependencies` selects, and -`current + CompiledDependencies` is what a plain run does today. The harness flag keeps meaning what -it says for the same reason — the measurements stay comparable across the change. - -| corpus | shape | `.go` files | emitted | -|---|---|---|---| -| `dockerctl` | generated client for a container-engine API | ~1490 | 198 defs / 0 paths | -| `kubeapi` | larger generated server, module, not vendored | ~2350 | 222 defs / 260 paths | -| `kubeapi-vendored` | the same tree with vendored dependencies | +646 (14 MB) | 222 defs / 260 paths | - -### Warm build cache (n=3) - -| configuration | wall | Δ | peak RSS | Δ | total alloc | Δ | -|---|---|---|---|---|---|---| -| **dockerctl** — v0.36.3 | 1.228 s | — | 684 MB | — | 1103 MB | — | -| current | 0.995 s | −19% | 682 MB | 0% | 1103 MB | 0% | -| current + `ToolchainFreeLoader` | 1.131 s | −8% | 394 MB | −42% | 585 MB | −47% | -| current + `CompiledDependencies` | 0.452 s | −63% | 169 MB | −75% | 237 MB | −78% | -| **kubeapi** — v0.36.3 | 1.704 s | — | 768 MB | — | 1214 MB | — | -| current | 1.430 s | −16% | 753 MB | −2% | 1214 MB | 0% | -| current + `ToolchainFreeLoader` | 1.309 s | −23% | 426 MB | −45% | 644 MB | −47% | -| current + `CompiledDependencies` | 0.916 s | −46% | 307 MB | −60% | 446 MB | −63% | -| **kubeapi-vendored** — v0.36.3 | 1.763 s | — | 752 MB | — | 1214 MB | — | -| current | 1.546 s | −12% | 754 MB | 0% | 1214 MB | 0% | -| current + `ToolchainFreeLoader` | 1.333 s | −24% | 422 MB | −44% | 640 MB | −47% | -| current + `CompiledDependencies` | 0.966 s | −45% | 303 MB | −60% | 446 MB | −63% | - -### Cold build cache (n=1) - -| configuration | cold wall | Δ vs baseline | build cache it produces | -|---|---|---|---| -| **dockerctl** — v0.36.3 | 1.987 s | — | 7.4 MB | -| current | 1.793 s | −10% | 7.4 MB | -| current + `ToolchainFreeLoader` | **1.228 s** | **−38%** | **4 KB** | -| current + `CompiledDependencies` | 10.695 s | **+438%** | 216 MB | -| **kubeapi** — v0.36.3 | 2.472 s | — | 7.6 MB | -| current | 2.204 s | −11% | 7.6 MB | -| current + `ToolchainFreeLoader` | **1.482 s** | **−40%** | **4 KB** | -| current + `CompiledDependencies` | 13.838 s | **+460%** | 229 MB | -| **kubeapi-vendored** — v0.36.3 | 2.495 s | — | 7.4 MB | -| current | 2.260 s | −9% | 7.4 MB | -| current + `ToolchainFreeLoader` | **1.391 s** | **−44%** | **4 KB** | -| current + `CompiledDependencies` | 15.631 s | **+527%** | 229 MB | - -Memory is unaffected by cache state and is omitted from the cold table; it matches the warm figures. - -## Analysis - -### 1. The default path does not prune, and that is the whole explanation - -Default to default, the current tree is 12–19% faster with **total allocation identical to four -digits** and peak RSS inside the run-to-run spread. - -That is not a prefilter defeated by annotation-dense corpora — that would show *partial* savings. -It is that pruning lives in the toolchain-free importer (which chooses source or export data per -import) and in `CompiledDependencies` (which takes dependency types from export data), and the -default configuration selects **neither**. Ask for a pruning loader and the pruning appears at once: -−47% allocation, on every corpus, invariably. - -So the default's gain is **CPU, not memory** — the same bytes allocated, less time spent on them. -A user who upgrades and changes nothing should expect a modest time win and no memory win. The −75% -figure belongs to an opt-in option and must never be quoted against the default path. - -### 2. Corpus size decides whether the pure-Go loader is also faster - -Its memory advantage is flat — −42 to −46%, every corpus, both vendoring modes. Wall clock is not: -on the smaller generated client the toolchain-free route is *slower* than the default (1.131 s vs -0.995 s, the concurrency gap its implementation notes call out), and on the larger server it is -*faster* (1.309 s vs 1.430 s). Scale does not change the memory story; it decides whether the -wall-clock deficit survives. It does not. - -### 3. The toolchain-free loader is cache-independent — the strongest result here - -It produces **4 KB** of build cache and its cold time equals its warm time, because it never invokes -the go command: there is no metadata to populate and nothing to compile. Every other configuration is -priming a toolchain it depends on. - -This inverts the ranking, and no configuration wins both states: - -| cache | fastest → slowest | -|---|---| -| **warm** | `CompiledDependencies` · toolchain-free · default · v0.36.3 | -| **cold** | **toolchain-free** · default · v0.36.3 · `CompiledDependencies` | - -Cold, the toolchain-free route is 38–44% faster than the baseline and ~25% faster than the current -default, while holding 45% less memory. It is the only configuration whose cost is *predictable*, -which for a tool that runs in CI is worth more than a warm best case. - -### 4. Compiled dependencies: the default, and when to turn it off - -Fastest warm by a wide margin (−45 to −63%), and **4.4× to 6.3× slower than the baseline cold**, -materialising up to 229 MB of build cache to get there — `go list -export` must *compile* the -closure, not merely type-check it. A first scan of a freshly generated tree is precisely its worst -case, and CI is cold by definition. - -What it does not cost is meaning: it emits the same document on every corpus measured here and -across the whole fixture corpus. Whatever the spec needs out of a dependency — its own annotations, -or a declaration the scanned code names — is read, the second kind at the lookup that wants it. That -is what made it the default in v0.36.4. See `internal/scanner/README.md#compiled-dependencies`. - -Nor does it cost the ability to scan code that does not compile: needing `go list` to build the -scanned packages would resurrect go-swagger#2874, so a load that fails that way is retried from -source. The retry costs a second load, and only on a tree that was not going to build. - -`SkipCompiledDependencies` opts out, and the cold column is the reason to reach for it. - -## Caveats - -- **Vendoring is inconclusive here, not a win.** Comparing the vendored tree against the same tree - unvendored, vendoring moves everything by a few percent and helps peak RSS only on the - toolchain-free route. 14 MB of vendored source against a 2350-file module is too small a share of - the closure to magnify anything. A corpus whose dependency closure dominates its own source would - be needed to make this axis speak. -- **The cold pass is n=1** — cold is a single-shot state by definition. Its wall-clock figures carry - the run-to-run spread visible in the warm tables (a few percent), which does not threaten - conclusions drawn from 4× gaps but does mean small cold deltas should not be read closely. -- **The workload looks RAM-bound rather than CPU-bound.** Figures taken on a much older CPU with - comparable RAM and SSD reproduce the cold numbers closely. Treat CPU model as a weak predictor - here, and memory bandwidth as the thing to hold constant when comparing machines. -- **A corpus that fails to scan still measures a load.** The harness reports the error and marks the - row `SCAN FAILED` rather than dropping it — but such a row measures load-then-abort, not a scan, - and must not be compared against a clean one. diff --git a/internal/benchmarks/README.md b/internal/benchmarks/README.md new file mode 100644 index 00000000..cf4c2ff3 --- /dev/null +++ b/internal/benchmarks/README.md @@ -0,0 +1,297 @@ +# benchmarks — what a scan costs + +What codescan pays to turn a Go tree into a Swagger document, measured on two real generated +projects that ship with this repository. + +Two questions, and the whole of this document answers them: + +1. **Which loader should I use?** They differ by 3× in memory and by an order of magnitude in time, + and no single one wins in every state of the build cache. +2. **Has this got better?** Yes — a scan of the larger corpus allocates **3.7× less** than it did in + v0.33.3 and takes **4.7× less** wall clock, emitting the identical document. + +## The corpus + +Both trees are go-swagger-generated projects. They ship as one archive +(`testdata/corpus.tgz`) and unpack on demand, so a measurement needs a clone and nothing else — no +external checkout, no environment variable naming somebody's home directory, no network. Both +vendor their dependencies, which is what makes that last one true. + +| corpus | shape | own `.go` files | LOC | vendored | emits | +|---|---|---|---|---|---| +| `dockerctl` | generated **client** for the docker engine API | 501 | 161k | 985 files | 198 definitions, 0 paths | +| `kubeapi` | generated **server** for the kubernetes API | 2352 | 344k | 646 files | 222 definitions, 260 paths | + +Two shapes on purpose. `dockerctl` is light code over a reasonably large API and carries its +annotations on models only; `kubeapi` is heavy code over a very large API, and 260 route/operation +blocks means the annotation grammar has real work to do. A conclusion that only holds on one of them +is not a conclusion — and, as the history section shows, the two disagree sharply. + +Every table below carries the emitted **shape** (`198d/0p`) alongside the cost, because a +configuration that got faster by scanning less is not an improvement. Across every row of every +table, each corpus emits its identical document. + +## Choosing a loader + +Three configurations of the working tree, the same document out of each. + +### Warm build cache (n=3) + +| configuration | wall | peak RSS | allocated | +|---|---|---|---| +| **dockerctl** — source dependencies | 0.972 s | 679 MB | 1103 MB | +| pure-Go loader (`ToolchainFreeLoader`) | 1.134 s | **403 MB** | **589 MB** | +| compiled dependencies *(the default)* | **0.438 s** | **170 MB** | **237 MB** | +| **kubeapi** — source dependencies | 1.506 s | 751 MB | 1215 MB | +| pure-Go loader (`ToolchainFreeLoader`) | 1.331 s | 412 MB | 643 MB | +| compiled dependencies *(the default)* | **0.970 s** | **306 MB** | **447 MB** | + +### Cold build cache (n=1, private empty `GOCACHE`) + +| configuration | wall | build cache it writes | +|---|---|---| +| **dockerctl** — source dependencies | 1.490 s | 7.5 MB | +| pure-Go loader | **1.198 s** | **4 KB** | +| compiled dependencies | 9.361 s | 216 MB | +| **kubeapi** — source dependencies | 2.208 s | 7.7 MB | +| pure-Go loader | **1.359 s** | **4 KB** | +| compiled dependencies | 14.511 s | 231 MB | + +Memory does not depend on cache state and is omitted here; it matches the warm figures. + +### Reading it + +**Compiled dependencies win warm and lose cold, by a lot.** Taking dependency types from the +compiler instead of their source is 35–55% faster and 2.5–4× smaller on a warm cache. On a cold one +it is **6× slower** than reading source, because `go list -export` must *compile* the closure rather +than type-check it, and it materialises up to 231 MB of build cache doing so. It is the default +since v0.36.4, and `SkipCompiledDependencies` is the opt-out; CI regenerating a spec from a clean +checkout is the case that wants it. + +**The pure-Go loader is the only one whose cost is predictable.** It writes 4 KB of build cache and +its cold time equals its warm time, because it never invokes the go command — there is no metadata +to populate and nothing to compile. Everything else is priming a toolchain it depends on. That +inverts the ranking, and no configuration wins both states: + +| cache | fastest → slowest | +|---|---| +| **warm** | compiled dependencies · source dependencies · pure-Go loader | +| **cold** | **pure-Go loader** · source dependencies · compiled dependencies | + +Its memory advantage is flat — 45–47% less allocated on both corpora — while its wall clock is +corpus-dependent: *slower* than the standard loader warm on the smaller client, *faster* on the +larger server. Scale decides whether the wall-clock deficit survives; on the larger corpus it does +not. + +**Where the memory goes.** The load ladders (`bench_loaders_test.go`) report the shape behind those +figures — how much of the closure ends up as parsed syntax: + +| corpus | configuration | closure | from source | export-served | AST files | retained | +|---|---|---|---|---|---|---| +| dockerctl | source dependencies | 335 | 334 | 1 | 2353 | 513 MB | +| dockerctl | compiled dependencies | 335 | 19 | 22 | 510 | 106 MB | +| dockerctl | pure-Go loader | 321 | 321 | 0 | 2286 | 277 MB | +| kubeapi | source dependencies | 297 | 296 | 1 | 3908 | 548 MB | +| kubeapi | compiled dependencies | 297 | 31 | 37 | 2368 | 188 MB | +| kubeapi | pure-Go loader | 283 | 283 | 0 | 3842 | 278 MB | + +Reading 19 packages from source instead of 334 is the entire compiled-dependencies win, and it costs +no meaning: a dependency whose source carries annotations is read anyway, and a declaration the spec +needs is fetched at the lookup that wants it. See +[`internal/scanner/README.md#compiled-dependencies`](../scanner/README.md#compiled-dependencies). + +The pure-Go loader resolves a slightly smaller closure than `go list` (321 vs 335, 283 vs 297). That +divergence is real and reproducible, belongs to the loader, and does not change the emitted document +on either corpus. + +**Which one to pick** is answered in prose, with the same figures, at +[`internal/scanner/README.md#loader`](../scanner/README.md#loader). + +## Six months of scans + +The same measurement against the working tree and against two released versions, each built into its +own throwaway module so it links the released library rather than this checkout. + +| version | landed | what changed | +|---|---|---| +| `v0.33.3` | the state codescan was extracted from go-swagger in | regexp-based annotation parsing | +| `v0.35.1` | the grammar parser, debugged | annotations parsed by a real grammar, with diagnostics | +| `current` | the loader work | dependency types from export data; a second loader | + +### Warm, same document at every point + +| configuration | dockerctl wall | alloc | kubeapi wall | alloc | +|---|---|---|---|---| +| v0.33.3 | 1.593 s | 1316 MB | 7.087 s | 4555 MB | +| v0.35.1 | 1.226 s | 1103 MB | 1.758 s | 1217 MB | +| current, source dependencies | 0.972 s | 1103 MB | 1.506 s | 1215 MB | +| current, compiled dependencies *(default)* | **0.438 s** | **237 MB** | **0.970 s** | **447 MB** | + +End to end, six months moved the larger corpus from 7.1 s / 4555 MB to 0.97 s / 447 MB — **7.3× +faster, 10× less allocated** — for the identical 222 definitions and 260 paths. + +### The two halves moved for different reasons, on different corpora + +The rows above split cleanly in two, and the split is the interesting part. + +**v0.33.3 → v0.35.1 is the annotation parser, and it is only visible where there are routes.** On +`dockerctl` (no paths) the change is 16%; on `kubeapi` (260 paths) it is 3.7×. Isolating it with two +sub-patterns of the same corpus settles what that means: + +| scan | emits | v0.33.3 | current | +|---|---|---|---| +| `kubeapi/./models/...` — nothing to build | 0 defs / 0 paths | 764.6 MB | 764.1 MB | +| `kubeapi/./restapi/...` — the whole document | 222 defs / 260 paths | 4475.6 MB | 1215.8 MB | + +With nothing to emit, the two versions allocate the same to within 0.5 MB: the loading half is +unchanged, as it must be — both call the same `go/packages`. The whole 3.3 GB difference is the +phase that reads annotations and emits Swagger, and it appears only where route and operation bodies +do. The grammar migration was never a performance project — it was about correctness, diagnostics +and completeness — so this is a byproduct of dropping Go's `regexp` from the hot path, and it is +worth knowing that the byproduct is this large on route-heavy code. + +**v0.35.1 → current is the loader**, which changes what a scan *holds* rather than what it does with +it. + +### Reading the two gains + +Each step against its predecessor, except the last two, which are both against v0.35.1 — they are +alternative configurations of the same tree, not successive ones. + +| step | dockerctl wall / alloc / peak RSS | kubeapi wall / alloc / peak RSS | +|---|---|---| +| v0.33.3 → v0.35.1 — the parser | −0.367 s / **−213 MB** / −188 MB | **−5.329 s** / **−3338 MB** / −346 MB | +| v0.35.1 → current, source dependencies | −0.254 s / −0.5 MB / +13 MB | −0.252 s / −1.5 MB / −56 MB | +| v0.35.1 → pure-Go loader | −0.092 s / −514 MB / −263 MB | −0.427 s / −573 MB / −395 MB | +| v0.35.1 → compiled dependencies | **−0.788 s** / −866 MB / **−497 MB** | −0.788 s / −769 MB / **−501 MB** | + +Three readings, because the headline table on its own looks like a contradiction. + +**Which change won "on memory" depends on which memory column.** The parser's win is overwhelmingly +*churn*: removing 3338 MB of allocation on kubeapi moved the high-water mark by 346 MB, because the +regexp engine allocated and discarded, and the collector kept the peak near the floor that the +retained package graph sets. The loader attacks that floor instead — about −500 MB of peak on both +corpora. So total allocation makes the parser the larger win, by 4× on kubeapi, and peak RSS makes +the loader the larger win everywhere. + +**The parser won wall clock too, and on route-heavy code it won most of it** — 5.33 s of kubeapi's +6.12 s. On dockerctl, with models to read and no routes, the ranking is the other way round (−0.37 s +against −0.79 s). "The parser bought memory and the loader bought time" is dockerctl's story, not a +general one. + +**The two scale with different things**, which is what makes them predictable on a corpus resembling +neither: + +- the **loader** acts on the dependency *closure*, similar in size for both (335 and 297 packages), + so its win is near-constant in absolute terms — −866 / −769 MB allocated, −497 / −501 MB of peak, + and −0.788 s on both, to three digits; +- the **parser** acts on the *annotated surface* — −213 MB where there are no paths, −3338 MB where + there are 260. + +So the loader's share of the total looks large on a small project and the parser's on an +annotation-dense one, and on a large generated server both matter, for unrelated reasons. + +The middle row deserves its own note: in the same configuration, v0.35.1 → current is +allocation-neutral to within 2 MB and still a quarter-second faster on both corpora. Nothing about +pruning reached the default path over that stretch — only speed. Every memory figure below that row +comes from *asking* for a loader that prunes. + +### What a scan is made of today + +Per phase, in the default configuration, warm: + +| corpus | Load | Build | Build's share | +|---|---|---|---| +| dockerctl | 415 ms · 223 MB | 33 ms · 14.7 MB | 7% of time, 6% of allocation | +| kubeapi | 802 ms · 390 MB | 156 ms · 56.9 MB | 16% of time, 13% of allocation | + +Load is resolving and type-checking the package graph; Build is reading annotations and emitting the +document. Loading still dominates, which is why the loader is where the remaining leverage is — and +why a parser micro-optimisation is worth at most a tenth of a scan. + +## Running it + +Everything here is an operator tool, run deliberately. None of it runs in CI: on a shared runner a +benchmark measures the runner. + +### The cross-version matrix + +```sh +cd internal/benchmarks/loader-benchmark +./run.sh # warm matrix, every corpus and configuration +CODESCAN_BENCH_COLD=1 ./run.sh # add the cold pass (slow: it compiles closures) +CODESCAN_BENCH_HISTORY="v0.33.3 v0.35.1" ./run.sh # which releases to compare against +``` + +| variable | meaning | default | +|---|---|---| +| `CODESCAN_BENCH_HISTORY` | released versions to build probes for, oldest first | `v0.33.3 v0.35.1` | +| `CODESCAN_BENCH_ROUNDS` | measured rounds per warm cell | `3` | +| `CODESCAN_BENCH_COLD` | `1` adds the cold-cache pass | off | +| `CODESCAN_BENCH_PATTERN` | package pattern handed to the scan | `./...` | +| `CODESCAN_BENCH_EXTRA_CORPUS` | `name:/path` — measure somebody else's tree alongside the shipped ones | unset | + +Raw measurements accumulate in `results/` as JSON lines and the table is a pure function of them, so +a run can be re-read without being repeated: + +```sh +go run . -summarize results/warm.jsonl,results/cold.jsonl -baseline-label v0.33.3 +``` + +`results/` and `.work/` (the generated version modules and built probes) are gitignored. + +### The in-tree harness + +Every report is behind `CODESCAN_BENCH=1`, so a plain `go test ./...` — CI's included — runs none +of it, and the corpus is not even unpacked. + +```sh +# per-phase cost of one scan of each corpus +CODESCAN_BENCH=1 go test ./internal/benchmarks/ -run TestScanPhases -v + +# the load ladders: what each way of loading charges, and the shape it leaves behind +CODESCAN_BENCH=1 go test ./internal/benchmarks/ -run TestBenchLoadLadders -v -timeout 30m +CODESCAN_BENCH=1 CODESCAN_BENCH_COLD=1 go test ./internal/benchmarks/ -run TestBenchLoadLaddersCold -v -timeout 60m + +# as Go benchmarks +go test ./internal/benchmarks/ -run x -bench BenchmarkScan -benchtime=1x -benchmem +go test ./internal/benchmarks/ -run x -bench BenchmarkLoadStrategies -benchtime=1x +``` + +| variable | meaning | +|---|---| +| `CODESCAN_BENCH=1` | required for the reporting tests — they load whole package graphs repeatedly | +| `CODESCAN_BENCH_COLD=1` | adds the cold-cache pass, each rung with a private empty `GOCACHE` | +| `CODESCAN_BENCH_EXPORTDATA_DOCKERCTL` / `_KUBEAPI` | an export-data blob per corpus, for the bundled-export-data rung | +| `CODESCAN_BENCH_SPOTLIGHT` | one dependency to describe in detail under each rung (default `github.com/go-openapi/strfmt`) | +| `CODESCAN_BENCH_CPUPROFILE` / `_MEMPROFILE` | write a profile of one scan (`TestScanProfile`) | +| `CODESCAN_BENCH_PHASE` / `_ITERS` / `_CORPUS` | what that profile covers, how many replays, and of which corpus | + +An export-data blob is not checked in — the format is tied to the Go release that produced it, and a +blob covers one corpus's closure and no other. Produce one with `go run ./hack/genexportdata`; the +skipped rung prints the exact command. + +### How the numbers above were taken, and what would invalidate them + +Measured 2026-08-16, go1.26.5, AMD Ryzen 7 5800X / 31 GB, one machine, warm page cache. Reproduce +with the commands above. + +- **One scan per process.** Peak RSS is a process-wide high-water mark, so a second scan in the same + process would report the first one's peak plus its own garbage. +- **A discarded warm-up per configuration, then alternating rounds.** Measuring config-by-config + would let cache and thermal drift land entirely on whichever ran last. +- **Wall clock needs a quiet machine; memory does not.** Under load the timings here move by 3× + while allocation and peak RSS reproduce to four digits. Distrust a wall-clock delta under 10%; + the conclusions above rest on gaps of 2× and more. +- **The cold pass is n=1**, cold being a single-shot state by definition. It isolates the *build* + cache and nothing else: each cell gets a private empty `GOCACHE`, `GOMODCACHE` is left alone, and + the operator's own cache is never touched. +- **`GOWORK=off` everywhere.** The corpora unpack inside this repository, whose workspace does not + list them, and a version probe built inside the workspace would link the working tree instead of + the release it claims to measure. Both are handled; anything reimplementing this must handle them + too. +- **A version probe refuses options younger than itself.** `options_baseline.go` accepts the same + flags as the current build and *exits* rather than ignoring them — a probe that silently accepted + `-compiled-deps` would report a default-configuration measurement under a label saying otherwise, + the one measurement error that cannot be spotted in the output. +- **Peak RSS is read from `/proc`**, so that column reads zero off Linux. diff --git a/internal/integration/bench_corpus_test.go b/internal/benchmarks/bench_corpus_test.go similarity index 52% rename from internal/integration/bench_corpus_test.go rename to internal/benchmarks/bench_corpus_test.go index 75008b65..66ab60e9 100644 --- a/internal/integration/bench_corpus_test.go +++ b/internal/benchmarks/bench_corpus_test.go @@ -1,18 +1,18 @@ // SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers // SPDX-License-Identifier: Apache-2.0 -package integration_test +package benchmarks_test // Corpus resolution and measurement primitives shared by the scanner measurement harness. // // The harness answers one question in several dimensions: what does a scan pay, and what does it -// get for it. Its axes are laid out in bench_scanner_axes.md, which is also where the reference -// figures live. This file holds only what every axis needs: +// get for it. This file holds only what every measurement needs: // -// - which external corpus to measure, and how to skip cleanly when it is not checked out; +// - which trees to measure, which is no longer a question — they ship with the repository and +// unpack on demand (see internal/benchmarks/corpus); // - how to time a phase and read the heap it RETAINS rather than the bytes it churned; -// - how to describe the SHAPE of a loaded graph, which is the axis that makes the on-demand -// stream's central state visible — a package with files and types but no AST. +// - how to describe the SHAPE of a loaded graph, which is what makes a package with files and +// types but no AST visible at all. // // Everything here is test-only and imports nothing that production code does not already use. @@ -20,201 +20,110 @@ import ( "fmt" "os" "runtime" - "strings" "testing" "time" "golang.org/x/tools/go/packages" + + "github.com/go-openapi/codescan/internal/benchmarks/corpus" + "github.com/go-openapi/testify/v2/require" ) -// Environment knobs. Every one of them has a working default; they exist so the harness can be -// pointed at a different checkout without editing source. +// Environment knobs. Every one of them has a working default; they exist so a run can be steered +// without editing source. const ( - // envBenchEnabled gates the reporting tests, which take minutes and need an external corpus. - // They are tests rather than benchmarks because what they produce is a table, not a scalar - // (see bench_scanner_axes.md, "Why some axes are tests"). + // envBenchEnabled gates the reporting tests, which load whole package graphs repeatedly. They + // are tests rather than benchmarks because what they produce is a table, not a scalar. envBenchEnabled = "CODESCAN_BENCH" - // envDockerctlDir overrides where the dockerctl corpus is checked out. - // Named without a corpus suffix for compatibility: it predates the second corpus. - envDockerctlDir = "CODESCAN_BENCH_DIR" - - // envGoSwaggerDir overrides where the go-swagger corpus is checked out. - envGoSwaggerDir = "CODESCAN_BENCH_GOSWAGGER_DIR" -) - -// Per-corpus export-data blobs (Axis C). A blob covers one corpus's dependency closure and no -// other, so pointing every corpus at the same file would silently measure a fallback-to-source run -// and call it export data. envExportData is honoured as a fallback for the single-corpus case. -const ( + // Per-corpus export-data blobs. A blob covers one corpus's dependency closure and no other, so + // pointing every corpus at the same file would silently measure a fallback-to-source run and + // call it export data. envDockerctlExportData = "CODESCAN_BENCH_EXPORTDATA_DOCKERCTL" - envGoSwaggerExportData = "CODESCAN_BENCH_EXPORTDATA_GOSWAGGER" -) - -// Default checkout locations of the two external corpora. Neither is vendored: they are large -// third-party trees, and the point of measuring them is that they are real. -const ( - // defaultDockerctlDir is the go-swagger-generated docker engine API client — ~500 Go files - // over a 336-package closure, of which two packages carry annotations. - defaultDockerctlDir = "/home/fred/src/github.com/go-swagger/dockerctl" - - // defaultGoSwaggerDir is go-swagger itself — a hand-written tree with a 440-package closure, - // and a second shape to check any conclusion against. - defaultGoSwaggerDir = "/home/fred/src/github.com/go-swagger/go-swagger" + envKubeapiExportData = "CODESCAN_BENCH_EXPORTDATA_KUBEAPI" ) // bytesPerMB is the only unit any of these measurements is legible in. const bytesPerMB = 1 << 20 -// benchCorpus is one external tree to measure, and everything needed to measure it. +// benchCorpus is one tree to measure, and everything needed to measure it. type benchCorpus struct { - name string - dir string - pattern string - exportEnv string // env var naming this corpus's export-data blob (Axis C) -} + corpus.Corpus -// knownCorpora is the whole corpus table. Adding a third tree is adding a row. -func knownCorpora() []benchCorpus { - return []benchCorpus{ - { - name: "dockerctl", - dir: envOr(envDockerctlDir, defaultDockerctlDir), - pattern: "./...", - exportEnv: envDockerctlExportData, - }, - { - name: "go-swagger", - dir: envOr(envGoSwaggerDir, defaultGoSwaggerDir), - pattern: "./...", - exportEnv: envGoSwaggerExportData, - }, - } + exportEnv string // env var naming this corpus's export-data blob } -func envOr(env, fallback string) string { - if v := os.Getenv(env); v != "" { - return v - } - - return fallback +// exportEnvFor names a corpus's blob variable. A corpus with no entry simply has no export-data +// rung. +func exportEnvFor(name string) string { + return map[string]string{ + "dockerctl": envDockerctlExportData, + "kubeapi": envKubeapiExportData, + }[name] } -// dockerctlCorpus resolves the dockerctl checkout and SKIPS the caller when it is absent. Absence -// is the normal case in CI and must never be an error. -func dockerctlCorpus(tb testing.TB) benchCorpus { +// benchCorpora unpacks the shipped corpora and returns them. +// +// Absence used to be the normal case, back when these were external checkouts named through the +// environment. It is now a failure: the archive is in the repository, so a corpus that cannot be +// resolved means something is wrong with the checkout rather than with the machine. +func benchCorpora(tb testing.TB) []benchCorpus { tb.Helper() - corpus := knownCorpora()[0] - if _, err := os.Stat(corpus.dir); err != nil { - tb.Skipf("corpus not found at %q (set %s): %v", corpus.dir, envDockerctlDir, err) - } - - return corpus -} + found, err := corpus.Ensure() + require.NoError(tb, err, "unpacking the benchmark corpora") -// dockerctlDir is the same resolution as a bare directory, for the callers that only need a path. -func dockerctlDir(tb testing.TB) string { - tb.Helper() + out := make([]benchCorpus, 0, len(found)) + for _, c := range found { + out = append(out, benchCorpus{Corpus: c, exportEnv: exportEnvFor(c.Name)}) + } - return dockerctlCorpus(tb).dir + return out } -// benchCorpora returns every corpus present on this machine, skipping the caller when none is. +// benchEnv is the environment every load runs under. // -// Callers that need one specific tree use dockerctlCorpus directly; this is for the axes whose -// whole point is that a conclusion holds on more than one shape of project. -func benchCorpora(tb testing.TB) []benchCorpus { - tb.Helper() - - var found []benchCorpus - for _, c := range knownCorpora() { - if _, err := os.Stat(c.dir); err == nil { - found = append(found, c) - } else { - tb.Logf("corpus %s absent at %q — not measured", c.name, c.dir) - } - } +// GOWORK=off is not optional. The corpora unpack INSIDE this repository, and the workspace at its +// root does not list them, so a load that inherits it fails outright: +// +// directory prefix . does not contain modules listed in go.work or their selected dependencies +// +// Turning it off is also what keeps each corpus resolving its own dependency versions, which a +// workspace spanning both would merge. Everything else is inherited, so the operator's GOMODCACHE +// and toolchain still apply. +func benchEnv() []string { + return append(os.Environ(), "GOWORK=off") +} - if len(found) == 0 { - tb.Skipf("no measurement corpus checked out (set %s and/or %s)", envDockerctlDir, envGoSwaggerDir) +// envOr reads an environment variable, falling back when it is unset or empty. +func envOr(env, fallback string) string { + if v := os.Getenv(env); v != "" { + return v } - return found + return fallback } // requireBenchEnabled skips unless the operator asked for the measurement explicitly. // -// These reporting tests load whole third-party package graphs repeatedly; on a machine where the -// corpora ARE checked out, letting them run under a plain `go test ./...` would add minutes to -// every unrelated run. +// These reporting tests load whole third-party package graphs repeatedly; letting them run under a +// plain `go test ./...` would add minutes to every unrelated run. func requireBenchEnabled(tb testing.TB) { tb.Helper() if os.Getenv(envBenchEnabled) != "1" { - tb.Skipf("%s!=1: skipping the scanner measurement report (see bench_scanner_axes.md)", envBenchEnabled) - } -} - -// mbOf renders a byte count in MB. -func mbOf(v uint64) float64 { return float64(v) / bytesPerMB } - -// retainedHeap returns the currently retained heap, after a GC so the figure reflects what is HELD -// rather than what was allocated on the way. -// -// Retained heap is the number this stream is steered by: the scanner's problem is that it keeps the -// whole package graph alive, not that it churns. Callers take a reading before and after and report -// the delta, so the figure means "what this rung holds" rather than "what this process holds". -func retainedHeap() uint64 { - runtime.GC() - - var m runtime.MemStats - runtime.ReadMemStats(&m) - - return m.HeapAlloc -} - -// retainedSince returns the retained-heap growth since a baseline reading, floored at zero. -// -// The floor is not cosmetic: a rung can end with LESS live heap than it started with, because the -// fencing GC also collects whatever the previous rung had just dropped. A negative "retained" is -// noise, not information. -func retainedSince(base uint64) uint64 { - now := retainedHeap() - if now < base { - return 0 + tb.Skipf("%s!=1: skipping the scanner measurement report (see README.md)", envBenchEnabled) } - - return now - base -} - -// allocBaseline takes the cumulative-allocation reading that allocatedSince is measured against. -func allocBaseline() uint64 { - runtime.GC() - - var m runtime.MemStats - runtime.ReadMemStats(&m) - - return m.TotalAlloc -} - -// allocatedSince returns the bytes allocated since a baseline — the CHURN figure, which is the -// right one for phases whose whole point is that they retain nothing (discovery, the prefilter). -func allocatedSince(base uint64) uint64 { - var m runtime.MemStats - runtime.ReadMemStats(&m) - - return m.TotalAlloc - base } // loadShape describes what a load produced, not what it cost. // -// This is the axis that makes the on-demand stream's central state visible, and since the -// per-dependency export-data policy landed it is a THREE-way state per package rather than two: +// Since the per-dependency export-data policy landed this is a THREE-way state per package rather +// than two: // // - fromSource — types and syntax, the ordinary load; // - exportServed — complete types, no AST. The package said nothing about its own types, so its -// source was never parsed. This is what a scan holds instead of a package; +// source was never parsed; // - exportOnly — served from export data AND its source could not be found at all, so nothing // could ever be read from it. The loader names this state itself // ([packages.ExportOnly] / WithOnExportOnly); the count is collected by the caller, @@ -242,8 +151,7 @@ type loadShape struct { // a package served from export data has no Imports map to walk (internal/packages.exportedPackage // builds an empty one — nothing downstream reads it). So under WithExportData graph does not mean // "the size of the closure"; it means "how many packages the loader materialised", which is the -// figure the retained heap corresponds to and is worth knowing on its own. The closure size for -// that corpus comes from a rung that walks the whole graph. +// figure the retained heap corresponds to and is worth knowing on its own. func shapeOf(roots []*packages.Package) loadShape { all := walkGraph(roots) @@ -299,13 +207,6 @@ func walkGraph(roots []*packages.Package) map[string]*packages.Package { return all } -// isStdlibPath reports whether an import path belongs to the standard library — no dot in its first -// segment, the same test cmd/go uses. Worth splitting out in reports because stdlib is the bulk of -// any closure and carries no annotations. -func isStdlibPath(pkgPath string) bool { - return !strings.Contains(strings.SplitN(pkgPath, "/", 2)[0], ".") -} - // reportRow is the one row format every cost table in the harness shares: what it was, what it // cost in wall clock, and what it retained. func reportRow(tb testing.TB, label string, elapsed time.Duration, retained uint64, tail string) { @@ -313,3 +214,35 @@ func reportRow(tb testing.TB, label string, elapsed time.Duration, retained uint tb.Logf("%-44s %7.2fs retained=%6.0f MB %s", label, elapsed.Seconds(), mbOf(retained), tail) } + +// mbOf renders a byte count in MB. +func mbOf(v uint64) float64 { return float64(v) / bytesPerMB } + +// retainedHeap returns the currently retained heap, after a GC so the figure reflects what is HELD +// rather than what was allocated on the way. +// +// Retained heap is the number a load is steered by: the scanner's problem is that it keeps the +// whole package graph alive, not that it churns. Callers take a reading before and after and report +// the delta, so the figure means "what this rung holds" rather than "what this process holds". +func retainedHeap() uint64 { + runtime.GC() + + var m runtime.MemStats + runtime.ReadMemStats(&m) + + return m.HeapAlloc +} + +// retainedSince returns the retained-heap growth since a baseline reading, floored at zero. +// +// The floor is not cosmetic: a rung can end with LESS live heap than it started with, because the +// fencing GC also collects whatever the previous rung had just dropped. A negative "retained" is +// noise, not information. +func retainedSince(base uint64) uint64 { + now := retainedHeap() + if now < base { + return 0 + } + + return now - base +} diff --git a/internal/integration/bench_loaders_test.go b/internal/benchmarks/bench_loaders_test.go similarity index 80% rename from internal/integration/bench_loaders_test.go rename to internal/benchmarks/bench_loaders_test.go index 348e74b5..f65baa01 100644 --- a/internal/integration/bench_loaders_test.go +++ b/internal/benchmarks/bench_loaders_test.go @@ -1,21 +1,22 @@ // SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers // SPDX-License-Identifier: Apache-2.0 -package integration_test +package benchmarks_test -// Loading axes of the scanner measurement harness. +// How a package graph is loaded, and what each way of loading it costs. // -// Axis A — loader strategy: go/packages vs internal/packages (ToolchainFreeLoader) -// Axis B — dependency types: read from source vs taken from compiled export data -// Axis C — bundled export data: a pre-computed blob, the toolchain-free route -// Axis D — the raw LoadMode ladder: what go/packages charges for each capability -// Axis E — the SHAPE of a load: closure size, files, types, syntax — not just the cost +// A — loader strategy: go/packages vs internal/packages (ToolchainFreeLoader) +// B — dependency types: read from source vs taken from compiled export data +// C — bundled export data: a pre-computed blob, the toolchain-free route +// D — the raw LoadMode ladder: what go/packages charges for each capability +// E — the SHAPE of a load: closure size, files, types, syntax — not just the cost // -// Axis E is not a separate rung: every rung reports it, because the cost of a load is only half of -// what distinguishes one rung from another. The state this stream exists to serve — a package with -// complete types and NO AST — is invisible in a timing and obvious in a shape. +// E is not a separate rung: every rung reports it, because the cost of a load is only half of what +// distinguishes one rung from another. The state the loader work turns on — a package with complete +// types and NO AST — is invisible in a timing and obvious in a shape. // -// See bench_scanner_axes.md for how to run these and how to read the output. +// This is the evidence behind the loader tables in README.md; the whole-scan figures reported there +// come from loader-benchmark, which measures released versions this package cannot see. import ( "archive/zip" @@ -116,7 +117,7 @@ func goPackagesLadder() []loadRung { // rawLoad builds a rung that calls go/packages directly, bypassing internal/packages. func rawLoad(mode packages.LoadMode) func(benchCorpus, []string) (loadResult, error) { return func(corpus benchCorpus, env []string) (loadResult, error) { - pkgs, err := packages.Load(&packages.Config{Dir: corpus.dir, Mode: mode, Env: env}, corpus.pattern) + pkgs, err := packages.Load(&packages.Config{Dir: corpus.Dir, Mode: mode, Env: env}, corpus.Pattern) return loadResult{pkgs: pkgs}, err } @@ -169,7 +170,7 @@ func ownLoad(opts ...ownpackages.Option) func(benchCorpus, []string) (loadResult }, opts...) pkgs, err := ownpackages.NewLoader(all...). - Load(&packages.Config{Dir: corpus.dir, Env: env}, corpus.pattern) + Load(&packages.Config{Dir: corpus.Dir, Env: env}, corpus.Pattern) res.pkgs = pkgs return res, err @@ -190,18 +191,18 @@ func exportDataBlob(tb testing.TB, corpus benchCorpus) fs.FS { tb.Logf("Axis C not measured for %s: no export-data blob. Produce one and set %s:\n"+ " go run ./hack/genexportdata -dir %s -out /tmp/%s-exportdata.zip %s\n"+ " %s=/tmp/%s-exportdata.zip", - corpus.name, corpus.exportEnv, - corpus.dir, corpus.name, corpus.pattern, - corpus.exportEnv, corpus.name) + corpus.Name, corpus.exportEnv, + corpus.Dir, corpus.Name, corpus.Pattern, + corpus.exportEnv, corpus.Name) return nil } - // The path is operator-supplied, on purpose: this whole harness is pointed at trees outside the - // repo. There is nothing to sanitise it against. + // The path is operator-supplied, on purpose: a blob is produced outside the repo, in a directory + // of the operator's choosing. There is nothing to sanitise it against. info, err := os.Stat(path) //nolint:gosec // an operator-supplied measurement input, not user data if err != nil { - tb.Logf("Axis C not measured for %s: %s=%q is unreadable: %v", corpus.name, corpus.exportEnv, path, err) + tb.Logf("Axis C not measured for %s: %s=%q is unreadable: %v", corpus.Name, corpus.exportEnv, path, err) return nil } @@ -214,7 +215,7 @@ func exportDataBlob(tb testing.TB, corpus benchCorpus) fs.FS { // still serves per-package reads. r, err := zip.OpenReader(path) if err != nil { - tb.Logf("Axis C not measured for %s: %q is neither a directory nor a readable zip: %v", corpus.name, path, err) + tb.Logf("Axis C not measured for %s: %q is neither a directory nor a readable zip: %v", corpus.Name, path, err) return nil } @@ -284,14 +285,14 @@ func TestBenchLoadLadders(t *testing.T) { requireBenchEnabled(t) for _, corpus := range benchCorpora(t) { - t.Run(corpus.name, func(t *testing.T) { - t.Logf("=== load ladders, WARM cache — %s (%s %s) ===", corpus.name, corpus.dir, corpus.pattern) + t.Run(corpus.Name, func(t *testing.T) { + t.Logf("=== load ladders, WARM cache — %s (%s %s) ===", corpus.Name, corpus.Dir, corpus.Pattern) for _, rung := range goPackagesLadder() { - measureRung(t, rung, corpus, nil, true) + measureRung(t, rung, corpus, benchEnv(), true) } for _, rung := range loaderLadder(t, corpus) { - measureRung(t, rung, corpus, nil, true) + measureRung(t, rung, corpus, benchEnv(), true) } }) } @@ -318,7 +319,7 @@ func TestBenchLoadLaddersCold(t *testing.T) { } for _, corpus := range benchCorpora(t) { - t.Run(corpus.name, func(t *testing.T) { + t.Run(corpus.Name, func(t *testing.T) { // Only the rungs whose cost depends on the build cache. A load that reads dependency // source compiles nothing, so a fresh GOCACHE would measure the same thing twice. raw := goPackagesLadder() @@ -332,49 +333,51 @@ func TestBenchLoadLaddersCold(t *testing.T) { } } - t.Logf("=== load ladders, COLD (private empty GOCACHE) — %s ===", corpus.name) + t.Logf("=== load ladders, COLD (private empty GOCACHE) — %s ===", corpus.Name) for _, rung := range coldRungs { - env := append(os.Environ(), "GOCACHE="+t.TempDir()) + env := append(benchEnv(), "GOCACHE="+t.TempDir()) measureRung(t, rung, corpus, env, false) } }) } } -// BenchmarkLoadStrategies is Axes A–C as a true benchmark, for the times a like-for-like comparison +// BenchmarkLoadStrategies is A–C as a true benchmark, for the times a like-for-like comparison // across a code change is what is wanted rather than a table. // // Run it with -benchtime=1x: these rungs cost the better part of a second each, and the retained // figure only means anything for a single iteration. // -// go test ./internal/integration/ -run x -bench BenchmarkLoadStrategies -benchtime=1x +// go test ./internal/benchmarks/ -run x -bench BenchmarkLoadStrategies -benchtime=1x func BenchmarkLoadStrategies(b *testing.B) { - corpus := dockerctlCorpus(b) - - for _, rung := range loaderLadder(b, corpus) { - b.Run(benchName(rung.name), func(b *testing.B) { - // Warm the cache outside the timer: the first CompiledDependencies load compiles the - // closure's export data, which is a one-off and not what this benchmark is comparing. - _, err := rung.load(corpus, nil) - require.NoError(b, err) - - var shape loadShape - base := retainedHeap() - - b.ResetTimer() - for range b.N { - res, err := rung.load(corpus, nil) - require.NoError(b, err) - b.StopTimer() - shape = shapeOf(res.pkgs) - b.StartTimer() + for _, corpus := range benchCorpora(b) { + b.Run(corpus.Name, func(b *testing.B) { + for _, rung := range loaderLadder(b, corpus) { + b.Run(benchName(rung.name), func(b *testing.B) { + // Warm the cache outside the timer: the first CompiledDependencies load compiles + // the closure's export data, a one-off and not what this benchmark compares. + _, err := rung.load(corpus, benchEnv()) + require.NoError(b, err) + + var shape loadShape + base := retainedHeap() + + b.ResetTimer() + for range b.N { + res, err := rung.load(corpus, benchEnv()) + require.NoError(b, err) + b.StopTimer() + shape = shapeOf(res.pkgs) + b.StartTimer() + } + b.StopTimer() + + b.ReportMetric(mbOf(retainedSince(base)), "MB-retained") + b.ReportMetric(float64(shape.graph), "pkgs") + b.ReportMetric(float64(shape.withSyntax), "pkgs-from-source") + }) } - b.StopTimer() - - b.ReportMetric(mbOf(retainedSince(base)), "MB-retained") - b.ReportMetric(float64(shape.graph), "pkgs") - b.ReportMetric(float64(shape.withSyntax), "pkgs-from-source") }) } } diff --git a/internal/integration/bench_dockerctl_profile_test.go b/internal/benchmarks/bench_profile_test.go similarity index 56% rename from internal/integration/bench_dockerctl_profile_test.go rename to internal/benchmarks/bench_profile_test.go index e0732e7c..a20f420e 100644 --- a/internal/integration/bench_dockerctl_profile_test.go +++ b/internal/benchmarks/bench_profile_test.go @@ -1,51 +1,48 @@ // SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers // SPDX-License-Identifier: Apache-2.0 -package integration_test +package benchmarks_test -// Profiling wrapper for the dockerctl scan benchmark. +// Profiling wrapper for a corpus scan. // -// `go test -bench` cannot collect a usable -cpuprofile/-memprofile when more -// than one package is selected (the flags are rejected / the files get -// clobbered). This wrapper sidesteps that by driving runtime/pprof directly -// from a single test, so the profile is dumped regardless of how many -// packages the surrounding `go test` invocation matched. It uses only the -// standard library — no extra module dependency (e.g. github.com/pkg/profile). +// `go test -bench` cannot collect a usable -cpuprofile/-memprofile when more than one package is +// selected (the flags are rejected / the files get clobbered). This wrapper sidesteps that by +// driving runtime/pprof directly from a single test, so the profile is dumped regardless of how +// many packages the surrounding `go test` invocation matched. Standard library only — no extra +// module dependency. // -// It profiles the Build phase by default — the regexp→grammar delta — by -// loading the ScanCtx once and replaying Build() in a loop so the CPU -// sampler (100 Hz) gathers enough samples on a ~115 ms workload. Set the -// phase to "full" to instead profile the whole Run (Load dominates there). +// It profiles the Build phase by default, loading the ScanCtx once and replaying Build() in a loop +// so the CPU sampler (100 Hz) gathers enough samples on a workload measured in milliseconds. Set +// the phase to "full" to profile the whole Run instead, where Load dominates. // // Knobs (env vars): // // CODESCAN_BENCH_CPUPROFILE=cpu.out write a CPU profile // CODESCAN_BENCH_MEMPROFILE=mem.out write a heap profile (after the run) -// CODESCAN_BENCH_PHASE=build|full what to profile (default build) -// CODESCAN_BENCH_ITERS=30 replay count for sampling (default 30) -// CODESCAN_BENCH_DIR=/path/to/client reference corpus location +// CODESCAN_BENCH_PHASE=build|full what to profile (default build) +// CODESCAN_BENCH_ITERS=30 replay count for sampling (default 30) +// CODESCAN_BENCH_CORPUS=dockerctl which corpus to profile (default dockerctl) // // The test skips unless at least one of CPUPROFILE/MEMPROFILE is set. // // Examples: // -// # CPU profile of the grammar parser (Build phase), then study it +// # CPU profile of the parser (Build phase), then study it // CODESCAN_BENCH_CPUPROFILE=cpu.out \ -// go test ./internal/integration/ -run TestDockerctlProfile -v +// go test ./internal/benchmarks/ -run TestScanProfile -v // go tool pprof -http=: cpu.out // // # heap profile; inspect total bytes allocated -// CODESCAN_BENCH_MEMPROFILE=mem.out \ -// go test ./internal/integration/ -run TestDockerctlProfile -v +// CODESCAN_BENCH_MEMPROFILE=mem.out CODESCAN_BENCH_ITERS=200 \ +// go test ./internal/benchmarks/ -run TestScanProfile -v // go tool pprof -sample_index=alloc_space -http=: mem.out // -// Caveat — the CPU profile brackets only the replay loop, so it cleanly -// isolates the chosen phase (Load runs before StartCPUProfile and is -// excluded). The heap profile, by contrast, is process-cumulative: for -// phase=build its alloc_space still includes the one-time Load (go/types) -// allocations, which can dwarf the parser. To study Build allocations, raise -// CODESCAN_BENCH_ITERS so the replayed Build dominates the single Load (e.g. -// 200), or compare the same iters across worktrees and read the delta. +// Caveat — the CPU profile brackets only the replay loop, so it cleanly isolates the chosen phase +// (Load runs before StartCPUProfile and is excluded). The heap profile, by contrast, is +// process-cumulative: for phase=build its alloc_space still includes the one-time Load (go/types) +// allocations, which dwarf the parser. To study Build allocations, raise CODESCAN_BENCH_ITERS so +// the replayed Build dominates the single Load (e.g. 200), or compare the same iters across two +// trees and read the delta. import ( "os" @@ -62,12 +59,14 @@ import ( ) const ( - envCPUProfile = "CODESCAN_BENCH_CPUPROFILE" - envMemProfile = "CODESCAN_BENCH_MEMPROFILE" - envPhase = "CODESCAN_BENCH_PHASE" - envIters = "CODESCAN_BENCH_ITERS" - - defaultProfileIters = 30 + envCPUProfile = "CODESCAN_BENCH_CPUPROFILE" + envMemProfile = "CODESCAN_BENCH_MEMPROFILE" + envPhase = "CODESCAN_BENCH_PHASE" + envIters = "CODESCAN_BENCH_ITERS" + envProfileCorps = "CODESCAN_BENCH_CORPUS" + + defaultProfileIters = 30 + defaultProfileCorpus = "dockerctl" ) // envIterCount reads CODESCAN_BENCH_ITERS, falling back to def. @@ -86,6 +85,22 @@ func envIterCount(tb testing.TB, def int) int { return n } +// profileCorpus resolves the corpus to profile. +func profileCorpus(tb testing.TB) benchCorpus { + tb.Helper() + + want := envOr(envProfileCorps, defaultProfileCorpus) + for _, c := range benchCorpora(tb) { + if c.Name == want { + return c + } + } + + tb.Fatalf("no corpus named %q (set %s)", want, envProfileCorps) + + return benchCorpus{} +} + // createProfile opens path for writing, failing the test on error. func createProfile(tb testing.TB, path string) *os.File { tb.Helper() @@ -96,30 +111,25 @@ func createProfile(tb testing.TB, path string) *os.File { return f } -// TestDockerctlProfile drives a CPU and/or heap profile of the dockerctl scan -// so the parser hot path can be studied with `go tool pprof`. It is a no-op -// (skipped) unless a profile output path is requested via env. -func TestDockerctlProfile(t *testing.T) { +// TestScanProfile drives a CPU and/or heap profile of one corpus scan so a hot path can be studied +// with `go tool pprof`. It is a no-op (skipped) unless a profile output path is requested via env. +func TestScanProfile(t *testing.T) { cpuPath := os.Getenv(envCPUProfile) memPath := os.Getenv(envMemProfile) if cpuPath == "" && memPath == "" { t.Skipf("no profile requested; set %s and/or %s", envCPUProfile, envMemProfile) } - dir := dockerctlDir(t) - opts := benchOptions(dir) + corpus := profileCorpus(t) + opts := benchOptions(corpus) - phase := os.Getenv(envPhase) - if phase == "" { - phase = "build" - } + phase := envOr(envPhase, "build") // workload is one replayable unit of the phase under study. var workload func() switch phase { case "build": - // Load once; replay Build so the CPU sampler sees the parser, not the - // go/packages front end. + // Load once; replay Build so the CPU sampler sees the parser, not the go/packages front end. ctx, err := scanner.NewScanCtx(opts) require.NoError(t, err) workload = func() { @@ -139,7 +149,7 @@ func TestDockerctlProfile(t *testing.T) { } iters := envIterCount(t, defaultProfileIters) - t.Logf("profiling phase=%s iters=%d corpus=%s", phase, iters, dir) + t.Logf("profiling phase=%s iters=%d corpus=%s (%s)", phase, iters, corpus.Name, corpus.Dir) if cpuPath != "" { f := createProfile(t, cpuPath) @@ -158,8 +168,8 @@ func TestDockerctlProfile(t *testing.T) { (time.Since(start) / time.Duration(iters)).Round(time.Microsecond), ) - // Stop CPU profiling before writing the heap profile so the heap dump's - // own allocations are not attributed to the workload. + // Stop CPU profiling before writing the heap profile so the heap dump's own allocations are not + // attributed to the workload. if cpuPath != "" { pprof.StopCPUProfile() t.Logf("wrote CPU profile: %s", cpuPath) diff --git a/internal/benchmarks/bench_scan_test.go b/internal/benchmarks/bench_scan_test.go new file mode 100644 index 00000000..9c75206d --- /dev/null +++ b/internal/benchmarks/bench_scan_test.go @@ -0,0 +1,163 @@ +// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers +// SPDX-License-Identifier: Apache-2.0 + +package benchmarks_test + +// What one whole scan costs, split into the two phases that answer different questions. +// +// - Load — scanner.NewScanCtx: resolving and type-checking the package graph. This is where a +// loader configuration shows up, and it is the bulk of a scan. +// - Build — spec.NewBuilder(...).Build(): reading annotations and emitting Swagger. This is where +// the parser shows up, and it is where a corpus with routes differs from one without. +// +// The split is the point. An end-to-end number moves for either reason and says which only by +// accident, and the two phases have moved in opposite directions over this project's history. +// +// Run: +// +// go test ./internal/benchmarks/ -run TestScanPhases -v +// go test ./internal/benchmarks/ -run x -bench BenchmarkScan -benchtime=1x -benchmem + +import ( + "runtime" + "testing" + "time" + + "github.com/go-openapi/codescan" + "github.com/go-openapi/codescan/internal/builders/spec" + "github.com/go-openapi/codescan/internal/scanner" + "github.com/go-openapi/testify/v2/require" +) + +// benchOptions is the configuration every phase measurement shares: scan everything, models +// included, and leave the loader at its default so the phase split is not about a loader. +func benchOptions(c benchCorpus) *codescan.Options { + return &codescan.Options{ + WorkDir: c.Dir, + Packages: []string{c.Pattern}, + ScanModels: true, + // The corpora unpack inside this repository, whose workspace does not list them — see + // benchEnv. + GOWORK: "off", + } +} + +// phaseStats captures the cost of one measured phase. +type phaseStats struct { + name string + elapsed time.Duration + allocPB uint64 // bytes allocated during the phase (cumulative TotalAlloc delta) + allocsN uint64 // number of heap allocations during the phase (Mallocs delta) +} + +// measure runs fn once, fencing it with a GC so the heap counters reflect only fn's own +// allocations, and records elapsed time + allocation deltas. +func measure(name string, fn func()) phaseStats { + var before, after runtime.MemStats + + runtime.GC() + runtime.ReadMemStats(&before) + + start := time.Now() + fn() + elapsed := time.Since(start) + + runtime.ReadMemStats(&after) + + return phaseStats{ + name: name, + elapsed: elapsed, + allocPB: after.TotalAlloc - before.TotalAlloc, + allocsN: after.Mallocs - before.Mallocs, + } +} + +func (p phaseStats) report(tb testing.TB) { + tb.Helper() + tb.Logf("%-6s time=%-12s alloc=%6.1f MB allocs=%d", + p.name, + p.elapsed.Round(time.Microsecond), + float64(p.allocPB)/bytesPerMB, + p.allocsN, + ) +} + +// TestScanPhases runs a single scan of every corpus and prints the per-phase cost. +// +// One iteration is enough: the ratios reproduce, and the figures worth quoting are the allocation +// ones, which do not move with machine load the way wall clock does. +// +// Gated like every other report here. CI runs `go test work ./...` with -race on six job +// combinations, and a scan that peaks at 800 MB is not something to instrument six times over on a +// shared runner — where the timing would measure the runner anyway. +func TestScanPhases(t *testing.T) { + requireBenchEnabled(t) + + for _, corpus := range benchCorpora(t) { + t.Run(corpus.Name, func(t *testing.T) { + opts := benchOptions(corpus) + + // Load then Build, sharing one ScanCtx so Build sees exactly what Run would feed it. + var ctx *scanner.ScanCtx + loadStats := measure("Load", func() { + var err error + ctx, err = scanner.NewScanCtx(opts) + require.NoError(t, err) + }) + + var sp any + buildStats := measure("Build", func() { + builder := spec.NewBuilder(opts.InputSpec, ctx, opts.ScanModels) + built, err := builder.Build() + require.NoError(t, err) + sp = built + }) + require.NotNil(t, sp) + + // Full end-to-end, fresh, for a single combined number. + fullStats := measure("Full", func() { + built, err := codescan.Run(opts) + require.NoError(t, err) + require.NotNil(t, built) + }) + + t.Logf("=== %s scan phases (single iteration, %s) ===", corpus.Name, corpus.Dir) + loadStats.report(t) + buildStats.report(t) + fullStats.report(t) + }) + } +} + +// BenchmarkScan is the same workload exposed to the Go benchmark framework. Run with +// -benchtime=1x -benchmem for a one-shot allocation profile, or with the default settings for a +// stabilized timing. +func BenchmarkScan(b *testing.B) { + for _, corpus := range benchCorpora(b) { + opts := benchOptions(corpus) + + b.Run(corpus.Name+"/Full", func(b *testing.B) { + b.ReportAllocs() + for range b.N { + built, err := codescan.Run(opts) + require.NoError(b, err) + require.NotNil(b, built) + } + }) + + b.Run(corpus.Name+"/Build", func(b *testing.B) { + b.ReportAllocs() + for range b.N { + b.StopTimer() + ctx, err := scanner.NewScanCtx(opts) + require.NoError(b, err) + b.StartTimer() + + builder := spec.NewBuilder(opts.InputSpec, ctx, opts.ScanModels) + built, err := builder.Build() + require.NoError(b, err) + require.NotNil(b, built) + } + }) + } +} diff --git a/internal/benchmarks/corpus/corpus.go b/internal/benchmarks/corpus/corpus.go new file mode 100644 index 00000000..5736e75a --- /dev/null +++ b/internal/benchmarks/corpus/corpus.go @@ -0,0 +1,299 @@ +// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers +// SPDX-License-Identifier: Apache-2.0 + +// Package corpus resolves the trees the benchmarks measure. +// +// The corpora ship with the repository as one archive +// (internal/benchmarks/testdata/corpus.tgz) and are unpacked on demand, so a +// measurement needs a clone and nothing else — no external checkout, no +// environment variable naming somebody's home directory, and no network. Both +// trees carry their dependencies in vendor/, which is what makes that last one +// true. +package corpus + +import ( + "archive/tar" + "compress/gzip" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "io" + "os" + "path" + "path/filepath" + "runtime" + "slices" + "strings" +) + +// Archive layout. The archive holds one directory per corpus at its root, and +// unpacking it is what produces Dir for each of them. +const ( + archiveName = "corpus.tgz" + unpackedDir = "corpus" + stampName = ".stamp" + + dirMode = 0o750 + fileMode = 0o600 +) + +// The three ways this package refuses to produce a corpus. +var ( + // errUnsafePath guards the extraction against an archive entry that would + // write outside the destination. Ours never does; an archive is still + // untrusted input and archive/tar hands over whatever name it was given. + errUnsafePath = errors.New("archive entry escapes the destination directory") + + errNoSuchCorpus = errors.New("no such corpus") + errNoCaller = errors.New("cannot locate the corpus archive: no caller information") +) + +// Corpus is one tree to measure. +// +// Two shapes on purpose, because a conclusion that only holds on one of them is +// not a conclusion: +// +// - dockerctl — a generated CLIENT for a reasonably large API. Light code, +// 198 definitions, no paths: the annotations are all on models. +// - kubeapi — a generated SERVER for a very large API. Heavy code, four times +// the source, 222 definitions and 260 paths, so the route and operation +// rendering paths carry real weight. +type Corpus struct { + Name string + Dir string // absolute path to the unpacked tree + Pattern string // package pattern to scan +} + +// names is the whole corpus table, in reporting order. Adding a third tree is +// adding a row here and a directory to the archive. +func names() []string { return []string{"dockerctl", "kubeapi"} } + +// Ensure unpacks the archive if needed and returns the corpora. +// +// The unpacked tree is stamped with the archive's digest, so a re-rolled archive +// is noticed and unpacked again rather than measured stale — the failure mode +// that would otherwise be invisible, since a stale tree scans perfectly well. +func Ensure() ([]Corpus, error) { + archive, err := archivePath() + if err != nil { + return nil, err + } + + digest, err := fileDigest(archive) + if err != nil { + return nil, err + } + + dest := filepath.Join(filepath.Dir(archive), unpackedDir) + if err := unpackIfStale(archive, dest, digest); err != nil { + return nil, err + } + + all := names() + corpora := make([]Corpus, 0, len(all)) + + for _, name := range all { + dir := filepath.Join(dest, name) + if _, err := os.Stat(dir); err != nil { + return nil, fmt.Errorf("corpus %q missing from %s: %w", name, archiveName, err) + } + + corpora = append(corpora, Corpus{Name: name, Dir: dir, Pattern: "./..."}) + } + + return corpora, nil +} + +// Find returns one corpus by name. +// +// The name is checked before anything is unpacked. Otherwise a typo pays for the +// whole archive before being told it was a typo — and, less obviously, the test +// covering that refusal would unpack 55 MB on every CI job of every platform, +// which is exactly what the measurement gate exists to prevent. +func Find(name string) (Corpus, error) { + if !slices.Contains(names(), name) { + return Corpus{}, fmt.Errorf("%w: %q (have %s)", errNoSuchCorpus, name, strings.Join(names(), ", ")) + } + + corpora, err := Ensure() + if err != nil { + return Corpus{}, err + } + + for _, c := range corpora { + if c.Name == name { + return c, nil + } + } + + return Corpus{}, fmt.Errorf("%w: %q missing from %s", errNoSuchCorpus, name, archiveName) +} + +// archivePath locates the archive from this file's own position in the tree. +// +// The alternative — resolving against the working directory — would mean one +// answer under `go test` (the package directory) and another under `go run` +// (wherever the operator stands). +func archivePath() (string, error) { + _, self, _, ok := runtime.Caller(0) + if !ok { + return "", errNoCaller + } + + // this file is internal/benchmarks/corpus/corpus.go; the archive is a + // sibling of the package directory, under testdata. + path := filepath.Join(filepath.Dir(filepath.Dir(self)), "testdata", archiveName) + if _, err := os.Stat(path); err != nil { + return "", fmt.Errorf("corpus archive: %w", err) + } + + return path, nil +} + +// unpackIfStale unpacks the archive unless the destination already carries a +// stamp matching its digest. +func unpackIfStale(archive, dest, digest string) error { + stamp := filepath.Join(dest, stampName) + if got, err := os.ReadFile(stamp); err == nil && string(got) == digest { + return nil + } + + if err := os.RemoveAll(dest); err != nil { + return fmt.Errorf("clearing %s: %w", dest, err) + } + + if err := unpack(archive, dest); err != nil { + return err + } + + return os.WriteFile(stamp, []byte(digest), fileMode) +} + +// unpack extracts a gzipped tar into dest. +func unpack(archive, dest string) error { + f, err := os.Open(archive) + if err != nil { + return err + } + + defer func() { _ = f.Close() }() + + gz, err := gzip.NewReader(f) + if err != nil { + return fmt.Errorf("reading %s: %w", archive, err) + } + + defer func() { _ = gz.Close() }() + + tr := tar.NewReader(gz) + for { + hdr, err := tr.Next() + if errors.Is(err, io.EOF) { + return nil + } + + if err != nil { + return fmt.Errorf("reading %s: %w", archive, err) + } + + if err := extractEntry(tr, hdr, dest); err != nil { + return err + } + } +} + +// extractEntry writes one archive entry. Only directories and regular files are +// honoured: the corpora are generated Go source and vendored dependencies, so +// anything else is not something to reproduce silently. +func extractEntry(tr *tar.Reader, hdr *tar.Header, dest string) error { + target, err := safeJoin(dest, hdr.Name) + if err != nil { + return err + } + + switch hdr.Typeflag { + case tar.TypeDir: + return os.MkdirAll(target, dirMode) + case tar.TypeReg: + if err := os.MkdirAll(filepath.Dir(target), dirMode); err != nil { + return err + } + + return writeFile(tr, target, hdr.Size) + default: + return nil + } +} + +// writeFile copies exactly size bytes out of the archive. +// +// The bound is the point: io.Copy over an untrusted archive is a decompression +// bomb waiting to happen, and the header already says how long the entry is. +func writeFile(r io.Reader, target string, size int64) error { + out, err := os.OpenFile(target, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, fileMode) + if err != nil { + return err + } + + if _, err := io.CopyN(out, r, size); err != nil && !errors.Is(err, io.EOF) { + _ = out.Close() + + return fmt.Errorf("writing %s: %w", target, err) + } + + return out.Close() +} + +// safeJoin resolves an archive entry name against dest, refusing anything that +// is not a plain relative path inside it. +// +// The name is judged in the SLASH form the tar format defines, never in the +// host's. A host-shaped check accepts on one platform what it refuses on another: +// filepath.IsAbs("/x") is false on Windows, which needs a volume — so the entry +// that Linux refused as absolute was silently accepted there. Backslash and colon +// go the same way, being a separator and a drive marker on Windows and ordinary +// filename characters here, so both are refused everywhere rather than in one +// place. +// +// Refusing rather than clamping: the usual sanitisation rewrites `../x` to `x` +// and carries on, which for an archive this repository rolls itself would turn a +// broken corpus into a quietly misplaced one. +func safeJoin(dest, name string) (string, error) { + if name == "" || strings.ContainsAny(name, `\:`) || strings.HasPrefix(name, "/") { + return "", fmt.Errorf("%w: %q", errUnsafePath, name) + } + + clean := path.Clean(name) + if clean == "." || clean == ".." || strings.HasPrefix(clean, "../") { + return "", fmt.Errorf("%w: %q", errUnsafePath, name) + } + + target := filepath.Join(dest, filepath.FromSlash(clean)) + + // The same refusal restated on the joined path. Redundant after the checks + // above, and worth keeping: it is the containment property this function + // exists to guarantee, asserted on the value it actually returns. + if !strings.HasPrefix(target, filepath.Clean(dest)+string(os.PathSeparator)) { + return "", fmt.Errorf("%w: %q", errUnsafePath, name) + } + + return target, nil +} + +// fileDigest returns the SHA-256 of a file, hex-encoded. +func fileDigest(path string) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + + defer func() { _ = f.Close() }() + + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + return "", fmt.Errorf("digesting %s: %w", path, err) + } + + return hex.EncodeToString(h.Sum(nil)), nil +} diff --git a/internal/benchmarks/corpus/corpus_test.go b/internal/benchmarks/corpus/corpus_test.go new file mode 100644 index 00000000..60916e42 --- /dev/null +++ b/internal/benchmarks/corpus/corpus_test.go @@ -0,0 +1,186 @@ +// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers +// SPDX-License-Identifier: Apache-2.0 + +package corpus + +import ( + "archive/tar" + "bytes" + "compress/gzip" + "errors" + "io/fs" + "os" + "path/filepath" + "testing" + + "github.com/go-openapi/testify/v2/assert" + "github.com/go-openapi/testify/v2/require" +) + +// envBenchEnabled gates the test that unpacks the real corpus — 5000 files and 56 MB, which is not +// something to write on every CI job of every platform. The extraction itself is covered below +// against a synthetic archive, which exercises the same code for a few hundred bytes. +const envBenchEnabled = "CODESCAN_BENCH" + +// TestEnsure covers the property the benchmarks depend on: after Ensure, every corpus is a real +// module tree on disk, and asking again is cheap rather than a second extraction. +func TestEnsure(t *testing.T) { + if os.Getenv(envBenchEnabled) != "1" { + t.Skipf("%s!=1: skipping the unpacking of the real corpus", envBenchEnabled) + } + + corpora, err := Ensure() + require.NoError(t, err) + require.Len(t, corpora, len(names())) + + for _, c := range corpora { + t.Run(c.Name, func(t *testing.T) { + assert.Equal(t, "./...", c.Pattern) + assert.True(t, filepath.IsAbs(c.Dir), "a corpus directory is resolved, not relative") + + // A go.mod, because every measurement scans a module; a vendor directory, because that + // is what makes the corpus readable with no network. + assert.FileExists(t, filepath.Join(c.Dir, "go.mod")) + assert.DirExists(t, filepath.Join(c.Dir, "vendor")) + }) + } + + t.Run("is idempotent", func(t *testing.T) { + again, err := Ensure() + require.NoError(t, err) + assert.Equal(t, corpora, again) + }) + + t.Run("finds one by name", func(t *testing.T) { + c, err := Find(names()[0]) + require.NoError(t, err) + assert.Equal(t, names()[0], c.Name) + }) +} + +// TestUnpackIfStale covers the extraction against a synthetic archive: what it writes, that it +// skips a destination whose stamp already matches, and that it starts over when it does not. +// +// The staleness half is the one that matters. A re-rolled archive measured against the tree the +// previous one left behind would report perfectly plausible figures for the wrong corpus. +func TestUnpackIfStale(t *testing.T) { + dir := t.TempDir() + archive := filepath.Join(dir, "synthetic.tgz") + dest := filepath.Join(dir, "unpacked") + + writeArchive(t, archive, map[string]string{ + "acorpus/go.mod": "module example.com/acorpus\n", + "acorpus/vendor/dep/x.go": "package dep\n", + }) + + digest, err := fileDigest(archive) + require.NoError(t, err) + + require.NoError(t, unpackIfStale(archive, dest, digest)) + assert.FileExists(t, filepath.Join(dest, "acorpus", "go.mod")) + assert.FileExists(t, filepath.Join(dest, "acorpus", "vendor", "dep", "x.go")) + + t.Run("leaves a matching destination alone", func(t *testing.T) { + // A file the archive does not carry survives only if nothing was re-extracted. + witness := filepath.Join(dest, "witness") + require.NoError(t, os.WriteFile(witness, []byte("kept"), fileMode)) + + require.NoError(t, unpackIfStale(archive, dest, digest)) + assert.FileExists(t, witness) + }) + + t.Run("starts over when the stamp differs", func(t *testing.T) { + require.NoError(t, unpackIfStale(archive, dest, "a different digest")) + assertAbsent(t, filepath.Join(dest, "witness")) + assert.FileExists(t, filepath.Join(dest, "acorpus", "go.mod")) + + stamp, err := os.ReadFile(filepath.Join(dest, stampName)) + require.NoError(t, err) + assert.Equal(t, "a different digest", string(stamp)) + }) +} + +// TestUnpackRefusesAnEscapingEntry pins the refusal on a whole archive rather than on safeJoin +// alone: an entry that escapes must fail the extraction, not be quietly relocated inside it. +func TestUnpackRefusesAnEscapingEntry(t *testing.T) { + dir := t.TempDir() + archive := filepath.Join(dir, "hostile.tgz") + + writeArchive(t, archive, map[string]string{"../escaped.go": "package escaped\n"}) + + err := unpack(archive, filepath.Join(dir, "unpacked")) + require.ErrorIs(t, err, errUnsafePath) + assertAbsent(t, filepath.Join(dir, "escaped.go")) +} + +// TestFindRejectsAnUnknownName covers what an operator sees after a typo. +func TestFindRejectsAnUnknownName(t *testing.T) { + _, err := Find("nosuchcorpus") + require.ErrorIs(t, err, errNoSuchCorpus) + assert.Contains(t, err.Error(), names()[0], "the error names what there is to choose from") +} + +// TestSafeJoin pins the path rule directly. +// +// The rule has to be the SAME on every platform, which is what the last three cases are about: a +// leading `/` is not "absolute" to Windows, and `\` and `:` are separators and drive markers there +// while being ordinary filename characters here. Judging the name in the host's terms let Windows +// accept what Linux refused — the divergence CI caught. +func TestSafeJoin(t *testing.T) { + dest := t.TempDir() + + t.Run("resolves an ordinary entry", func(t *testing.T) { + got, err := safeJoin(dest, "dockerctl/go.mod") + require.NoError(t, err) + assert.Equal(t, filepath.Join(dest, "dockerctl", "go.mod"), got) + }) + + for name, why := range map[string]string{ + "../escaped.go": "climbs out of the destination", + "dockerctl/../../escaped.go": "climbs out after cleaning", + "..": "is the parent itself", + "": "names nothing", + "/absolute/escaped.go": "is rooted (not IsAbs on Windows)", + `dockerctl\..\..\escaped.go`: "climbs out through Windows separators", + "C:/escaped.go": "carries a Windows volume", + } { + t.Run("refuses "+name+" — "+why, func(t *testing.T) { + _, err := safeJoin(dest, name) + require.ErrorIs(t, err, errUnsafePath) + }) + } +} + +// assertAbsent fails when path exists. testify/v2 has FileExists and no negative form of it. +func assertAbsent(tb testing.TB, path string) { + tb.Helper() + + _, err := os.Stat(path) + assert.Truef(tb, errors.Is(err, fs.ErrNotExist), "expected %q not to exist, stat says: %v", path, err) +} + +// writeArchive builds a gzipped tar of the given entries, so the extraction is tested against an +// archive rather than against a mock of one. +func writeArchive(tb testing.TB, path string, entries map[string]string) { + tb.Helper() + + var buf bytes.Buffer + + gz := gzip.NewWriter(&buf) + tw := tar.NewWriter(gz) + + for name, content := range entries { + require.NoError(tb, tw.WriteHeader(&tar.Header{ + Name: name, + Mode: int64(fileMode), + Size: int64(len(content)), + Typeflag: tar.TypeReg, + })) + _, err := tw.Write([]byte(content)) + require.NoError(tb, err) + } + + require.NoError(tb, tw.Close()) + require.NoError(tb, gz.Close()) + require.NoError(tb, os.WriteFile(path, buf.Bytes(), fileMode)) +} diff --git a/internal/benchmarks/corpus/unpack/main.go b/internal/benchmarks/corpus/unpack/main.go new file mode 100644 index 00000000..a0b6afeb --- /dev/null +++ b/internal/benchmarks/corpus/unpack/main.go @@ -0,0 +1,31 @@ +// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers +// SPDX-License-Identifier: Apache-2.0 + +// Command unpack makes the benchmark corpora available on disk and prints where +// they are, one `namedirectory` line per corpus. +// +// It exists so run.sh does not have to grow its own copy of the layout: where +// the archive is, where it unpacks to and when it is stale are decided once, in +// the corpus package the Go harness uses. +// +// go run ./internal/benchmarks/corpus/unpack +package main + +import ( + "fmt" + "os" + + "github.com/go-openapi/codescan/internal/benchmarks/corpus" +) + +func main() { + corpora, err := corpus.Ensure() + if err != nil { + fmt.Fprintln(os.Stderr, "unpack:", err) + os.Exit(1) + } + + for _, c := range corpora { + fmt.Fprintf(os.Stdout, "%s\t%s\n", c.Name, c.Dir) + } +} diff --git a/internal/benchmarks/doc.go b/internal/benchmarks/doc.go new file mode 100644 index 00000000..494062ba --- /dev/null +++ b/internal/benchmarks/doc.go @@ -0,0 +1,10 @@ +// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers +// SPDX-License-Identifier: Apache-2.0 + +// Package benchmarks measures what a scan costs, on two generated projects that +// ship with this repository. +// +// The package itself holds no production code: it is a test harness plus +// loader-benchmark, a command that measures released versions of the library as +// well as this one. README.md carries the results and how to run them. +package benchmarks diff --git a/hack/loader-benchmark/main.go b/internal/benchmarks/loader-benchmark/main.go similarity index 90% rename from hack/loader-benchmark/main.go rename to internal/benchmarks/loader-benchmark/main.go index 4bb7c101..03431d35 100644 --- a/hack/loader-benchmark/main.go +++ b/internal/benchmarks/loader-benchmark/main.go @@ -6,15 +6,15 @@ // It exists because the in-repo benchmarks cannot answer "how much did we improve": they live inside // the module, so they do not exist in any earlier release. This one talks only to the public // [codescan.Run] API, which every version has, so the same source compiles against a released tag and -// against the working tree — and run.sh builds it both ways to put the two side by side. +// against the working tree — and run.sh builds one probe per version to put them side by side. // // One scan per process, deliberately. Peak RSS is a process-wide high-water mark, so a second scan in // the same process would report the first one's peak plus its own garbage. // -// Usage is through run.sh. Running this directly measures a single cell: +// Usage is through run.sh; see ../README.md. Running this directly measures a single cell: // -// loader-benchmark -dir /path/to/corpus -corpus mycorpus -label master -// loader-benchmark -summarize results/*.jsonl +// loader-benchmark -dir /path/to/corpus -corpus mycorpus -label current +// loader-benchmark -summarize results/warm.jsonl -baseline-label v0.33.3 package main import ( @@ -72,19 +72,21 @@ func main() { func run() error { var ( dir, pattern, label, corpus, cache string - summarize, baselineLabel string + summarize, baselineLabel, gowork string scanModels bool extra extraOptions ) flag.StringVar(&dir, "dir", "", "corpus checkout to scan") + flag.StringVar(&gowork, "gowork", "off", + "GOWORK for the scan; \"off\" is the default because the shipped corpora unpack inside a workspace that does not list them") flag.StringVar(&pattern, "pattern", "./...", "package pattern to scan") flag.StringVar(&label, "label", "unlabelled", "version/configuration label for the report") flag.StringVar(&corpus, "corpus", "corpus", "corpus name for the report") flag.StringVar(&cache, "cache", "warm", "build-cache state this run was made in: warm or cold") flag.BoolVar(&scanModels, "scan-models", true, "emit swagger:model definitions too") flag.StringVar(&summarize, "summarize", "", "comma-separated result files to tabulate instead of measuring") - flag.StringVar(&baselineLabel, "baseline-label", "v0.36.3", "label deltas are computed against") + flag.StringVar(&baselineLabel, "baseline-label", "v0.33.3", "label deltas are computed against") extra.register() flag.Parse() @@ -96,6 +98,15 @@ func run() error { return errNoCorpus } + // Through the environment rather than through Options, because this source has to compile + // against every release being measured and Options.GOWORK is younger than some of them. The + // go command reads it from the environment either way. + if gowork != "" { + if err := os.Setenv("GOWORK", gowork); err != nil { + return fmt.Errorf("setting GOWORK: %w", err) + } + } + m := measure(dir, pattern, label, corpus, cache, scanModels, extra) return json.NewEncoder(os.Stdout).Encode(m) diff --git a/hack/loader-benchmark/options_baseline.go b/internal/benchmarks/loader-benchmark/options_baseline.go similarity index 100% rename from hack/loader-benchmark/options_baseline.go rename to internal/benchmarks/loader-benchmark/options_baseline.go diff --git a/hack/loader-benchmark/options_current.go b/internal/benchmarks/loader-benchmark/options_current.go similarity index 100% rename from hack/loader-benchmark/options_current.go rename to internal/benchmarks/loader-benchmark/options_current.go diff --git a/hack/loader-benchmark/run.sh b/internal/benchmarks/loader-benchmark/run.sh similarity index 54% rename from hack/loader-benchmark/run.sh rename to internal/benchmarks/loader-benchmark/run.sh index c5923301..177e7831 100755 --- a/hack/loader-benchmark/run.sh +++ b/internal/benchmarks/loader-benchmark/run.sh @@ -2,14 +2,14 @@ # SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers # SPDX-License-Identifier: Apache-2.0 # -# Runs the loader benchmark matrix: a released baseline against the working tree, over every corpus -# that is present, warm and cold. +# Runs the benchmark matrix over the corpora shipped with the repository: every configuration of the +# working tree, plus one probe per historic release, warm and cold. # -# See README.md. The short version: +# See ../README.md. The short version: # -# CODESCAN_BENCH_DOCKERCTL_DIR=... ./run.sh # warm matrix over whatever is configured -# CODESCAN_BENCH_COLD=1 ./run.sh # add the cold pass (slow: it compiles closures) -# CODESCAN_BENCH_BASELINE=v0.36.3 ./run.sh # compare against a different release +# ./run.sh # warm matrix +# CODESCAN_BENCH_COLD=1 ./run.sh # add the cold pass (slow: it compiles closures) +# CODESCAN_BENCH_HISTORY="v0.33.3 v0.35.1" ./run.sh # which releases to compare against # # Results land in results/ as JSON lines and are tabulated at the end; re-running -summarize over # them costs nothing, so a run can be re-read without being repeated. @@ -17,11 +17,13 @@ set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -repo="$(cd "$here/../.." && pwd)" +repo="$(cd "$here/../../.." && pwd)" work="$here/.work" results="$here/results" -baseline="${CODESCAN_BENCH_BASELINE:-v0.36.3}" +# The releases the working tree is compared against, oldest first. The first one is what deltas are +# computed against, so it is the start of the story rather than an arbitrary reference. +read -r -a history <<<"${CODESCAN_BENCH_HISTORY:-v0.33.3 v0.35.1}" rounds="${CODESCAN_BENCH_ROUNDS:-3}" pattern="${CODESCAN_BENCH_PATTERN:-./...}" @@ -29,69 +31,73 @@ mkdir -p "$work" "$results" # --- corpora ----------------------------------------------------------------------------------- # -# Every corpus is external and large; none is vendored into this repo, because the point of measuring -# them is that they are real trees. Name them through the environment — there is deliberately no -# default path, so a missing corpus is skipped by name rather than silently measuring someone else's -# checkout. -corpora=() -add_corpus() { # name env-var - local dir="${!2:-}" - if [[ -n "$dir" && -d "$dir" ]]; then - corpora+=("$1:$dir") - else - echo " skipping corpus '$1' (set $2 to a checkout to include it)" - fi -} - +# Both trees ship in internal/benchmarks/testdata/corpus.tgz and unpack on demand, so a measurement +# needs a clone and nothing else. They vendor their dependencies, so no download either. echo "corpora:" -add_corpus dockerctl CODESCAN_BENCH_DOCKERCTL_DIR -add_corpus kubeapi CODESCAN_BENCH_KUBEAPI_DIR -add_corpus kubeapi-vendored CODESCAN_BENCH_KUBEAPI_VENDORED_DIR +corpora=() +while IFS=$'\t' read -r name dir; do + [[ -n "$name" ]] || continue + corpora+=("$name:$dir") + echo " $name -> $dir" +done < <(cd "$repo" && go run ./internal/benchmarks/corpus/unpack) + +# An extra tree to measure alongside them, for the times the question is about somebody else's code. +if [[ -n "${CODESCAN_BENCH_EXTRA_CORPUS:-}" ]]; then + corpora+=("${CODESCAN_BENCH_EXTRA_CORPUS}") + echo " ${CODESCAN_BENCH_EXTRA_CORPUS%%:*} -> ${CODESCAN_BENCH_EXTRA_CORPUS#*:} (extra)" +fi if [[ ${#corpora[@]} -eq 0 ]]; then - echo "no corpus configured; nothing to measure" >&2 + echo "no corpus resolved; nothing to measure" >&2 exit 1 fi # --- builds ------------------------------------------------------------------------------------ # -# The current build comes from this module. The baseline needs its own module requiring the released -# library, and it MUST be built with GOWORK=off: the workspace at the repo root would substitute the -# local tree for the released one and quietly measure the working tree twice. +# The current build comes from this module. Each historic probe needs its own throwaway module +# requiring that release, and it MUST be built with GOWORK=off: the workspace at the repo root would +# substitute the local tree for the released one and quietly measure the working tree every time. echo echo "building current tree..." -(cd "$repo" && go build -o "$work/loader-benchmark-current" ./hack/loader-benchmark) +(cd "$repo" && go build -o "$work/probe-current" ./internal/benchmarks/loader-benchmark) -echo "building baseline $baseline..." -mkdir -p "$work/baseline" -cp "$here"/*.go "$work/baseline/" -cat >"$work/baseline/go.mod" <"$work/$version/go.mod" </dev/null 2>&1 - GOWORK=off go build -tags baseline -o "$work/loader-benchmark-baseline" . -) + ( + cd "$work/$version" + GOWORK=off go mod tidy >/dev/null 2>&1 + GOWORK=off go build -tags baseline -o "$work/probe-$version" . + ) + configs+=("$version:$work/probe-$version:") +done -current="$work/loader-benchmark-current" -base="$work/loader-benchmark-baseline" +current="$work/probe-current" -# --- the matrix -------------------------------------------------------------------------------- -# -# label:binary:flags. The baseline has exactly one row because the release has exactly one way to -# load and scan — which is what makes it a baseline. -configs=( - "$baseline:$base:" +# The working tree, in the configurations worth telling apart. `current` is the source-loading scan, +# which is what SkipCompiledDependencies selects today; `current+compiled-deps` is what a plain run +# does since v0.36.4. Rows are labelled by configuration rather than by default, so they stay +# comparable across a release that moves the default. +configs+=( "current:$current:" "current+toolchain-free:$current:-toolchain-free" "current+compiled-deps:$current:-compiled-deps" ) +baseline="${history[0]:-current}" + measure() { # corpus dir label binary flags cache out local corpus="$1" dir="$2" label="$3" bin="$4" flags="$5" cache="$6" out="$7" # shellcheck disable=SC2086 # flags is a deliberately word-split option list diff --git a/internal/benchmarks/testdata/corpus.tgz b/internal/benchmarks/testdata/corpus.tgz new file mode 100644 index 00000000..a10c018c Binary files /dev/null and b/internal/benchmarks/testdata/corpus.tgz differ diff --git a/internal/integration/bench_discovery_test.go b/internal/integration/bench_discovery_test.go deleted file mode 100644 index 9443d98a..00000000 --- a/internal/integration/bench_discovery_test.go +++ /dev/null @@ -1,275 +0,0 @@ -// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers -// SPDX-License-Identifier: Apache-2.0 - -package integration_test - -// Axis F — discovery cost, separated from loading. -// -// Finding every annotation in a closure and loading that closure are the same operation today. They -// need not be: `go list` gives the package graph and every package's GoFiles without types or -// syntax, a byte scan over those files rejects ~95% of them, and only the survivors are worth -// parsing. This measures each of those three steps on its own. -// -// Discovery has to stay EXHAUSTIVE — stdlib and all dependencies included — which is why it is -// worth making cheap rather than making narrow. Discriminator subtypes are discovered backwards (a -// subtype $refs its base and nothing $refs the subtype), so an annotated type in a dependency is -// never reached by following types from the root packages. -// -// See bench_scanner_axes.md. - -import ( - "bytes" - "go/ast" - "go/parser" - "go/token" - "os" - "sort" - "strings" - "testing" - "time" - - "golang.org/x/tools/go/packages" - - "github.com/go-openapi/testify/v2/require" -) - -// annotationNeedle is the byte-level pre-filter, and is also the exact test the loader's own -// per-dependency export-data policy uses (internal/packages.scanForAnnotations). -// -// It can only over-admit: a `swagger:` inside a string literal, or inside prose ABOUT annotations, -// costs one wasted parse. It cannot under-admit, because a real annotation contains this literal by -// definition — which is what makes the cheap pass safe to run INSTEAD of loading the package. -const annotationNeedle = "swagger:" - -// isAnnotationLine is a regexp-free restatement of parsers.rxCommentPrefix followed by the keyword: -// strip the leading comment noise, allow one markdown table pipe, and require `swagger:` to START -// what remains. -// -// The line-start rule is the whole subtlety. A naive substring test counts 14 annotated packages on -// dockerctl where this counts 2 — codescan's own godoc, read as a dependency, discusses `swagger:` -// constantly. The gap between the two rules is measured directly by Axis H, because the loader -// routes dependencies on the naive rule and pays a full source load for each over-admission. -func isAnnotationLine(text string) bool { - i := 0 - for i < len(text) && strings.ContainsRune(" \t/*-", rune(text[i])) { - i++ - } - if i < len(text) && text[i] == '|' { - i++ - for i < len(text) && text[i] == ' ' { - i++ - } - } - - return strings.HasPrefix(text[i:], annotationNeedle) -} - -// fileList is the cheapest useful load: the package graph and every package's GoFiles, with no -// types and no syntax. -// -// It is also what makes on-demand syntax possible later — GoFiles is populated for the whole -// closure here, so materialising a package's source afterwards is plain parser.ParseFile on known -// paths, with no second `go list`. -func fileList(corpus benchCorpus) (map[string]*packages.Package, time.Duration, error) { - const mode = packages.NeedName | packages.NeedFiles | packages.NeedImports | packages.NeedDeps - - start := time.Now() - pkgs, err := packages.Load(&packages.Config{Dir: corpus.dir, Mode: mode}, corpus.pattern) - if err != nil { - return nil, 0, err - } - - return walkGraph(pkgs), time.Since(start), nil -} - -// annotatedDecls is the annotation index for one package: the type names whose doc comment carries -// an annotation. -type annotatedDecls struct { - pkgPath string - names []string -} - -// discoveryResult is what the two discovery passes produce: the index itself, the two package sets -// the two admission rules yield, and what each pass cost. -type discoveryResult struct { - // annotated holds the packages with at least one annotated TYPE declaration, and their names. - // These are the roots the working-set walk starts from (Axis G). - annotated []annotatedDecls - - // markerPkgs holds the packages admitted by the naive substring rule — the loader's own - // per-dependency test. A superset of realPkgs, and the difference is what Axis H prices. - markerPkgs map[string]bool - - // realPkgs holds the packages with at least one comment line that genuinely STARTS an - // annotation, anywhere in the package (not only above a type declaration, so routes and meta - // blocks count too). - realPkgs map[string]bool - - files int - survivors int - sourceBytes int64 - prefilter time.Duration - parse time.Duration -} - -// discoverAnnotations runs the two cheap discovery passes over every file in the closure and -// returns the annotated declarations it found, plus the cost of each pass. -// -// Deliberately exhaustive: see the file comment on why discovery cannot be demand-driven. -func discoverAnnotations(all map[string]*packages.Package) discoveryResult { - type fileRef struct{ pkgPath, path string } - - res := discoveryResult{markerPkgs: map[string]bool{}, realPkgs: map[string]bool{}} - - var files []fileRef - for pkgPath, p := range all { - for _, f := range p.GoFiles { - files = append(files, fileRef{pkgPath, f}) - } - } - res.files = len(files) - - // Pass 1 — read every file, keep the ones containing the literal. - start := time.Now() - // A starting capacity, sized from the ~5% hit rate the prefilter has on both corpora. - const expectedHitRate = 16 - - kept := make([]fileRef, 0, len(files)/expectedHitRate) - contents := make(map[string][]byte, len(files)/expectedHitRate) - for _, fr := range files { - blob, err := os.ReadFile(fr.path) - if err != nil { - continue - } - res.sourceBytes += int64(len(blob)) - if bytes.Contains(blob, []byte(annotationNeedle)) { - kept = append(kept, fr) - contents[fr.path] = blob - res.markerPkgs[fr.pkgPath] = true - } - } - res.prefilter = time.Since(start) - res.survivors = len(kept) - - // Pass 2 — parse only the survivors and apply the real, line-start rule. - start = time.Now() - fset := token.NewFileSet() - byPkg := map[string][]string{} - for _, fr := range kept { - af, err := parser.ParseFile(fset, fr.path, contents[fr.path], parser.ParseComments|parser.SkipObjectResolution) - if err != nil { - continue - } - if fileCarriesAnnotation(af) { - res.realPkgs[fr.pkgPath] = true - } - // Guarded: a file can survive the prefilter on a `swagger:` mention in prose and yield no - // annotated decl at all. An unguarded append would still create the map key and count the - // package as a root. - if names := annotatedTypeNames(af); len(names) > 0 { - byPkg[fr.pkgPath] = append(byPkg[fr.pkgPath], names...) - } - } - res.parse = time.Since(start) - - for pkgPath, names := range byPkg { - res.annotated = append(res.annotated, annotatedDecls{pkgPath: pkgPath, names: names}) - } - sort.Slice(res.annotated, func(i, j int) bool { return res.annotated[i].pkgPath < res.annotated[j].pkgPath }) - - return res -} - -// fileCarriesAnnotation reports whether any comment in af genuinely starts an annotation. -// -// Every comment, not just doc comments above type declarations: `swagger:route` and `swagger:meta` -// are free-standing, and a package carrying only those still has to be read from source. -func fileCarriesAnnotation(af *ast.File) bool { - for _, group := range af.Comments { - for _, c := range group.List { - if isAnnotationLine(c.Text) { - return true - } - } - } - - return false -} - -// annotatedTypeNames returns the type names in af whose doc comment carries an annotation line. -func annotatedTypeNames(af *ast.File) []string { - var names []string - for _, d := range af.Decls { - gd, ok := d.(*ast.GenDecl) - if !ok { - continue - } - for _, sp := range gd.Specs { - ts, ok := sp.(*ast.TypeSpec) - if !ok { - continue - } - doc := ts.Doc - if doc == nil { - doc = gd.Doc - } - if doc == nil { - continue - } - for _, c := range doc.List { - if isAnnotationLine(c.Text) { - names = append(names, ts.Name.Name) - - break - } - } - } - } - - return names -} - -// TestBenchDiscovery is Axis F: what a complete annotation index over the whole closure costs when -// nothing is loaded to build it. -// -// The figure to compare it against is the first rung of Axis D — today's load, which produces the -// same index as a side effect of type-checking everything. -func TestBenchDiscovery(t *testing.T) { - requireBenchEnabled(t) - - for _, corpus := range benchCorpora(t) { - t.Run(corpus.name, func(t *testing.T) { - base := allocBaseline() - - all, listed, err := fileList(corpus) - require.NoError(t, err, "go list") - - res := discoverAnnotations(all) - - t.Logf("=== discovery without loading — %s ===", corpus.name) - t.Logf("go list (graph + GoFiles) %7.2fs graph=%d packages, %d Go files, %.1f MB of source", - listed.Seconds(), len(all), res.files, float64(res.sourceBytes)/bytesPerMB) - t.Logf("byte prefilter over ALL files %7.2fs -> %d/%d files survive (%.1f%%)", - res.prefilter.Seconds(), res.survivors, res.files, percent(res.survivors, res.files)) - t.Logf("parse + classify survivors only %7.2fs -> %d packages with an annotated type decl", - res.parse.Seconds(), len(res.annotated)) - t.Logf("TOTAL discovery %7.2fs allocated=%.0f MB (transient), retained=%.0f MB", - (listed + res.prefilter + res.parse).Seconds(), mbOf(allocatedSince(base)), mbOf(retainedHeap())) - t.Logf("admission rules: substring=%d packages, line-start=%d packages", - len(res.markerPkgs), len(res.realPkgs)) - - for _, a := range res.annotated { - t.Logf(" %-70s %d annotated type decls", a.pkgPath, len(a.names)) - } - }) - } -} - -// percent renders n/total as a percentage, tolerating a zero total. -func percent(n, total int) float64 { - if total == 0 { - return 0 - } - - return 100 * float64(n) / float64(total) -} diff --git a/internal/integration/bench_dockerctl_design_notes.md b/internal/integration/bench_dockerctl_design_notes.md deleted file mode 100644 index 84a5c090..00000000 --- a/internal/integration/bench_dockerctl_design_notes.md +++ /dev/null @@ -1,258 +0,0 @@ -# Scanner performance — findings & redesign notes - -Companion to [`bench_dockerctl_report.md`](./bench_dockerctl_report.md). The -report measures *what is*; this records *what we concluded* and *where the next -win is*, so it is on hand when we start designing the new scanner. - -> [!NOTE] -> Written before the loader work started, and several of its predictions have -> since been measured rather than estimated — notably §6, which asked for a -> retention-aware harness. That harness now exists: -> [`bench_scanner_axes.md`](./bench_scanner_axes.md) reports retained heap per -> load configuration, the shape each one leaves behind, and the working set -> (§2's "what fraction is skippable" question, answered: 4 packages of 336). -> The estimates in §3 and §4 stand as estimates; the axes doc has the figures. - -Corpus throughout: the go-swagger **dockerctl** client — 502 non-test `.go` -files, ~160k LOC, 121 annotated files (~36k LOC), 19 packages (13 annotated). - -## 1. Where the cost actually is - -A full `codescan.Run` of dockerctl, by phase: - -| Phase | Time | Alloc (cumulative) | Allocs | -|-------|------|--------------------|--------| -| **Load** (`scanner.NewScanCtx`, go/packages full type-check) | ~4.1–5.4 s | ~1088 MB | ~12.19 M | -| **Build** (`spec.NewBuilder().Build()`, grammar parse + emit) | ~116 ms | ~28.8 MB | ~298 k | -| **Full** (fresh, Load+Build) | ~4.4 s | ~1115 MB | ~12.5 M | - -**Load dominates: ~95%+ of time and ~97% of allocations.** The comment parser -is not the bottleneck — type resolution is. - -Profiling confirms it (heap, `alloc_space`): - -- `go/types` (type-checking) ≈ **73%** of the cumulative heap. -- A single function, `go/types.(*Checker).recordTypeAndValue`, ≈ **20%** (316 MB) - on its own — it records type+value for *every typed expression*, including - function bodies. -- Our `grammar` parser ≈ **6%** of total / ~18% of *Build-only* allocations / - ~10% of Build CPU. - -The grammar parser looking "invisible" in the profile is the **good** result: -it dropped below the type-resolution floor. The migration win is real but only -visible as the cross-worktree Build *delta* (everything else is byte-identical): -alloc −57%, allocs −50%, time −49% vs regexp v0.34.0. - -**Yardstick:** further micro-optimization of the grammar parser is worth at most -~10–15% of *total* scan time. The lever that matters is type resolution. - -## 2. The opportunity: AST-first discovery, targeted resolution - -Replace the systematic full type-check of every package with: - -1. a cheap **AST-only** pass over all files to discover annotations (finding - `swagger:` comments is pure text/AST — no type info needed), then -2. type resolution **only on the interesting surface**. - -### Granularity reality - -`go/types` resolves at **package** granularity — you cannot type-check a single -decl without its package. Discovery is file/decl-level; resolution bottoms out -at the package. "Finer than file" buys nothing on the resolution side. The way -you go *finer than a package* is not sub-package checking — it's turning off -func-body checking and trimming the `types.Info` maps (below). - -### dockerctl LOC split (what's skippable) - -| Bucket | LOC | Share | Type-check needed? | -|--------|-----|-------|--------------------| -| `cli/` (cobra wrappers, generated) | 77,582 | **48%** | No — zero annotations | -| `models/` | 21,082 | 13% | Yes | -| `client/*` (16 sub-pkgs) | ~61,900 | 38% | Yes | -| `cmd/`, misc | ~80 | <1% | No | - -`cli/` is ~half the code and entirely skippable. But 13/19 packages *are* -annotated, so at package granularity the unskippable surface is still ~52% of -LOC — package-skipping alone is bounded. - -### Three levers (independently landable & measurable) - -1. **`types.Config{IgnoreFuncBodies: true}`** — the scanner reads type decls, - struct fields, and comments; it never analyzes statement bodies. Generated - client/CLI code is overwhelmingly function bodies, which are - *expression-dense*, so this saves **more than proportionally** to LOC and - directly attacks `recordTypeAndValue`. Smallest diff, likely best ROI — do - it first. -2. **Trim `types.Info`** — `recordTypeAndValue` only populates `Info.Types` if - that map is provided. If fields/enums can be resolved from `Defs`/`Uses` - alone, dropping the per-expression `Types` map largely evaporates that 20%. - (Validate against the alias/enum/const paths.) -3. **AST-discovery + selective package type-checking** — skip the unannotated - packages (`cli/`) entirely. The big structural change; do it last, once 1–2 - have de-risked the resolution semantics. - -## 3. Gross gain estimate — TIME - -- Time floor = **parse-all** (must read every file's comments). Unavoidable; - caps the time win. -- Package-skipping alone: ~1.6–1.8×. Full scan ~4.4 s → ~2.6 s. -- Combined (skip + `IgnoreFuncBodies` + trimmed `Info`): ~3–4×. Full scan - plausibly ~4.4 s → **~1.2–1.5 s**. - -**Realistic: ½–⅓ of original time.** Not a screamer, not bad. At that point the -parser becomes the new ~10–15% tail and grammar work starts to matter to the -total. - -Corpus-dependent: dockerctl is lucky (half is unannotated `cli/`). A typical -generated *server* (handlers + models, nearly all annotated) has little to skip -→ package-skipping ~1.2×. The `IgnoreFuncBodies` / Info-trim levers are the ones -that **generalize** — every corpus has fat function bodies. - -## 4. Gross gain estimate — MEMORY (the bigger prize) - -Memory has **more headroom than time**, because the two have different floors: - -- Time floor = parse-all (CPU you pay regardless). -- Memory floor = the **retained** set. Parsing is *transient* — parse a file, - extract annotations + a lean descriptor, then **free the AST**. The current - design can't: `go/packages` pins every AST, every `types.Info` map, and the - whole type graph alive for the entire scan. That retention *is* the 1 GB. - -So a "**pull-as-you-go, cache, and free-as-you-go**" design attacks retention — -the thing that dominates memory — and can beat the time multiplier. - -Decisive fork on what "pull a type" means: - -1. **Lazy `go/types` per package, cache its `Info`** — still materializes a - whole package's heavyweight objects + expression `Info` on first touch. Saves - only fully-untouched packages. **Memory ≈ time ≈ ~½×.** -2. **Lean AST-derived type model, fall back to `go/types` only for hard cases** - — common cases (basic type, local named type, well-known format) hold a slim - descriptor we control; never instantiate `types.Named`/`Struct`/`Signature` - or per-expression `Info`. **Memory ~⅓ to ⅕× — beats the time win.** This is - the variant to build. - -Favorable for go-swagger corpora: -- **Well-known formats resolve by name** (`time.Time→date-time`, - `strfmt.UUID→uuid`, …) — string match, no resolution, never pulls `strfmt`'s - types into the graph. A large fraction of field types short-circuit. -- **Models reference models** — the reachable subgraph is fairly self-contained. - -Erosion risks — the cases that force the lean model back into `go/types`, each -potentially dragging a whole package's memory in (and the **correctness-risk -hotspots**: validate against goldens): -- Embedded field promotion (underlying-type query). -- Type aliases (`RefAliases` / `TransparentAliases`). -- Cross-package field types that aren't well-known formats. -- Enum/const values needing evaluated expression info. - -## 5. Bounding peak high-water (the invariant to design to) - -"Bounded" doesn't mean constant — it means **decoupled from total source size -and coupled to spec complexity**: - -``` -peak ≈ |reachable type cache| + max_over_packages( transient working set ) -``` - -- The first term grows **monotonically** (can't safely evict — a type - referenced by the last annotation may have been first seen early), so it's - bounded *by the reachable set*, not capped to a constant. -- The second term is **freeable per package**. -- Neither scales with `cli/`'s unreferenced bloat — peak tracks how big the spec - actually is, not how big the module is. That's the property to enforce. - -Two consequences: - -1. **The lever is per-entry weight, not eviction.** Since the cache can't be - evicted, minimize peak by making each entry a *lean descriptor* (kind, name, - field refs, format), not a retained `types.Named` + `Info`. "Bounded - reachable set × small constant per entry" is how you reach ⅓–⅕ rather than ½. -2. **Two phases keep the transient term tiny.** Phase A streams packages: parse - → extract annotation records + root-type identities → **free the AST** (peak - = one package's AST). Phase B resolves roots transitively into the lean cache - (peak = reachable cache + one resolution's working set). Without the split you - risk holding everything concurrently and peak collapses back to eager. - -**The trap:** a cache only wins if `reachable ⊊ whole`, strictly. If the -annotated surface transitively touches most types, the cache grows to ≈ eager -**plus** bookkeeping — strictly worse. - -## 6. Measurement - -The current harness captures cumulative `TotalAlloc`, which is **blind to -retention** — the exact thing the redesign optimizes. Before prototyping: - -- Add a **peak high-water sampler**: a background goroutine reading - `runtime.MemStats.HeapInuse` (or `HeapSys` / maxRSS via `getrusage`) on a - ticker, reporting the **max** across the scan. That gives the current eager - design's peak as the number to beat. -- Re-run it after each lever (`IgnoreFuncBodies`, package-skip, lean-model) to - confirm peak is actually falling, not just cumulative allocs. -- Use the heap-profile **baseline diff** (`-base=load.out`) to isolate Build / - resolution allocations from the one-time Load. - -**De-risking measurement to take first:** what fraction of field types are -well-known-format / local-named (short-circuit) vs. forced into `go/types` -(embedded / alias / cross-pkg / enum)? That ratio *is* the memory multiplier. - -## 7. Context & invariants to preserve - -Framing that is obvious now and easy to lose by the time the redesign starts. - -### The grammar was not a performance project - -It was driven by **readability, correctness, and completeness** — it does a lot -more than the regexp parser, notably **diagnostics tracking** (new). The speed -win is a *byproduct* of Go's `regexp` being slow, not a design goal. Two -consequences for reading the benchmark: - -- Don't regression-gate the grammar on raw parse throughput; gate it on - correctness/diagnostics. The parser is ~6% of the scan — it is not the budget. -- Feature work on the parser (diagnostics, completeness, the provenance feature - below) is **nearly free** against the type-resolution baseline. The x2–x3 - redesign win comes from *resolution*, which is orthogonal to parser features — - they don't compete for the same budget. - -### Invariant 1 — the comment-parse cache must survive the redesign - -Parsed annotation blocks are **cached and reused** as the same type/field is -reached from multiple places while drilling down the type graph. This makes -grammar cost `O(distinct annotations)`, not `O(type-graph traversals)` — it is -*why* the parser stays invisible. - -A lazy "pull-types-as-you-go" resolver **re-drills the same types repeatedly**. -If the parse cache is keyed on the traversal path rather than the comment block, -lazy resolution will re-trigger parsing and the grammar's footprint stops being -invisible. **Constraint: key the parse cache on the comment block (declaration -identity), not on the resolution path, and carry it across the lazy resolver.** - -### Invariant 2 — provenance must store positions, not AST handles - -A feature in parallel development tracks **mappings between originating code and -spec node** (source ↔ spec provenance). Its cost scales as `O(spec size)` (≈ one -position per emitted node), not `O(source size)` — i.e. on the *well-behaved* -axis (output complexity, not module bloat). It will be a real but **bounded** -regression. The one trap: - -- If provenance stores `ast.Node` handles or raw `token.Pos` into live - `*ast.File`s, it **pins ASTs alive** and silently defeats the free-as-you-go - goal of §5. -- If it stores resolved `file:line:col` (or a compact interned position), it - stays cheap and retention-friendly. - -**Constraint: provenance records compact resolved positions, never live AST -node references.** Cheap to get right early, expensive to retrofit. - -## Summary - -| Axis | Conservative | Realistic | Why | -|------|--------------|-----------|-----| -| Time | ~½× | **⅓–½×** | parse-all is the floor | -| Memory | ~½× (lazy over go/types) | **⅓–⅕×** (lean model + free-as-you-go) | retention, not CPU, is the floor | -| Grammar micro-opt | — | ~10–15% of total | it's no longer the bottleneck | - -Sequencing: `IgnoreFuncBodies` → trim `types.Info` → AST-discovery + selective -checking → lean-model lazy resolver with bounded reachable cache. Land each -independently, re-baseline goldens between them (the alias/enum/embedded paths -are where correctness breaks), and measure **peak**, not cumulative. diff --git a/internal/integration/bench_dockerctl_report.md b/internal/integration/bench_dockerctl_report.md deleted file mode 100644 index 1c1848ca..00000000 --- a/internal/integration/bench_dockerctl_report.md +++ /dev/null @@ -1,113 +0,0 @@ -# Grammar vs. regexp parser — dockerctl scan benchmark - -Comparison of the comment-annotation parser before and after the -regexp→grammar migration, run against a sizable real-world corpus. - -> [!NOTE] -> This report measures the **Build** phase — the parser. It is no longer the -> whole story: [`bench_scanner_axes.md`](./bench_scanner_axes.md) measures the -> **Load** phase, which is 94% of a scan, along the axes the on-demand scanner -> work is steered by. Read that one for anything about loading, dependency -> types or memory retention; this one stands as the parser's own record. - -## Method - -- **Harness:** [`bench_dockerctl_test.go`](./bench_dockerctl_test.go) — a - self-contained program that touches only the public `Run` entry point plus - the two internal phases (`scanner.NewScanCtx` + `spec.NewBuilder`). These - exist identically in both trees, so the file compiles **verbatim** in each - worktree. -- **Corpus:** the go-swagger **dockerctl** client - (`github.com/go-swagger/dockerctl`) — ~502 Go files, ~160k LOC, 121 files - carrying `swagger:` annotations. -- **Worktrees compared:** - - **regexp** — tag `v0.34.0` (after the package-layout refactor landed, - before the grammar parser; still fully regexp-based). - - **grammar** — `master` (grammar-based parser). -- **Phase split:** the scan is measured in two parts so the parser change is - isolated from the unchanged front end: - - **Load** — `scanner.NewScanCtx`: `go/packages` type-checking. *Identical* - work in both trees; a stable baseline. It dominates wall-clock, which is - why the end-to-end `Full` number barely moves. - - **Build** — `spec.NewBuilder(...).Build()`: annotation parsing + Swagger - emission. **This is the phase that changed**, and where the win shows up. -- **Measurement:** wall-clock time plus heap allocations (`TotalAlloc` bytes - and `Mallocs` count deltas, GC-fenced). Figures below are the stabilized - 10-iteration `-benchmem` run; a single iteration is enough and reproduces - the same ratios. - -## Results — Build phase (the regexp→grammar delta) - -| Metric | Regexp (v0.34.0) | Grammar (master) | Improvement | -| ------- | ---------------- | ---------------- | ---------------------- | -| Time | 225.8 ms/op | 115.8 ms/op | **1.95× faster** (−49%) | -| Memory | 67.7 MB/op | 28.8 MB/op | **2.35× less** (−57%) | -| Allocs | 601,674/op | 298,105/op | **2.02× fewer** (−50%) | - -Single-iteration figures agree: regexp 212 ms / 65 MB / 601k allocs vs. -grammar 128 ms / 28 MB / 298k allocs. - -## Load phase (baseline — unchanged front end) - -| Metric | Regexp | Grammar | -| ------- | ------------- | ------------- | -| Memory | ~1088 MB | ~1088 MB | -| Allocs | ~12.19M | ~12.19M | - -Effectively identical, as expected — both call the same `go/packages` -loader. This is the bulk of total cost, so it is the right thing to factor -out when judging the parser. - -## Verdict - -The grammar parser **wins decisively** on this corpus: it nearly halves -parse time and more than halves both bytes allocated and allocation count. -The regexp engine's repeated pattern-matching and intermediate string -allocation is exactly what the grammar-based single-pass approach removes. - -## Reproduce - -From either worktree: - -```sh -# one-shot human-readable report -go test ./internal/integration/ -run TestDockerctlBenchReport -v - -# stabilized Go benchmark -go test ./internal/integration/ -run x -bench BenchmarkDockerctl/Build -benchtime=10x -benchmem -``` - -Point the harness at the client with `CODESCAN_BENCH_DIR` if it is not at the -default path; the program skips (rather than fails) when the corpus is absent. - -## Profiling - -`go test -bench` cannot collect a `-cpuprofile`/`-memprofile` across multiple -packages, so [`bench_dockerctl_profile_test.go`](./bench_dockerctl_profile_test.go) -drives `runtime/pprof` directly (stdlib only — no `github.com/pkg/profile` -dependency). It profiles the **Build** phase by default, loading the -`ScanCtx` once and replaying `Build()` in a loop so the 100 Hz CPU sampler -gathers enough samples on the ~115 ms workload. - -```sh -# CPU profile of the grammar parser (Build phase) -CODESCAN_BENCH_CPUPROFILE=cpu.out \ - go test ./internal/integration/ -run TestDockerctlProfile -v -go tool pprof -http=: cpu.out - -# heap profile (alloc_space) -CODESCAN_BENCH_MEMPROFILE=mem.out CODESCAN_BENCH_ITERS=200 \ - go test ./internal/integration/ -run TestDockerctlProfile -v -go tool pprof -sample_index=alloc_space -http=: mem.out -``` - -Knobs: `CODESCAN_BENCH_PHASE=build|full` (default `build`), -`CODESCAN_BENCH_ITERS=N` (default 30). The test skips unless at least one -profile path is set. - -**Caveat:** the CPU profile brackets only the replay loop and so cleanly -isolates the chosen phase (Load runs before `StartCPUProfile`). The heap -profile is process-cumulative, so for `phase=build` its `alloc_space` still -includes the one-time Load (`go/types`) allocations — raise -`CODESCAN_BENCH_ITERS` so the replayed Build dominates, or compare the same -iter count across worktrees and read the delta. diff --git a/internal/integration/bench_dockerctl_test.go b/internal/integration/bench_dockerctl_test.go deleted file mode 100644 index 80d2bbb2..00000000 --- a/internal/integration/bench_dockerctl_test.go +++ /dev/null @@ -1,173 +0,0 @@ -// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers -// SPDX-License-Identifier: Apache-2.0 - -package integration_test - -// Cross-worktree comparison benchmark. -// -// This program scans a sizable, real-world generated client (the docker -// engine API client produced by go-swagger, at github.com/go-swagger/dockerctl) -// and reports the wall-clock time and heap allocations for a single full scan. -// -// It is intentionally self-contained: it only touches the public Run entry -// point plus the two internal phases (scanner.NewScanCtx + spec.NewBuilder), -// all of which exist identically in both the grammar-based worktree and the -// pre-grammar (v0.34.0, regexp-based) worktree. Copy this file verbatim into -// the other worktree's internal/integration directory and run the same -// command to compare. -// -// The scan splits into two phases: -// -// - Load: scanner.NewScanCtx — go/packages type-checking. Identical work -// in both worktrees; serves as a stable baseline. -// - Build: spec.NewBuilder(...).Build() — comment-annotation parsing and -// Swagger emission. THIS is the phase that changed from regexp to -// grammar; it is where any win or regression shows up. -// -// - Full: codescan.Run — the whole thing, fresh (Load+Build), for a -// single end-to-end number. -// -// Run it (single iteration is enough): -// -// go test ./internal/integration/ -run TestDockerctlBenchReport -v -// -// or as a true Go benchmark: -// -// go test ./internal/integration/ -run x -bench BenchmarkDockerctl -benchtime=1x -benchmem -// -// Point it at the reference client with CODESCAN_BENCH_DIR if it is not at the -// default location below; the program skips (rather than fails) when absent. - -import ( - "runtime" - "testing" - "time" - - "github.com/go-openapi/codescan" - "github.com/go-openapi/codescan/internal/builders/spec" - "github.com/go-openapi/codescan/internal/scanner" - "github.com/go-openapi/testify/v2/require" -) - -// dockerctlDir, and the corpus locations it resolves, live in bench_corpus_test.go — they are -// shared with the loader/discovery axes added alongside this benchmark. - -func benchOptions(dir string) *codescan.Options { - return &codescan.Options{ - WorkDir: dir, - Packages: []string{"./..."}, - ScanModels: true, - } -} - -// phaseStats captures the cost of one measured phase. -type phaseStats struct { - name string - elapsed time.Duration - allocPB uint64 // bytes allocated during the phase (cumulative TotalAlloc delta) - allocsN uint64 // number of heap allocations during the phase (Mallocs delta) -} - -// measure runs fn once, fencing it with two GCs so the heap counters reflect -// only fn's own allocations, and records elapsed time + allocation deltas. -func measure(name string, fn func()) phaseStats { - var before, after runtime.MemStats - - runtime.GC() - runtime.ReadMemStats(&before) - - start := time.Now() - fn() - elapsed := time.Since(start) - - runtime.ReadMemStats(&after) - - return phaseStats{ - name: name, - elapsed: elapsed, - allocPB: after.TotalAlloc - before.TotalAlloc, - allocsN: after.Mallocs - before.Mallocs, - } -} - -func (p phaseStats) report(tb testing.TB) { - tb.Helper() - tb.Logf("%-6s time=%-12s alloc=%6.1f MB allocs=%d", - p.name, - p.elapsed.Round(time.Microsecond), - float64(p.allocPB)/(1024*1024), - p.allocsN, - ) -} - -// TestDockerctlBenchReport runs a single scan of the reference client and -// prints a human-readable timing + allocation report for each phase. This is -// the primary cross-worktree comparison vehicle: one iteration is enough. -func TestDockerctlBenchReport(t *testing.T) { - dir := dockerctlDir(t) - opts := benchOptions(dir) - - t.Logf("scanning reference client at %s", dir) - - // Phase split: Load then Build, sharing one ScanCtx so Build sees exactly - // what Run would feed it. - var ctx *scanner.ScanCtx - loadStats := measure("Load", func() { - var err error - ctx, err = scanner.NewScanCtx(opts) - require.NoError(t, err) - }) - - var sp any - buildStats := measure("Build", func() { - builder := spec.NewBuilder(opts.InputSpec, ctx, opts.ScanModels) - built, err := builder.Build() - require.NoError(t, err) - sp = built - }) - require.NotNil(t, sp) - - // Full end-to-end, fresh, for a single combined number. - fullStats := measure("Full", func() { - built, err := codescan.Run(opts) - require.NoError(t, err) - require.NotNil(t, built) - }) - - t.Log("=== dockerctl scan benchmark (single iteration) ===") - loadStats.report(t) - buildStats.report(t) - fullStats.report(t) -} - -// BenchmarkDockerctl is the same workload exposed to the Go benchmark -// framework. Run with -benchtime=1x -benchmem for a one-shot allocation -// profile, or with the default settings for a stabilized timing. -func BenchmarkDockerctl(b *testing.B) { - dir := dockerctlDir(b) - opts := benchOptions(dir) - - b.Run("Full", func(b *testing.B) { - b.ReportAllocs() - for range b.N { - built, err := codescan.Run(opts) - require.NoError(b, err) - require.NotNil(b, built) - } - }) - - b.Run("Build", func(b *testing.B) { - b.ReportAllocs() - for range b.N { - b.StopTimer() - ctx, err := scanner.NewScanCtx(opts) - require.NoError(b, err) - b.StartTimer() - - builder := spec.NewBuilder(opts.InputSpec, ctx, opts.ScanModels) - built, err := builder.Build() - require.NoError(b, err) - require.NotNil(b, built) - } - }) -} diff --git a/internal/integration/bench_scanner_axes.md b/internal/integration/bench_scanner_axes.md deleted file mode 100644 index 2d4ca2b5..00000000 --- a/internal/integration/bench_scanner_axes.md +++ /dev/null @@ -1,347 +0,0 @@ -# Scanner measurement harness — the axes, how to run them, how to read them - -This is the control loop for the on-demand scanner work: what a scan pays, and what it gets for it. -It exists because none of the figures that steer that work were reproducible from the repo. - -Two halves, and only the pair is a control loop: - -| half | what it answers | where | -|---|---|---| -| **cost** | what a load costs and what shape it leaves behind | `bench_corpus_test.go`, `bench_loaders_test.go`, `bench_discovery_test.go`, `bench_workingset_test.go` | -| **output** | whether a cheaper configuration still produces the same spec | `loader_agreement_test.go` | - -A lazy load that is faster *and* silently drops a `swagger:strfmt` format is precisely what must not -ship, so a cost improvement is only reportable alongside the output half. - -The older [`bench_dockerctl_report.md`](./bench_dockerctl_report.md) and -[`bench_dockerctl_design_notes.md`](./bench_dockerctl_design_notes.md) measure a different thing — -the regexp→grammar parser migration — and still stand. They are no longer the whole story: they -measure the *build* phase, and everything here measures the *load*. - -## Corpora - -Two external checkouts, neither vendored. Both are skipped (never failed) when absent, which is the -normal case in CI. - -| corpus | default location | override | shape | -|---|---|---|---| -| dockerctl | `/home/fred/src/github.com/go-swagger/dockerctl` | `CODESCAN_BENCH_DIR` | generated client: 336-package closure, **2** annotated packages | -| go-swagger | `/home/fred/src/github.com/go-swagger/go-swagger` | `CODESCAN_BENCH_GOSWAGGER_DIR` | hand-written: 440-package closure, **14** annotated packages | - -Two shapes on purpose. dockerctl is lucky — nearly all of it is unannotated generated CLI code — and -a conclusion that only holds there is not a conclusion. - -## Running it - -Everything cost-related is gated behind `CODESCAN_BENCH=1`. The gate is not about CI (the corpora -are absent there anyway); it is so that a plain `go test ./...` on a machine that *does* have them -does not grow by minutes. - -```sh -# the whole cost half -CODESCAN_BENCH=1 go test ./internal/integration/ -run 'TestBench' -v -timeout 30m - -# one axis -CODESCAN_BENCH=1 go test ./internal/integration/ -run TestBenchLoadLadders -v -``` - -| axis | what it measures | entry point | -|---|---|---| -| **A** | loader strategy: `go/packages` vs the toolchain-free loader | `TestBenchLoadLadders` | -| **B** | dependency types: from source vs from compiled export data, **warm and cold** | `TestBenchLoadLadders`, `TestBenchLoadLaddersCold` | -| **C** | a bundled export-data blob (the toolchain-free route) | `TestBenchLoadLadders` | -| **D** | the raw `go/packages` LoadMode ladder | `TestBenchLoadLadders` | -| **E** | the *shape* of a load, not its cost | every rung of the above | -| **F** | discovery cost, separated from loading | `TestBenchDiscovery` | -| **G** | the working set: annotated roots → type-reference closure | `TestBenchWorkingSet` | -| **H** | what the per-dependency export-data policy loads vs what a scan reaches | `TestBenchDependencyRouting` | - -Plus `BenchmarkLoadStrategies`, the same A/B/C rungs as a real benchmark for `benchstat`: - -```sh -go test ./internal/integration/ -run x -bench BenchmarkLoadStrategies -benchtime=1x -``` - -### Why some axes are tests rather than benchmarks - -What most of these produce is a *table* — a ladder of rungs with four or five columns each — not a -scalar that gets faster or slower. `go test -bench` has one output slot per iteration, and forcing -Axis G ("4 packages of 336") through it would lose the thing worth reading. Axes A–C are also a -like-for-like comparison, so those additionally exist as `BenchmarkLoadStrategies`. - -### Cold vs warm build cache - -This distinction matters more than any other single knob, and the harness refuses to leave it -implicit. - -- **Warm** is the default, and it is *enforced*: every rung in `TestBenchLoadLadders` runs once with - the result discarded before the measured run. A number in that report is never accidentally cold. -- **Cold** is `TestBenchLoadLaddersCold`, opt-in via `CODESCAN_BENCH_COLD=1`. It gives each rung a - private, empty `GOCACHE` (a `t.TempDir()`), so "cold" means a well-defined extreme rather than - "whatever had not been built yet". `GOMODCACHE` is left alone, so nothing is downloaded. - -Only the rungs whose cost depends on the build cache are run cold; a load that reads dependency -source compiles nothing, so a fresh `GOCACHE` would measure the same thing twice. - -```sh -CODESCAN_BENCH=1 CODESCAN_BENCH_COLD=1 \ - go test ./internal/integration/ -run TestBenchLoadLaddersCold -v -timeout 60m -``` - -### Axis C — the export-data blob - -`Options.ExportData` needs a blob, which is not checked in: the export format is tied to the Go -release that produced it, and a blob covers one corpus's dependency closure and no other. Produce -one per corpus and point the harness at it: - -```sh -go run ./hack/genexportdata -dir /path/to/dockerctl -out /tmp/dockerctl-exportdata.zip ./... -export CODESCAN_BENCH_EXPORTDATA_DOCKERCTL=/tmp/dockerctl-exportdata.zip - -go run ./hack/genexportdata -dir /path/to/go-swagger -out /tmp/goswagger-exportdata.zip ./... -export CODESCAN_BENCH_EXPORTDATA_GOSWAGGER=/tmp/goswagger-exportdata.zip -``` - -`CODESCAN_BENCH_EXPORTDATA` is honoured as a fallback for whichever corpus has no variable of its -own. Without a blob the axis is skipped with the command printed — it is never silently omitted. - -Handing a corpus somebody else's blob does not fail: every package the blob does not cover falls -back to source. That is exactly why the variables are per corpus, because the result would look -like a measurement and be a fallback. Measured, the difference is large — go-swagger under -dockerctl's blob reads 168 packages from source in 0.70 s / 123 MB; under its own, 31 packages in -0.33 s / 30 MB. - -## Reading the output - -Every cost row shares one format: - -``` -[B] loader: go/packages + CompiledDependencies 0.42s retained= 105 MB graph=336 goFiles=336 types= 42 fromSource= 19 exportServed= 23 exportOnly= 0 astFiles=502 -``` - -| column | meaning | -|---|---| -| `retained` | heap **held** after the load, GC-fenced, measured as a delta against a pre-rung reading. Not `TotalAlloc`: the scanner's problem is retention, not churn | -| `graph` | packages reachable through `Package.Imports` | -| `goFiles` | of those, how many have `GoFiles` — i.e. whose source is *locatable* | -| `types` | of those, how many have complete type information | -| `fromSource` | of those, how many have an AST | -| `exportServed` | of those, how many have complete types and **no** AST | -| `exportOnly` | how many the loader reported (`WithOnExportOnly`) as having no reachable source at all | -| `astFiles` | parsed files held | - -`fromSource` / `exportServed` / `exportOnly` are the three-way state the on-demand work turns on: -loaded, served-without-syntax, and unreadable. The middle one is the interesting one — source -locatable, AST absent — and each rung also spotlights one dependency by name -(`CODESCAN_BENCH_SPOTLIGHT`, default `github.com/go-openapi/strfmt`) so the state has a face: - -``` -github.com/go-openapi/strfmt: GoFiles=9 Types=true Syntax=0 -``` - -### Two caveats worth knowing before quoting a number - -1. **`graph` under `ExportData` is not the closure size.** A package served from export data has no - `Imports` map to walk (`internal/packages.exportedPackage` builds an empty one — nothing - downstream reads it), so the walk stops there. Under that rung `graph` means "packages the loader - materialised", which is the figure the retained heap corresponds to. Closure size comes from a - rung that walks the whole graph. -2. **`retained` is a whole-process delta.** It is fenced with GCs and taken before/after, so it is - the heap the rung *holds*, but a rung that ends with less live heap than it started reports 0 - rather than a negative. - -## Reference figures - -Measured on this branch, go1.26.5, warm build and page cache, single runs. Reproduce with the -commands above; they move by ±10% run to run. - -### Axis D — the LoadMode ladder (dockerctl, 336 packages) - -| mode | wall | retained | astFiles | -|---|---|---|---| -| current (`Deps+Types+Syntax+TypesInfo`) | 0.91 s | 512 MB | 2354 | -| all types, no syntax | 0.69 s | 80 MB | 0 | -| `&^ NeedDeps` (= `CompiledDependencies`) | 0.39 s | 105 MB | 502 | -| syntax only, no types | 0.16 s | 38 MB | 502 | -| `go list` only | 0.08 s | 1 MB | 0 | - -### Axes A–C — what the scanner actually runs - -dockerctl: - -| loader | wall | retained | graph | fromSource | exportServed | -|---|---|---|---|---|---| -| go/packages (default) | 0.96 s | 512 MB | 336 | 335 | 1 | -| go/packages + `CompiledDependencies` | 0.42 s | 105 MB | 336 | 19 | 23 | -| toolchain-free | 1.12 s | 277 MB | **322** | 322 | 0 | -| toolchain-free + `ExportData` | 0.39 s | 87 MB | 61¹ | 20 | 41 | - -go-swagger: - -| loader | wall | retained | graph | fromSource | exportServed | -|---|---|---|---|---|---| -| go/packages (default) | 0.81 s | 550 MB | 440 | 439 | 1 | -| go/packages + `CompiledDependencies` | 0.22 s | 30 MB | 440 | 30 | 75 | -| toolchain-free | 1.22 s | 288 MB | **433** | 433 | 0 | -| toolchain-free + `ExportData` | 0.33 s | 30 MB | 119¹ | 31 | 88 | - -¹ see caveat 1 above — not a closure size. - -The **graph divergence between strategies is real and reproducible**: 336 vs 322 on dockerctl, -440 vs 433 on go-swagger. It belongs to the loader branch, not to this harness, but it is visible -from here. - -### Axis B — cold vs warm, the distinction that matters most - -Each rung with its own empty `GOCACHE`: - -| rung | dockerctl warm | dockerctl cold | go-swagger warm | go-swagger cold | -|---|---|---|---|---| -| source deps (`Deps+Types+Syntax+TypesInfo`) | 0.93 s | 1.46 s | 0.81 s | 1.54 s | -| compiled deps | **0.40 s** | **9.08 s** | **0.22 s** | **8.40 s** | - -Warm, compiled dependencies are 2.3× *faster*. Cold, 6× *slower*. This is not "faster"; it is -"faster once the compiler has already done the work", and quoting one figure without the other is -how a benchmark misleads. - -Rows are named by configuration rather than by default: the second is what a plain scan does since -v0.36.4, and the first is what `SkipCompiledDependencies` selects. The cold column is the whole of the -reason that opt-out exists. - -The shape is identical in both states — 23 packages export-served either way — so nothing about the -result changes, only when the compilation is paid for. - -Note this is a harder "cold" than the plan's earlier 4.06 s reading, which was taken on a partially -warm cache. An empty `GOCACHE` is the well-defined extreme; a working developer's cache sits -somewhere between the two columns. - -### Axis F — discovery without loading - -| step | dockerctl | go-swagger | -|---|---|---| -| `go list` (graph + `GoFiles`) | 0.08 s — 335 pkgs, 2341 files, 22.3 MB | 0.10 s — 439 pkgs, 2378 files, 23.4 MB | -| byte prefilter over **all** files | 0.05 s — 127/2341 survive (5.4%) | 0.06 s — 142/2378 survive (6.0%) | -| parse + classify survivors | 0.03 s | 0.03 s | -| **total** | **0.16 s**, 38 MB transient, **1 MB retained** | **0.18 s**, 38 MB transient, **1 MB retained** | - -A complete annotation index over the whole closure, retaining nothing, for under a fifth of a -second — against 0.93 s and 512 MB for the load that produces the same index as a side effect. (The -`go list` row is the one that moves most between runs; an earlier reading put it at 0.16 s.) - -The prefilter can only over-admit (a `swagger:` in a string literal costs one wasted parse) and -never under-admit, since a real annotation contains that literal by definition. - -### Axis G — the working set - -| corpus | closure | annotated roots | type-reference closure | never needed | -|---|---|---|---|---| -| dockerctl | 336 | 2 (168 decls) | **4** | 332 (98.8%) | -| go-swagger | 440 | 14 (144 decls) | **19** | 421 (95.7%) | - -Packages reached purely by following a field's type, carrying no annotation at all — dockerctl: -`github.com/oklog/ulid`, `time`; go-swagger: `strfmt/internal/countries`, -`scan-repo-boundary/makeplans`, `oklog/ulid/v2`, `golang.org/x/text/currency`, `time`. - -The walk approximates the schema builder (struct fields, elems, map keys/values, type args, -interface methods, signature results). It is good enough to *size* the closure and not good enough -to prove an incremental loader reaches the same set — that is what the output half is for. - -### Axis H — what is parsed vs what is needed - -The per-dependency export-data policy routes a dependency on a **substring** test for `swagger:`. -The scanner's real rule requires the literal to *start* the comment line. The gap between those two -rules, and between both and the closure a scan actually reaches, is the cost still on the table: - -dockerctl (336 packages, 2351 files): - -| set | wall | retained | packages | files | -|---|---|---|---|---| -| everything in the closure | 0.39 s | 175 MB | 336 (100%) | 2351 | -| admitted by substring (**the loader's rule**) | 0.04 s | 16 MB | 14 (4.2%) | 299 | -| admitted by line-start (a real annotation) | 0.01 s | 5 MB | 2 (0.6%) | 91 | -| reached by the type-reference closure | 0.02 s | 6 MB | 4 (1.2%) | 102 | - -go-swagger (440 packages, 2388 files): - -| set | wall | retained | packages | files | -|---|---|---|---|---| -| everything in the closure | 0.43 s | 188 MB | 440 (100%) | 2388 | -| admitted by substring | 0.02 s | 8 MB | 35 (8.0%) | 197 | -| admitted by line-start | 0.02 s | 6 MB | 22 (5.0%) | 147 | -| reached by the type-reference closure | 0.02 s | 6 MB | 19 (4.3%) | 111 | - -Read it this way: the per-dependency policy already captures nearly the whole prize (0.39 s → 0.04 s -on dockerctl). What is left is the **substring → line-start** step, worth ~0.03 s and ~11 MB on -dockerctl and ~nothing on go-swagger. That is small, and worth reporting precisely *because* it is -small: it says the next win is not here. - -The 14-vs-2 gap on dockerctl is the hazard the plan predicted, now measured: codescan's own godoc, -read as a dependency, discusses `swagger:` constantly, so twelve packages are loaded from source for -prose. `isAnnotationLine` in `bench_discovery_test.go` is the tighter rule, restated without a -regexp. - -## The output half — spec agreement across configurations - -[`loader_agreement_test.go`](./loader_agreement_test.go) A/Bs whole marshalled documents across the -configurations that change *how dependency types arrive*, over the fixture corpus. It extends -`TestLoaderChoice_AgreeOnTheRealFilesystem`, which had the right instinct (whole documents, not spot -checks) on one fixture and one axis. - -Configurations compared against the source scan: `ToolchainFreeLoader` (the control — it changes the -loader, not where types come from), `compiled-dependencies` (the default), and `ExportData` when a -blob is given. - -Two tiers: - -```sh -# default: the curated targets — the fixtures whose meaning comes from a dependency -go test ./internal/integration/ -run TestLoaderConfigs_AgreeOnTheFixtureCorpus - -# the whole corpus (minutes) -CODESCAN_AB_CORPUS=1 go test ./internal/integration/ -run TestLoaderConfigs -timeout 60m - -# with the export-data configuration too -go run ./hack/genexportdata -dir testdata -out /tmp/testdata-exportdata.zip std -CODESCAN_AB_EXPORTDATA=/tmp/testdata-exportdata.zip go test ./internal/integration/ -run TestLoaderConfigs -``` - -A configuration that legitimately differs is recorded in `abExpected` with a reason, and asserted as -a **difference**. When the stream closes one of those gaps the assertion fails — which is the point; -an expectation that quietly survives its own fix is not an expectation. Regenerate the table with -`CODESCAN_AB_REPORT=1`. - -### What it finds - -Nothing. Over all 306 fixture bundles, no (configuration, target) pair diverges, and `abExpected` is -an empty map whose documentation says staying empty is the contract. How dependency types arrive is -a question about cost, not about meaning — which is what let compiled dependencies become the -default in v0.36.4. - -The reference is deliberately the **source** scan (`abBaseline` sets `SkipCompiledDependencies`) -rather than the default one. Now that the default takes the shortcut, holding it as the reference -would put the shortcut on both sides of the comparison. - -### What it found before, and why that mattered - -Worth keeping, because it is the measurement that decided the design. Twelve pairs diverged, in -three families: - -| family | what happened | affected | -|---|---|---| -| **the declaration contract** | the scan **failed**, it did not degrade — a builder asked for the declaration of a type whose package had types and no AST | 4 targets under `compiled-dependencies`, 3 of them also under `export-data` | -| **a dependency's own annotations were lost** | `format: date-time` / `email` / `uuid` vanished, because the marks live in strfmt's source | 3 targets, `compiled-dependencies` only | -| **a dependency-declared model collapsed** | the definition itself went, not one keyword — `Booking` is declared in `scan-repo-boundary/makeplans` | 2 targets, `compiled-dependencies` only | - -Two things came out of that: - -1. **It did not merely lose formats — on four targets the scan errored.** `unable to find package - and source file for: time.Duration`, `can't find source file for type: interface{Write(p []byte) - (n int, err error)}` (io.Writer), the same for `reflect.Type`, and `unable to find package and - source file for: github.com/go-openapi/strfmt.DateTime`. A stronger statement than "quietly - poorer", and it made the declaration contract a shipping-blocker rather than a tidiness item. - It is what the on-demand read-back and the name-and-line field bridge were built to close; see - [§compiled-dependencies](../scanner/README.md#compiled-dependencies). -2. **The per-dependency policy already fixed the annotation families.** `export-data` lost no format - and no definition, because a dependency whose source carries `swagger:` is read from source. Its - three remaining divergences were all the declaration contract, all on **stdlib** types, where - there is no annotated source to fall back to. diff --git a/internal/integration/bench_workingset_test.go b/internal/integration/bench_workingset_test.go deleted file mode 100644 index b79a974d..00000000 --- a/internal/integration/bench_workingset_test.go +++ /dev/null @@ -1,280 +0,0 @@ -// SPDX-FileCopyrightText: Copyright 2015-2025 go-swagger maintainers -// SPDX-License-Identifier: Apache-2.0 - -package integration_test - -// Axis G — the working set, and Axis H — what the per-dependency export-data policy leaves on the -// table. -// -// Axis G asks how much of a loaded closure anything ever needs: follow types outward from the -// annotated declarations and count the PACKAGES reached. The annotated set is not the working set — -// a package carrying no annotation at all is pulled in purely because a field is typed that way. -// -// Axis H prices the gap between three sets of dependencies, which under the per-dependency -// export-data policy is where the remaining cost lives: -// -// all every dependency in the closure -// substring those whose source contains the literal `swagger:` — the loader's own admission -// rule, and therefore what it loads from source today -// line-start those that genuinely carry an annotation (parsers.rxCommentPrefix's rule) -// reached those the type-reference closure actually lands in -// -// substring ⊇ line-start, and reached is a different set again — it is neither a subset nor a -// superset, since a package can be reached by a field type while saying nothing itself. What the -// harness reports is the cost of parsing each set's source, so the gap between "what is loaded" and -// "what is needed" is a number rather than an argument. -// -// See bench_scanner_axes.md. - -import ( - "fmt" - "go/parser" - "go/token" - "go/types" - "runtime" - "sort" - "testing" - "time" - - "golang.org/x/tools/go/packages" - - "github.com/go-openapi/testify/v2/require" -) - -// maxFollowDepth bounds the structural walk. Recursive types are cut by the seen-set, not by this; -// the bound only guards pathological nesting. -const maxFollowDepth = 64 - -// typeCloser walks outward from a declaration the way a schema builder does, recording which -// PACKAGES the walk lands in. That set — not the annotated set — is the scanner's real working set. -type typeCloser struct { - seenTypes map[string]bool - touched map[string]bool -} - -func newTypeCloser() *typeCloser { - return &typeCloser{seenTypes: map[string]bool{}, touched: map[string]bool{}} -} - -// follow records t's declaring package and recurses through its structure. -// -// This APPROXIMATES the schema builder rather than replaying it: struct fields, element types, map -// keys and values, type arguments, interface methods and signature results. Good enough to size the -// closure; NOT good enough to prove an incremental loader reaches the same set — that is what the -// spec A/B in loader_agreement_test.go is for. -func (c *typeCloser) follow(t types.Type, depth int) { - if t == nil || depth > maxFollowDepth { - return - } - - switch tt := t.(type) { - case *types.Named: - if obj := tt.Obj(); obj != nil && obj.Pkg() != nil { - key := obj.Pkg().Path() + "." + obj.Name() - if c.seenTypes[key] { - return - } - c.seenTypes[key] = true - c.touched[obj.Pkg().Path()] = true - } - for arg := range tt.TypeArgs().Types() { - c.follow(arg, depth+1) - } - c.follow(tt.Underlying(), depth+1) - case *types.Alias: - if obj := tt.Obj(); obj != nil && obj.Pkg() != nil { - c.touched[obj.Pkg().Path()] = true - } - c.follow(types.Unalias(tt), depth+1) - case *types.Struct: - for field := range tt.Fields() { - c.follow(field.Type(), depth+1) - } - case *types.Pointer: - c.follow(tt.Elem(), depth+1) - case *types.Slice: - c.follow(tt.Elem(), depth+1) - case *types.Array: - c.follow(tt.Elem(), depth+1) - case *types.Map: - c.follow(tt.Key(), depth+1) - c.follow(tt.Elem(), depth+1) - case *types.Interface: - for method := range tt.Methods() { - c.follow(method.Type(), depth+1) - } - case *types.Signature: - if res := tt.Results(); res != nil { - for v := range res.Variables() { - c.follow(v.Type(), depth+1) - } - } - } -} - -// workingSet is one corpus measured: the loaded graph, the discovery result over it, and the -// type-reference closure from the annotated roots. -type workingSet struct { - all map[string]*packages.Package - discovery discoveryResult - reached map[string]bool - roots int -} - -// measureWorkingSet loads the closure with types but no syntax — the cheapest load the walk can -// follow — then discovers annotations and closes over the types they name. -func measureWorkingSet(tb testing.TB, corpus benchCorpus) workingSet { - tb.Helper() - - const mode = packages.NeedName | packages.NeedFiles | packages.NeedImports | - packages.NeedDeps | packages.NeedTypes - - pkgs, err := packages.Load(&packages.Config{Dir: corpus.dir, Mode: mode}, corpus.pattern) - require.NoError(tb, err, "load") - - ws := workingSet{all: walkGraph(pkgs)} - ws.discovery = discoverAnnotations(ws.all) - - closer := newTypeCloser() - for _, a := range ws.discovery.annotated { - p := ws.all[a.pkgPath] - if p == nil || p.Types == nil { - continue - } - closer.touched[a.pkgPath] = true - for _, name := range a.names { - obj := p.Types.Scope().Lookup(name) - if obj == nil { - continue - } - ws.roots++ - closer.follow(obj.Type(), 0) - } - } - ws.reached = closer.touched - - return ws -} - -// TestBenchWorkingSet is Axis G: how much of the loaded closure anything ever needs. -func TestBenchWorkingSet(t *testing.T) { - requireBenchEnabled(t) - - for _, corpus := range benchCorpora(t) { - t.Run(corpus.name, func(t *testing.T) { - ws := measureWorkingSet(t, corpus) - - std, external := 0, 0 - reached := make([]string, 0, len(ws.reached)) - for p := range ws.reached { - reached = append(reached, p) - if isStdlibPath(p) { - std++ - } else { - external++ - } - } - sort.Strings(reached) - - t.Logf("=== the working set — %s ===", corpus.name) - t.Logf("graph loaded %d packages", len(ws.all)) - t.Logf("annotated packages (roots) %d (%d annotated type decls)", - len(ws.discovery.annotated), ws.roots) - t.Logf("type-reference closure %d packages (%d non-stdlib, %d stdlib)", - len(ws.reached), external, std) - t.Logf("NEVER NEEDED %d / %d (%.1f%%)", - len(ws.all)-len(ws.reached), len(ws.all), - percent(len(ws.all)-len(ws.reached), len(ws.all))) - - annotated := make(map[string]bool, len(ws.discovery.annotated)) - for _, a := range ws.discovery.annotated { - annotated[a.pkgPath] = true - } - for _, p := range reached { - mark := " " // reached only by following a type: invisible to the annotation prefilter - if annotated[p] { - mark = " *" - } - t.Logf(" %s %s", mark, p) - } - }) - } -} - -// parseCost parses the source of a set of packages the way the loader would when it decides to read -// one, and reports what that costs. -// -// ParseComments is the point: comments are the only reason a dependency's source is worth reading -// at all, and they are also what makes parsing expensive. -func parseCost(all map[string]*packages.Package, want map[string]bool) (elapsed time.Duration, retained uint64, pkgs, files int) { - base := retainedHeap() - - start := time.Now() - fset := token.NewFileSet() - parsed := make([]any, 0, len(want)) - for path := range want { - p, ok := all[path] - if !ok || len(p.GoFiles) == 0 { - continue - } - pkgs++ - for _, name := range p.GoFiles { - af, err := parser.ParseFile(fset, name, nil, parser.ParseComments|parser.SkipObjectResolution) - if err != nil { - continue - } - files++ - parsed = append(parsed, af) - } - } - elapsed = time.Since(start) - retained = retainedSince(base) - - runtime.KeepAlive(parsed) - runtime.KeepAlive(fset) - - return elapsed, retained, pkgs, files -} - -// TestBenchDependencyRouting is Axis H: what the per-dependency export-data policy loads from -// source, against what a scan actually reaches. -// -// The policy routes a dependency on the naive substring test, so it pays a full source load for -// every package that merely MENTIONS `swagger:` — codescan's own godoc among them. Two tighter sets -// exist: the packages that genuinely carry an annotation, and the packages the type walk reaches. -// The distance between them is what is still on the table, and on a corpus with a large annotated -// dependency it will not be small. -func TestBenchDependencyRouting(t *testing.T) { - requireBenchEnabled(t) - - for _, corpus := range benchCorpora(t) { - t.Run(corpus.name, func(t *testing.T) { - ws := measureWorkingSet(t, corpus) - - everything := make(map[string]bool, len(ws.all)) - for path := range ws.all { - everything[path] = true - } - - t.Logf("=== dependency routing: what is parsed vs what is needed — %s ===", corpus.name) - for _, set := range []struct { - name string - want map[string]bool - }{ - {"all packages in the closure", everything}, - {"admitted by substring (the loader's rule)", ws.discovery.markerPkgs}, - {"admitted by line-start (a real annotation)", ws.discovery.realPkgs}, - {"reached by the type-reference closure", ws.reached}, - } { - elapsed, retained, pkgs, files := parseCost(ws.all, set.want) - reportRow(t, set.name, elapsed, retained, - sprintfPkgFiles(pkgs, files, percent(pkgs, len(ws.all)))) - } - }) - } -} - -// sprintfPkgFiles renders the tail columns of an Axis H row. -func sprintfPkgFiles(pkgs, files int, share float64) string { - return fmt.Sprintf("pkgs=%3d (%.1f%% of the closure) files=%d", pkgs, share, files) -} diff --git a/internal/scanner/README.md b/internal/scanner/README.md index 07ab21f7..08c82306 100644 --- a/internal/scanner/README.md +++ b/internal/scanner/README.md @@ -610,10 +610,11 @@ Go installation at all (a WASI guest, a browser): `StubStdlib`, which pays for t reach in fidelity, and `ExportData`, which pays in preparation. See [What it costs](#what-it-costs). -Figures are from `hack/loader-benchmark`, which compares a release against the -working tree over external corpora; its README carries the tables and the method. -They are indicative, not a promise — the balance moves with corpus size, and on a -small tree the pure-Go loader is *slower* warm than the standard one. +Figures are from [`internal/benchmarks`](../benchmarks/README.md), which measures two +generated projects that ship with this repository, warm and cold, against the +released versions as well as the working tree. That document carries the tables and +the method. They are indicative, not a promise — the balance moves with corpus size, +and on a small tree the pure-Go loader is *slower* warm than the standard one. ### Choosing