benchmark: compile time & serialization vs. popular Scala libraries - #47
Merged
Conversation
Adds a standalone benchmark module (benchmark/, a separate scala-cli build) that
measures the two axes that matter for a derivation codec against circe,
jsoniter-scala, uPickle, zio-json, borer, play-json and AVSystem GenCodec (the
design mcodec is modelled on):
- serialization/deserialization throughput (JMH) over 4 models: flat primitives,
deep nesting, a recursive GeoJSON-style ADT, and a 500-element collection batch
- clean-compile time swept over N derived codecs, with a scalac phase breakdown
and emitted-bytecode size
GenCodec has no Scala 3 release, so benchmark/gencodec/ is a nested Scala 2.13
build with the same models; its numbers feed both the runtime tables and the
compile sweep (the latter flagged as a cross-ecosystem, not like-for-like, point).
Pipeline: scripts/{aggregate,plot}.py turn the raw JMH/compile output into CSVs,
charts and the tables in docs/benchmarks.md; scripts/run_all.sh drives the lot.
There is no CI for the benchmark module — run it locally. scala-cli has no
directive to exclude the benchmark tree from `scala-cli .`, so the main
CI/publish/mima/docs workflows and the README build commands pass
--exclude benchmark.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCfiQMwXigsoCeMcR3jXP6
halotukozak
force-pushed
the
benchmarks
branch
from
September 2, 2026 11:27
1eb3bf2 to
db51456
Compare
halotukozak
added a commit
that referenced
this pull request
Sep 2, 2026
The MiMa baseline is now a worktree of a commit that contains benchmark/ (everything since #47), so `scala-cli package "$RUNNER_TEMP/old-src"` picks up benchmark/gencodec (Scala 2.13) and fails to resolve its deps. Completes the --exclude benchmark rollout started in #47. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TCfiQMwXigsoCeMcR3jXP6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A standalone benchmark suite (
benchmark/, a separate scala-cli build) comparing mcodec's compile time and serialization throughput against the field: circe, jsoniter-scala, uPickle, zio-json, borer, play-json, and AVSystem GenCodec (the design mcodec is modelled on).Full write-up + charts:
docs/benchmarks.md.Layout
benchmark/src/models/benchmark/src/codecs/JsonCodecfacadebenchmark/src/bench/@Param)benchmark/gencodec/benchmark/compile/benchmark/scripts/aggregate.py/plot.py→ CSVs, charts, tables injected into the report;run_all.shdrives it allNo CI — run it locally with
benchmark/scripts/run_all.sh.Headline numbers (laptop run — indicative, not citable)
transparent inlinederivation is the steepest-scaling of the group — ~10 s for 50 + 12 codecs vs. ~4–6 s for the Scala 3 macro libraries and ~4 s for GenCodec's Scala 2 macro — and emits ~3.5× more bytecode than circe/jsoniter. Cost concentrated in typer + inlining.Notes for review
scala-clihas no directive to exclude a subtree fromscala-cli ., soci.yml,docs.yaml,mima.yml,publish.ymlnow pass--exclude benchmark(andbenchmark/gencodecbuilds separately).ci.yml/docs.yaml/mima.ymlare synced fromshared/— these edits need upstreaming to the templates or they revert on the next repo-sync.scala-cli --power publish local . --exclude benchmark --project-version 0.0.0-BENCH --doc=false.benchmark/gencodec/Bench.scalastarts with// format: off— the repo scalafmt applies Scala-3-only rewrites that 2.13 can't parse.🤖 Generated with Claude Code