Skip to content

Document how to compare an engine against the reference implementation - #27

Draft
Jim Garrison (garrison) wants to merge 1 commit into
mainfrom
faq-comparing-implementations
Draft

Jim Garrison (garrison) wants to merge 1 commit into
mainfrom
faq-comparing-implementations

Conversation

@garrison

Copy link
Copy Markdown
Member

Someone trying to verify that an engine agrees numerically with the implementation it replaces runs into the activation contract: the reference implementation and an engine cannot both be active in one process, and there is no way to relax that. The Design FAQ already explained the restriction from several angles, but every explanation was framed as why the framework forbids something — none answered the question a user actually arrives with, which is what to do instead.

This adds an entry to the activation contract section giving three reasons that one process per implementation is the right shape rather than a consolation prize:

  • A same-process comparison would be a confounded experiment. Enabling an engine can change denormal or flush-to-zero flags, the FPU control word, thread counts, or which BLAS is loaded, so code running after activation is the reference implementation as observed in a process an engine has already altered. The tighter the bar, the more this matters.
  • Benchmarking needs separate processes regardless, since timings taken after an engine has warmed caches or spun up a thread pool measure the state of the process. A comparison harness therefore needs this shape whether or not it also checks numerical agreement.
  • Output recorded as an artifact can be compared against a previous release, another machine, or a build that cannot be installed locally, and can be attached to a bug report or bisected against.

The entry also records why the restriction applies uniformly even though the first reason is weakest for an engine that touches no global numerical state: Coheriq is built for the setting where non-Python code enters the picture, and a contract holding only for pure-Python engines would not be worth much in the HPC settings it targets.

Two smaller points, both new to the published docs:

  • It discourages reaching for pytest-forked downstream. Coheriq's own suite needs --forked to exercise the activation machinery repeatedly, but a domain or engine suite runs under a single activation and can share one process; depending on os.fork() would only cost it Windows support. This was previously recorded only in contributor-facing files. The entry links to the existing threads-and-fork question so the two do not appear to contradict each other.
  • It names pytest-regressions as one worked example of comparing recorded artifacts, rather than adding comparison helpers to Coheriq itself.

Docs-only; no source, test, or packaging changes. tox -e docs and tox -e lint pass.


This pull request was generated by Claude Opus 5 under my guidance.

@coveralls

Coveralls (coveralls) commented Oct 2, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 37057668831

Coverage remained the same at 86.047%

Details

  • Coverage remained the same as the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 215
Covered Lines: 185
Line Coverage: 86.05%
Coverage Strength: 0.86 hits per line

💛 - Coveralls

@garrison Jim Garrison (garrison) added the documentation Improvements or additions to documentation label Oct 2, 2026
Someone trying to check an engine for numerical agreement with the
implementation it replaces runs into the activation contract: the
reference implementation and an engine cannot both be active in one
process, and there is no way to relax that.  The FAQ explained the
restriction from several angles but never answered the question a user
actually arrives with, which is what to do instead.

Add an entry to the activation contract section covering it, giving
three reasons one process per implementation is the right shape rather
than a consolation prize.  First, a same-process comparison would be a
confounded experiment: enabling an engine can change denormal flags,
the FPU control word, thread counts, or which BLAS is loaded, so code
running after activation is the reference implementation as observed in
a process an engine has already altered, and the tighter the bar, the
more that matters.  Second, benchmarking needs separate processes
regardless, since timings taken after an engine has warmed caches or
spun up a thread pool measure the state of the process.  Third, output
recorded as an artifact can be compared against a previous release,
another machine, or a build that cannot be installed locally.

The entry records why the restriction applies uniformly even though the
first reason is weakest for an engine touching no global numerical
state: Coheriq is built for the setting where non-Python code enters
the picture, and a contract holding only for pure-Python engines would
not be worth much in the HPC settings it targets.

Also discourages reaching for pytest-forked downstream.  Coheriq's own
suite needs it to exercise the activation machinery repeatedly, but a
domain or engine suite runs under a single activation and can share one
process; depending on os.fork() would only cost it Windows support.

Assisted-by: Claude Opus 5
@garrison
Jim Garrison (garrison) force-pushed the faq-comparing-implementations branch from b139b13 to 767b92a Compare October 2, 2026 19:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants