Repository navigation
Document how to compare an engine against the reference implementation - #27
Draft
Jim Garrison (garrison) wants to merge 1 commit into
Draft
Jim Garrison (garrison) wants to merge 1 commit into
Jim Garrison (garrison) wants to merge 1 commit into
Conversation
Coverage Report for CI Build 37057668831Coverage remained the same at 86.047%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
Someone trying to check an engine for numerical agreement with the implementation it replaces runs into the activation contract: the reference implementation and an engine cannot both be active in one process, and there is no way to relax that. The FAQ explained the restriction from several angles but never answered the question a user actually arrives with, which is what to do instead. Add an entry to the activation contract section covering it, giving three reasons one process per implementation is the right shape rather than a consolation prize. First, a same-process comparison would be a confounded experiment: enabling an engine can change denormal flags, the FPU control word, thread counts, or which BLAS is loaded, so code running after activation is the reference implementation as observed in a process an engine has already altered, and the tighter the bar, the more that matters. Second, benchmarking needs separate processes regardless, since timings taken after an engine has warmed caches or spun up a thread pool measure the state of the process. Third, output recorded as an artifact can be compared against a previous release, another machine, or a build that cannot be installed locally. The entry records why the restriction applies uniformly even though the first reason is weakest for an engine touching no global numerical state: Coheriq is built for the setting where non-Python code enters the picture, and a contract holding only for pure-Python engines would not be worth much in the HPC settings it targets. Also discourages reaching for pytest-forked downstream. Coheriq's own suite needs it to exercise the activation machinery repeatedly, but a domain or engine suite runs under a single activation and can share one process; depending on os.fork() would only cost it Windows support. Assisted-by: Claude Opus 5
Jim Garrison (garrison)
force-pushed
the
faq-comparing-implementations
branch
from
October 2, 2026 19:59
b139b13 to
767b92a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Someone trying to verify that an engine agrees numerically with the implementation it replaces runs into the activation contract: the reference implementation and an engine cannot both be active in one process, and there is no way to relax that. The Design FAQ already explained the restriction from several angles, but every explanation was framed as why the framework forbids something — none answered the question a user actually arrives with, which is what to do instead.
This adds an entry to the activation contract section giving three reasons that one process per implementation is the right shape rather than a consolation prize:
The entry also records why the restriction applies uniformly even though the first reason is weakest for an engine that touches no global numerical state: Coheriq is built for the setting where non-Python code enters the picture, and a contract holding only for pure-Python engines would not be worth much in the HPC settings it targets.
Two smaller points, both new to the published docs:
pytest-forkeddownstream. Coheriq's own suite needs--forkedto exercise the activation machinery repeatedly, but a domain or engine suite runs under a single activation and can share one process; depending onos.fork()would only cost it Windows support. This was previously recorded only in contributor-facing files. The entry links to the existing threads-and-fork question so the two do not appear to contradict each other.Docs-only; no source, test, or packaging changes.
tox -e docsandtox -e lintpass.This pull request was generated by Claude Opus 5 under my guidance.