Hold the odds calculator's error bar to the width it claims - #146
Merged
Merged
Conversation
The containment check only asks whether the sampled answer lands inside the band. A band twice as wide as it should be passes that and every check like it, while telling the reader the calculator is half as good as it is. 24 seeded runs at 1,500 iterations, spread measured against the standard error the printed band claims. 36,000 showdowns, 4.5s, deterministic. scripts/odds-band-coverage.ts is the full sweep behind the published coverage table: 100 runs at each of 1,500, 5,000 and 20,000 against an enumerated ground truth.
Deploying pip-web with
|
| Latest commit: |
8545026
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8d255736.pip-web-9oj.pages.dev |
| Branch Preview URL: | https://guard-the-odds-band.pip-web-9oj.pages.dev |
The comment justifying the 0.7 and 1.4 bounds said all four measured ratios were inside the +/-9% that 60 seeds buy. 1.11 is not: 1/sqrt(2(n-1)) is 9.2% and that is one standard error, so 1.11 is 1.2 of them out. The 95% interval is 0.82 to 1.18, which every row is inside, so the conclusion holds and the sentence did not.
Contributor
Author
|
Ready. Gate green on One commit added since the first review: the comment justifying the 0.7 and 1.4 bounds said all four measured ratios were inside the ±9% that 60 seeds buy. 1.11 is not. 1/sqrt(2(n-1)) is 9.2% and that is one standard error on a standard deviation, not an interval, so the right sentence is the 95% interval, 0.82 to 1.18, which every row is inside. Bounds unchanged, conclusion unchanged. The gate covers the new test and the script's types. It does not cover anything on a page: no component, route or string changed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The calculator prints a 95% band on every sampled answer. Until now the tests only asked whether the answer landed inside that band. A band twice as wide as it should be passes that check and every check like it, while telling the reader the calculator is half as good as it is.
tests/oddsQuote.test.tsnow measures the estimator's spread across 60 fixed seeds at 1,500 iterations and compares it to the standard error the printed band claims. 90,000 showdowns, 6.8s, deterministic.scripts/odds-band-coverage.ts(pnpm odds-band-coverage) is the full sweep: 100 runs at each of 1,500, 5,000 and 20,000 against an enumerated ground truth. It stays a script rather than a test because 100 runs puts a +/-4.3 point band on a coverage rate, so an assertion on the rate either flakes or passes with a badly broken band.What the gate covered.
pnpm test:allgreen, 682 tests. Three mutants killed: band doubled (caught, 0.56x), band halved (caught, 2.23x), andestimateEquitysilently doing a quarter of the showdowns it reports while the band stays correct (caught at 2.24x, and this is the only one the existing width tests miss).What it did not cover. No UI. Nothing on
/poker-odds-calculatorchanged.A measurement, for the record. Running the sweep found no defect: the ratio of measured spread to claimed standard error is 1.03 at 500 iterations, 1.11 at 1,500, 1.05 at 5,000 and 0.92 at 20,000, all inside the +/-9% precision 60 seeds buy. Bias against the enumerated truth is at most 0.12 points at every size. The band is the right width.