Skip to content

Updated Multisession3ItMemoryThr flaky test checks - #1758

Merged
matlabbe merged 1 commit into
masterfrom
bayes_memthr_flaky_test
Aug 30, 2026
Merged

Updated Multisession3ItMemoryThr flaky test checks#1758
matlabbe merged 1 commit into
masterfrom
bayes_memthr_flaky_test

Conversation

@matlabbe

Copy link
Copy Markdown
Member

No description provided.

@matlabbe
matlabbe marked this pull request as ready for review August 29, 2026 22:03
@codecov

codecov Bot commented Aug 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.36%. Comparing base (d494063) to head (cf2c3a1).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1758      +/-   ##
==========================================
- Coverage   61.37%   61.36%   -0.01%     
==========================================
  Files         291      291              
  Lines       51231    51231              
==========================================
- Hits        31441    31436       -5     
- Misses      19790    19795       +5     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@fallenmi fallenmi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new adjacent-variant margin addresses the observed cross-run reversal without removing the regression signal. At exact head cf2c3a1, the old assertion rejects the documented local=0.0956 / none=0.0946 pair, while the new strict +0.015 bound accepts that 0.001 reversal and still rejects a reversal at or beyond the margin. The two independent both-retrieval < ... assertions are retained, so a collapse of the configured retrieval benefit remains a failure.

I compiled and ran boundary cases against the exact base/head assertion logic. I also inspected the exact-head macOS Sequoia run: 18/18 CTest targets passed, and Multisession3ItMemoryThr ran (not skipped) and passed in 175.981 seconds, with mean value differences of 0.0461745 (both retrieval), 0.075264 (local only), and 0.0932199 (none).

AI disclosure: I used OpenAI Codex to assist this review; I verified the source and evidence against exact head cf2c3a156659426e1fd713e78c3f7ce9022f82ff.

@matlabbe matlabbe self-assigned this Aug 30, 2026
@matlabbe
matlabbe merged commit 8732a2c into master Aug 30, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants