perf: make the small-object allocation fast path a leaf - #15011
Draft
Kha wants to merge 1 commit into
Draft
Conversation
This PR shortens `lean_alloc_small_object_core` from 18 to 14 instructions by keeping mimalloc's generic allocation path out of line, so that the fast path needs no stack frame. The entry point now tests `page->free` itself and tail-jumps to a cold helper on a miss, instead of calling `mi_theap_malloc_small` and checking its result. Only the generic routine can fail, so both the OOM check and the `m_cs_sz` store move to that helper; leaving either at the call site would make the cold call one that has to return, and the fast path would pay a prologue and epilogue on every allocation to serve a path it almost never takes. Restating the callers' precondition on `sz` (word-aligned, non-zero, at most `MI_SMALL_SIZE_MAX`) additionally folds mimalloc's bin-index rounding into a plain byte offset and lets the header store drop its mask. That bound is a safety condition in its own right, as a larger `sz` would index `pages_free_direct` out of bounds, so it is also asserted. ```diff -push rbx ; frame, only for the generic call below mov rax, [rip+...] ; &lean_g_tls inc QWORD PTR fs:[rax] ; heartbeat++ mov rcx, QWORD PTR fs:[rax+0x8] ; theap mov esi, edi -lea rax, [rsi+0x7] ; round sz up to a bin index -and rax, 0xfffffffffffffff8 -mov rdx, [rcx+rax*1+0x120] ; theap->pages_free_direct[..] +mov rdx, [rcx+rsi*1+0x120] ; sz is the byte offset already mov rax, [rdx+0x8] ; page->free test rax, rax -je .Lcold ; inline generic path, ending in pop rbx; ret +je lean_alloc_small_generic_core ; tail jump, nothing to tear down mov rcx, [rax] ; pop the block mov [rdx+0x8], rcx inc WORD PTR [rdx+0x10] ; page->used++ mov QWORD PTR [rax], 0x0 ; mimalloc's free-list scrub -mov WORD PTR [rax+0x4], di ; m_cs_sz, masked to 16 bits -pop rbx +mov DWORD PTR [rax+0x4], edi ; m_cs_sz, no mask needed ret ``` Measured by swapping `libleanshared.so` between two full builds, single-threaded with `-DElab.async=false` on pinned cores: `tests/elab_bench/big_omega.lean` goes from 18.781G to 18.650G user instructions (-0.694%) and `import Lean` from 1.9356G to 1.9215G (-0.729%), each reproducible to within 0.02%. LLVM's hot-cold splitting does not do this: with `-mllvm -hot-cold-split=true` the pass fires 27 times elsewhere in the translation unit and leaves this function byte-identical, because it outlines a cold region into a function the original still calls and returns from, which does not remove the frame. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Member
Author
|
!bench |
|
Benchmark results for cc86abf against 137a88d are in. There are significant results. @Kha
Large changes (15✅)
Medium changes (7✅)
Small changes (1020✅)
|
github-actions
Bot
added
the
toolchain-available
|
Mathlib CI status (docs):
|
Collaborator
|
Reference manual CI status:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR shortens
lean_alloc_small_object_corefrom 18 to 14 instructions by keeping mimalloc's generic allocation path out of line, so that the fast path needs no stack frame.The entry point now tests
page->freeitself and tail-jumps to a cold helper on a miss, instead of callingmi_theap_malloc_smalland checking its result. Only the generic routine can fail, so both the OOM check and them_cs_szstore move to that helper; leaving either at the call site would make the cold call one that has to return, and the fast path would pay a prologue and epilogue on every allocation to serve a path it almost never takes. Restating the callers' precondition onsz(word-aligned, non-zero, at mostMI_SMALL_SIZE_MAX) additionally folds mimalloc's bin-index rounding into a plain byte offset and lets the header store drop its mask. That bound is a safety condition in its own right, as a largerszwould indexpages_free_directout of bounds, so it is also asserted.