Skip to content

feat(allocator): add allocator_api and allocator_api2 support and add more test cases - #30

Open
al8n wants to merge 28 commits into
mainfrom
perf/optimize-atomic-orderings-and-hot-paths
Open

al8n wants to merge 28 commits into
mainfrom
perf/optimize-atomic-orderings-and-hot-paths

Conversation

@al8n

@al8n al8n commented Mar 13, 2026

Copy link
Copy Markdown
Owner

No description provided.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the rarena / rarena-allocator workspace with additional allocator tests (including concurrency stress tests), benchmark coverage, and a set of documentation/version/CI workflow updates.

Changes:

  • Add new sync/unsync allocator tests to exercise slow-path allocation and freelist edge cases under contention.
  • Update the sync freelist implementation and atomic/CAS behavior (plus minor perf-oriented inlines/cleanups).
  • Add a Criterion benchmark suite and refresh READMEs, versions, and CI/coverage configuration.

Reviewed changes

Copilot reviewed 25 out of 28 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
rarena/README.md Updates umbrella-crate docs, quick-start, features, and license links.
rarena/Cargo.toml Bumps crate version to 0.7.0.
rarena-allocator/src/unsync/tests.rs Adds new slow-path allocation and debug/printing tests.
rarena-allocator/src/unsync.rs Adds #[inline] annotations and adjusts comments/flow.
rarena-allocator/src/sync/tests.rs Adds slow-path, segment list, and high-contention concurrency tests.
rarena-allocator/src/sync.rs Modifies freelist traversal/unlinking logic and atomic/CAS usage/orderings.
rarena-allocator/src/object.rs Simplifies as_mut_ptr handling for MaybeUninit storage.
rarena-allocator/src/memory.rs Removes redundant mmap initialization logic during create-new mapping.
rarena-allocator/src/lib.rs Minor internal write/put helpers adjustments.
rarena-allocator/src/allocator.rs Simplifies get_u8/get_i8 implementations to direct pointer reads.
rarena-allocator/benches/alloc.rs Adds Criterion benchmarks for sync/unsync allocation scenarios.
rarena-allocator/README.md Expands documentation (features, examples, safety notes) and bumps version strings.
rarena-allocator/Cargo.toml Bumps version, updates deps, adds Criterion + bench target.
cobertura.xml Adds a generated coverage XML report.
ci/sanitizer_generic.sh Removes generic sanitizer script.
ci/sanitizer.sh Updates sanitizer commands/flags and test selection.
ci/miri_tb_generic.sh Removes generic miri TB script.
ci/miri_tb.sh Switches to target-parameterized miri execution and adds cross-toolchain install logic.
ci/miri_sb_generic.sh Removes generic miri SB script.
ci/miri_sb.sh Switches to target-parameterized miri execution and adds cross-toolchain install logic.
README.md Expands top-level project docs, features, and quick start; bumps version strings.
Cargo.toml Bumps workspace dependency version for rarena-allocator.
.github/workflows/loc.yml Updates action versions and gist upload steps.
.github/workflows/ci.yml Updates schedules, action versions, adds matrix targets for miri, and revises coverage/sanitizer jobs.
.codecov.yml Adjusts Codecov ignore patterns.
.clippy.toml Updates MSRV for clippy configuration.
Comments suppressed due to low confidence (3)

.github/workflows/ci.yml:47

  • Under rustfmt, the step list items are indented at the same level as steps: (e.g. steps: then - uses ...). YAML requires the list items to be nested under steps:; as written, this job will fail to parse. Indent the - uses / - name entries two more spaces under steps:.
    steps:
    - uses: actions/checkout@v6
    - name: Install Rust
      # --no-self-update is necessary because the windows environment cannot self-update rustup.exe.
      run: rustup update stable --no-self-update && rustup default stable
    - name: Check formatting
      run: cargo fmt --all -- --check

.github/workflows/ci.yml:68

  • Under clippy, the step entries (- uses, - name, etc.) are not indented beneath steps:. This makes the workflow YAML invalid for this job. Indent the list items under steps: consistently (like the cross job does).
    steps:
    - uses: actions/checkout@v6
    - name: Install Rust
      # --no-self-update is necessary because the windows environment cannot self-update rustup.exe.
      run: rustup update stable --no-self-update && rustup default stable
    - name: Install cargo-hack
      run: cargo install cargo-hack
    - name: Apply clippy lints
      run: cargo hack clippy --each-feature --exclude-no-default-features

rarena-allocator/src/sync.rs:406

  • dealloc uses compare_exchange_weak(...) only once. Because compare_exchange_weak may fail spuriously even when the expected value matches, this can incorrectly fall through into the freelist path and leave allocated unchanged (leaking reclaimable space / breaking stack-like deallocation). Use compare_exchange here, or loop/retry on spurious failure.
    // if the offset + size is the current allocated size, then we can deallocate the memory back to the main memory.
    if header
      .allocated
      .compare_exchange_weak(offset + size, offset, Ordering::Release, Ordering::Relaxed)
      .is_ok()
    {
      return true;
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread cobertura.xml Outdated
Comment thread Cargo.toml
Comment thread rarena-allocator/src/object.rs Outdated
Comment thread rarena-allocator/src/object.rs Outdated
Comment thread ci/miri_sb.sh
Comment thread .codecov.yml Outdated
Comment thread .github/workflows/ci.yml
Comment thread .github/workflows/ci.yml
@al8n
al8n requested a review from Copilot March 13, 2026 20:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands the rarena / rarena-allocator workspace with additional allocator API support, varint/error handling refactors, more concurrency/slow-path tests, and corresponding CI/doc/version updates.

Changes:

  • Add extensive new test coverage for slow-path allocation, freelist traversal/printing, allocator API integrations, and high-contention concurrency scenarios.
  • Introduce optional allocator_api / allocator_api2 support (nightly-gated) and refactor varint encoding/decoding + error types to use varing.
  • Update CI scripts/workflows, docs/READMEs, and bump crate/workspace versions to 0.7.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
rarena/src/lib.rs Changes feature gating for the umbrella re-export of rarena_allocator.
rarena/README.md Updates crate overview/quickstart, feature flags, and version strings.
rarena/Cargo.toml Bumps version and adds feature passthroughs for allocator API support.
rarena-allocator/src/unsync/tests.rs Adds additional unsync tests for slow paths, freelist printing, debug impls, and allocator APIs.
rarena-allocator/src/unsync.rs Adds inlining annotations and implements allocator API traits for unsync::Arena.
rarena-allocator/src/sync/tests.rs Adds additional sync tests for slow paths, freelist printing, contention, and allocator APIs.
rarena-allocator/src/sync.rs Adjusts freelist traversal/unlinking logic, some atomic orderings, and implements allocator API traits for sync::Arena.
rarena-allocator/src/object.rs Simplifies as_mut_ptr handling for MaybeUninit-backed storage.
rarena-allocator/src/memory.rs Removes eager zero-init on new mmap creation; adjusts safety annotation/comments in unmount.
rarena-allocator/src/lib.rs Adds nightly feature gate for allocator_api, refactors buffer utilities to use varing, and adds impl_core_allocator! macro.
rarena-allocator/src/error.rs Switches exported error types and varint decode error plumbing to varing.
rarena-allocator/src/bytes.rs Removes re-exports tied to dbutils error/leb128 types.
rarena-allocator/src/allocator.rs Switches varint decoding to varing, optimizes byte getters, and adds non_zero_offset.
rarena-allocator/benches/alloc.rs Adds criterion benchmarks for sync/unsync allocation patterns under varying contention/freelist modes.
rarena-allocator/README.md Expands documentation (overview, quickstart, memory layouts, feature flags, safety notes) and updates versions.
rarena-allocator/Cargo.toml Bumps version, adds varing, adds allocator API feature flags and bench config, updates deps.
ci/sanitizer_generic.sh Removes the generic sanitizer script.
ci/sanitizer.sh Updates sanitizer runs/flags and adjusts which targets/features are exercised.
ci/miri_tb_generic.sh Removes the generic miri tree-borrows script.
ci/miri_tb.sh Parameterizes miri tree-borrows script by target and adds allocator API feature coverage.
ci/miri_sb_generic.sh Removes the generic miri stacked-borrows script.
ci/miri_sb.sh Parameterizes miri stacked-borrows script by target and adds allocator API feature coverage.
README.md Updates workspace-level overview/quickstart, versions, and feature documentation.
Cargo.toml Bumps workspace dependency version for rarena-allocator to 0.7.
.github/workflows/loc.yml Updates workflow action versions and simplifies Rust install step.
.github/workflows/ci.yml Updates workflow action versions, expands miri matrix/targets, adds allocator API job, refactors caches and coverage steps.
.codecov.yml Updates ignore patterns for coverage collection.
.clippy.toml Bumps MSRV used by clippy configuration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread rarena-allocator/src/object.rs Outdated
Comment thread rarena-allocator/src/lib.rs
Comment thread .codecov.yml Outdated
Comment thread rarena/src/lib.rs
Comment thread rarena-allocator/src/lib.rs Outdated
Comment thread rarena-allocator/src/lib.rs Outdated
Comment thread rarena-allocator/src/object.rs Outdated
@codecov

codecov Bot commented Mar 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.36530% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.76%. Comparing base (52ded10) to head (d536924).

Files with missing lines Patch % Lines
rarena-allocator/src/sync.rs 67.03% 30 Missing ⚠️
rarena-allocator/src/unsync.rs 63.15% 7 Missing ⚠️
rarena-allocator/src/lib.rs 96.29% 3 Missing ⚠️
rarena-allocator/src/object.rs 0.00% 2 Missing ⚠️
rarena-allocator/src/error.rs 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main      #30       +/-   ##
===========================================
+ Coverage   67.90%   83.76%   +15.86%     
===========================================
  Files          15       12        -3     
  Lines        3159     2624      -535     
===========================================
+ Hits         2145     2198       +53     
+ Misses       1014      426      -588     
Files with missing lines Coverage Δ
rarena-allocator/src/allocator.rs 96.07% <100.00%> (+31.11%) ⬆️
rarena-allocator/src/bytes.rs 93.28% <100.00%> (+73.88%) ⬆️
rarena-allocator/src/memory.rs 77.35% <100.00%> (+13.12%) ⬆️
rarena-allocator/src/options.rs 96.20% <100.00%> (+11.27%) ⬆️
rarena-allocator/src/error.rs 87.50% <66.66%> (+87.50%) ⬆️
rarena-allocator/src/object.rs 90.62% <0.00%> (+45.84%) ⬆️
rarena-allocator/src/lib.rs 93.10% <96.29%> (+22.97%) ⬆️
rarena-allocator/src/unsync.rs 86.13% <63.15%> (+38.00%) ⬆️
rarena-allocator/src/sync.rs 77.20% <67.03%> (+13.51%) ⬆️

... and 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52ded10...d536924. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

al8n and others added 3 commits March 14, 2026 11:20
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@al8n al8n changed the title test: add more test cases feat(allocator): add allocator_api and allocator_api2 support and add more test cases Mar 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands the rarena workspace with broader allocator API support, updated CI tooling, and significantly more test/benchmark coverage around freelist slow paths and concurrency behavior.

Changes:

  • Add allocator_api / allocator_api2 feature support (including core::alloc::Allocator impls) and refactor related encoding/decoding utilities.
  • Add many new sync/unsync tests (slow-path, debug/printing, contention) and introduce Criterion benchmarks.
  • Update READMEs, crate versions, and CI scripts/workflows (miri/sanitizers/coverage).

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
rarena/src/lib.rs Changes feature gating for the umbrella re-export.
rarena/README.md Adds overview/quick start and feature table updates.
rarena/Cargo.toml Version bump + new feature flags passthrough.
rarena-allocator/src/unsync/tests.rs Adds new unsync tests (slow path, debug, allocator APIs).
rarena-allocator/src/unsync.rs Adds inlines + Allocator-trait impls behind feature flags.
rarena-allocator/src/sync/tests.rs Adds new sync tests (slow path, contention, allocator APIs).
rarena-allocator/src/sync.rs Adjusts freelist traversal/unlinking + memory ordering tweaks + allocator API impls.
rarena-allocator/src/object.rs Simplifies pointer access for MaybeUninit storage.
rarena-allocator/src/memory.rs Minor mmap init/unmount annotation/comment adjustments.
rarena-allocator/src/lib.rs Adds allocator_api gating, varint/error refactors, and core allocator helper macro.
rarena-allocator/src/error.rs Switches exported error types to varing-based ones.
rarena-allocator/src/bytes.rs Removes legacy re-exports and relies on updated crate errors/utilities.
rarena-allocator/src/allocator.rs Switches varint decode helpers + adds non_zero_offset.
rarena-allocator/benches/alloc.rs New Criterion benchmarks for sync/unsync allocation patterns.
rarena-allocator/README.md Expanded docs, quick start, memory layout, and feature flag table.
rarena-allocator/Cargo.toml Version bump + new dependencies/features (varing, allocator-api2, benches).
ci/sanitizer_generic.sh Removed.
ci/sanitizer.sh Reworked sanitizer execution commands.
ci/miri_tb_generic.sh Removed.
ci/miri_tb.sh Reworked miri script to accept target + install cross toolchains.
ci/miri_sb_generic.sh Removed.
ci/miri_sb.sh Reworked miri script to accept target + install cross toolchains.
README.md Expanded workspace-level overview, quick start, and install guidance.
Cargo.toml Bumps workspace dependency version for rarena-allocator.
.github/workflows/loc.yml Workflow updates for LoC publishing.
.github/workflows/ci.yml CI matrix, caching, sanitizer/miri scripts, and coverage updates.
.codecov.yml Updates ignore patterns.
.clippy.toml MSRV bump.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread Cargo.toml
Comment thread rarena-allocator/src/lib.rs Outdated
Comment thread rarena/src/lib.rs
Comment thread rarena-allocator/README.md Outdated
Comment thread rarena/README.md Outdated
Comment on lines +125 to +134
let mut offsets = Vec::new();
for _ in 0..500 {
let bytes = arena.alloc_bytes(size).unwrap();
offsets.push((bytes.offset() as u32, size));
}
for (offset, sz) in offsets {
unsafe {
arena.dealloc(offset, sz);
}
}
Comment thread rarena-allocator/benches/alloc.rs Outdated
let mut offsets = Vec::new();
for _ in 0..500 {
let bytes = arena.alloc_bytes(size).unwrap();
offsets.push((bytes.offset() as u32, size));
Comment on lines +158 to +161
for _ in 0..500 {
let bytes = arena.alloc_bytes(size).unwrap();
offsets.push((bytes.offset() as u32, size));
}
Comment thread rarena-allocator/benches/alloc.rs Outdated
Comment on lines +196 to +206
// Pre-populate freelist
let mut offsets = Vec::new();
for _ in 0..2000 {
let bytes = arena.alloc_bytes(size).unwrap();
offsets.push((bytes.offset() as u32, size));
}
for (offset, sz) in offsets {
unsafe {
arena.dealloc(offset, sz);
}
}
al8n and others added 3 commits March 14, 2026 11:26
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the rarena workspace to add allocator_api / allocator_api2 integration for sync::Arena and unsync::Arena, modernizes varint/error handling, expands concurrency/slow-path test coverage, and refreshes CI + documentation for the 0.7 release.

Changes:

  • Add core::alloc::Allocator (nightly) and allocator_api2::alloc::Allocator (stable) implementations for arenas, plus feature plumbing.
  • Replace LEB128/insufficient-buffer error plumbing with varing + NonZeroUsize-based APIs and update related utilities.
  • Add extensive slow-path and concurrency regression tests; update CI scripts/workflows and docs for the new release.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 15 comments.

Show a summary per file
File Description
rarena/src/lib.rs Gates the umbrella re-export of rarena_allocator behind std/alloc features.
rarena/README.md Adds overview/quick-start, feature table, version bump.
rarena/Cargo.toml Bumps version; adds allocator_api / allocator_api2 feature forwarding.
rarena-allocator/src/unsync/tests.rs Adds slow-path, segment list, Debug, and allocator API tests.
rarena-allocator/src/unsync.rs Adds Allocator trait impls via impl_core_allocator!; adds some #[inline].
rarena-allocator/src/sync/tests.rs Adds slow-path, contention, segment list, Debug, and allocator API tests.
rarena-allocator/src/sync.rs Refactors freelist traversal/unlinking and updates atomic orderings; adds Allocator trait impls.
rarena-allocator/src/object.rs Clarifies pointer-return docs and simplifies MaybeUninit pointer handling.
rarena-allocator/src/memory.rs Tweaks mmap init behavior and documents unmount safety block.
rarena-allocator/src/lib.rs Enables nightly allocator_api, switches to varing APIs, adds impl_core_allocator! macro.
rarena-allocator/src/error.rs Switches error re-exports and varint decode error mapping to varing.
rarena-allocator/src/bytes.rs Removes old dbutils error re-exports; aligns with new error types.
rarena-allocator/src/allocator.rs Switches varint decoding to varing; adds non_zero_offset.
rarena-allocator/benches/alloc.rs Adds Criterion benchmarks for alloc/read/write and freelist scenarios.
rarena-allocator/README.md Major documentation refresh (features, quick start, safety, memory layout).
rarena-allocator/Cargo.toml Bumps version; adds allocator-api features, varing, Criterion bench config, dependency updates.
ci/sanitizer_generic.sh Removed legacy generic sanitizer script.
ci/sanitizer.sh Updates sanitizer runs/flags and which test suites are executed.
ci/miri_tb_generic.sh Removed legacy generic miri TB script.
ci/miri_tb.sh Adds target parameterization and cross toolchain install; runs allocator-api feature tests.
ci/miri_sb_generic.sh Removed legacy generic miri SB script.
ci/miri_sb.sh Adds target parameterization and cross toolchain install; runs allocator-api feature tests.
README.md Workspace-level docs refresh (overview, crates table, quick start, install).
Cargo.toml Bumps workspace rarena-allocator version reference.
.github/workflows/loc.yml Updates GitHub Actions versions and simplifies Rust install step.
.github/workflows/ci.yml Updates workflow schedule, actions versions, adds allocator_api job, refactors miri/sanitizer/coverage steps.
.codecov.yml Updates ignore patterns and formatting.
.clippy.toml Updates MSRV setting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread rarena/src/lib.rs
Comment thread rarena/Cargo.toml Outdated
Comment on lines +84 to +86
for _ in 0..3 {
let _ = arena.alloc_aligned_bytes::<u64>(8);
}
Comment on lines +143 to +145
for _ in 0..3 {
let _ = unsafe { arena.alloc::<u32>() };
}
Comment thread rarena-allocator/benches/alloc.rs Outdated
Comment on lines +124 to +133
// Pre-allocate and deallocate to populate freelist
let mut offsets = Vec::new();
for _ in 0..500 {
let mut bytes = arena.alloc_bytes(size).unwrap();
offsets.push((bytes.buffer_offset() as u32, bytes.buffer_capacity() as u32));
unsafe {
bytes.detach();
}
}
for (offset, sz) in offsets {
Comment on lines +165 to +167
for _ in 0..3 {
let _ = arena.alloc_aligned_bytes::<u64>(8);
}
Comment on lines +195 to +205
_ => unreachable!(),
};
group.bench_with_input(BenchmarkId::from_parameter(name), &freelist, |b, &fl| {
let size = 128u32;
b.iter_batched(
|| {
let arena = make_sync_arena(256 << 20, fl);
// Pre-populate freelist
let mut offsets = Vec::new();
for _ in 0..2000 {
let mut bytes = arena.alloc_bytes(size).unwrap();
Comment on lines +39 to +42
// Now allocate typed objects (goes through alloc_in slow path)
for _ in 0..3 {
let _ = unsafe { arena.alloc::<u32>() };
}
Comment on lines +62 to +64
for _ in 0..3 {
let _ = unsafe { arena.alloc::<u32>() };
}
Comment thread rarena-allocator/src/sync/tests.rs
al8n and others added 3 commits March 14, 2026 12:11
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants