Skip to content

fix: support MemoryArena allocations up to 4GB #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eeeebbbbrrrr merged 4 commits into from
Sep 19, 2025
+40 −6
Merged

fix: support MemoryArena allocations up to 4GB #62

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a676611
fix: support MemoryArena allocations up to 4GB
Sep 19, 2025
e81c6e7
Update stacker/src/memory_arena.rs
eeeebbbbrrrr Sep 19, 2025
65f3d8d
add a test to make sure we panic if we try to go over 4G
Sep 19, 2025
997a1b8
Merge branch 'eebbrr/memory-arena-limit' of github.com:paradedb/tanti…
Sep 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 40 additions & 6 deletions stacker/src/memory_arena.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@
const NUM_BITS_PAGE_ADDR: usize = 19;
const PAGE_SIZE: usize = 1 << NUM_BITS_PAGE_ADDR; // pages are 512k large

// We use 32-bits addresses.
// - 19 bits for the in-page addressing
// - 13 bits for the page id.
// This limits us to 2^13 - 1=8191 for the page id.
const MAX_PAGES: usize = 1 << (32 - NUM_BITS_PAGE_ADDR);

/// Represents a pointer into the `MemoryArena`
/// .
/// Pointer are 32-bits and are split into
Expand Down Expand Up @@ -102,7 +108,7 @@
capacity: usize,
}

static ARENA_POOL: Lazy<Arc<Mutex<Vec<MemoryArena>>>> = Lazy::new(|| Default::default());

Check warning on line 111 in stacker/src/memory_arena.rs

View workflow job for this annotation

GitHub Actions / clippy

redundant closure 

warning: redundant closure
   --> stacker/src/memory_arena.rs:111:67
    |
111 | static ARENA_POOL: Lazy<Arc<Mutex<Vec<MemoryArena>>>> = Lazy::new(|| Default::default());
    |                                                                   ^^^^^^^^^^^^^^^^^^^^^ help: replace the closure with the function itself: `Default::default`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_closure
    = note: `#[warn(clippy::redundant_closure)]` on by default

impl Default for MemoryArena {
fn default() -> MemoryArena {
Expand All @@ -118,7 +124,7 @@

impl Drop for MemoryArena {
fn drop(&mut self) {
let my_pages = std::mem::replace(&mut self.pages, Vec::new());

Check warning on line 127 in stacker/src/memory_arena.rs

View workflow job for this annotation

GitHub Actions / clippy

replacing a value of type `T` with `T::default()` is better expressed using `std::mem::take` 

warning: replacing a value of type `T` with `T::default()` is better expressed using `std::mem::take`
   --> stacker/src/memory_arena.rs:127:24
    |
127 |         let my_pages = std::mem::replace(&mut self.pages, Vec::new());
    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider using: `std::mem::take(&mut self.pages)`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#mem_replace_with_default
    = note: `#[warn(clippy::mem_replace_with_default)]` on by default
let my_capacity = self.capacity;
let mut recycled = MemoryArena {
pages: my_pages,
Expand Down Expand Up @@ -248,11 +254,7 @@

impl Page {
fn new(page_id: usize) -> Page {
// We use 32-bits addresses.
// - 20 bits for the in-page addressing
// - 12 bits for the page id.
// This limits us to 2^12 - 1=4095 for the page id.
assert!(page_id < 4096);
assert!(page_id < MAX_PAGES);
Page {
page_id,
len: 0,
Expand Down Expand Up @@ -301,7 +303,7 @@
#[cfg(test)]
mod tests {

use super::MemoryArena;
use super::{MemoryArena, MAX_PAGES};
use crate::memory_arena::PAGE_SIZE;

#[test]
Expand Down Expand Up @@ -376,4 +378,36 @@
assert_eq!(arena.read::<MyTest>(addr_a), a);
assert_eq!(arena.read::<MyTest>(addr_b), b);
}

#[test]
fn test_can_allocate_4GB() {
Comment thread
eeeebbbbrrrr marked this conversation as resolved.
use super::NUM_BITS_PAGE_ADDR;

let mut arena = MemoryArena::default();
for i in 0..MAX_PAGES {
let addr = arena.allocate_space(PAGE_SIZE - 1); // -1 to ensure we don't cross page boundary
assert_eq!(addr.page_id(), i);
}
assert_eq!(arena.pages.len(), 1 << (32 - NUM_BITS_PAGE_ADDR));
assert_eq!(
arena.mem_usage(),
PAGE_SIZE * (1 << (32 - NUM_BITS_PAGE_ADDR))
);
}
#[test]
#[should_panic]
fn test_cannot_allocate_more_than_4GB() {
use super::NUM_BITS_PAGE_ADDR;

let mut arena = MemoryArena::default();
for i in 0..MAX_PAGES + 1 {
let addr = arena.allocate_space(PAGE_SIZE - 1); // -1 to ensure we don't cross page boundary
assert_eq!(addr.page_id(), i);
}
assert_eq!(arena.pages.len(), 1 << (32 - NUM_BITS_PAGE_ADDR));
assert_eq!(
arena.mem_usage(),
PAGE_SIZE * (1 << (32 - NUM_BITS_PAGE_ADDR))
);
}
}
Loading