High memory allocation in pendulum 3.0

[cody-scott]

• [x ] I am on the latest Pendulum version.

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: Debian GNU/Linux 12 (bookworm)

• Pendulum version: 3.0.0

Issue

In pendulum 3.0.0 it is allocating a large amount of memory for the library (up to 1gb). In prior versions, memory allocation was usually less than 2 mb for the same import. Possibly due to rust re-write?

tested/validated using memray with both pendulum==3.0.0 and pendulum==2.1.2

flame_graphs.zip

[Secrus]

That is troubling. I don't know too much about Rust, but I am open to accepting PRs that implement improvements on that front.

[cody-scott]

I think one first step would be to see if anyone else sees the same results I get when profiling. I tested in windows/linux and saw similar results, but would be great to confirm its not just a "me" issue.

[protoss-player]

I think one first step would be to see if anyone else sees the same results I get when profiling

Actually, I got the exactly same memory consumption issue with memray and ended up here.

[s-rigaud]

I'm getting segfault inside a docker container as fastapi_cache is based over pendulum and memory consumption is way higher than before.

[Secrus]

@cody-scott any chance you could test the latest master branch? I have merged some changes to the Rust-Python code and I suspect it might be the answer to this issue (at least partially)

[rudolfix]

this is because mimalloc is used instead of of default memory allocator. but I bet memray is giving you wrong results.
bloomberg/memray#577

I can't see increased memory usage otherwise, not in the process stat, not when using valgrind.

[GriffinBoris]

I am seeing the same issue in version 3.0.0

I am running macOS Sequoia 15.3.1.
I am using python 3.12

I have tested with versions 3.0.0a1, 3.0.0b1, and 3.0.0. The high memory usage appears to correlate with the new Rust extension introduced in 3.0.0b1.

I've attached an archive file containing flamegraphs for each version, with comparisons across different malloc implementations, along with a test Python script used for profiling.

Archive.zip

For a quick reference here is 3.0.0a1 with malloc:

Here is 3.0.0b1 with malloc:

Here is the reference of the script:

def regular_pendulum_import():
    import pendulum
    pass


if __name__ == '__main__':
    regular_pendulum_import()

This continues into 3.0.0 regardless of allocator.