Potential Memory Leak after 0.27

[ARizzo35]

Describe the bug

Some time after version 0.27, my applications that use Rerun grow in memory usage indefinitely. I've observed this on 0.28 and now on 0.29 as well.

To Reproduce
Steps to reproduce the behavior:
In one terminal, run the following snippet:

use anyhow::Result;
use rerun::{RecordingStreamBuilder, Scalars};

fn main() -> Result<()> {
    let rec = RecordingStreamBuilder::new("test")
        .batcher_config(rerun::log::ChunkBatcherConfig::ALWAYS)
        .set_sinks((
            rerun::sink::GrpcSink::default(),
            rerun::sink::FileSink::new("test.rrd")?,
        ))?;

    loop {
        for i in 0..10 {
            rec.log("signal", &Scalars::single(i as f32 * 0.1))?;
        }
    }
}

In another terminal, run rerun to run the server and open up the viewer.

Expected behavior

The app using the Rerun recorder shouldn't be continuously growing in memory and this wasn't the case in 0.27 versions. Also, the memory usage does not go up as fast if I don't set the batcher config to ALWAYS, but it still seems to go up indefinitely albeit much slower. There could be something related here but I'm not sure.

Screenshots

Backtrace

Desktop (please complete the following information):

• OS: I've observed this on both MacOS and Linux.

Rerun version

rerun-cli 0.29.1 (nasm native_viewer web_viewer) [rustc 1.92.0 (ded5c06cf 2025-12-08) (built from a source tarball), LLVM 21.1.8] aarch64-apple-darwin
Video features: av1 ffmpeg nasm

Additional context

[emilk]

TL;DR: please try Rerun 0.29.1 and report back :)

What I observe on my computer, running on main, is that the logging application (your code) uses around 250MB of memory. The viewer (the rerun binary) on the other hand, uses more and more memory (unless you set a memory limit). The latency panel tells the story well:

We can see that the ingestion is the bottleneck here. And the reason is ChunkBatcherConfig::ALWAYS creates a lot of tiny, tiny chunks, with enormous overhead.

So far, this is all expected behavior. If you're creating many tiny tiny chunks, then you are creating a lot of extra work for the viewer.

But you seem to say that is the other way around for you?

BTW, we did do some optimizations in 0.29.1 which might help things a bit.

[ARizzo35]

TL;DR: please try Rerun 0.29.1 and report back :)

What I observe on my computer, running on main, is that the logging application (your code) uses around 250MB of memory. The viewer (the rerun binary) on the other hand, uses more and more memory (unless you set a memory limit). The latency panel tells the story well:

We can see that the ingestion is the bottleneck here. And the reason is ChunkBatcherConfig::ALWAYS creates a lot of tiny, tiny chunks, with enormous overhead.

So far, this is all expected behavior. If you're creating many tiny tiny chunks, then you are creating a lot of extra work for the viewer.

But you seem to say that is the other way around for you?

BTW, we did do some optimizations in 0.29.1 which might help things a bit.

I was using 0.29.1 as shown in the rerun version output I added to my original post.

I can take a video of htop showing the application memory usage growing very fast if that helps.

[emilk]

Thanks for the response!

I'm starting to suspect that maybe you are using an old logging SDK, even though your viewer is new.

What would be very helpful for me would be:

• confirmation that the logging SDK you are using is ALSO 0.29.1 (check your Cargo.toml!)
• a screenshot of the latency panel, like the one I showed
• confirmation if "the application" with the rapid memory use is the logger (caller of rec.log) or the receiver (viewer / rerun)
• explaining exactly how much memory the logging application uses (many gigabytes?)

[ARizzo35]

Here is exactly what I'm seeing, using the below code snippet:

use anyhow::Result;
use rerun::{RecordingStreamBuilder, Scalars};

fn main() -> Result<()> {
    let rec = RecordingStreamBuilder::new("test")
        .batcher_config(rerun::log::ChunkBatcherConfig::ALWAYS)
        .set_sinks((
            rerun::sink::GrpcSink::default(),
            rerun::sink::FileSink::new("test.rrd")?,
        ))?;

    loop {
        for i in 0..10 {
            rec.log("signal", &Scalars::single(i as f32 * 0.1))?;
            std::thread::sleep(std::time::Duration::from_millis(10));
        }
    }
}

In the video below, running 0.27.3 in the logging SDK and 0.27.2 server/viewer, the memory usage of the logging app stabilizes at around 23MiB:

https://streamable.com/4b4vbc

However, in the following video below, running 0.29.1 in the logging SDK and 0.29.1 server/viewer, the memory usage continues to grow up to 28MiB before I stop it, and would have continued to grow if I left it running:

https://streamable.com/tu1c7d

This is only logging a single Scalars value at 100Hz in this example, and the memory usage gets much worse with more data being logged.

[ARizzo35]

cc @Wumpf

[emilk]

Oh, so we are talking about kilobytes here? That would be have been good to know from the start.

The logging SDK has several channels where it puts data before it reached the gRPC client endpoint. These can grow in case the server cannot keep up with the logger. These channels all have hard-coded max sizes, so they can grow up to be a few hundred MiB in size, but this is not a leak. Again, the latency panel tells the story:

Here you can see there is latency induced on the SDK side of things (because of back-pressure, which was introduced in 0.29). That latency is messages in a queue, taking up memory. It's not a leak. It's by design.

[ARizzo35]

Oh, so we are talking about kilobytes here? That would be have been good to know from the start.

The logging SDK has several channels where it puts data before it reached the gRPC client endpoint. These can grow in case the server cannot keep up with the logger. These channels all have hard-coded max sizes, so they can grow up to be a few hundred MiB in size, but this is not a leak. Again, the latency panel tells the story:

Here you can see there is latency induced on the SDK side of things (because of back-pressure, which was introduced in 0.29). That latency is messages in a queue, taking up memory. It's not a leak. It's by design.

NO! This isn't just kilobytes! I mentioned at the end of my last comment that if I left this running it would continue to grow indefinitely. There is also no latency build up in the example I showed.

This simple example was for illustration purposes. The real issue I'm seeing is that I am using rerun on a cloud server which logs data to multiple rerun server instances via rerun loggers in multiple applications within containers. If I use rerun version 0.27, this cloud server can run for weeks with no issues. If I use rerun version 0.29, the cloud server runs out of memory and completely halts in less than a day.

Even if this was latency related, the memory usage should not be unbounded like this.

[ARizzo35]

@emilk I let my simple test app run for about 1.5 hours. The memory usage continuously increased from 19MiB to 161MiB before I stopped it. See the screenshots below.

Also the rerun viewer grew in memory usage up to 1.0GiB. I ran it with rerun --server-memory-limit 64MiB --memory-limit 64MiB.