Cannot send 2GiB+ chunks over gRPC

[nathan-b-flya]

Describe the bug

When sending too much data, Rerun crashes with the error:

[... ERROR re_grpc_client::write] Failed to flush gRPC messages during shutdown: gRPC connection is open, but flush still failed. Probably a bug in the Rerun SDK
[... ERROR re_grpc_client::write] Failed to gracefully shut down message proxy client: channel closed
[... ERROR re_grpc_server] Error while receiving messages: gRPC error, message: "h2 protocol error: error reading a body from connection"

To Reproduce
Here is the C++ code to reproduce the issue easily rerun_test.cpp:

#include <cstdlib>
#include <numeric>
#include <random>
#include <string>
#include <vector>

#include <rerun.hpp>
#include <rerun/components/color.hpp>
#include <rerun/components/position3d.hpp>

rerun::components::Color gray(uint8_t value) { return {value, value, value}; }

int main()
{
    const auto rec = rerun::RecordingStream("github_issue");
    rec.spawn().exit_on_failure();

    const std::size_t nb_scans = 2200;  // does not work
    // const std::size_t nb_scans = 2100;  // works
    const std::size_t nb_points_per_scan = 1024 * 64;
    const std::size_t nb_points = nb_scans * nb_points_per_scan;

    std::vector<uint64_t> times(nb_scans);
    std::iota(times.begin(), times.end(), 0);
    const auto timeline = rerun::TimeColumn::from_sequence("times", std::move(times));

    std::vector<uint32_t> lengths(nb_scans, nb_points_per_scan);

    std::mt19937 gen{42};
    std::uniform_real_distribution<float> distrib(0., 1.);
    std::uniform_int_distribution<int> distrib_bis(0, 255);

    std::vector<rerun::components::Position3D> points(nb_points);
    std::generate(points.begin(), points.end(),
                  [&]() {
                      return rerun::components::Position3D{distrib(gen), distrib(gen),
                                                           distrib(gen)};
                  });

    [[maybe_unused]] const rerun::components::Color BLUE{0, 0, 255};
    std::vector<rerun::components::Color> colors(nb_points);
    std::generate(colors.begin(), colors.end(), [&]()
                  { return gray(static_cast<uint8_t>(distrib_bis(gen))); });  // does not work
                  // { return BLUE; });  // works

    rec.send_columns("scans", timeline,
                     rerun::Points3D()
                         .with_positions(points)
                         .with_colors(colors)
                         .columns(lengths));
}

This code sends many scans with random points x,y,z and random gray color.
One scan is 1024*64=65536 points.

If 2100 scans are sent, it works as expected:

But if 2200 scans are sent, the executable fails with gRPC error, message: "h2 protocol error: error reading a body from connection" and Rerun does not show the pointcloud.

Note that if the random gray colors are replaced by one constant color (like {0,0,255}), it does not fail.
Same if the random points are replaced by one dummy point (like {1,2,3}).

Desktop

• OS: Ubuntu 24
• RAM: 32GB, ~25GB is free

Rerun version
0.27.2

[emilk]

I see two things here:

A) A bad error message
B) An avoidable error

The error likely comes from us trying to send 2GiB+ chunks over gRPC, which is afaict unsupported by gRPC/protobuf. We should split those chunks up into smaller pieces.

The workaround for you (as you might have guessed) is to manually split your data into smaller chunks (keep nb_scans small).

Btw: we run LZ4 on our data, which is why homogenous data will fit the 2GiB+ limit, but heterogeneous data (which compresses worse) will not.

• Related, but distinct: videos larger than 2.1GB cause rerun to crash #10929
• Code for splitting chunks added in JIT compaction in ChunkStore can now split chunks in addition to merging them #11921
  • I'm moving the code to Chunk::split_chunk_if_needed for more easy re-use

[nathan-b-flya]

The workaround for you (as you might have guessed) is to manually split your data into smaller chunks (keep nb_scans small).

Yep, that's what I'm doing. Instead of processing a full dataset of like 5min, I tweak the start and end of the dataset to make it smaller. Feasible, but a bit annoying.

Btw: we run LZ4 on our data, which is why homogenous data will fit the 2GiB+ limit, but heterogeneous data (which compresses worse) will not.

Ok, that's what I was suspecting when trying to cook you a minimal reproducible example!

Thanks for the answer!
I will keep an eye on this issue and the one you mentioned.