Enable cloud MCAP loading

[oceanusxiv]

Related

• Part of MCAP MessageLayers should only look at relevant chunks #10862

What

This is a first stab at enabling more sophisticated ingestion of MCAP files by rerun. It provides the following features

1. Allowing mcap file paths to be from AWS and GCP as well as local ones.
2. Allowing special MCAP URL syntax to specify start and end timestamps for loading partial mcap files.

In many ways this is kind of a hack, considering it hijacks the file loading pathway for remote loading. However, I considered this to be less intrusive compared to writing a bespoke cloud mcap data source pathway.

[github-actions]

Hi! Thanks for opening this pull request.

Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.

[grtlr]

I'm sorry that it took me a while to respond, where currently doing our Rerun offsite meetup.

First of all thank you very much for your contribution! We know that this is a very important feature, which we eventually will need in Rerun!

But, I took a look at your PR, and at the current stage I'm a bit concerned with adding these dependencies, since they can significantly increase the size of our binary and complicate our web viewer builds.

I tried to gauge the effect on our WASM blob, unfortunately, I could not get your PR to compile our web viewer using:

pixi run rerun-build-web

As an aside, you could also explore writing an external data loader: https://github.com/rerun-io/rerun/tree/main/examples/rust/custom_data_loader

[oceanusxiv]

@grtlr Yeah I can get the wasm build working sorry about that.

I did also consider doing this as an external data loader, however, there are two options

1. Doing it in rust, in order to take advantage of re_mcap which your team has done a lot of work on for bridging mcap and rerun.
2. Doing it in python, which would entail basically rewriting re_mcap in python since there is no re_mcap python bindings.

If doing option 1, unfortunately when using rerun I'm mostly using it in a python based environment, without a rust setup, and it's very complicated to try to spin up a rust setup just for the purposes of producing a rust based external data loader.

If doing option 2, well now I'd have to re-implement re_mcap which does not seem super feasible nor advisable since re_mcap is undergoing a lot of development at this moment.

This was the reason why I thought trying to upstream this change into rerun itself was best, it makes downstream user experience a lot less complicated.