Ways to organize Rust data model

[nyurik]

I keep thinking how to best organize Rust data model. Data can be in multiple states:

• Encoded (raw bytes as received), with minimal processing, with lifetime tied to the input: struct RawFoo<'a>
• Encoded but owned, as the result of encoding, no lifetimes: struct OwnedFoo
• Fully decoded, i.e. the user is constructing a new value, with the expectation to later encode it and send: struct DecodedFoo
• Partially decoded, with some fields decoded and some still raw. This is the tricky one, as it can be any combination of decoded and raw fields - so at each level the value type must be an enum of raw/decoded.

The expected usage patterns are:

• parse input and later partially decode only what's needed. This is what MapLibre Native could use as a data model to quickly access what's needed.
• create new owned instance, encode it, and write out
• parse input, partially decode, modify some fields, re-encode, write out

Current Code Structure

Given the raw input bytes, the parser quickly runs over the input slice and only stores references to the streams and their metadata. Later, decoding can be done on-demand, either for all columns, or just for the specific ones. This example is for Id, but the same idea applies to Geometry, Property, v01::Layer, and root Layer entities.

• RawId struct contains references into the original input data. The values are not decoded, just some metadata is parsed. Most data is stored as Stream<'a> instances, which hold references to parts of the original input and tie to input lifetime.
• OwnedRawId struct is auto-generated with the borrowme crate - it has the same fields as the RawId struct, but owns its data. This is useful when you want to store a RawId struct beyond the lifetime of the original input slice, or when you want to modify it or store the result of the encoding before storing it into a file.
• DecodedId struct is used to store the decoded value. At the moment, only DecodedGeometry is implemented, but the same idea applies to other entities. The decoded values are stored in standard Rust types, e.g. Vec<u64> for IDs.
• Id enum contains Raw(RawId) and Decoded(DecodedId) variants, with values described above. This allows in-place decoding, e.g. it is possible to decode just one property column / ID / Geometry, while keeping the rest in their raw form. The enum also has a corresponding borrowme-generated OwnedId.

[nyurik]

It seems the simpler is better path solves this -- if I don't try to do this whole thing recursively, but instead focus on just one layer - i.e. columns, it makes the whole thing much easier. In other words, there will be only one type for a layer, but each column could be in either raw or decoded state. I just hope the extra cost of checking each column on each access if it has already been decoded is negligible.

[ianthetechie]

I just hope the extra cost of checking each column on each access if it has already been decoded is negligible.

What does the "is this decoded" check look like? I'd assume an enum variant? Checks like this are usually so fast as to be essentially zero cost.

[nyurik]

yeah, its just an enum variant check, so might not be that much of a cost, even if we do this on every feature access

[ianthetechie]

huh, TIL about the borrowme crate! That's pretty cool!

This smells like a similar set of decisions that I faced when designing the data model for Valinor, where we want to have zero-copy loading of data in pointers. The main difference is that it sounds like you're needing to actually decode data. Specifically, we mostly cared about mapping into a structure using zerocopy just as you would do a reinterpret_cast in C++. Zerocopy lets us do this safely with essentially zero overhead (one-time bounds checks).

For cases where we want to read data out, we do this lazily. It can be thought of as similar to decoding actually, except that in your case it sounds like you always want to decode the whole column (I think you got that right in your follow-up comment!).

parse input, partially decode, modify some fields, re-encode, write out

This is exactly what we needed to do as well with Valinor. Basically we want to be able to interpret some bytes in memory as a tile, but don't actually copy anything unless we need to modify it. We used the standard library Cow rather than borrowme; I'm sure there is a case where borrowme enables something new, but straight up Cow worked for my use case since it was almost entirely dealing with slices which can be converted to owned as Vecs only as needed for modification.

You can find the full code for the builder here: https://github.com/stadiamaps/valinor/blob/main/valhalla-graphtile/src/graph_tile/builder.rs.

TL;DR it sounds like you've probably found good enough modeling principles for a first pass.