Support nested arrays

[seanlinsley]

When monitoring pg_stat_activity, we track which processes are blocking or blocked by other processes. Each backend has an array of PIDs for these fields which have a variable length. In order to flatten those so that pco can compress them, we can split it up into two arrays: the first array tracks the length of each nested array, and the second array is a flat concatenation of all values.

So this data:

[[1,2,3],[4,5],[6]]

Is transformed into this before compression:

[3,2,1]
[1,2,3,4,5,6]

Allowing wrapped structs to have array fields:

#[pco_store::store(timestamp = collected_at, group_by = [server_id])]
pub struct BackendSnapshot {
    pub server_id: i64,
    pub collected_at: SystemTime,
    pub pid: i64,
    pub blocking_pids: Vec<i64>,
    pub blocked_by_pids: Vec<i64>,
}

Note: both array lengths and flat values would be concatenated to a single byte array after compression. Does pco support partial decompression of bytes? If not, every pco-compressed blob is prefixed with an identifier so we can at least split the bytes using that. The design should keep #18 in mind; is there a common pattern or crate we can use?

[seanlinsley]

The implementation would be more complicated when combined with #16. Each field needs its own iterator so it can decompress as many nested data points as exist for a single row, and would need to temporarily retain the decompressed data that doesn't belong to this row so it's available for the next row.

[seanlinsley]

Note that this intermediate structure requires two pco compress & decompress calls. If the data types are compatible, there's no reason we couldn't encode it as a single vec: with the first element being the length of the first vec, and then flattening both vecs into this one. So instead of:

[3,2,1]
[1,2,3,4,5,6]

It'd be:

[3,3,2,1,1,2,3,4,5,6]

But there's the issue of compatibility. A usize is a u64, which e.g. cannot be contained in the full -/+ range of i32. So to support i32 we'd have to at runtime ensure that no row can contain that much data. It might be reasonable to limit row size to 100,000 for this, with the added benefit of avoiding out of memory crashes. It may even make sense to use i16's max of 32,767. Note that each nested Vec also needs to be checked that it doesn't go over this limit.

pco just added support for i8 and u8: pcodec/pcodec#362. For that to support this model we'd have to encode them as a larger numeric type.

[seanlinsley]

Martin (pco author) agrees that upcasting is relatively cheap, so e.g. we'd want to do that for u8. But he warned that concatenating both arrays may lead to significantly worse compression, and noted that version 1.0.0 should significantly improve decompression times for small arrays.

So, first we should confirm if a single flat array actually results in worse compression. If it does, we'll want to compress the two arrays separately, which is blocked on deciding on a binary serialization format.