Support text and JSON

[seanlinsley]

Pco is designed for numeric data, so to compress strings we will want to pull in a binary compression algorithm like zstd.

JSON can also be compressed as if it's a string, but when the JSON all has the same structure it's worth considering converting it to protobuf or flattening it out to individual fields like pco_store does for structured data.

[seanlinsley]

I've split the struct idea out to #18, since that's much more complicated than supporting JSON.

[seanlinsley]

For binary compression, it seems like the current best option is zstd with the default compression level (3). Note that we can use zeekstd for incrmemental decompression.

Options to consider when adding support for JSON:

• jsonb
• jsonbb
• jsonm
• simple text JSON that uses zeekstd to avoid high memory usage (from decompressing everything at once)
• a custom Rust structure that is compatible with JSON, and uses a top-level array for all strings that are lazily decompressed when needed

Binary serialization formats to consider:

• serde-brief
• protobuf
• some other self-describing format?

[seanlinsley]

For text, we may want to optionally store a compressed index so read queries can filter by a text pattern, and then if the compression algorithm we use supports random access we might be able to decompress only the rows that matched the filter, improving performance and reducing memory pressure.

[seanlinsley]

I have a working branch using serde-brief, but that crate was archived just a few weeks ago... It also wouldn't have avoided large allocations: it incrementally decompresses the data but then deserializes it all at once, so there's still a large initial allocation.

In order to incrementally deserialize each row it looks like we'll need to incrementally serialize them in the first place. I was looking at bincode but it was also archived recently... postcard then?

postcard example:

use std::io::{Read, Write};
use zstd::stream::{read::Decoder, write::Encoder};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let data = vec!["apple".to_string(), "banana".to_string(), "cherry".to_string()];
    let mut compressed_buffer = Vec::new();

    {
        let mut encoder = Encoder::new(&mut compressed_buffer, 3)?;
        postcard::to_io(&(data.len() as u64), &mut encoder)?;
        for s in &data {
            postcard::to_io(s, &mut encoder)?;
        }
        encoder.finish()?;
    }

    {
        let mut decoder = Decoder::new(&compressed_buffer[..])?;
        let count: u64 = postcard::from_io(&mut decoder)?;
        for _ in 0..count {
            let s: String = postcard::from_io(&mut decoder)?;
        }
    }
    Ok(())
}

rkyv seems to be the fastest at (de)serialization so it's also worth considering:

use rkyv::{Archive, Serialize, access};
use zstd::stream::{Encoder, Decoder};
use std::io::{Write, Read};

fn incremental_serialize(strings: Vec<String>, writer: &mut impl Write) -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Encoder::new(writer, 3)?;
    for s in strings {
        // 1. Serialize individual string with rkyv
        let bytes = rkyv::to_bytes::<_, 256>(&s)?;
        
        // 2. Write length prefix for framing
        let len = bytes.len() as u32;
        encoder.write_all(&len.to_le_bytes())?;
        
        // 3. Write compressed data
        encoder.write_all(&bytes)?;
    }
    encoder.finish()?;
    Ok(())
}

fn incremental_deserialize(reader: &mut impl Read) -> Result<(), Box<dyn std::error::Error>> {
    let mut decoder = Decoder::new(reader)?;
    loop {
        let mut len_buf = [0u8; 4];
        if decoder.read_exact(&mut len_buf).is_err() { break; } // End of stream
        
        let len = u32::from_le_bytes(len_buf) as usize;
        let mut buf = vec![0u8; len];
        decoder.read_exact(&mut buf)?;
        
        // Zero-copy access to the archived string
        let archived = unsafe { access::<String>(&buf) };
        println!("Value: {}", archived.as_str());
    }
    Ok(())
}

The incremental and non-incremental versions should be benchmarked in terms of compressed size, CPU time, and memory usage. If serializing each string separately is slow, maybe serializing them in batches would help?

We should also consider JSON support: it would be nice to use the same dependencies for both text and JSON. Perhaps we could represent JSON using facet_value and facet-postcard?

Or for rkyv would it make sense to implement our own JSON-like struct?

use rkyv::{Archive, Deserialize, Serialize};
use std::collections::HashMap;

#[derive(Archive, Deserialize, Serialize, Debug, Clone)]
#[rkyv(check_bytes)] 
pub enum JsonLike {
    Null,
    Bool(bool),
    I64(i64),
    U64(u64),
    F64(f64),
    String(String),
    Array(Vec<JsonLike>),
    Object(HashMap<String, JsonLike>),
}

fn main() {
    // 1. Create data
    let mut map = HashMap::new();
    map.insert("name".to_string(), JsonLike::String("rkyv".to_string()));
    map.insert("version".to_string(), JsonLike::U64(0));
    
    let my_data = JsonLike::Object(map);

    // 2. Serialize to bytes
    let bytes = rkyv::to_bytes::<_, 256>(&my_data).unwrap();

    // 3. Zero-copy deserialize (access directly from bytes)
    let archived = rkyv::access::<JsonLike, rkyv::rancor::Error>(&bytes).unwrap();
    
    match archived {
        ArchivedJsonLike::Object(map) => {
            if let Some(ArchivedJsonLike::String(name)) = map.get("name") {
                println!("Name: {}", name.as_str());
            }
        }
        _ => println!("Not an object"),
    }
}

A point against rkyv is that it doesn't support adding new fields to a struct, but with postcard you can just serde(default) that field.