Skip to content

Commit 10ced00

Browse files
refactor: one codec type per block size for u32 and u64
Consolidate the separate FastPForWide128/256 types into the existing FastPFor128/FastPFor256: each now implements AnyLenCodec (u32) and BlockCodec64 (u64), mirroring the C++ codecs that expose both widths from one type. The u64 engine is retained as a private field, so buffer reuse is preserved and the public API drops the extra FastPForWide* names. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 53e4c19 commit 10ced00

8 files changed

Lines changed: 109 additions & 33 deletions

File tree

README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -60,14 +60,14 @@ assert_eq!(decoded, input);
6060

6161
### 64-bit integers (`u64`)
6262

63-
`FastPForWide128` / `FastPForWide256` compress `u64` values via the `BlockCodec64`
64-
trait. The wire format is byte-compatible with the C++ `CppFastPFor128` /
65-
`CppFastPFor256` `encode64` / `decode64` paths.
63+
The same `FastPFor128` / `FastPFor256` codecs also compress `u64` values via the
64+
`BlockCodec64` trait (`encode64` / `decode64`). The wire format is byte-compatible
65+
with the C++ `CppFastPFor128` / `CppFastPFor256` 64-bit paths.
6666

6767
```rust
68-
use fastpfor::{BlockCodec64, FastPForWide256};
68+
use fastpfor::{BlockCodec64, FastPFor256};
6969

70-
let mut codec = FastPForWide256::default();
70+
let mut codec = FastPFor256::default();
7171
let input: Vec<u64> = (0..600).map(|i| i * 1_000_000_000).collect();
7272

7373
let mut encoded = Vec::new();

fuzz/fuzz_targets/fastpfor_u64.rs

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22

33
//! Differential fuzz of the 64-bit FastPFOR codec.
44
//!
5-
//! For any `u64` input, `FastPForWide` and the C++ codec must produce identical compressed bytes.
5+
//! For any `u64` input, the Rust and C++ codecs must produce identical compressed bytes.
66
//! Every decoder must reproduce the original input.
77
//! Each side must also decode the other's output.
88
99
use fastpfor::cpp::{CppFastPFor128, CppFastPFor256};
10-
use fastpfor::{BlockCodec64, FastPForWide128, FastPForWide256};
10+
use fastpfor::{BlockCodec64, FastPFor128, FastPFor256};
1111
use libfuzzer_sys::fuzz_target;
1212

1313
#[derive(arbitrary::Arbitrary, Debug)]
@@ -44,17 +44,17 @@ fn check(rust: &mut impl BlockCodec64, cpp: &mut impl BlockCodec64, data: &[u64]
4444
fuzz_target!(|input: Input| {
4545
if input.use_256 {
4646
check(
47-
&mut FastPForWide256::default(),
47+
&mut FastPFor256::default(),
4848
&mut CppFastPFor256::default(),
4949
&input.data,
50-
"FastPForWide256",
50+
"FastPFor256",
5151
);
5252
} else {
5353
check(
54-
&mut FastPForWide128::default(),
54+
&mut FastPFor128::default(),
5555
&mut CppFastPFor128::default(),
5656
&input.data,
57-
"FastPForWide128",
57+
"FastPFor128",
5858
);
5959
}
6060
});

src/codec.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ pub trait BlockCodec: Default {
9393

9494
/// Codec that supports compressing 64-bit integers into a 32-bit word stream.
9595
///
96-
/// Implemented by the pure-Rust [`FastPForWide`](crate::FastPForWide) codecs.
96+
/// Implemented by the pure-Rust [`FastPFor128`](crate::FastPFor128) and [`FastPFor256`](crate::FastPFor256) codecs.
9797
/// With the `cpp` feature, `CppFastPFor128`, `CppFastPFor256`, and `CppVarInt` also implement it.
9898
/// For simple use, call `encode64` / `decode64` directly on the struct.
9999
///

src/lib.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ pub use bytemuck::Pod;
3030
#[cfg(feature = "rust")]
3131
pub use rust::{
3232
CompositeCodec, FastPFor, FastPFor128, FastPFor256, FastPForBlock128, FastPForBlock256,
33-
FastPForWide, FastPForWide128, FastPForWide256, JustCopy, VariableByte,
33+
JustCopy, VariableByte,
3434
};
3535

3636
// `src/test_utils.rs` uses `fastpfor::...`; alias this crate for unit tests only.

src/rust/composite.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ use crate::helpers::AsUsize;
4040
/// codec.decode(&encoded, &mut decoded, None).unwrap();
4141
/// assert_eq!(decoded, data);
4242
/// ```
43+
#[derive(Debug)]
4344
pub struct CompositeCodec<Blocks: BlockCodec, Tail: AnyLenCodec> {
4445
block: Blocks,
4546
tail: Tail,

src/rust/fastpfor_codec.rs

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
//! Public any-length `FastPFOR` codecs supporting both 32- and 64-bit integers.
2+
//!
3+
//! [`FastPFor128`] and [`FastPFor256`] are the primary entry points.
4+
//! Each implements [`AnyLenCodec`] for `u32` and [`BlockCodec64`] for `u64`.
5+
//! Aligned blocks are coded with `FastPFOR` and the sub-block remainder with variable-byte coding.
6+
7+
use crate::codec::{AnyLenCodec, BlockCodec64};
8+
use crate::rust::VariableByte;
9+
use crate::rust::composite::CompositeCodec;
10+
use crate::rust::integer_compression::fastpfor::{FastPForBlock128, FastPForBlock256};
11+
use crate::rust::integer_compression::fastpfor64::FastPForWide;
12+
use crate::FastPForResult;
13+
14+
macro_rules! define_fastpfor {
15+
($(#[$meta:meta])* $name:ident, $block:ty, $n:literal) => {
16+
$(#[$meta])*
17+
#[derive(Debug, Default)]
18+
pub struct $name {
19+
narrow: CompositeCodec<$block, VariableByte>,
20+
wide: FastPForWide<$n>,
21+
}
22+
23+
impl AnyLenCodec for $name {
24+
fn encode(&mut self, input: &[u32], out: &mut Vec<u32>) -> FastPForResult<()> {
25+
self.narrow.encode(input, out)
26+
}
27+
28+
fn decode(
29+
&mut self,
30+
input: &[u32],
31+
out: &mut Vec<u32>,
32+
expected_len: Option<u32>,
33+
) -> FastPForResult<()> {
34+
self.narrow.decode(input, out, expected_len)
35+
}
36+
}
37+
38+
impl BlockCodec64 for $name {
39+
fn encode64(&mut self, input: &[u64], out: &mut Vec<u32>) -> FastPForResult<()> {
40+
self.wide.encode64(input, out)
41+
}
42+
43+
fn decode64(&mut self, input: &[u32], out: &mut Vec<u64>) -> FastPForResult<()> {
44+
self.wide.decode64(input, out)
45+
}
46+
}
47+
};
48+
}
49+
50+
define_fastpfor! {
51+
/// Any-length `FastPFOR` codec with 128-value blocks.
52+
///
53+
/// Compresses `u32` via [`AnyLenCodec`] and `u64` via [`BlockCodec64`].
54+
FastPFor128, FastPForBlock128, 128
55+
}
56+
57+
define_fastpfor! {
58+
/// Any-length `FastPFOR` codec with 256-value blocks.
59+
///
60+
/// Compresses `u32` via [`AnyLenCodec`] and `u64` via [`BlockCodec64`].
61+
FastPFor256, FastPForBlock256, 256
62+
}
63+
64+
#[cfg(test)]
65+
mod tests {
66+
use super::*;
67+
68+
#[test]
69+
fn one_codec_handles_both_widths() {
70+
let mut codec = FastPFor256::default();
71+
72+
let data32: Vec<u32> = (0..600).collect();
73+
let mut enc32 = Vec::new();
74+
codec.encode(&data32, &mut enc32).unwrap();
75+
let mut dec32 = Vec::new();
76+
codec.decode(&enc32, &mut dec32, None).unwrap();
77+
assert_eq!(dec32, data32);
78+
79+
let data64: Vec<u64> = (0..600).map(|i| i * 1_000_000_000).collect();
80+
let mut enc64 = Vec::new();
81+
codec.encode64(&data64, &mut enc64).unwrap();
82+
let mut dec64 = Vec::new();
83+
codec.decode64(&enc64, &mut dec64).unwrap();
84+
assert_eq!(dec64, data64);
85+
}
86+
}

src/rust/integer_compression/fastpfor64.rs

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
//! 64-bit ([`u64`]) `FastPFOR` codec.
1+
//! 64-bit ([`u64`]) `FastPFOR` engine.
22
//!
33
//! This is the widened counterpart of the 32-bit [`FastPFor`](super::fastpfor::FastPFor).
44
//! Values, exceptions, and the exception bitmap are 64 bits wide instead of 32.
55
//! The output is byte-compatible with the C++ `CppFastPFor128` / `CppFastPFor256` 64-bit paths.
66
//!
7-
//! `FastPFOR` only handles complete blocks.
8-
//! A [`VariableByte`] tail encodes the sub-block remainder.
9-
//! [`FastPForWide`] bundles both and implements [`BlockCodec64`].
7+
//! [`FastPForWide`] is the 64-bit half of the public [`FastPFor128`](crate::FastPFor128) /
8+
//! [`FastPFor256`](crate::FastPFor256) codecs and is not exported on its own.
9+
//! It handles complete blocks, then a [`VariableByte`] tail encodes the sub-block remainder.
1010
1111
use std::cmp::min;
1212
use std::io::Cursor;
@@ -29,20 +29,14 @@ const DEFAULT_PAGE_SIZE: u32 = 65536;
2929
/// Number of frequency/exception buckets: one per possible bit width `0..=64`.
3030
const WIDTHS: usize = 65;
3131

32-
/// [`FastPForWide`] with 128-value blocks.
33-
pub type FastPForWide128 = FastPForWide<128>;
34-
35-
/// [`FastPForWide`] with 256-value blocks.
36-
pub type FastPForWide256 = FastPForWide<256>;
37-
3832
fn bits64(value: u64) -> usize {
3933
64 - value.leading_zeros().as_usize()
4034
}
4135

42-
/// 64-bit `FastPFOR` codec: `FastPFOR`-packed blocks plus a variable-byte tail.
36+
/// 64-bit `FastPFOR` engine: `FastPFOR`-packed blocks plus a variable-byte tail.
4337
///
4438
/// `N` is the block size (128 or 256 values).
45-
/// Use [`FastPForWide128`] or [`FastPForWide256`], or call [`encode64`](BlockCodec64::encode64) / [`decode64`](BlockCodec64::decode64).
39+
/// This is the internal `u64` engine behind [`FastPFor128`](crate::FastPFor128) and [`FastPFor256`](crate::FastPFor256).
4640
#[derive(Debug)]
4741
pub struct FastPForWide<const N: usize> {
4842
exception_buffers: [Vec<u64>; WIDTHS],

src/rust/mod.rs

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,14 @@
11
mod composite;
22
mod cursor;
3+
mod fastpfor_codec;
34
mod integer_compression;
45

56
pub use composite::CompositeCodec;
7+
/// Any-length `FastPFOR` codecs supporting both `u32` and `u64`.
8+
pub use fastpfor_codec::{FastPFor128, FastPFor256};
69
/// Type-safe block codec with block size encoded in the type.
710
pub use integer_compression::fastpfor::{FastPFor, FastPForBlock128, FastPForBlock256};
8-
/// 64-bit ([`u64`]) `FastPFOR` codec implementing [`BlockCodec64`](crate::BlockCodec64).
9-
pub use integer_compression::fastpfor64::{FastPForWide, FastPForWide128, FastPForWide256};
1011
/// Pass-through codec — implements [`AnyLenCodec`](crate::codec::AnyLenCodec).
1112
pub use integer_compression::just_copy::JustCopy;
1213
/// Variable-byte codec — implements [`AnyLenCodec`](crate::codec::AnyLenCodec).
1314
pub use integer_compression::variable_byte::VariableByte;
14-
15-
/// `FastPForBlock256` blocks + `VariableByte` remainder — the most common composite.
16-
pub type FastPFor256 = CompositeCodec<FastPForBlock256, VariableByte>;
17-
18-
/// `FastPForBlock128` blocks + `VariableByte` remainder.
19-
pub type FastPFor128 = CompositeCodec<FastPForBlock128, VariableByte>;

0 commit comments

Comments
 (0)