restructuring and renaming

Author: dbeck

tsl/src/compression/algorithms/CMakeLists.txt

@@ -8,3 +8,5 @@ set(SOURCES
     ${CMAKE_CURRENT_SOURCE_DIR}/null.c
     ${CMAKE_CURRENT_SOURCE_DIR}/uuid_compress.c)
 target_sources(${TSL_LIBRARY_NAME} PRIVATE ${SOURCES})
+
+add_subdirectory(fastlanes)

tsl/src/compression/algorithms/fastlanes/CMakeLists.txt

@@ -0,0 +1,5 @@
+set(SOURCES
+    ${CMAKE_CURRENT_SOURCE_DIR}/fastlanes_ffor.c
+    ${CMAKE_CURRENT_SOURCE_DIR}/fastlanes_pack.c  
+    ${CMAKE_CURRENT_SOURCE_DIR}/fastlanes_sizing.c)
+target_sources(${TSL_LIBRARY_NAME} PRIVATE ${SOURCES})

tsl/src/compression/algorithms/fastlanes/README.md

@@ -2,18 +2,19 @@
 
 ## What is this library
 
-This is a header only, C library that implements Fastlanes inspired bit-packing using C macros.
-The library only uses portable C and the main objective is to allow compilers generate efficient
-SIMD code from this. This is not a compression library. It is a building block for higher level
-compression algorithms.
+This is mostly a header only, C library that implements FastLanes inspired bit-packing using C
+macros and a very thin C wrapper over these. The library only uses portable C and the main
+objective is to allow compilers generate efficient SIMD code from this. This is not a compression
+library. It is a building block for higher level compression algorithms.
 
 ## Credits and introduction
 
 [The FastLanes Compression Layout:
 Decoding >100 Billion Integers per Second with Scalar Code](https://doi.org/10.14778/3598581.3598587)
 
-That is the basis of the implementation here. There are other C++ and Rust implementations out there, like
-[cwida/fastlanes](https://github.com/cwida/fastlanes) and [spiraldb/fastlanes](https://github.com/spiraldb/fastlanes)
+That paper is the main inspiration basis of the implementation. There are other C++ and Rust
+implementations out there, like [cwida/fastlanes](https://github.com/cwida/fastlanes) and
+[spiraldb/fastlanes](https://github.com/spiraldb/fastlanes).
 
 FastLanes is used as a building block for higher level compression algorithms, but not as a standalone
 compression algorithm. The main use of FastLanes is to pack integers. The callers need to do extra work
@@ -39,8 +40,8 @@ In case of odd bitwidths, for example 5 bits, and we want to pack them into a 64
  1) either we pack 12 integers and waste 4 bits, or
  2) we split the 13th integer and handle the overflow
 
-Both options are not ideal. The beauty of the FastLanes approach is that we can pack odd width integers when we
-have enough of them. In the compression context we tend to have many integers so the numbers are not an issue.
+None of the options are ideal. The beauty of the FastLanes approach is that we can pack odd width integers when we
+have 'enough' of them. In the compression context we tend to have many integers so the numbers are not an issue.
 
 The second advantage of the FastLanes approach is that it takes advantage of the 'enough' numbers to pack, and
 developed a set of instructions that allows to transpose the data with SIMD instructions. This is a huge
@@ -61,10 +62,10 @@ NULL values, in which case the underlying compression will only store the non-NU
 up with way less than 1000 non-NULL values. Finally, there are situations specific to TimescaleDB where we
 naturally have smaller batch sizes.
 
-The problem with targeting a fixed size of 1024 elements when we often have much smaller batches is that it leads
-to a lot of wasted space which will erode the packing advantages of FastLanes. The natural pack boundary of FL1024
-is 128 bytes, so the smallest amount we can truncate from the output is 128 bytes. This causes too much waste
-for smaller batches.
+The problem with targeting a fixed size of 1024 elements (as in the original paper) when we often have much
+smaller batches is that it leads to a lot of wasted space which will erode the packing advantages of FastLanes.
+The natural pack boundary of FL1024 is 128 bytes, so the smallest amount we can truncate from the output is
+128 bytes. This causes too much waste for smaller batches.
 
 ### Supporting vectorized execution
 
@@ -77,8 +78,8 @@ some level of cooperation from the caller. The three areas where it is required
 - the input buffer needs to have space for a certain number of elements, potentially more than the actual number of elements being packed
 - the packed buffer needs to be allocated with a certain size, potentially more than the actual size of the packed data, and the caller needs to use the returned truncated size to determine the result size
 
-Imagine that we have 255 integers to pack. It is faster to pad it to 256 elements and use AVX2 instructions
-than use SSE2 to pack 128 and handle the remaining 127 separately. The requirements the library has are
+Imagine that we have 249 integers to pack. It is faster to pad it to 256 elements and use AVX2 instructions
+than use SSE2 to pack 128 and handle the remaining 121 separately. The requirements the library has are
 designed along these lines, so we maximize speed at the expense of some extra buffer capacity.
 
 The library provides functions to inform the user about these requirements, and there is a section below, that
@@ -89,7 +90,8 @@ these three parameters:
 - the width of the input elements (T)
 - the actual bit-width of the packed elements (W)
 
-Where W is <= T, and N is <= 256. These determine the actual packing method the library uses and the requirements for the buffers.
+Where W is <= T, and N is <= 256. These determine the actual packing method the library uses and the requirements
+for the buffers.
 
 'N', 'W' and 'T' are the central concepts in the FastLanes library and they are often referred to by only these
 initials. The number of parallel lanes (S) is also a central concept but it is derived from T and the tier bits,
@@ -114,10 +116,10 @@ The tiered FL packing is used in many places in the compression algorithm, not o
 values but also for smaller integer arrays, for example the new dictionary compression has a smaller array of the
 dictionary entries, or in the PFOR compression the exceptions are stored as small integer arrays. In these cases
 we easily end up storing only a handful of elements and this is why the very small tiers are useful. The small
-tiers don't offer much SIMD performance but they save space significantly.
+tiers don't offer much SIMD performance but they save significant amount of space.
 
-The library provides wrappers over the FL tiers in the `fl/fl.h` file and it is strongly encouraged to use
-these instead of calling the specific tiers directly.
+The library provides wrappers over the FL tiers in the `fastlanes/fastlanes.h` file and the internal
+tiers are not exposed.
 
 ## The C implementation
 
@@ -146,36 +148,36 @@ truncated_size = ceil(rows * W / T) * tier_bytes
 ## Allocation, alignment and input sizing
 
 The SIMD operations mandate certain alignment and padding of the data. This includes both the input and the
-output buffers. In case of the input buffer the operations we use may over-read the input data if not properly
-sized. For the output data, the encoder may overwrite adjacent memory if not properly sized. Similarly, during
-unpacking we must carefully size the output buffer and ensure that the input buffer is aligned and padded.
+output buffers. For example, in case of the input buffer for the packing operation we may over-read the
+input data beyond the useful elementts and it needs to be properly sized. For the output data, the encoder
+may overwrite adjacent memory if not properly sized. Similarly, during unpacking we must carefully size the
+output buffer and ensure that the input buffer is aligned and padded by zeros.
 
 The exact amount of alignment and padding depends on the FL tier. For example FL256 operates on 32 byte aligned
 data and it requires the tail padding to be 32 bytes as well. The other tiers have different (smaller)
 requirements. The natural truncation point of the output also depends on the tier.
 
-The [fl/fl.h](./fl.h) header has functions to wrap the FL tiers and forward the sizing/packing/unpacking calls
-to the right tier based on the input parameters. This is for convenience. The users may choose to call
-a given tier directly as well.
+The [fastlanes/fastlanes.h](./fastlanes.h) header provides functions to size, pack, and unpack data.
+The right internal tier is automatically selected based on the input parameters.
 
-The 'fl/fl.h' header has functions to determine the required alignment and allocation size based on the
-(N, W, T) triple. These functions are:
+The 'fastlanes/fastlanes.h' header has functions to determine the required alignment and allocation
+size based on the (N, W, T) triple. These functions are:
 
-```C
+``` C
 /*
  * Tells how many bytes to allocate for the pack/unpack buffer based on the (N, W, T) triple.
- * This function selects the right FL tier based on the input parameters and returns the required
- * allocation size.
+ * This function determines the right FL tier based on the input parameters and returns the required
+ * allocation size based on the parameters.
  */
-size_t fl_alloc_bytes(uint32_t n, uint8_t w, fl_elem_width_t t);
+size_t fl_required_bytes(uint32_t n, uint8_t w, fl_elem_width_t t);
 
 /*
- * The truncated size allows to determine the result size in the output buffer. With
- * this function we can determine the part of the output buffer that is actually used. Remember,
- * this is needed because the FL tiers operate on a fixed number of elements which is larger or
- * equal to the actual number of elements we pack (for SIMD efficiency).
+ * The function determines the result size in the output buffer. With this we can determine the
+ * part of the output buffer that is actually used. Remember, this is needed because the FL tiers
+ * operate on a fixed number of elements which is larger or equal to the actual number of elements
+ * we pack (for SIMD efficiency).
  */
-size_t fl_truncated_bytes(uint32_t n, uint8_t w, fl_elem_width_t t);
+size_t fl_result_bytes(uint32_t n, uint8_t w, fl_elem_width_t t);
 
 /*
  * The fl_alignment function returns the required alignment for the buffers based on the
@@ -188,7 +190,7 @@ size_t fl_truncated_bytes(uint32_t n, uint8_t w, fl_elem_width_t t);
 size_t fl_alignment(uint32_t n, fl_elem_width_t t);
 
 /*
- * The number of input elements of the `values` to be packed. The library expect
+ * The number of input elements of the `values` to be packed. The library expects
  * this many elements and it is the caller's responsibility to provide it. As the
  * library uses SIMD instructions, it needs to operate on fixed sized element chunks.
  */
@@ -204,12 +206,12 @@ size_t fl_input_bytes(uint32_t n, fl_elem_width_t t);
 | Buffer         | Min size                        | Alignment                                   |
 |----------------|---------------------------------|---------------------------------------------|
 | values (input) | `fl_input_count` elements       | `fl_alignment` recommended, minimum natural |
-| packed         | `fl_alloc_bytes` bytes          | `fl_alignment` mandatory                    |
+| packed         | `fl_required_bytes` bytes       | `fl_alignment` mandatory                    |
 | unpack output  | `fl_input_count` elements       | `fl_alignment` recommended, minimum natural |
 
 ## Unpack tail padding
 
-The input of the unpack functions must be padded with zeros up to `fl_alloc_bytes`. This
+The input of the unpack functions must be padded with zeros up to `fl_required_bytes`. This
 is because the SIMD instructions will read beyond the actual packed data, up to the allocated size.
 
 ## Interface contract
@@ -219,13 +221,10 @@ This is a concise recap of the previous sections.
 - the buffer for the input values MUST have at least `fl_input_count` elements (not bytes) and it SHOULD be aligned to `fl_alignment` bytes, and it MUST be aligned naturally
 - the packed buffer (pack output / unpack input) MUST be aligned to `fl_alignment` bytes
 - the output of the unpack function holding the unpacked values SHOULD be aligned to `fl_alignment` bytes, and MUST be aligned naturally
-- the size of the pack/unpack buffer must be `fl_alloc_bytes` bytes
-- the unpack input buffer must be padded with zeros up to `fl_alloc_bytes` bytes
-- `fl_pack` and `fl_pack_ffor` return the packed size in bytes, which is the same as `fl_truncated_bytes`
-- the input size `n` MUST NOT exceed 256, if using the fl/fl.h provided interface, and
-- if you use a specific tiered packer directly (in addition to the above):
-  - `n` MUST NOT exceed the tier's limit
-  - `w` MUST be <= `T`
+- the size of the pack/unpack buffer must be `fl_required_bytes` bytes
+- the unpack input buffer must be padded with zeros up to `fl_required_bytes` bytes
+- `fl_pack` and `fl_pack_ffor` return the packed size in bytes, which is the same as `fl_result_bytes`
+- the input size `n` MUST NOT exceed 256
 
 ## Simple packing vs FFOR packing
 
@@ -258,10 +257,13 @@ caller's responsibility to store the `base` value, similar to `w` and `n`.
 
 ## Batch sizing
 
-The various FL tiers have a maximum capacity. In case of FL256, it is 256 elements. It is the
-caller's responsibility to manage the inputs and potentially split it such that it fits into
-one of the available FL tiers. This is a deliberate decision, because in practice, using the
-same bitwidth for more than 256 elements is rarely economical because of the outliers.
+The maximum number of elements the FL tiers can handle is 256 elements (FL256 tier). It is the
+caller's responsibility to manage the inputs and potentially split it such that it fits. This
+is a deliberate decision, because in practice, using the same bitwidth for more than 256 elements
+is rarely economical because of the outliers.
+
+When passing less than 256 elements, the library chooses the best tier based on performance
+and packing efficiency.
 
 ## Signed integers and input validation
 
@@ -271,8 +273,7 @@ is chosen such that the subtracted values are non-negative. The library does not
 the inputs, so if the caller passes negative values, the behavior is undefined.
 
 Similarly, the library doesn't validate the 'W' parameter to be <= T, or N <= 256, so
-it is the caller's responsibility to ensure that these conditions are met. If one
-calls a tiered FL function directly, the constraints for N can be tighter.
+it is the caller's responsibility to ensure that these conditions are met.
 
 ## The W=0 special case
 
@@ -285,11 +286,11 @@ The pack functions in both cases will return 0, and they handle this special cas
 and with that, the functions ignore the buffers and passing NULLs as buffers is safe. The
 unpack functions need a valid result buffer. Also the below holds:
 
-```C
-fl_alloc_bytes(n, 0, t) == fl_truncated_bytes(n, 0, t) == 0
+``` C
+fl_required_bytes(n, 0, t) == fl_result_bytes(n, 0, t) == 0
 ```
 
 ## Test and usage examples
 
-Example usage is provided in the `test_fl.c` file. The purpose of this file is to
+Example usage is provided in the `test_fastlanes.c` file. The purpose of this file is to
 demonstrate the usage of the library.

tsl/src/compression/algorithms/fastlanes/fastlanes.h

@@ -0,0 +1,98 @@
+/*
+ * This file and its contents are licensed under the Timescale License.
+ * Please see the included NOTICE for copyright information and
+ * LICENSE-TIMESCALE for a copy of the license.
+ */
+
+/*
+ * fastlanes/fastlanes.h -- public interface for the FastLanes pack/unpack
+ * layer.
+ *
+ *   This is the ONLY header callers should include. All functions are
+ *   declared here; their definitions live in fastlanes_sizing.c
+ *   (tier selection + sizing API), fastlanes_pack.c (plain pack and
+ *   unpack), and fastlanes_ffor.c (FFOR pack and unpack).
+ *
+ * Conventions:
+ *
+ *   - 'N' is the number of values to pack (0..256).
+ *   - 'T' is the element bit width (8, 16, 32, or 64), expressed
+ *     via fl_elem_width_t.
+ *   - 'W' is the actual bit width of the packed elements (0..T).
+ *
+ *   See README.md for the full caller contract: alignment and
+ *   sizing rules.
+ */
+#pragma once
+
+#include <postgres.h>
+
+#include "fastlanes_types.h" /* fl_elem_width_t, fl_tier_width_t */
+
+/*
+ * Tier selection.
+ *
+ *   | N range  | T = 8 | T = 16 | T = 32 | T = 64 |
+ *   |   0..8   | FL8   | FL16   | FL32   | FL128  |
+ *   |   9..16  | FL16  | FL16   | FL32   | FL128  |
+ *   |  17..32  | FL32  | FL32   | FL32   | FL128  |
+ *   |  33..64  | FL64  | FL64   | FL64   | FL128  |
+ *   |  65..128 | FL128 | FL128  | FL128  | FL128  |
+ *   | 129..256 | FL256 | FL256  | FL256  | FL256  |
+ */
+fl_tier_width_t fl_tier_select(uint32 n, fl_elem_width_t t);
+
+/* Required size for the tier (selected by fl_tier_select(N, T)).
+ * Determines the sizes of pack output and unpack input buffers exactly.
+ */
+size_t fl_required_bytes(uint32 n, uint8 w, fl_elem_width_t t);
+
+/* Bytes the encoded output carries for N elements (<= fl_required_bytes).
+ * Matches the return of fl_pack / fl_pack_ffor; useful for sizing the
+ * truncated prefix without running the encoder. */
+size_t fl_result_bytes(uint32 n, uint8 w, fl_elem_width_t t);
+
+/* Required alignment for the _packed_ input and output buffers.
+ * Recommended alignment for the input and output _values_.
+ */
+size_t fl_alignment(uint32 n, fl_elem_width_t t);
+
+/* Number of input ELEMENTS the kernel reads -- callers must provide
+ * at least this many readable elements at `values` (positions past N
+ * are read but only the [0..N) outputs are meaningful). */
+uint32 fl_input_count(uint32 n, fl_elem_width_t t);
+
+/* Byte size of the input buffer (= fl_input_count() * t / 8). */
+size_t fl_input_bytes(uint32 n, fl_elem_width_t t);
+
+/*
+ * Plain pack / unpack.
+ *
+ *   fl_pack returns the truncated byte count (<= fl_required_bytes) --
+ *   the size of the `packed` data in bytes. The kernel reads
+ *   fl_input_count() elements from `values` and writes fl_required_bytes
+ *   to `packed`.
+ *
+ *   fl_unpack reverses the operation. The caller MUST pre-zero
+ *   packed[truncated_bytes..alloc_bytes) before calling unpack.
+ *
+ *   W = 0 (constant block) is a special case: fl_pack returns 0 and
+ *   writes nothing; fl_unpack fills outputs [0..N) with 0.
+ */
+size_t fl_pack(const void *values, void *packed, uint32 n, uint8 w, fl_elem_width_t t);
+void fl_unpack(const void *packed, void *values, uint32 n, uint8 w, fl_elem_width_t t);
+
+/*
+ * FFOR (Frame Of Reference) variants.
+ *
+ *   `base` is subtracted from each value before packing and added back
+ *   after unpacking. Equivalent to packing `values - base` with the
+ *   plain functions, but combined into the same loop.
+ *
+ *   W = 0: fl_pack_ffor returns 0 and writes nothing; fl_unpack_ffor
+ *   fills outputs [0..N) with `base`.
+ */
+size_t fl_pack_ffor(const void *values, void *packed, uint32 n, uint8 w, fl_elem_width_t t,
+					uint64 base);
+void fl_unpack_ffor(const void *packed, void *values, uint32 n, uint8 w, fl_elem_width_t t,
+					uint64 base);

…ression/algorithms/fastlanes/fl_common.h …/algorithms/fastlanes/fastlanes_common.htsl/src/compression/algorithms/fastlanes/fl_common.h renamed to tsl/src/compression/algorithms/fastlanes/fastlanes_common.h tsl/src/compression/algorithms/fastlanes/fl_common.h renamed to tsl/src/compression/algorithms/fastlanes/fastlanes_common.h

@@ -5,7 +5,7 @@
  */
 
 /*
- * fl/fl_common.h -- shared macro vocabulary for the FL tier headers.
+ * fastlanes/fastlanes_common.h -- shared macros for the FL tier headers.
  *
  * Two pieces every tier needs:
  *
@@ -22,7 +22,7 @@
 
 #pragma once
 
-#include "fl_types.h"
+#include "fastlanes_types.h"
 
 /*
  * Bit-position helpers -- compile-time when (row, W, T) are constants.