Check manual

.github/workflows/dev-long-tests.yml

@@ -222,16 +222,13 @@ jobs:
         sudo apt-get install clang
         CC=clang FUZZER_FLAGS="--long-tests" make clean msan-fuzztest
 
-  # lasts ~24mn
   clang-msan-testzstd:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # tag=v4.1.1
     - name: clang + MSan + Test Zstd
       run: |
-        sudo apt-get update
-        sudo apt-get install clang
-        CC=clang make msan-test-zstd HAVE_ZLIB=0 HAVE_LZ4=0 HAVE_LZMA=0 V=1
+        CC=clang make -j msan-test-zstd HAVE_ZLIB=0 HAVE_LZ4=0 HAVE_LZMA=0 V=1
 
   armfuzz:
       runs-on: ubuntu-latest

.github/workflows/release_check.yml

@@ -0,0 +1,26 @@
+name: Checks before release
+
+on:
+  push:
+    branches:
+      - release
+
+jobs:
+  verify-manual:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v3
+
+      - name: Save current manual
+        run: mv doc/zstd_manual.html doc/zstd_manual_saved.html
+
+      - name: Generate new manual
+        run: make manual
+
+      - name: Compare manuals
+        run: |
+          if ! cmp -s doc/zstd_manual.html doc/zstd_manual_saved.html; then
+            echo "The manual was not updated before release !"
+            exit 1
+          fi

README.md

@@ -30,9 +30,9 @@ a list of known ports and bindings is provided on [Zstandard homepage](https://f
 
 For reference, several fast compression algorithms were tested and compared
 on a desktop featuring a Core i7-9700K CPU @ 4.9GHz
-and running Ubuntu 20.04 (`Linux ubu20 5.15.0-101-generic`),
+and running Ubuntu 24.04 (`Linux 6.8.0-53-generic`),
 using [lzbench], an open-source in-memory benchmark by @inikep
-compiled with [gcc] 9.4.0,
+compiled with [gcc] 14.2.0,
 on the [Silesia compression corpus].
 
 [lzbench]: https://github.com/inikep/lzbench
@@ -41,16 +41,16 @@ on the [Silesia compression corpus].
 
 | Compressor name         | Ratio | Compression| Decompress.|
 | ---------------         | ------| -----------| ---------- |
-| **zstd 1.5.6 -1**       | 2.887 |   510 MB/s |  1580 MB/s |
-| [zlib] 1.2.11 -1        | 2.743 |    95 MB/s |   400 MB/s |
-| brotli 1.0.9 -0         | 2.702 |   395 MB/s |   430 MB/s |
-| **zstd 1.5.6 --fast=1** | 2.437 |   545 MB/s |  1890 MB/s |
-| **zstd 1.5.6 --fast=3** | 2.239 |   650 MB/s |  2000 MB/s |
-| quicklz 1.5.0 -1        | 2.238 |   525 MB/s |   750 MB/s |
-| lzo1x 2.10 -1           | 2.106 |   650 MB/s |   825 MB/s |
-| [lz4] 1.9.4             | 2.101 |   700 MB/s |  4000 MB/s |
-| lzf 3.6 -1              | 2.077 |   420 MB/s |   830 MB/s |
-| snappy 1.1.9            | 2.073 |   530 MB/s |  1660 MB/s |
+| **zstd 1.5.7 -1**       | 2.896 |   510 MB/s |  1550 MB/s |
+| [zlib] 1.3.1 -1         | 2.743 |   105 MB/s |   390 MB/s |
+| brotli 1.1.0 -0         | 2.702 |   400 MB/s |   425 MB/s |
+| **zstd 1.5.7 --fast=1** | 2.439 |   545 MB/s |  1850 MB/s |
+| **zstd 1.5.7 --fast=3** | 2.241 |   635 MB/s |  1980 MB/s |
+| quicklz 1.5.0 -1        | 2.238 |   520 MB/s |   750 MB/s |
+| lzo1x 2.10 -1           | 2.106 |   650 MB/s |   780 MB/s |
+| [lz4] 1.10.0            | 2.101 |   675 MB/s |  3850 MB/s |
+| snappy 1.2.1            | 2.089 |   520 MB/s |  1500 MB/s |
+| lzf 3.6 -1              | 2.077 |   410 MB/s |   820 MB/s |
 
 [zlib]: https://www.zlib.net/
 [lz4]: https://lz4.github.io/lz4/

doc/zstd_manual.html

@@ -110,6 +110,7 @@ <h3>Decompression helper functions</h3><pre></pre><b><pre></pre></b><BR>
            - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
            - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
   note 1 : a 0 return value means the frame is valid but "empty".
+           When invoking this method on a skippable frame, it will return 0.
   note 2 : decompressed size is an optional field, it may not be present (typically in streaming mode).
            When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
            In which case, it's necessary to use streaming mode to decompress data.
@@ -128,9 +129,8 @@ <h3>Decompression helper functions</h3><pre></pre><b><pre></pre></b><BR>
 </p></pre><BR>
 
 <pre><b>ZSTD_DEPRECATED("Replaced by ZSTD_getFrameContentSize")
-ZSTDLIB_API
 unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
-</b><p>  NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize().
+</b><p>  This function is now obsolete, in favor of ZSTD_getFrameContentSize().
   Both functions work the same way, but ZSTD_getDecompressedSize() blends
   "empty", "unknown" and "error" results to the same return value (0),
   while ZSTD_getFrameContentSize() gives them separate return values.
@@ -142,7 +142,12 @@ <h3>Decompression helper functions</h3><pre></pre><b><pre></pre></b><BR>
  `srcSize` must be >= first frame size
  @return : the compressed size of the first frame starting at `src`,
            suitable to pass as `srcSize` to `ZSTD_decompress` or similar,
-        or an error code if input is invalid 
+           or an error code if input is invalid
+  Note 1: this method is called _find*() because it's not enough to read the header,
+          it may have to scan through the frame's content, to reach its end.
+  Note 2: this method also works with Skippable Frames. In which case,
+          it returns the size of the complete skippable frame,
+          which is always equal to its content size + 8 bytes for headers. 
 </p></pre><BR>
 
 <h3>Compression helper functions</h3><pre></pre><b><pre></pre></b><BR>
@@ -166,8 +171,7 @@ <h3>Compression helper functions</h3><pre></pre><b><pre></pre></b><BR>
 
 </p></pre><BR>
 
-<h3>Error helper functions</h3><pre></pre><b><pre>#include "zstd_errors.h" </b>/* list of errors */<b>
-</b>/* ZSTD_isError() :<b>
+<h3>Error helper functions</h3><pre></pre><b><pre></b>/* ZSTD_isError() :<b>
  * Most ZSTD_* functions returning a size_t value can be tested for error,
  * using ZSTD_isError().
  * @return 1 if error, 0 otherwise
@@ -1239,7 +1243,7 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
 </p></pre><BR>
 
 <pre><b>ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
-</b><p>  srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX.
+</b><p>  srcSize must be large enough, aka >= ZSTD_FRAMEHEADERSIZE_PREFIX.
  @return : size of the Frame Header,
            or an error code (if srcSize is too small) 
 </p></pre><BR>
@@ -1252,20 +1256,20 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
     unsigned blockSizeMax;
     ZSTD_FrameType_e frameType;          </b>/* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */<b>
     unsigned headerSize;
-    unsigned dictID;
+    unsigned dictID;                     </b>/* for ZSTD_skippableFrame, contains the skippable magic variant [0-15] */<b>
     unsigned checksumFlag;
     unsigned _reserved1;
     unsigned _reserved2;
-} ZSTD_frameHeader;
+} ZSTD_FrameHeader;
 </b></pre><BR>
-<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader(ZSTD_FrameHeader* zfhPtr, const void* src, size_t srcSize);   </b>/**< doesn't consume input */<b>
+<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader(ZSTD_FrameHeader* zfhPtr, const void* src, size_t srcSize);
 </b>/*! ZSTD_getFrameHeader_advanced() :<b>
  *  same as ZSTD_getFrameHeader(),
  *  with added capability to select a format (like ZSTD_f_zstd1_magicless) */
 ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader_advanced(ZSTD_FrameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format);
-</b><p>  decode Frame Header, or requires larger `srcSize`.
- @return : 0, `zfhPtr` is correctly filled,
-          >0, `srcSize` is too small, value is wanted `srcSize` amount,
+</b><p>  decode Frame Header into `zfhPtr`, or requires larger `srcSize`.
+ @return : 0 => header is complete, `zfhPtr` is correctly filled,
+          >0 => `srcSize` is too small, @return value is the wanted `srcSize` amount, `zfhPtr` is not filled,
            or an error code, which can be tested using ZSTD_isError() 
 </p></pre><BR>
 
@@ -1421,7 +1425,7 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
 ZSTD_compressSequencesAndLiterals(ZSTD_CCtx* cctx,
                       void* dst, size_t dstCapacity,
                 const ZSTD_Sequence* inSeqs, size_t nbSequences,
-                const void* literals, size_t litSize, size_t litCapacity,
+                const void* literals, size_t litSize, size_t litBufCapacity,
                 size_t decompressedSize);
 </b><p> This is a variant of ZSTD_compressSequences() which,
  instead of receiving (src,srcSize) as input parameter, receives (literals,litSize),
@@ -1435,20 +1439,22 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
  - Not compatible with frame checksum, which must be disabled
  - If any block is incompressible, will fail and return an error
  - @litSize must be == sum of all @.litLength fields in @inSeqs. Any discrepancy will generate an error.
- - the buffer @literals must have a size @litCapacity which is larger than @litSize by at least 8 bytes.
+ - @litBufCapacity is the size of the underlying buffer into which literals are written, starting at address @literals.
+   @litBufCapacity must be at least 8 bytes larger than @litSize.
  - @decompressedSize must be correct, and correspond to the sum of all Sequences. Any discrepancy will generate an error.
  @return : final compressed size, or a ZSTD error code.
 
 </p></pre><BR>
 
 <pre><b>ZSTDLIB_STATIC_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity,
-                                const void* src, size_t srcSize, unsigned magicVariant);
+                                 const void* src, size_t srcSize,
+                                       unsigned magicVariant);
 </b><p> Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
 
  Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
  ranging from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15.
- As such, the parameter magicVariant controls the exact skippable frame magic number variant used, so
- the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant.
+ As such, the parameter magicVariant controls the exact skippable frame magic number variant used,
+ so the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant.
 
  Returns an error if destination buffer is not large enough, if the source size is not representable
  with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore invalid).
@@ -1457,21 +1463,22 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
 
 </p></pre><BR>
 
-<pre><b>size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity, unsigned* magicVariant,
-                                const void* src, size_t srcSize);
-</b><p> Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer.
+<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity,
+                                      unsigned* magicVariant,
+                                      const void* src, size_t srcSize);
+</b><p> Retrieves the content of a zstd skippable frame starting at @src, and writes it to @dst buffer.
 
- The parameter magicVariant will receive the magicVariant that was supplied when the frame was written,
- i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START.  This can be NULL if the caller is not interested
- in the magicVariant.
+ The parameter @magicVariant will receive the magicVariant that was supplied when the frame was written,
+ i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START.
+ This can be NULL if the caller is not interested in the magicVariant.
 
  Returns an error if destination buffer is not large enough, or if the frame is not skippable.
 
  @return : number of bytes written or a ZSTD error.
 
 </p></pre><BR>
 
-<pre><b>unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size);
+<pre><b>ZSTDLIB_STATIC_API unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size);
 </b><p>  Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame.
 
 </p></pre><BR>