Compression fails on linux/i386 with ZSTD_maxCLevel + ZSTD_c_nbWorkers

[bradking]

Describe the bug

Compression fails on 32-bit architectures at level 22 with threading.
I've traced the problem to a malloc(1<<31) failure.

To Reproduce

On linux/i386, such as docker run -it --platform=linux/i386 debian:testing, create the following source file:

example.c (click to expand/collapse)

#include <assert.h>
#include <zstd.h>

int main(void)
{
  size_t zret;

  unsigned char const buf_in[] = "Sample Input Buffer Content";
  ZSTD_inBuffer zin = { buf_in, sizeof(buf_in), 0 };
  unsigned char buf_out[4096];
  ZSTD_outBuffer zout = { buf_out, sizeof(buf_out), 0 };

  ZSTD_CStream* zcs = ZSTD_createCStream();

  /* Activate ZSTD_MULTITHREAD code paths.  */
  ZSTD_CCtx_setParameter(zcs, ZSTD_c_nbWorkers, 1);

  /* Set maximum compression level (22).  */
  zret = ZSTD_initCStream(zcs, ZSTD_maxCLevel());
  assert(!ZSTD_isError(zret) && "ZSTD_initCStream failed");

  /* Try compressing.  */
  zret = ZSTD_compressStream(zcs, &zout, &zin);
  assert(!ZSTD_isError(zret) && "ZSTD_compressStream failed");

  ZSTD_freeCStream(zcs);

  return 0;
}

Compile and link against a distro-packaged zstd 1.5.7. Run the binary:

$ gcc -g example.c -o example -lzstd && ./example
example: example.c:24: main: Assertion `!ZSTD_isError(zret) && "ZSTD_compressStream failed"' failed.

Compression fails.

Expected behavior

Compression succeeds.

Additional context

The failure occurs on this line with capacity == 0x80000000, sectionsSize == 0x20000000, mtctx->targetSectionSize == 0x20000000, nbWorkers == 1, nbSlackBuffers == 3.

[bradking]

The problem seems to be that the combination of nbWorkers == 1, nbSlackBuffers == 3, and mtctx->targetSectionSize == 0x20000000 leads to an attempt to allocate 4*0x20000000 (aka 0x80000000) bytes, which fails. I think the targetSectionSize set here is too big due to this line using a ZSTDMT_JOBLOG_MAX that is too large for a 32-bit architecture.

The following patch seems to resolve the problem, and adds a relevant debug log message:

patch (click to expand/collapse)

diff --git a/lib/compress/zstdmt_compress.c b/lib/compress/zstdmt_compress.c
index 0f1fe6d7..ca600cd2 100644
--- a/lib/compress/zstdmt_compress.c
+++ b/lib/compress/zstdmt_compress.c
@@ -1330,6 +1330,7 @@ size_t ZSTDMT_initCStream_internal(
                 ZSTD_customFree(mtctx->roundBuff.buffer, mtctx->cMem);
             mtctx->roundBuff.buffer = (BYTE*)ZSTD_customMalloc(capacity, mtctx->cMem);
             if (mtctx->roundBuff.buffer == NULL) {
+                DEBUGLOG(4, "ZSTD_customMalloc(0x%zx, ...) failed", capacity);
                 mtctx->roundBuff.capacity = 0;
                 return ERROR(memory_allocation);
             }
diff --git a/lib/compress/zstdmt_compress.h b/lib/compress/zstdmt_compress.h
index 91b489b9..acd4d124 100644
--- a/lib/compress/zstdmt_compress.h
+++ b/lib/compress/zstdmt_compress.h
@@ -32,8 +32,8 @@
 #ifndef ZSTDMT_JOBSIZE_MIN   /* a different value can be selected at compile time */
 #  define ZSTDMT_JOBSIZE_MIN (512 KB)
 #endif
-#define ZSTDMT_JOBLOG_MAX   (MEM_32bits() ? 29 : 30)
-#define ZSTDMT_JOBSIZE_MAX  (MEM_32bits() ? (512 MB) : (1024 MB))
+#define ZSTDMT_JOBLOG_MAX   (MEM_32bits() ? 28 : 30)
+#define ZSTDMT_JOBSIZE_MAX  (MEM_32bits() ? (256 MB) : (1024 MB))


 /* ========================================================