cross platform support (e.g. with CMake)?

[pieper]

We'd be interested in using pigz as a library in our application and would want to support Windows, Linux, and Mac with a uniform build system that supports native compilers (e.g. Visual Studio, not mingw). CMake has been our go-to solution for that so it would be great if pigz had CMake support.

Has anyone looked into CMake for pigz before?

[neurolabusc]

@pieper can you try the branch below which adapts @ningfei's Cmake super build scripts. These scripts have been used for years with dcm2niix. One clever feature is you can specify the ZLIB_IMPLEMENTATION to be System or Cloudflare (the default). The Cloudflare zlib accelerates pigz 40%, though it requires an x86-64 CPU built since 2009.

You can try this out with the following command:

git clone https://github.com/neurolabusc/pigz.git
cd pigz
mkdir build && cd build

I would be happy if @madler would consider including this Cmake script in the main pigz repository (though he might choose whether the System zlib should be the default).

Since the Cmake uses the glibc of your system, it would be great to test if these address issue 68. None of my computers have ever exhibited that issue, it would be great to know that it has been addressed.

[neurolabusc]

@pieper the current CMake scripts support Unix (MacOS and Linux), but not Windows. There is a fork of pigz that supports Windows, albeit v2.3.1 versus v2.4. @ningfei's dcm2niix scripts do support Windows, so maybe he can can extend this to pigz. It would be nice to have Windows support be part of the main trunk and even better if users can leverage the Cloudflare zlib.

[pieper]

See also the discussion here: https://discourse.vtk.org/t/cloudflare-zlib/2330/7

[dzenanz]

pigz is a program and not a library, therefore it cannot really replace zlib.

[neurolabusc]

@dzenanz, you are correct that pigz is a program and not a library. However, on Unix you can use it piped mode. After calling popen you can write to pigz using the same routines that you would to write to an uncompressed file, closing it with pclose. Please take a look at my modification of znzlib where a few lines of code allows one to select pigz output using the same functions designed to write uncompressed data. That project demonstrates how a copy of pigz compiled with CloudFlare zlib provides optimized parallel compression with very little changes to the code and no changes to the make script. In my example, I simply detect an environment variable to see if the user wants to write the data to pigz.

[dzenanz]

About 50% of programmers use Windows, and that percentage is much greater for ordinary users. If you don't have to support Windows (for whatever reason), PIGZ is fine. If you do (as all multi-platform libraries aim to do), PIGZ is out of the question.

[neurolabusc]

@dzenanz point taken. I evaluated znzlib since it is used by several of the most popular tools in my field (AFNI, FSL, FreeSurfer) and those tools only target Unix (though many Windows users run them via Windows Subsystem for Linux). I do not people have ported pigz to Windows, though I do not know if they have added modern CMake support or if pipes work with those. Using a pipe does have performance benefits. @ningfei is a Cmake guru who developed the dcm2niix scripts to work on Windows, Linux and MacOS as well as the Unix pigz CMake from my repository. Your comments might give him the incentive to look at making a CMake script for pigz that includes all the operating systems.

[ningfei]

Not really a CMake guru :D As I searched, currently there are only binaries available for the latest pigz for Windows. The open source port might be not up-to-date. Will have a look when I have time.

[neurolabusc]

@ningfei thanks for looking into this. As you note, forks of pigz that support the Microsoft compiler are based on pigz 2.3.1, which predates piped storage and the notes go up to Visual Studio 2012 (and remake instead of CMake). There are Windows compatible executables for pigz 2.4 built using gcc on MSYS2. I know the recent versions of Microsoft compilers have improved their compatibility with GCC conventions. Popular tools in my field (FSL, AFNI) only support Unix (though they run fine on Windows Subsystem for Linux), and the vtk thread linked above suggests the developers of that tool prefer libraries over stand alone applications.

@madler can you tell us how to proceed. To summarize @pieper asked for an enhancement for pigz to support CMake for deploying to MacOS, Linux and Windows. @ningfei and I provide a CMake script that supports MacOS and Linux (and optionally optimized versions of zlib). However, this script does not support Windows. The options are:

1. If you request, I can generate a pull request to add the CMake script to your repository.
2. You could relabel this issue as an enhancement and keep it open until some developer steps up to extend the CMake to support Windows.
3. You could close the issue, as it is an enhancement and not a bug, and outside of the remit of your project.

[madler]

Sure, you can generate a pull request for a CMake script, and I'll include it. However it sounds like your script does not resolve the original request for Windows support.

[neurolabusc]

You are correct, the CMake script currently only supports MacOS and Linux. It has the general benefits of CMake (detecting capabilities, installing missing components) and liabilities (requiring CMake), plus the added benefit of simplifying building with optimized zlib. So it would be a simple alternative to the current make file.

Before I make a pull request, I make a call out to Windows developers to see if anyone wants to extend the CMake script to Windows. @kjk do you have any interest in this?

[neurolabusc]

@madler I need some guidance on how to proceed with the pull request. There are three options:

1. The existing CMake makes minimal changes to pigz, targets Unix and MacOS. Allows the user to select CloudFlare-zlib, System or zlib-ng. Since the core code is virtually untouched, this has low risk of unintended consequences.

2. The windows branch includes the minimal changes to compile to Windows (while retaining compatibility with MacOS and Linux). At the moment, this can be built from the command line, but I think @ningfei could get a CMake working with Windows pretty easily. Since the core code is virtually untouched, this also has a low risk of unintended consequences. However, it does have a clear known limitation: it requires that filenames use Latin characters, and will fail with non-Latin values, e.g. pigz Описание.pdf returns pigz.exe: skipping: ????????.pdf does not exist.

3. I am willing to create a more thorough modification that will handle Windows files. But this course is only worth pursuing if you are willing to thoroughly vet the changes and would consider incorporating the changes in the main release. Given the large Windows community, I do think bugs could be quickly identified, but the code will have a lot of ifdefs that will make it harder to maintain. Specifically, the char * type will need to be wchar_t * . I think this is the simplest robust solution, but it will create widespread changes throughout the code. Below is a minimal example to demonstrate the changes.

// arg.c
// compile with: 
// cl /EHsc /TP arg.c
// arg  Описание.pdf

#include <io.h> // _setmode()
#include <fcntl.h> //_O_U16TEXT
#include <stdio.h>
                        
// For local functions and globals.
#define local static

#if defined(_MSC_VER)
local void process(wchar_t* arg) {
 wprintf(L"processing '%wS'\n", arg);
#else
local void process(char *path) {
 printf("processing '%s'\n", arg);
#endif
}

#if defined(_MSC_VER)
int wmain(int argc, wchar_t* argv[]) {
#else
int main(int argc, char **argv) {
#endif
    _setmode(_fileno(stdout), _O_U16TEXT);
    for (int i = 1; i < argc; ++i) {
    	process(argv[i]);
    }
 }

I am happy to follow any path you recommend, would just like to know your preference. I appreciate all your work to the community, and respect that ultimately you must maintain this code.