[C++] Be able to replace default Codec implementation

[HuaHuaY]

Describe the enhancement requested

For example, allow users to replace zlib with zlib-ng. Maybe we can implement a registration mechanism for Codec and use the default implementation without registration.

Component(s)

C++

[wgtmac]

Is this something worth doing? The general idea is to allow users to register custom Codec implementations to override the default one provided by Arrow for better performance. @pitrou @kou @zanmato1984

[kou]

Is there any other use case?
If you have only zlib-ng use case, it may be better that we have zlib-ng support instead of implementing a registration mechanism.

[zanmato1984]

It might make sense to allow compression library customization. There are two practical approaches to expose such customization point imo:

1. Allow specifying zlib distribution via cmake option. This way you can only customize at build time.
2. Allow enable/disable certain zlib distributions via cmake options. They are all built into the final binary if enabled. Expose the customization via an env var. This is essentially what we did for our memory pool. More complicated, but also more flexible in runtime.

What I don't think quite practical is: fully runtime registration and picking. Might be too much engineering complexity.

But before all of that, I'm also wondering about possible use cases, as @kou asked.

[wgtmac]

Runtime registration is useful when Arrow is used as a library dependency by a system where proprietary zlib|zstd|lz4 implementations are available and preferable.

[HuaHuaY]

We use arrow through conan2 and use the parquet writer to write data. We found that a lot of time is spent on compression, and we also want to decrease the size of the output file. It is currently not convenient to debug arrow compression. If custom implementations are allowed, we can use other implementations without modifying and re-compiling arrow's source code. We do not have to provide a mechanism to register Codec. We can provide some interfaces that allow us to pass some subclasses, and so on.