[Feature Request] Zstandard Codecs to core OpenSearch

[sarthakaggarwal97]

Is your feature request related to a problem? Please describe

Zstandard codecs (zstd & zstd_no_dict) were released in OS 2.9, and moved to a separate custom-codecs plugin repository.

We have had a stable GA for past five minor version releases for the Zstandard codecs, we should discuss on moving the codecs to core OpenSearch. This would help the users to use Zstandard codecs without having to install an additional plugin.
We also have the required library as a part of OpenSearch libs over here

Given the performance benefits we have seen, we can also explore the option of making codecs defaults for specific workloads (like timeseries etc.).

Describe the solution you'd like

Supporting Zstandard codecs in the core OpenSearch.

Related component

Indexing:Performance

Describe alternatives you've considered

No response

Additional context

No response

[lukas-vlcek]

Out of curiosity, are there any perf tests results available?

[dblock]

Out of curiosity, are there any perf tests results available?

Yes, https://opensearch.org/docs/latest/im-plugin/index-codecs/#benchmarking.

[mgodwan]

+1 on making this available within core (maybe even the default?).

We've observed ZSTD to provide better performance in quite a good amount of workloads while compressing better, and we already have required dependencies being loaded as part of the OpenSearch core. Also, since this has been part of the distribution we release for the past few versions, and we've not seen any issues, it makes a strong case to keep this available in core itself.

@backslasht @shwetathareja @Bukhtawar @dblock Thoughts on this?

[backslasht]

Zstd codecs clearly emerge as winner optimizing for both compression and speed.

+1 on moving this to core.

[shwetathareja]

+1 on moving to core