Add support for Gzip compression for the JSON format on Filesystem connector (V1 and V2) (ArroyoSystems#1007)

* Add support for Gzip compression for the JSON format on Filesystem connector (V1 and V2)

This patch adds support for compression when using the JSON format with the
Filsystem connector (V1 and V2). The compression algorithm can be specified
using the `json.compression` configuration knob.

At the moment, only `gzip` or `uncompressed` are supported; the default is
`uncompressed`.

Note that local filesystem is currently **not** supported.

* Make JsonCompression an Option and add config validation for JSON compression

The `JsonCompression` enum now only contains actual compression algorithms,
no more `Uncompressed`. To convey that something shouldn't be compressed, we
now use the `Option::None`.

This patch also adds some config validation; that ensures that the connector
is `filesystem` when `json.compression` is set.

* Address comments and move to a streaming compressor

This patch addresses the comments from ArroyoSystems#1007. It also moves to compressing
the JSON in a streaming fashion, instead of buffering in-memory.

* Address comments

1. Add comment to explain why expect won't panic
2. Model the buffer as an enum. This provides better typing support

Author: QnJ1c2kNCg

Cargo.lock

@@ -327,6 +327,7 @@ impl IntoResponse for HttpError {
         SchemaDefinition,
         TestSourceMessage,
         JsonFormat,
+        JsonCompression,
         AvroFormat,
         ProtobufFormat,
         ParquetFormat,

crates/arroyo-api/src/lib.rs

@@ -95,6 +95,7 @@ object_store = { workspace = true }
 deltalake = { workspace = true, features = ["s3", "gcs"] }
 delta_kernel = { workspace = true, features = ["arrow-55", "arrow-conversion"] }
 async-compression = { version = "0.4.3", features = ["tokio", "zstd", "gzip"] }
+flate2 = "1.0.30"
 iceberg = { version = "0.6.0" }
 iceberg-catalog-rest = "0.6.0"
 apache-avro = {workspace = true}