Problem
Currently when user have option to choose the compression during creating connections. for example
conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{"127.0.0.1:8123"},
Protocol: clickhouse.HTTP,
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionGZIP,
},
})
The problem is, "who" compresses/decompresses is always the driver itself. And driver expects the plain uncompressed format all the time.
What if user say want to insert already compressed (gzip) parquet file?
Today that is not possible.
Reproduction program
- Create gzip(ed) parquet file
$ clickhouse-local -q "SELECT number as id, concat('user-', toString(number)) as name FROM numbers(1000) FORMAT Parquet" > /tmp/example.parquet
$ gzip /tmp/example.parquet
$
- Try to insert already compressed parquet
Go program
func main() {
ctx := context.Background()
conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{"127.0.0.1:8123"},
Protocol: clickhouse.HTTP,
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionGZIP,
},
})
if err != nil {
panic(err)
}
if err := conn.Exec(ctx, `
CREATE TABLE if NOT EXISTS example_format_parquet (
id UInt64, name String
) Engine = MergeTree
`); err != nil {
panic(err)
}
parquetFile, err := os.Open("/tmp/example.parquet.gz")
if err != nil {
panic(err)
}
defer parquetFile.Close()
if err := conn.InsertFormat(ctx, "Parquet", "INSERT INTO example_format_parquet", parquetFile); err != nil {
panic(err)
}
}
Run
$ go run format_play.go
Parquet: [HTTP 400] code: 117, message: Not a Parquet file (wrong magic bytes at the start): While executing ParquetV3BlockInputFormat. (INCORRECT_DATA) (version 25.12.3.21 (official build))
goroutine 1 [running]:
main.main()
/home/kavi/src/play-go2/format_play.go:38 +0x1bb
exit status 2
Proposal
Have a way to transparently accept already compressed input formats. May be with something like WithContentEncoding("gzip") .
Which when set the driver sets the Content-Encoding: gzip header like it does now, but skip picking gzip compression writer from the pool and send it directory to the server with just the header set.
Problem
Currently when user have option to choose the compression during creating connections. for example
The problem is, "who" compresses/decompresses is always the driver itself. And driver expects the plain uncompressed format all the time.
What if user say want to insert already compressed (gzip) parquet file?
Today that is not possible.
Reproduction program
Go program
Run
Proposal
Have a way to transparently accept already compressed input formats. May be with something like
WithContentEncoding("gzip").Which when set the driver sets the
Content-Encoding: gzipheader like it does now, but skip picking gzip compression writer from the pool and send it directory to the server with just the header set.