Skip to content

Feature: Support inserting "compressed" data #1936

Description

@kavirajk

Problem

Currently when user have option to choose the compression during creating connections. for example

	conn, err := clickhouse.Open(&clickhouse.Options{
		Addr:     []string{"127.0.0.1:8123"},
		Protocol: clickhouse.HTTP,
		Compression: &clickhouse.Compression{
			Method: clickhouse.CompressionGZIP,
		},
	})

The problem is, "who" compresses/decompresses is always the driver itself. And driver expects the plain uncompressed format all the time.

What if user say want to insert already compressed (gzip) parquet file?

Today that is not possible.

Reproduction program

  1. Create gzip(ed) parquet file
$ clickhouse-local -q "SELECT number as id, concat('user-', toString(number)) as name FROM numbers(1000) FORMAT Parquet" > /tmp/example.parquet
$ gzip /tmp/example.parquet
$
  1. Try to insert already compressed parquet

Go program

func main() {
	ctx := context.Background()

	conn, err := clickhouse.Open(&clickhouse.Options{
		Addr:     []string{"127.0.0.1:8123"},
		Protocol: clickhouse.HTTP,
		Compression: &clickhouse.Compression{
			Method: clickhouse.CompressionGZIP,
		},
	})
	if err != nil {
		panic(err)
	}

	if err := conn.Exec(ctx, `
		CREATE TABLE if NOT EXISTS example_format_parquet (
			  id   UInt64, name String
		) Engine = MergeTree
	`); err != nil {
		panic(err)
	}
	parquetFile, err := os.Open("/tmp/example.parquet.gz")
	if err != nil {
		panic(err)
	}
	defer parquetFile.Close()

	if err := conn.InsertFormat(ctx, "Parquet", "INSERT INTO example_format_parquet", parquetFile); err != nil {
		panic(err)
	}

}

Run

$ go run format_play.go
 Parquet: [HTTP 400] code: 117, message: Not a Parquet file (wrong magic bytes at the start): While executing ParquetV3BlockInputFormat. (INCORRECT_DATA) (version 25.12.3.21 (official build))

goroutine 1 [running]:
main.main()
	/home/kavi/src/play-go2/format_play.go:38 +0x1bb
exit status 2

Proposal

Have a way to transparently accept already compressed input formats. May be with something like WithContentEncoding("gzip") .

Which when set the driver sets the Content-Encoding: gzip header like it does now, but skip picking gzip compression writer from the pool and send it directory to the server with just the header set.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions