Scanning large arrays is very expensive

[bobrik]

Here's an example query that takes ~450ms with curl to execute and drain the results:

$ time curl -s -G http://localhost:8123/ --data-urlencode "query=SELECT stackMap.stack as stack, sum(stackMap.value) as value FROM profiles_v3 ARRAY JOIN stackMap WHERE profileType = 'process_cpu:cpu:nanoseconds:cpu:nanoseconds' AND timestamp >= now() - 3600 * 3 AND serviceName = 'grafana.alloy.ebpf' AND label_app_name = '/clickhouse' GROUP BY stackMap.stack" | wc -c
124736917

real    0m0.456s
user    0m0.041s
sys     0m0.105s

Note almost 125MiB of data here, it's not a tiny result.

We can do the same query with Go:

package main

import (
	"context"
	"log"
	"time"

	"github.com/ClickHouse/clickhouse-go/v2"
)

func main() {
	db, err := clickhouse.Open(&clickhouse.Options{
		Protocol: clickhouse.HTTP,
		Addr:     []string{"127.0.0.1:8123"},
	})
	if err != nil {
		log.Fatal(err)
	}

	started := time.Now()

	rows, err := db.Query(context.Background(), "SELECT stackMap.stack as stack, sum(stackMap.value) as value FROM profiles_v3 ARRAY JOIN stackMap WHERE profileType = 'process_cpu:cpu:nanoseconds:cpu:nanoseconds' AND timestamp >= now() - 3600 * 3 AND serviceName = 'grafana.alloy.ebpf' AND label_app_name = '/clickhouse' GROUP BY stackMap.stack")
	if err != nil {
		log.Fatal(err)
	}

	log.Printf("query returned in %.2fs", time.Since(started).Seconds())

	started = time.Now()

	// for rows.Next() {}

	stacks := []stack{}
	var frames []string
	var value uint64

	for rows.Next() {
		err := rows.Scan(&frames, &value)
		if err != nil {
			log.Fatal(err)
		}

		stacks = append(stacks, stack{frames: frames, value: value})
	}

	log.Printf("row scanning finished in %.2fs", time.Since(started).Seconds())
}

type stack struct {
	frames []string
	value  uint64
}

If we just iterate rows without any scanning (uncomment for rows.Next() {} and comment the actual loop):

$ go build -o /tmp/whoa ./cmd/huh && time GOMAXPROCS=1 /tmp/whoa
2026/01/04 06:53:50 query returned in 0.25s
2026/01/04 06:53:50 row scanning finished in 0.22s

real    0m0.469s
user    0m0.160s
sys     0m0.054s

That's pretty close to curl. We're also actually parsing columns, so that's not too bad:

If we add scanning (using the unchanged code above), wall time to execute pretty much doubles, while on-CPU user time goes up 3.625x:

$ go build -o /tmp/whoa ./cmd/huh && time GOMAXPROCS=1 /tmp/whoa
2026/01/04 06:54:37 query returned in 0.24s
2026/01/04 06:54:38 row scanning finished in 0.72s

real    0m0.968s
user    0m0.578s
sys     0m0.149s

Here's the flamegraph:

Zoomed into scanning specifically:

It would be nice for this to be a bit faster.

Details

Environment

• clickhouse-go version: v2.41.0
• Interface: ClickHouse API
• Go version: go1.25.4
• Operating system: Debian running Linux v6.19.0-rc1
• ClickHouse version: v25.10.3.100
• Is it a ClickHouse Cloud? It is not
• ClickHouse Server non-default settings, if any: none
• CREATE TABLE statements for tables involved: not necessarily relevant
• Sample data for all these tables, use clickhouse-obfuscator if necessary