When executing a COPY command with a jsonb/json column, 1 core out of 16 is loaded.

[tsorokin36]

I'm running the following query on an ec2 m7g.4xlarge with postgres 15 and pg_duckdb 1.0.0 installed :

SET duckdb.force_execution = true;
SET duckdb.preserve_insertion_order = false;
SET duckdb.max_workers_per_postgres_scan = 16;
SET duckdb.threads_for_postgres_scan = 16;
COPY (
	SELECT
	id, -- type int
    inputs, -- type jsonb,
    extra, -- type json,
    condition -- type jsonb,
FROM schema_name.table_name
)
TO 's3://bucket_name/path'
WITH (
	FORMAT 'PARQUET',
	PER_THREAD_OUTPUT true,
	OVERWRITE_OR_IGNORE true,
	FILE_SIZE_BYTES '500MB'
);

I see that out of 16 cores, one is running at 100%, and there's minimal reading from EBS.
My table weighs more than 1 TB, but I don't think it matters.
When I run a query without JSON/JSONB columns, htop shows the expected 16 cores are running at 50-70%, and maximum reading from EBS.
Why are other cores being ignored when working with JSON/JSONB? Are there ways to achieve the same parallelism with these columns?

[YuweiXiao]

Hi, this is expected. We rely on Postgres utility functions for JSON/JSONB (also list, varbit, bit type) conversion (from pg layout to duckdb's). Since those utility functions are not thread safe, we limit the conversion to single thread.

The related PR for parallel pg scan: #762 (comment)