DuckDB throws Invalid data or Snappy decompression failure while reading parquet from S3 after loading cache_httpfs

[ahmed-shameem]

Describe the bug
Large Parquet file in S3 using snappy compression (50M records) is created (size: 4GB)
writer = pq.ParquetWriter(output_file, schema, compression='snappy')

While invoking read_parquet to select from the created parquet from S3 on the above file, DuckDB throws either Invalid data error if we load cache_httpfs.

To Reproduce

1. Create parquet using snappy compression (50M records and size > 4GB):

writer = pq.ParquetWriter(output_file, schema, compression='snappy')

2. Install and Load cache_httpfs:

INSTALL cache_httpfs FROM community;
LOAD cache_httpfs;

3. Set the S3 credentials:

D SET s3_region='***';
D SET s3_access_key_id='***';
D SET s3_secret_access_key='***';

4. SELECT using read_parquet:

D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
TProtocolException: Invalid data
D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
TProtocolException: Invalid data
D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
TProtocolException: Invalid data
D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
TProtocolException: Invalid data
D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
TProtocolException: Invalid data
D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
don't know what type:
D explain analyze SELECT * FROM read_parquet('your-s3-parquet-path');
Invalid Error:
TProtocolException: Invalid data

Even after multiple retries, the query does NOT succeed.

Expected behavior
The query should be executed successfully

Screenshots
NA

Desktop (please complete the following information):
MacOS arm64

Smartphone (please complete the following information):
NA

DuckDB Version:
v1.4.3 (Andium) d1dc88f950

DuckDB Client:
CLI

Additional context
Initially created an issue in DuckDB side: duckdb/duckdb#20167

[dentiny]

Hi @ahmed-shameem thanks for the report! But somehow I cannot reproduce the issue on my end.
This is what I did:

• Setup local minio env as S3 bucket
• Use duckdb to dump parquet files and confirm it has enough rows and object size
• Query with cache httpfs extension and make sure query/result works correctly
• Make sure the on-disk cache does take affect
• Repeat the above procedures for multiple times

This is my sql test script:

# name: test/sql/parquet_snappy_compression.test
# description: Test reading parquet files with snappy compression through cache_httpfs
# group: [sql]

require cache_httpfs

require parquet

# Override the default behaviour of skipping HTTP errors and connection failures
# This test should fail on connection issues, not be skipped
set ignore_error_messages

# Configure S3/MinIO credentials for the devcontainer MinIO instance
statement ok
SET s3_endpoint='minio:9000';

statement ok
SET s3_access_key_id='minioadmin';

statement ok
SET s3_secret_access_key='minioadmin';

statement ok
SET s3_region='us-east-1';

statement ok
SET s3_use_ssl=false;

statement ok
SET s3_url_style='path';

statement ok
SET cache_httpfs_type='on_disk';

statement ok
SET cache_httpfs_cache_block_size=1048576;

statement ok
SET cache_httpfs_cache_directory='__TEST_DIR__/cache';

statement ok
SELECT cache_httpfs_clear_cache();

# Verify cache is empty before starting
query I
SELECT cache_httpfs_get_ondisk_data_cache_size();
----
0

# Create a large file (50M records, ~4GB size, similar to the bug report scenario)
# Add many columns with substantial data to ensure we reach ~4GB file size
statement ok
CREATE TABLE large_test_data AS 
SELECT 
    i::INTEGER as id, 
    (i * 2.5)::DOUBLE as value, 
    'large_test_string_' || i::VARCHAR as text_col,
    (i % 200)::BIGINT as category,
    'additional_text_data_' || (i % 1000)::VARCHAR || '_' || (i * 3)::VARCHAR || '_more_data_' || (i % 500)::VARCHAR as long_text_col,
    (i * 1.1)::DOUBLE as value2,
    (i * 1.2)::DOUBLE as value3,
    (i * 1.3)::DOUBLE as value4,
    (i * 1.4)::DOUBLE as value5,
    (i * 1.5)::DOUBLE as value6,
    'prefix_' || (i % 500)::VARCHAR || '_suffix_' || (i * 2)::VARCHAR || '_extra_' || (i % 100)::VARCHAR as another_text_col,
    'col_' || (i % 10000)::VARCHAR || '_data_' || (i * 5)::VARCHAR || '_end' as text_col2,
    'col_' || (i % 5000)::VARCHAR || '_data_' || (i * 7)::VARCHAR || '_end' as text_col3,
    (i % 10000)::BIGINT as category2,
    (i % 5000)::BIGINT as category3,
    (i % 2500)::BIGINT as category4,
    (i % 1250)::BIGINT as category5,
    (i * 0.9)::DOUBLE as value7,
    (i * 0.8)::DOUBLE as value8,
    (i * 0.7)::DOUBLE as value9,
    (i * 0.6)::DOUBLE as value10
FROM range(0, 50000000) t(i);

# Write large file with snappy compression on S3
statement ok
COPY large_test_data TO 's3://test-bucket/test_snappy_large.parquet' (FORMAT PARQUET, COMPRESSION 'snappy');

# Verify cache is still empty after write (write doesn't use cache)
query I
SELECT cache_httpfs_get_ondisk_data_cache_size();
----
0

# Read the large parquet file through cache_httpfs (first read - should populate cache)
# Verify core columns to ensure data integrity
query IIII nosort
SELECT COUNT(*), SUM(id), SUM(value), COUNT(DISTINCT category) 
FROM read_parquet('s3://test-bucket/test_snappy_large.parquet');
----
50000000	1249999975000000	3124999937500000.0	200

# Verify cache has been populated after first read
query I
SELECT cache_httpfs_get_ondisk_data_cache_size() > 0;
----
1

# Inspect cache directory contents in detail
# Show all cache files
query T
SELECT file FROM glob('__TEST_DIR__/cache/*') ORDER BY file;
----

# Show cache status with detailed information
query IIIII
SELECT * FROM cache_httpfs_cache_status_query();
----

# Read again to test cached read with large file
query IIII nosort
SELECT COUNT(*), SUM(id), SUM(value), COUNT(DISTINCT category) 
FROM read_parquet('s3://test-bucket/test_snappy_large.parquet');
----
50000000	1249999975000000	3124999937500000.0	200

# Verify cache size is still non-zero (should be using cached data)
query I
SELECT cache_httpfs_get_ondisk_data_cache_size() > 0;
----
1

# Inspect cache directory contents again after second read
# Show all cache files (no expected results - just display what's cached)
query T
SELECT file FROM glob('__TEST_DIR__/cache/*') ORDER BY file;
----

# Show cache status again
query IIIII
SELECT * FROM cache_httpfs_cache_status_query();
----

# Test with filters on cached file
query I
SELECT COUNT(*) FROM read_parquet('s3://test-bucket/test_snappy_large.parquet') WHERE category = 42;
----
250000

# Final inspection of cache directory contents
# Show all cache files (no expected results - just display what's cached)
query T
SELECT file FROM glob('__TEST_DIR__/cache/*') ORDER BY file;
----

# Show final cache status
query IIIII
SELECT * FROM cache_httpfs_cache_status_query();
----

# Show total cache size in bytes
query I
SELECT cache_httpfs_get_ondisk_data_cache_size();
----

# Cleanup
statement ok
DROP TABLE large_test_data;

Some of the cache files output:

duckdb_unittest_tempdir/32899/cache/7171658db5998067b62d9160b7f43d3607e0ba42390444eb149c66a089eb78ed-test_snappy_large.parquet-4697620480-1048576
duckdb_unittest_tempdir/32899/cache/7171658db5998067b62d9160b7f43d3607e0ba42390444eb149c66a089eb78ed-test_snappy_large.parquet-4698669056-1048576

Object stat:

[~/duck-read-cache-fs] (main) 
vscode@9a5948bd4884$ mc alias list
local    
  URL       : http://minio:9000
  AccessKey : minioadmin
  SecretKey : minioadmin
  API       : s3v4
  Path      : auto
  Src       : /home/vscode/.mc/config.json

[~/duck-read-cache-fs] (main) 
vscode@9a5948bd4884$ mc stat local/test-bucket/test_snappy_large.parquet
Name      : test_snappy_large.parquet
Date      : 2025-12-14 10:35:17 UTC 
Size      : 4.5 GiB 
ETag      : b3d1d7ba36953fcca0ce374f9d8685e9-61 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

[dentiny]

I'm wondering if there's any chance you could provide more stacktrace?
Or try some other ways and some other followup questions:

• Does it work without the extension?
• Does it work with in-memory cache? Ref:

  duck-read-cache-fs/src/cache_httpfs_extension.cpp

  Lines 535 to 539 in 5f1259c

  	config.AddExtensionOption("cache_httpfs_type",
  	"Type for cached filesystem. Currently there're two types available, one is `in_mem`, "
  	"another is `on_disk`. By default we use on-disk cache. Set to `noop` to disable, which "
  	"behaves exactly same as httpfs extension.",
  	LogicalType {LogicalTypeId::VARCHAR}, *ON_DISK_CACHE_TYPE, UpdateCacheType);

• Do you have concurrent read and write operations?

[2jungkook]

I can reproduce this even with a much smaller data set: TPC-H SF10 customer.parquet (~120MB) generated by tpch extension

Hardware

Model Name: MacBook Pro
Chip: Apple M4 Pro
Total Number of Cores: 14 (10 performance and 4 efficiency)
Memory: 48 GB

DuckDB setting and query

-- DuckDB v1.4.3 (Andium) d1dc88f950
-- no other concurrent read or write operations

set threads to 100;
INSTALL cache_httpfs FROM community;
LOAD cache_httpfs;
select cache_httpfs_clear_cache();
explain analyze select * from read_parquet('s3://.../tpch-sf10.db/customer.parquet');

Observed errors (randomly one of the following):

Snappy decompression failure
TProtocolException: Invalid data
runtime error: cast to virtual base of address ... which does not point to an object of type 'duckdb_parquet::PageHeader' (with UBSan enabled)

Some observations:

• Only happens with cache_httpfs extension loaded
• Does NOT happen when prefetch is disabled
• High thread count (e.g., SET threads TO 100) increases reproduction rate
• Adding a mutex around the Read call (https://github.com/dentiny/duck-read-cache-fs/blob/main/src/disk_cache_reader.cpp#L429-L433) in disk_cache_reader.cpp resolves the issue:

Question

Is sharing internal_file_handle across multiple threads in io_threads.Push intended to be thread-safe?

[dentiny]

Adding a mutex around the Read call (https://github.com/dentiny/duck-read-cache-fs/blob/main/src/disk_cache_reader.cpp#L429-L433) in disk_cache_reader.cpp resolves the issue:

Interesting, I already specify PARALLEL_ACCESS open option:

duck-read-cache-fs/src/cache_filesystem.cpp

Line 364 in 5f1259c

file_handle = internal_filesystem->OpenFile(path, flags | FileOpenFlags::FILE_FLAGS_PARALLEL_ACCESS, opener);

Thanks for the instruction! I will take another look.

[dentiny]

@2jungkook Thanks for the followup!

I just tried to repro with your instruction, but unfortunately still cannot reproduce. Here's my sql

# name: test/sql/test_s3_simple.test
# description: Simple test to read from local DB, write to S3, then read back
# group: [sql]

require cache_httpfs

require parquet

set ignore_error_messages

# Configure S3/MinIO credentials for the devcontainer MinIO instance
statement ok
SET s3_endpoint='minio:9000';

statement ok
SET s3_access_key_id='minioadmin';

statement ok
SET s3_secret_access_key='minioadmin';

statement ok
SET s3_region='us-east-1';

statement ok
SET s3_use_ssl=false;

statement ok
SET s3_url_style='path';

# I download the database to local filesystem first
statement ok
ATTACH DATABASE '/home/vscode/tpch-sf10.db.1' AS tpch_db (READ_ONLY);

# Check what tables exist in the attached database
query T
SHOW TABLES FROM tpch_db;
----
customer
lineitem
nation
orders
part
partsupp
region
supplier

# Read all rows from lineitem table and write to S3 as parquet
# Adjust table name based on what's in your database
statement ok
COPY (SELECT * FROM tpch_db.lineitem LIMIT 1000) TO 's3://test-bucket/lineitem_sample.parquet' (FORMAT PARQUET);

# Verify we can read it back from S3
query I
SELECT COUNT(*) FROM read_parquet('s3://test-bucket/lineitem_sample.parquet');
----
1000

# Read a few sample rows to verify data integrity
query IIII
SELECT l_orderkey, l_partkey, l_quantity, l_extendedprice 
FROM read_parquet('s3://test-bucket/lineitem_sample.parquet') 
LIMIT 5;
----
1   1551894 17.00   33078.94
1   673091  36.00   38306.16
1   636998  8.00    15479.68
1   21315   28.00   34616.68
1   240267  24.00   28974.00

[dentiny]

Hi both, if you don't mind you could reach me at here, we could schedule a quick meeting and check on the problem together.

[2jungkook]

@dentiny I think I found the root cause. In https://github.com/dentiny/duck-read-cache-fs/blob/main/src/disk_cache_reader.cpp#L344-L452, when ThreadPool::Push() throws an exception (e.g., due to file descriptor exhaustion), the exception remains in the returned future object. Since we never consume these futures, the main thread is unaware of the failures in worker threads. This causes the request to continue processing and eventually reach the snappy decompression failure.

How to reproduce:
In my case, I encountered this due to file descriptor exhaustion. You can likely reproduce it with:

# In your local machine:
ulimit -n 256

# Inside DuckDB:
set threads to 100;
SET cache_httpfs_cache_block_size=65536; 
EXPLAIN ANALYZE SELECT * FROM read_parquet('s3://mybucket/customer.parquet'); 
-- Note: the parquet file size should be > 100MB

[dentiny]

Hi @2jungkook thank you so much for the deep investigation! I think it completely makes sense to me.
I put a fix PR here: #363, and plan to make a release soon.
Let me know if you have any questions later on, feel free to contact any time!