Partial parquet reads

[pgarrison]

Context

The initial load of test file sample_file_large.parquet from S3 previously took 90s (AICS VPN) and downloaded the full 1GB of data.

Now, loading this test file takes 10-12s and loads just 100MB (the first row group).

Resolves #694.

Changes

• Enable partial reads of parquet files by setting forceFullHTTPReads: false
• Use parquet table statistics to replace the following full-table scan triggered by checkDataSourceForErrors when preparing a data source. See the detailed implementation notes in the source code comments.

SELECT A.row
FROM (
    SELECT ROW_NUMBER() OVER () AS row, "${column}"
    FROM "${dataSource}"
) AS A
WHERE TRIM(A."${column}") IS NULL

Testing

So far my manual testing has only been in the browser, with the following test files:

• sample_file_large.parquet, reduced load time from ~90s to ~11s
• cpg0001-cellpainting-protocol_draft.csv.parquet, reduced load time from ~10s to ~2s

Reviewing

For notes on follow-up performance improvements (could get load times down to 2s for sample_file_large), see my comment on the parent issue.

[pgarrison]

This db.open call is the only change from the code this replaces from DatabaseServiceWeb and DatabaseServiceElectron.

[BrianWhitneyAI]

Whats the significance of wrapping errors here? will this suppress the type?

[pgarrison]

You mean, why return [error] instead of error? I just kept that to be consistent with the previous code. The return type remains Promise<string[]>.

I replaced errors.push with return statements to have simple logic for exiting early if we get a useful result from parquetHasBlankValues.

[SeanDuHare]

I think the original returned multiple errors in some cases - looks like that isn't true anymore

[SeanDuHare]

Is this quick? Should we cache it?

[pgarrison]

I don't think it makes sense to cache this result. The second time this query runs, it is very fast (under 10ms), which I believe is due to DuckDB's cache for portions of the parquet file it has already read.