Allow r2:// URL scheme#365
Draft
timmclaughlin wants to merge 1 commit into
Draft
Conversation
The DuckDB S3FileSystem natively recognizes r2:// alongside s3://, s3a://,
s3n://, gcs://, and gs:// (see duckdb-httpfs s3fs.cpp::CanHandleFile), and
duckdb_pglake's RegionAwareS3FileSystem already short-circuits all non-
s3:// URLs straight to the underlying S3FileSystem (region auto-detection
is intentionally s3:// only — R2 always uses "auto"). The only thing
gating r2:// today is pg_lake's PG-side URL allowlist.
Add r2:// to both URL allowlists:
* pg_lake_engine IsSupportedURL — the project-wide check used by
pg_lake_copy (CREATE TABLE ... load_from/definition_from),
pg_lake_iceberg (lake_iceberg.metadata/files/snapshots, the
pg_lake_iceberg.default_location_prefix GUC),
pg_lake_table (file_size/file_exists/file_preview/delete_file,
foreign-table "path" and "location" options, CREATE TABLE
iceberg/pg_lake_table), and pg_lake_benchmark.
* pg_lake_table IsFileListSupportedUrl — the tighter list-only
check used by lake_file.list (excludes http/https since those
can't enumerate directories).
Refresh the now-stale "only s3://, gs://, az://, azure://, and abfss://"
error strings across the codebase so they accurately reflect what
IsSupportedURL accepts (this also picks up the missing hf:// that has
been accepted since HuggingFace support landed). Update the four
pytest assertions that match those strings exactly.
Signed-off-by: Tim McLaughlin <tim@gotab.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Allow Cloudflare R2 (`r2://`) URLs through pg_lake's PG-side URL allowlists.
Why this works
The DuckDB `S3FileSystem` natively recognizes `r2://` alongside `s3://`, `s3a://`, `s3n://`, `gcs://`, and `gs://` — see `duckdb-httpfs/src/s3fs.cpp` `S3FileSystem::CanHandleFile`. `duckdb_pglake`'s `RegionAwareS3FileSystem` already short-circuits all non-`s3://` URLs straight to the underlying `S3FileSystem` (region auto-detection is intentionally `s3://` only — R2 always uses `auto`), so no `duckdb_pglake` change is needed. The only thing gating `r2://` today is pg_lake's PG-side URL allowlist.
What this PR changes
Add `r2://` to both URL allowlists:
Refresh the now-stale `"only s3://, gs://, az://, azure://, and abfss://"` error strings across the codebase so they accurately reflect what `IsSupportedURL` accepts. This also picks up the previously-missing `hf://` — HuggingFace URLs have been accepted by `IsSupportedURL` for a while but the error strings were never updated. Updated the four pytest assertions that match those strings exactly.
Repro of the original failure
```sql
SELECT * FROM lake_file.list('r2://my-bucket/prefix/*');
-- ERROR: list_files: only s3://, gs://, az://, azure://, abfss://, and hf:// urls are supported
```
After this PR the same call falls through to `pgduck_server` / DuckDB which handles `r2://` via the standard R2 secret mechanism.
Checklist
Notes for reviewers