Bump duckdb-sources to vendored-vss commit (fix Windows build)#39
Merged
Conversation
Point the duckdb-sources submodule at spiceai/duckdb#17 (01dbffdd), which vendors the vss extension as committed files (extension/vss/src) instead of a git submodule. Cargo recursively checks out submodules of the duckdb-rs git dependency, so the vss submodule pulled in duckdb-vss's own nested `duckdb` checkout, whose deep Swift example path exceeded Windows' MAX_PATH and broke the Windows build. A vendored copy has nothing for Cargo to recurse into. Regenerated duckdb.tar.gz (vss sources now under extension/vss/src; DUCKDB_VERSION still v1.5.3; vss still statically linked). Updated SKILL.md for the vendored layout.
There was a problem hiding this comment.
Pull request overview
This PR updates the bundled DuckDB source input used by libduckdb-sys --features bundled by bumping the duckdb-sources submodule and regenerating the checked-in source tarball so the VSS extension is vendored as committed files (not a git submodule), avoiding Windows MAX_PATH build failures caused by recursive submodule checkouts.
Changes:
- Bump
duckdb-sourcesto a commit where VSS is vendored underextension/vss/src(no nested submodules). - Regenerate the bundled
duckdb.tar.gz(and embedded manifest) to reflect the new VSS source layout. - Update the maintenance guide (
SKILL.md) to reflect the new vendored VSS workflow and gotchas.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| crates/libduckdb-sys/SKILL.md | Updates the bundled VSS maintenance instructions for the new vendored extension/vss/src layout. |
| crates/libduckdb-sys/duckdb.tar.gz | Regenerated bundled DuckDB source archive used for bundled builds (includes VSS vendored sources). |
| crates/libduckdb-sys/duckdb-sources | Submodule pointer bump to the updated DuckDB fork commit used as regeneration input. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
29
to
+33
| after re-running `update_sources.py` and committing the regenerated `duckdb.tar.gz` + `manifest.json`.** | ||
|
|
||
| ## VSS wiring (what makes vss statically linked) | ||
|
|
||
| 1. `duckdb-sources/extension/vss` (in the C++ fork) vendors the VSS source — see | ||
| `duckdb-sources/extension/vss/SKILL.md` for that side (submodule pin, `vss_config.py`). | ||
| 1. `duckdb-sources/extension/vss` (in the C++ fork) vendors the VSS source as committed files |
Comment on lines
+51
to
55
| # Populate the duckdb-sources submodule. vss is vendored files (extension/vss/src), NOT a | ||
| # submodule, so no nested/recursive init is needed (and must not be re-introduced — see Gotchas). | ||
| git submodule update --init crates/libduckdb-sys/duckdb-sources | ||
| git -C crates/libduckdb-sys/duckdb-sources submodule update --init extension/vss/upstream | ||
|
|
||
| python3 crates/libduckdb-sys/update_sources.py # regenerates duckdb.tar.gz + manifest.json |
phillipleblanc
added a commit
to datafusion-contrib/datafusion-table-providers
that referenced
this pull request
Jun 3, 2026
…ild) (#660) Updates the duckdb-rs pin from b1cf1723 to 7648ff8c — spiceai/duckdb-rs#39, which vendors the vss extension as committed files instead of a git submodule. Cargo recursively checks out submodules of this git dependency, and the previous vss submodule pulled in duckdb-vss's nested duckdb checkout, whose deep path exceeded Windows' MAX_PATH and broke the Windows build. No DuckDB Rust API change; lock updated for the new rev only.
phillipleblanc
added a commit
to spiceai/spiceai
that referenced
this pull request
Jun 3, 2026
Bump duckdb-rs b1cf1723 -> 7648ff8c (spiceai/duckdb-rs#39) and datafusion-table-providers be0837e3 -> 4af85422 (datafusion-contrib/datafusion-table-providers#660). Those vendor the vss extension as committed files instead of a git submodule. The vss submodule dragged in duckdb-vss's own nested `duckdb` submodule via Cargo's recursive submodule checkout of the duckdb-rs git dependency, and that nested checkout's deep Swift example path exceeded Windows' MAX_PATH, breaking the Windows build. Vendored sources have no submodules to recurse into. No functional change to the bundled DuckDB (same v1.5.3 + statically-linked vss).
phillipleblanc
added a commit
to spiceai/spiceai
that referenced
this pull request
Jun 3, 2026
…mongodb API Bump duckdb-rs b1cf1723 -> 7648ff8c (spiceai/duckdb-rs#39) and datafusion-table-providers be0837e3 -> 4af85422 (datafusion-contrib/datafusion-table-providers#660). Those vendor the vss extension as committed files instead of a git submodule, fixing the Windows MAX_PATH break (Cargo recursively checked out the vss submodule's nested duckdb, whose deep Swift path exceeded Windows' MAX_PATH). The table-providers bump to the current spiceai-52 head also brings in #657 (mongodb schema merging), which added an optional declared-schema arg to MongoDBTableFactory::table_provider. Adapt crates/data_components/src/mongodb.rs to pass None (no declared schema at that layer). No functional change to the bundled DuckDB (same v1.5.3 + statically-linked vss).
phillipleblanc
added a commit
to spiceai/spiceai
that referenced
this pull request
Jun 3, 2026
…mongodb API (#11140) Bump duckdb-rs b1cf1723 -> 7648ff8c (spiceai/duckdb-rs#39) and datafusion-table-providers be0837e3 -> 4af85422 (datafusion-contrib/datafusion-table-providers#660). Those vendor the vss extension as committed files instead of a git submodule, fixing the Windows MAX_PATH break (Cargo recursively checked out the vss submodule's nested duckdb, whose deep Swift path exceeded Windows' MAX_PATH). The table-providers bump to the current spiceai-52 head also brings in #657 (mongodb schema merging), which added an optional declared-schema arg to MongoDBTableFactory::table_provider. Adapt crates/data_components/src/mongodb.rs to pass None (no declared schema at that layer). No functional change to the bundled DuckDB (same v1.5.3 + statically-linked vss).
phillipleblanc
added a commit
to spiceai/spiceai
that referenced
this pull request
Jun 3, 2026
…mongodb API (#11140) Bump duckdb-rs b1cf1723 -> 7648ff8c (spiceai/duckdb-rs#39) and datafusion-table-providers be0837e3 -> 4af85422 (datafusion-contrib/datafusion-table-providers#660). Those vendor the vss extension as committed files instead of a git submodule, fixing the Windows MAX_PATH break (Cargo recursively checked out the vss submodule's nested duckdb, whose deep Swift path exceeded Windows' MAX_PATH). The table-providers bump to the current spiceai-52 head also brings in #657 (mongodb schema merging), which added an optional declared-schema arg to MongoDBTableFactory::table_provider. Adapt crates/data_components/src/mongodb.rs to pass None (no declared schema at that layer). No functional change to the bundled DuckDB (same v1.5.3 + statically-linked vss).
github-actions Bot
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Jun 6, 2026
…mongodb API (#11140) Bump duckdb-rs b1cf1723 -> 7648ff8c (spiceai/duckdb-rs#39) and datafusion-table-providers be0837e3 -> 4af85422 (datafusion-contrib/datafusion-table-providers#660). Those vendor the vss extension as committed files instead of a git submodule, fixing the Windows MAX_PATH break (Cargo recursively checked out the vss submodule's nested duckdb, whose deep Swift path exceeded Windows' MAX_PATH). The table-providers bump to the current spiceai-52 head also brings in #657 (mongodb schema merging), which added an optional declared-schema arg to MongoDBTableFactory::table_provider. Adapt crates/data_components/src/mongodb.rs to pass None (no declared schema at that layer). No functional change to the bundled DuckDB (same v1.5.3 + statically-linked vss).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps the
duckdb-sourcessubmodule to spiceai/duckdb#17 (01dbffdd), which vendors the VSS extension as committed files (extension/vss/src) instead of a git submodule. Fixes the Windows build break.Why
Cargo recursively checks out all submodules of the duckdb-rs git dependency. The previous vss submodule (
extension/vss/upstream → duckdb-vss) dragged in duckdb-vss's own nestedduckdbsubmodule, whose deeply-nested Swift example path exceeds Windows'MAX_PATH:The vendored copy has no submodules to recurse into.
Change
duckdb-sources:6cdcae79→01dbffdd(Fix writing Arrow Decimal128 with zero scale #17 — vss vendored as files).duckdb.tar.gz+manifest.json: vss sources now underextension/vss/src(wasextension/vss/upstream/src); 12 cpp + header-only usearch/fp16/simsimd;DUCKDB_VERSIONstillv1.5.3; vss still statically linked.SKILL.md: updated for the vendored layout (no submodule init; "vss must stay vendored" gotcha with the Windows reason).Validation
update_sources.pyregenerates cleanly;manifest.jsonincludesvss(12 cpp). Build/symbol verification carries over from #37/#38 (source bytes unchanged — only the in-tree path differs). The real cross-check is the downstream spiceai Windows CI run after the re-chain.