Heads-up: release-mode crash when co-linked with vapor/postgres-nio
Filing as a courtesy / FYI — this looks like a Swift release-mode codegen issue, not a defect in swift-transformers. Upstream tracker: swiftlang/swift#88794.
Symptom
A Swift release build that links both huggingface/swift-transformers (Tokenizers product, 1.3.0) and vapor/postgres-nio (1.33.0) into the same executable crashes with freed pointer was not the last allocation on the second consecutive PostgresClient.query(...) call. The first query succeeds; the second aborts the process.
Either dependency on its own works fine in release. A swift build -c debug build with both deps also works fine.
The trigger is import Tokenizers itself — no Tokenizers API has to be called for the crash to fire. That's what points the finger at Tokenizers-side codegen / module-init being part of the interaction (alongside NIO's allocator paths).
Environment
- Swift 6.3.1 (
swiftlang-6.3.1.1.2) / Xcode 26.4.1 / macOS 26.4.1, Apple M4
- swift-transformers 1.3.0
- postgres-nio 1.33.0
- swift-nio 2.99.0
Minimal repro
The full minimal Package.swift + main.swift (≈30 lines) is in swiftlang/swift#88794. Two SELECT N queries against the bare PostgresClient are enough to reproduce, but only when import Tokenizers is also in the executable.
Why I'm posting here
Pure FYI in case:
- There's a known interaction between Tokenizers' module-init path (Hub/Tokenizers globals, lazy resource lookup, etc.) and Swift's allocator that the swift-transformers maintainers have context on, or
- you want this on your radar in case other users hit it.
I'm not asking swift-transformers to fix this; the diagnosis points at the Swift toolchain. Feel free to close after triage. If a workaround on the swift-transformers side is feasible (e.g. avoiding whatever module-init path triggers it in release mode), that'd be a nice-to-have, but a Swift compiler patch is the right durable fix.
Workarounds for users hitting this today
- Build the executable with
swift build -c debug when both libraries are linked.
- Avoid linking
Tokenizers and PostgresNIO into the same executable target (run tokenization in a separate process, or do tokenization Python-side and ship token IDs to the Swift side).
Heads-up: release-mode crash when co-linked with vapor/postgres-nio
Filing as a courtesy / FYI — this looks like a Swift release-mode codegen issue, not a defect in swift-transformers. Upstream tracker: swiftlang/swift#88794.
Symptom
A Swift release build that links both
huggingface/swift-transformers(Tokenizersproduct, 1.3.0) andvapor/postgres-nio(1.33.0) into the same executable crashes withfreed pointer was not the last allocationon the second consecutivePostgresClient.query(...)call. The first query succeeds; the second aborts the process.Either dependency on its own works fine in release. A
swift build -c debugbuild with both deps also works fine.The trigger is
import Tokenizersitself — no Tokenizers API has to be called for the crash to fire. That's what points the finger atTokenizers-side codegen / module-init being part of the interaction (alongside NIO's allocator paths).Environment
swiftlang-6.3.1.1.2) / Xcode 26.4.1 / macOS 26.4.1, Apple M4Minimal repro
The full minimal Package.swift +
main.swift(≈30 lines) is in swiftlang/swift#88794. TwoSELECT Nqueries against the barePostgresClientare enough to reproduce, but only whenimport Tokenizersis also in the executable.Why I'm posting here
Pure FYI in case:
I'm not asking swift-transformers to fix this; the diagnosis points at the Swift toolchain. Feel free to close after triage. If a workaround on the swift-transformers side is feasible (e.g. avoiding whatever module-init path triggers it in release mode), that'd be a nice-to-have, but a Swift compiler patch is the right durable fix.
Workarounds for users hitting this today
swift build -c debugwhen both libraries are linked.TokenizersandPostgresNIOinto the same executable target (run tokenization in a separate process, or do tokenization Python-side and ship token IDs to the Swift side).