Repository navigation
Size columnar text buffers for multi-byte characters - #409
Merged
Merged
Conversation
Columnar fetches bound each text column with a buffer of the driver's column
size + 1 bytes. Column sizes count characters, so a value with multi-byte
UTF-8 characters was truncated, and its reported full length was then read
from the buffer: the value picked up the next rows' bytes, and the last rows
read past the end of the buffer ('ééééé' in a VARCHAR(5) came back as
"éééabcd"). Text buffers now hold 4 bytes per character, and a value whose
length still exceeds its slot raises an error instead of being read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #409 +/- ##
==========================================
+ Coverage 83.19% 83.41% +0.22%
==========================================
Files 7 7
Lines 970 977 +7
==========================================
+ Hits 807 815 +8
+ Misses 163 162 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
quinnj
force-pushed
the
claude/columnar-text-buffers
branch
from
October 22, 2026 15:56
03d427f to
c39bae3
Compare
quinnj
referenced
this pull request
Oct 27, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default (columnar) fetch path binds each text column with a buffer of the driver's column size + 1 bytes per row. Column sizes for character types count characters, so values with multi-byte UTF-8 characters were truncated, and the loop then read each value's reported full length from the shared buffer. The value picked up bytes from the following rows, and the last rows read past the end of the buffer:
iterate_rows=truealready returned correct values, sincegetdatahandles truncation.SQL_C_CHARcolumn buffers (other than DECIMAL/NUMERIC) now hold 4 bytes per character plus the terminator, the UTF-8 maximum.SQL_NO_TOTAL) now raises an error naming the column and suggestingiterate_rows=true, instead of reading neighbouring memory.Cost: columnar text buffers are 4x larger. 100k rows of
VARCHAR(100)allocate 58.1 MB instead of 29.5 MB; fetch time is unchanged (0.079 s vs 0.081 s). Fetching in fixed-size batches would bound this if it matters.Verification: new test with Latin, CJK, and emoji values in a
VARCHAR(5)column in both fetch modes (fails on main with the output above, passes here); fullPkg.test()passes locally against MariaDB (latest, utf8mb4, as in CI) with the MariaDB_Connector_ODBC_jll driver.🤖 Generated with Claude Code