Skip to content

Size columnar text buffers for multi-byte characters - #409

Merged
quinnj merged 2 commits into
mainfrom
claude/columnar-text-buffers
Oct 27, 2026
Merged

quinnj merged 2 commits into
mainfrom
claude/columnar-text-buffers

Conversation

@quinnj

@quinnj quinnj commented Oct 22, 2026

Copy link
Copy Markdown
Member

The default (columnar) fetch path binds each text column with a buffer of the driver's column size + 1 bytes per row. Column sizes for character types count characters, so values with multi-byte UTF-8 characters were truncated, and the loop then read each value's reported full length from the shared buffer. The value picked up bytes from the following rows, and the last rows read past the end of the buffer:

# MariaDB Connector/ODBC, VARCHAR(5) CHARACTER SET utf8mb4
DBInterface.execute(conn, "SELECT s FROM utf8_check ORDER BY id") |> columntable
# s = ["éééabcd", "abcde", "日本😀\xf0\x9f\0\0\0", "😀\xf0\x9f\0\0\0\0\0\0x\0\0\0\0\0\0\0", missing, "x"]
# expected ["ééééé", "abcde", "日本語テキ", "😀😀😀😀😀", missing, "x"]

iterate_rows=true already returned correct values, since getdata handles truncation.

  • SQL_C_CHAR column buffers (other than DECIMAL/NUMERIC) now hold 4 bytes per character plus the terminator, the UTF-8 maximum.
  • A value whose reported length still doesn't fit its slot (a driver that under-reports column size, or SQL_NO_TOTAL) now raises an error naming the column and suggesting iterate_rows=true, instead of reading neighbouring memory.

Cost: columnar text buffers are 4x larger. 100k rows of VARCHAR(100) allocate 58.1 MB instead of 29.5 MB; fetch time is unchanged (0.079 s vs 0.081 s). Fetching in fixed-size batches would bound this if it matters.

Verification: new test with Latin, CJK, and emoji values in a VARCHAR(5) column in both fetch modes (fails on main with the output above, passes here); full Pkg.test() passes locally against MariaDB (latest, utf8mb4, as in CI) with the MariaDB_Connector_ODBC_jll driver.

🤖 Generated with Claude Code

Columnar fetches bound each text column with a buffer of the driver's column
size + 1 bytes. Column sizes count characters, so a value with multi-byte
UTF-8 characters was truncated, and its reported full length was then read
from the buffer: the value picked up the next rows' bytes, and the last rows
read past the end of the buffer ('ééééé' in a VARCHAR(5) came back as
"éééabcd"). Text buffers now hold 4 bytes per character, and a value whose
length still exceeds its slot raises an error instead of being read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.41%. Comparing base (965d743) to head (c39bae3).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #409      +/-   ##
==========================================
+ Coverage   83.19%   83.41%   +0.22%     
==========================================
  Files           7        7              
  Lines         970      977       +7     
==========================================
+ Hits          807      815       +8     
+ Misses        163      162       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quinnj
quinnj force-pushed the claude/columnar-text-buffers branch from 03d427f to c39bae3 Compare October 22, 2026 15:56
@quinnj
quinnj merged commit 6ec8e1e into main Oct 27, 2026
7 checks passed
@quinnj
quinnj deleted the claude/columnar-text-buffers branch October 27, 2026 09:06
quinnj referenced this pull request Oct 27, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant