Skip to content

fix: scan Enum8/Enum16 columns into integer destinations - #1921

Open
polyglotAI-bot wants to merge 2 commits into
mainfrom
polyglot/enum-scan-int-destinations
Open

fix: scan Enum8/Enum16 columns into integer destinations#1921
polyglotAI-bot wants to merge 2 commits into
mainfrom
polyglot/enum-scan-int-destinations

Conversation

@polyglotAI-bot

Copy link
Copy Markdown
Collaborator

Description

Fixes #1918.

Enum8.ScanRow / Enum16.ScanRow only accepted *string / **string
destinations (plus sql.Scanner). Any integer destination fell through to a
ColumnConverterError, so reading an enum column into its underlying numeric
ordinal was impossible — even though the write side has accepted integer
values since #802 ("Support inserting Enums as int8/int16/int values"). This
restores read/write symmetry.

proto.Enum8 is int8-based and proto.Enum16 is int16-based, so the raw
value returned by col.col.Row(row) is the ordinal. The fix adds integer
destination cases that return it.

Changes

  • lib/column/enum8.goScanRow now handles *int8, *int16, *int32,
    *int64, *int (and the ** pointer-to-pointer variants). All are lossless
    widenings of the int8 base.
  • lib/column/enum16.goScanRow now handles *int16, *int32, *int64,
    *int (and ** variants). int8 is intentionally not offered for
    Enum16
    : an Enum16 ordinal can exceed the int8 range (e.g. 1000) and
    would silently truncate, so an int8 destination for Enum16 stays an
    explicit ColumnConverterError rather than corrupting data (consistent with
    the "hard to use incorrectly" design principle). Only signed integer types are
    added, matching the signed enum ordinals and the write side.
  • Nullable(Enum) inherits the fix automatically — Nullable.ScanRow
    delegates element scanning to the base Enum column's ScanRow.

Test

tests/issues/1918_test.go (native round-trip against a real server):

  • Enum8 scanned into every signed width (int8/int16/int32/int64/int).
  • Enum16 scanned into int16/int32/int64/int, using ordinal 1000 (outside
    int8 range) to prove the wider destinations are not truncated.
  • Negative ordinals (-5, -300) preserve their sign.
  • Pointer-to-pointer destinations (**int8 etc.).
  • Contrast/regression guards: string destinations still work unchanged, and
    Enum16 -> int8 remains an error (no silent truncation).
  • Nullable(Enum) into integer destinations (value scanned; NULL leaves a
    pointer destination nil).

Each integer subtest fails on main with
clickhouse [ScanRow]: converting Enum8 to *int8 is unsupported and passes
with this change; the surrounding existing enum unit and integration tests
continue to pass unchanged.

Known limitation / follow-up (out of scope here)

Array(Enum)[]int and Map(String, Enum)map[string]int are not
addressed by this PR: they use a different, reflection-based element-scan path
(Array.scanSlicesetJSONFieldValue, and Map.ScanRow's whole-type match),
not Enum.ScanRow. Neither has ever worked, neither is a regression, and fixing
them touches shared code used by all container element types — so it belongs in a
separate change (and, for Map, may be intentional). Kept this PR to one focused
fix.

Pre-PR validation gate

  • Deterministic repro confirmed (integer subtests fail on main, pass here)
  • Root cause documented above
  • Fix targets the root cause (the ScanRow type switch)
  • Test fails without fix, passes with fix
  • No existing tests broken (enum unit + integration suites pass)
  • Convention compliance verified per AGENTS.md (regression test in
    tests/issues/, pointer receivers, t.Cleanup for connections)
  • No public API break; CHANGELOG is release-time generated from PR labels

Enum8/Enum16 ScanRow only accepted *string/**string destinations, so
scanning an enum column into an integer (its underlying ordinal) returned
a ColumnConverterError, even though the write side already accepts
int8/int16/int values (PR #802). Add integer destination cases that
return the numeric ordinal:

  Enum8  -> *int8/*int16/*int32/*int64/*int (+ **... variants)
  Enum16 -> *int16/*int32/*int64/*int       (+ **... variants)

int8 is intentionally not offered for Enum16 because Enum16 ordinals can
exceed the int8 range and would silently truncate; that destination
remains an explicit ColumnConverterError. Nullable(Enum) inherits the fix
via Nullable.ScanRow delegation.

Fixes: #1918

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes enum read/write symmetry in the native driver by allowing Enum8/Enum16 columns to be scanned into signed integer destinations (returning the underlying ordinal), matching existing insert-side support and addressing #1918.

Changes:

  • Extend Enum8.ScanRow to support *int8/*int16/*int32/*int64/*int (and ** variants).
  • Extend Enum16.ScanRow to support *int16/*int32/*int64/*int (and ** variants), intentionally excluding int8 to avoid truncation.
  • Add an integration regression test covering integer scans, negative ordinals, pointer-to-pointer destinations, and nullable enums.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
lib/column/enum8.go Adds integer destination handling in ScanRow for Enum8 ordinals.
lib/column/enum16.go Adds integer destination handling in ScanRow for Enum16 ordinals (excluding int8).
tests/issues/1918_test.go Adds regression coverage for scanning enums into signed integer destinations (incl. nullable + ** pointers).
Comments suppressed due to low confidence (1)

tests/issues/1918_test.go:135

  • The nullable-table subtest has the same idempotency issue as the outer table: CREATE can fail if test_1918_nullable already exists, and cleanup is only registered after a successful CREATE. Dropping first and using t.Cleanup prevents flakiness on reruns.
		require.NoError(t, conn.Exec(ctx, `
			CREATE TABLE test_1918_nullable (
				  n8  Nullable(Enum8 ('a' = -5, 'c' = 42))
				, n16 Nullable(Enum16('z' = 1000))
			) Engine MergeTree() ORDER BY tuple()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/issues/1918_test.go Outdated
Comment thread lib/column/enum8.go
Comment thread tests/issues/1918_test.go Outdated
Comment thread lib/column/enum16.go
Comment thread tests/issues/1918_test.go Outdated
@github-actions

Copy link
Copy Markdown

🤖 Claude review

Adds integer scan destinations to Enum8.ScanRow/Enum16.ScanRow so an enum column can be read back as its numeric ordinal, matching the write side. The direction is right and the deliberate Enum16int8 refusal is a good call, but the new *int/**int destinations break Nullable(Enum) NULL handling.

Key concerns:

  • Nullable.ScanRow has no **int case, so a NULL scanned into a *int silently leaves the previous row's value in the destination instead of nil — no error (see inline on lib/column/enum8.go).
  • The Nullable(Enum) subtest's assert.Nil(t, p8) is vacuous: p8 is never written before the assertion, so it passes whether or not the destination is cleared. That is why the **int hole slipped through.
  • *int/**int are also a departure from the rest of lib/columnInt64.ScanRow accepts only *int64/**int64, no widenings, and no column in the package accepts *int. Either add **int to nullable.go or drop the two int cases.

Surface coverage: the fix lands on the native driver.Conn path only. database/sql fills destinations from Column.Row(i, ptr) (clickhouse_std.go:432), which still returns the enum name, so sql.Rows.Scan(&myInt) keeps failing in convertAssign. The claimed read/write symmetry therefore holds for Open but not OpenDB.

Blind spots: no live ClickHouse here, so these come from tracing the scan path rather than running tests/issues/1918_test.go. LowCardinality(Enum8) delegates via index.ScanRow and should inherit the fix, but nothing covers it. Deferring Array(Enum)/Map(Enum) is reasonable and genuinely out of scope.

Verdict: ⚠️ Request changes

General findings

  • ⚠️ Should fixdatabase/sql surface still cannot scan an enum into an integer
    The PR frames the change as restoring read/write symmetry, but that only holds on the native API. stdRows.Next (clickhouse_std.go:432) fills driver.Value from Column.Row(i, ptr), not ScanRow, and Enum8.Row/Enum16.Row return col.vi[...] — the enum name. So db.QueryRow(...).Scan(&myInt) still fails inside database/sql's convertAssign with a strconv error, even though the std write path accepts integers.

    Per the impacted-surface gate this needs one of:

    • coverage proving the std behaviour is what you intend (a subtest asserting the current error, using TestDatabaseSQLClientWithDefaultSettings), or
    • an explicit note in the PR description / release notes that the fix is native-only.

    Separately, Test1918 uses TestClientWithDefaultSettings, which is useHTTP=false — native TCP only. ScanRow is protocol-independent so the HTTP risk is low, but no run of this test exercises the HTTP path.

Inline comments are attached to the relevant lines. This summary updates in place on re-review.

…hint Enum16->int8

- Nullable.ScanRow: add **int to the NULL clear-list. The PR added *int/**int
  as Enum scan destinations, but Nullable clears a NULL destination via an
  explicit **T list that omitted **int, so a NULL scanned into a reused *int
  left a stale non-nil pointer. All other widths added (**int8/16/32/64) were
  already in the list.
- Enum16.ScanRow: reject *int8/**int8 explicitly with an actionable Hint
  ("Enum16 values may exceed the int8 range; use *int16 or wider"), matching the
  generated numeric columns' convention.
- tests/issues/1918_test.go: DROP TABLE IF EXISTS before CREATE + register
  cleanup up-front (idempotent reruns); seed pointer destinations non-nil before
  the NULL scan so a missing nil-out is detectable, and add a *int NULL case
  (regression pin for the Nullable fix); assert the specific Enum16->int8
  rejection message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[backfill: ClickHouse/clickhouse-go] Enum columns cannot be scanned into integer destinations

3 participants