Skip to content

Postgres: unconstrained numeric column is rounded to the first row's scale (silent precision loss) #719

Description

@NubeDev

Describe the bug

For an unconstrained Postgres numeric column, the Arrow decimal scale is taken from the first non-null row of the result set and every subsequent value is rescaled down to it. rescale rounds, so values with more decimal places than the first row are silently corrupted.

Columns with a declared scale (numeric(10,3)) are unaffected — they take the schema-driven path in get_decimal_column_precision_and_scale. The problem is specific to unconstrained numeric, which is what max(), min(), avg() and arithmetic over a numeric column return, so aggregates are the common exposure.

To Reproduce

SELECT v FROM (VALUES (1.0::numeric),(2.5::numeric),(3.75::numeric)) t(v)
-- returns 1.0, 2.5, 3.8        ← 3.75 rounded to the first row's 1dp

SELECT v FROM (VALUES (3.75::numeric),(1.0::numeric),(2.5::numeric)) t(v)
-- returns 3.75, 1.0, 2.5       ← same values, reordered, all exact

Worst case, an integer first row gives the column zero decimal places:

SELECT v FROM (VALUES (20::numeric),(17.685::numeric),(15.334::numeric)) t(v)
-- returns 20.0, 18.0, 15.0     ← two values destroyed

A single-row result is always exact, so this survives casual testing.

Expected behavior

Every value returned exactly, independent of row order.

Source

src/sql/arrow_sql_gen/postgres.rs, the Type::NUMERIC arm (0.11.0). The existing comment identifies the cause:

// Record Batch Scale is determined by first row, while Postgres Numeric Type doesn't have fixed scale
// Resolve scale difference for incoming records
let dest_scale = postgres_numeric_scale.unwrap_or_default();
v.rescale(dest_scale);            // lossy when dest_scale < v.scale()
dec_builder.append_value(v.mantissa());

postgres_numeric_scale is set once from the first row (if postgres_numeric_scale.is_none()), and Decimal::rescale rounds when narrowing.

Suggested fix

The scale must never narrow below what a value needs. Options:

  1. Two-pass per batch — scan the batch for max(scale) and build the Decimal128Builder with that. Correct within a batch; still wrong across batches unless the field is fixed per stream.
  2. Pin unconstrained numeric to a fixed generous scale (e.g. the Decimal128(38, N) the builder already uses for precision), so rescale only ever widens. Simple and order-independent; costs nothing for typical values.
  3. Error rather than round if a value cannot be represented at the batch's scale, so the loss is at least visible.

Option 2 seems the smallest correct change. Whichever is chosen, a regression test needs the first row to be a whole number and later rows to carry more decimals — the bug is invisible if the fixture is ordered the other way.

Impact seen in practice

Found in a BMS deployment reading a 91 GB Timescale table whose value is numeric. A sensor frozen at 15.33 was reported as 15.0, and 17.68 as 18.0; a real temperature series 20.0, 19.82, 19.64, … came back as 20.0, 19.0, 19.0, …. Alerting logic that compares a transported value against a threshold makes wrong decisions, not merely wrong display.

Workaround for callers: cast ::float8 or ::text in the projection.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions