Describe the bug
For an unconstrained Postgres numeric column, the Arrow decimal scale is taken from the first non-null row of the result set and every subsequent value is rescaled down to it. rescale rounds, so values with more decimal places than the first row are silently corrupted.
Columns with a declared scale (numeric(10,3)) are unaffected — they take the schema-driven path in get_decimal_column_precision_and_scale. The problem is specific to unconstrained numeric, which is what max(), min(), avg() and arithmetic over a numeric column return, so aggregates are the common exposure.
To Reproduce
SELECT v FROM (VALUES (1.0::numeric),(2.5::numeric),(3.75::numeric)) t(v)
-- returns 1.0, 2.5, 3.8 ← 3.75 rounded to the first row's 1dp
SELECT v FROM (VALUES (3.75::numeric),(1.0::numeric),(2.5::numeric)) t(v)
-- returns 3.75, 1.0, 2.5 ← same values, reordered, all exact
Worst case, an integer first row gives the column zero decimal places:
SELECT v FROM (VALUES (20::numeric),(17.685::numeric),(15.334::numeric)) t(v)
-- returns 20.0, 18.0, 15.0 ← two values destroyed
A single-row result is always exact, so this survives casual testing.
Expected behavior
Every value returned exactly, independent of row order.
Source
src/sql/arrow_sql_gen/postgres.rs, the Type::NUMERIC arm (0.11.0). The existing comment identifies the cause:
// Record Batch Scale is determined by first row, while Postgres Numeric Type doesn't have fixed scale
// Resolve scale difference for incoming records
let dest_scale = postgres_numeric_scale.unwrap_or_default();
v.rescale(dest_scale); // lossy when dest_scale < v.scale()
dec_builder.append_value(v.mantissa());
postgres_numeric_scale is set once from the first row (if postgres_numeric_scale.is_none()), and Decimal::rescale rounds when narrowing.
Suggested fix
The scale must never narrow below what a value needs. Options:
- Two-pass per batch — scan the batch for
max(scale) and build the Decimal128Builder with that. Correct within a batch; still wrong across batches unless the field is fixed per stream.
- Pin unconstrained
numeric to a fixed generous scale (e.g. the Decimal128(38, N) the builder already uses for precision), so rescale only ever widens. Simple and order-independent; costs nothing for typical values.
- Error rather than round if a value cannot be represented at the batch's scale, so the loss is at least visible.
Option 2 seems the smallest correct change. Whichever is chosen, a regression test needs the first row to be a whole number and later rows to carry more decimals — the bug is invisible if the fixture is ordered the other way.
Impact seen in practice
Found in a BMS deployment reading a 91 GB Timescale table whose value is numeric. A sensor frozen at 15.33 was reported as 15.0, and 17.68 as 18.0; a real temperature series 20.0, 19.82, 19.64, … came back as 20.0, 19.0, 19.0, …. Alerting logic that compares a transported value against a threshold makes wrong decisions, not merely wrong display.
Workaround for callers: cast ::float8 or ::text in the projection.
Describe the bug
For an unconstrained Postgres
numericcolumn, the Arrow decimal scale is taken from the first non-null row of the result set and every subsequent value isrescaled down to it.rescalerounds, so values with more decimal places than the first row are silently corrupted.Columns with a declared scale (
numeric(10,3)) are unaffected — they take the schema-driven path inget_decimal_column_precision_and_scale. The problem is specific to unconstrainednumeric, which is whatmax(),min(),avg()and arithmetic over anumericcolumn return, so aggregates are the common exposure.To Reproduce
Worst case, an integer first row gives the column zero decimal places:
A single-row result is always exact, so this survives casual testing.
Expected behavior
Every value returned exactly, independent of row order.
Source
src/sql/arrow_sql_gen/postgres.rs, theType::NUMERICarm (0.11.0). The existing comment identifies the cause:postgres_numeric_scaleis set once from the first row (if postgres_numeric_scale.is_none()), andDecimal::rescalerounds when narrowing.Suggested fix
The scale must never narrow below what a value needs. Options:
max(scale)and build theDecimal128Builderwith that. Correct within a batch; still wrong across batches unless the field is fixed per stream.numericto a fixed generous scale (e.g. theDecimal128(38, N)the builder already uses for precision), sorescaleonly ever widens. Simple and order-independent; costs nothing for typical values.Option 2 seems the smallest correct change. Whichever is chosen, a regression test needs the first row to be a whole number and later rows to carry more decimals — the bug is invisible if the fixture is ordered the other way.
Impact seen in practice
Found in a BMS deployment reading a 91 GB Timescale table whose
valueisnumeric. A sensor frozen at15.33was reported as15.0, and17.68as18.0; a real temperature series20.0, 19.82, 19.64, …came back as20.0, 19.0, 19.0, …. Alerting logic that compares a transported value against a threshold makes wrong decisions, not merely wrong display.Workaround for callers: cast
::float8or::textin the projection.