This appears to be the same theme as #94, #437, #439, and tbh I'm not entirely certain if this fits better here, in nanoarrow, or in bigrquerystorage. The below reproduces similarly whether using bigint="character", numeric, or integer64, all of which should return similar values.
N.B., I'm not asking why bit64::as.integer64("9223372036854775806") and as.numeric("9223372036854775806") return different values.
Emprically, the max int64 supported by bit64 never works except as character, the lowest int64 in BQ that is returned non-null to R is 9223372036854775295, which is 512 below bit64's upper limit. I suspect that difference is not BQ but within the same internal conversion issue in nanoarrow.
bqcon <- DBI::dbConnect(...)
bqcon@bigint
# [1] "integer64"
bit64::lim.integer64()
# integer64
# [1] -9223372036854775807 9223372036854775807
bit64::as.integer64("9223372036854775806") # subtracted 1
# integer64
# [1] 9223372036854775806
tb <- bigrquery::bq_project_query(bqcon@billing, "select 9223372036854775296 as lost_int64, 9223372036854775295 as wrong_int64, cast(9223372036854775295 as string) as right_chr")
bigrquery::bq_table_download(tb, bigint = "integer64") |> as.data.frame()
# Warning in convert_array_stream(x, to) :
# 1 value(s) may have incurred loss of precision in conversion to double()
# Warning in convert_array_stream(x, to) :
# 1 value(s) may have incurred loss of precision in conversion to double()
# Warning in as.integer64.double(x) : NAs produced by integer64 overflow
# lost_int64 wrong_int64 right_chr
# 1 <NA> 9223372036854774784 9223372036854775295
### 9223372036854775295 9223372036854775295 <-- original numbers from the query, for visual reference
In my use-case, I have data in BQ that uses an int64 as a sequence value, and some of the values (well below bit64's limits) are triggering this warning. While I could cast(.. as string) to avoid the warning with the same end-effect, it is certainly unexpected and concerning that the bigint= intention is not carried through the whole pipe.
This appears to be the same theme as #94, #437, #439, and tbh I'm not entirely certain if this fits better here, in
nanoarrow, or inbigrquerystorage. The below reproduces similarly whether usingbigint="character", numeric, or integer64, all of which should return similar values.N.B., I'm not asking why
bit64::as.integer64("9223372036854775806")andas.numeric("9223372036854775806")return different values.Emprically, the max int64 supported by
bit64never works except as character, the lowest int64 in BQ that is returned non-null to R is9223372036854775295, which is 512 belowbit64's upper limit. I suspect that difference is not BQ but within the same internal conversion issue innanoarrow.In my use-case, I have data in BQ that uses an int64 as a sequence value, and some of the values (well below
bit64's limits) are triggering this warning. While I couldcast(.. as string)to avoid the warning with the same end-effect, it is certainly unexpected and concerning that thebigint=intention is not carried through the whole pipe.