Skip to content

bigint not passed to nanoarrow #689

Description

@r2evans

This appears to be the same theme as #94, #437, #439, and tbh I'm not entirely certain if this fits better here, in nanoarrow, or in bigrquerystorage. The below reproduces similarly whether using bigint="character", numeric, or integer64, all of which should return similar values.

N.B., I'm not asking why bit64::as.integer64("9223372036854775806") and as.numeric("9223372036854775806") return different values.

Emprically, the max int64 supported by bit64 never works except as character, the lowest int64 in BQ that is returned non-null to R is 9223372036854775295, which is 512 below bit64's upper limit. I suspect that difference is not BQ but within the same internal conversion issue in nanoarrow.

bqcon <- DBI::dbConnect(...)
bqcon@bigint
# [1] "integer64"

bit64::lim.integer64()
# integer64
# [1] -9223372036854775807 9223372036854775807 
bit64::as.integer64("9223372036854775806")            # subtracted 1
# integer64
# [1] 9223372036854775806

tb <- bigrquery::bq_project_query(bqcon@billing, "select 9223372036854775296 as lost_int64, 9223372036854775295 as wrong_int64, cast(9223372036854775295 as string) as right_chr")
bigrquery::bq_table_download(tb, bigint = "integer64") |> as.data.frame()
# Warning in convert_array_stream(x, to) :
#   1 value(s) may have incurred loss of precision in conversion to double()
# Warning in convert_array_stream(x, to) :
#   1 value(s) may have incurred loss of precision in conversion to double()
# Warning in as.integer64.double(x) : NAs produced by integer64 overflow
#   lost_int64         wrong_int64           right_chr
# 1       <NA> 9223372036854774784 9223372036854775295

###            9223372036854775295 9223372036854775295 <-- original numbers from the query, for visual reference

In my use-case, I have data in BQ that uses an int64 as a sequence value, and some of the values (well below bit64's limits) are triggering this warning. While I could cast(.. as string) to avoid the warning with the same end-effect, it is certainly unexpected and concerning that the bigint= intention is not carried through the whole pipe.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions