Unable to convert from existing arrow dataset from polars-st

[msmitherdc]

Datafile is in public s3 location

import polars as pl
import polars_st as st
import pyarrow.dataset as ds
import rustac

dataset = ds.dataset("s3://grid-partner-share/stac_11764_nzimagery_rasters.parquet", format="parquet")
df = pl.scan_pyarrow_dataset(dataset).collect()
gdf = st.GeoDataFrame(data=df)
ruarrow = rustac.from_arrow(gdf.to_arrow())

results in

thread '<unnamed>' (322187) panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-56.2.0/src/cast.rs:868:29:
binary array
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
Cell In[24], line 1
----> 1 rustac.from_arrow(gdf)

PanicException: binary array

[gadomski]

Any way you could share the source file (or a subset of it)?

[msmitherdc]

The s3 location is public. You can pull it down.

[gadomski]

Looking like it's some sort of arrow/parquet version mismatch, I got this from gpq:

$ gpq describe stac_11764_nzimagery_rasters.parquet
panic: invalid logical type

goroutine 1 [running]:
github.com/apache/arrow/go/v16/parquet/schema.getLogicalType(0x140009f40e0)
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/schema/logical_types.go:76 +0x360
github.com/apache/arrow/go/v16/parquet/schema.PrimitiveNodeFromThrift(0x14000115b00)
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/schema/node.go:273 +0x70
github.com/apache/arrow/go/v16/parquet/schema.FromParquet.func1()
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/schema/schema.go:92 +0xb8
github.com/apache/arrow/go/v16/parquet/schema.FromParquet.func1()
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/schema/schema.go:97 +0x1a8
github.com/apache/arrow/go/v16/parquet/schema.FromParquet({0x140009e4160?, 0x10451baa0?, 0x140009f5101?})
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/schema/schema.go:107 +0x13c
github.com/apache/arrow/go/v16/parquet/metadata.(*FileMetaData).initSchema(0x14000a34060)
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/metadata/file.go:308 +0x34
github.com/apache/arrow/go/v16/parquet/metadata.NewFileMetaData({0x140009f0000, 0x3a33, 0x3a33}, {0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/metadata/file.go:269 +0x150
github.com/apache/arrow/go/v16/parquet/file.(*Reader).parseMetaData(0x14000114c60)
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/file/file_reader.go:187 +0x620
github.com/apache/arrow/go/v16/parquet/file.NewParquetReader({0x12c70c0b8, 0x14000196e60}, {0x0, 0x0, 0x140009ed460?})
	/home/runner/go/pkg/mod/github.com/apache/arrow/go/v16@v16.1.0/parquet/file/file_reader.go:131 +0x230
github.com/planetlabs/gpq/cmd/gpq/command.(*DescribeCmd).Run(0x105677880)
	/home/runner/work/gpq/gpq/cmd/gpq/command/describe.go:59 +0x144
reflect.Value.call({0x1042c2360?, 0x105677880?, 0x1400093f8c8?}, {0x103979e4b, 0x4}, {0x1400093f9a8, 0x0, 0x102550e9c?})
	/opt/hostedtoolcache/go/1.22.9/x64/src/reflect/value.go:596 +0x970
reflect.Value.Call({0x1042c2360?, 0x105677880?, 0x10?}, {0x1400093f9a8?, 0x1400093f958?, 0x10245db30?})
	/opt/hostedtoolcache/go/1.22.9/x64/src/reflect/value.go:380 +0x94
github.com/alecthomas/kong.callAnyFunction({0x1042c2360?, 0x105677880?, 0x10426aa20?}, 0x140009ee8a0)
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v1.4.0/callbacks.go:112 +0x544
github.com/alecthomas/kong.callFunction({0x1042c2360?, 0x105677880?, 0x1400093fcd0?}, 0x140009ee8a0)
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v1.4.0/callbacks.go:79 +0x178
github.com/alecthomas/kong.(*Context).RunNode(0x1400081f380, 0x140009ce2d0, {0x1400093ff10, 0x2, 0x1400093fe68?})
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v1.4.0/context.go:798 +0x658
github.com/alecthomas/kong.(*Context).Run(0x1400081f380, {0x1400093ff10?, 0x0?, 0x102434358?})
	/home/runner/go/pkg/mod/github.com/alecthomas/kong@v1.4.0/context.go:825 +0x114
main.main()
	/home/runner/work/gpq/gpq/cmd/gpq/main.go:32 +0x10c

@msmitherdc can you share how you created the file? In particular, it'd be good to know which versions of arrow and friends were used.

[msmitherdc]

Yeah, its a django querry the uses python oracledb to create a dataframe, I then use polars-st (conda) to convert to a geodataframe and then write and it uses pyogrio. This is the polars geospatial lib https://github.com/oreilles/polars-st
pyarrow 23.0.1
oracledb 3.4.2
polars 1.38.1
polars-st 0.4.3

[gadomski]

I get a similar-but-different error from the command-line:

$ rustac translate stac_11764_nzimagery_rasters.parquet out.json
ERROR: Parquet error: Cannot annotate Geography from BYTE_ARRAY for field 'geometry'

I'm going to guess it has to do with native parquet geometry support, but I'm quickly running out of my own knowledge. I'll poke around it a bit, and then ask Kyle (not summoning yet) for help.

[msmitherdc]

I just recently changed to native parquet geometry support, I can try with wkb geometry. I've found better performance with the native geometry / geography format,

[gadomski]

Yeah, it'd be good to go to native geometry by default, but the tooling is still catching up. Try WKB and see if things work. I'd still like to get native working.

[msmitherdc]

Changed to wkb encoding and got

<ipython-input-2-eab3012ede2a>:2: UserWarning: Extension type 'geoarrow.wkb' is not registered; loading as its storage type.

To avoid this warning, register the extension type or set environment variable 'POLARS_UNKNOWN_EXTENSION_TYPE_BEHAVIOR' to 'load_as_storage' or 'load_as_extension'.

In Polars 2.0, the default behavior will change to 'load_as_extension'.
  df = pl.scan_pyarrow_dataset(dataset).collect()

In [3]: ruarrow = rustac.from_arrow(gdf.to_arrow())

thread '<unnamed>' (491285) panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-56.2.0/src/cast.rs:868:29:
binary array
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---------------------------------------------------------------------------
PanicException                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 ruarrow = rustac.from_arrow(gdf.to_arrow())

PanicException: binary array

Will push the updated file to replace the existing file i have shared

[gadomski]

Ok, I've got a fix (I think) into the Rust code in this branch. I think it's just a matter of updating our stuff to v58 of the Arrow ecosystem packages; it will be a minute to be able to release our Rust crates, but I should be able to get a Python wheel out sooner (stay tuned):

$ cargo run --release -- translate ~/Desktop/stac_11764_nzimagery_rasters.parquet ~/Desktop/out.parquet
$ gpq describe ~/Desktop/out.parquet | head
╭────────────────────────┬────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────────┬─────────────┬──────────┬────────────────┬──────────────────────────────────────────┬────────╮
│ COLUMN                 │ TYPE   │ ANNOTATION                                                                                                           │ REPETITION │ COMPRESSION │ ENCODING │ GEOMETRY TYPES │ BOUNDS                                   │ DETAIL │
├────────────────────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────┼─────────────┼──────────┼────────────────┼──────────────────────────────────────────┼────────┤
│ type                   │ binary │ string                                                                                                               │ 0..1       │ zstd        │          │                │                                          │        │
│ stac_version           │ binary │ string                                                                                                               │ 0..1       │ zstd        │          │                │                                          │        │
│ stac_extensions        │        │ list                                                                                                                 │ 0..1       │             │          │                │                                          │        │
│ id                     │ binary │ string                                                                                                               │ 0..1       │ zstd        │          │                │                                          │        │
│ data_classification    │ binary │ string                                                                                                               │ 0..1       │ zstd        │          │                │                                          │        │
│ links                  │        │ list                                                                                                                 │ 0..1       │             │          │                │                                          │        │
│ assets                 │        │ group                                                                                                                │ 0..1       │             │          │                │                                          │        │

I had to make a couple deserialization fixes to work with your data:

• Your id was an integer, not a string
• Your start_datetime and end_datetime fields don't have a Z or +00:00 timezone specifier
• Your stac_extensions was a JSON-encoded string, not a list of strings

This all suggests that we need a dedicated stac-geoparquet validator, since many parsers will break when converting to JSON, where most validation is implemented.

[gadomski]

@msmitherdc I've got things working in Python-land: #253. Note that your example still breaks, but if you want to go from a Parquet file to a dictionary, just go through rustac (no need to get polars involved):

import rustac

item_collection = rustac.read_sync(
    "s3://grid-partner-share/stac_11764_nzimagery_rasters.parquet"
)

I don't have an API yet to from an href to an arrow table, but we can add that too if it's useful? If you want an arrow table, not a dictionary, just use polars, no need to get rustac involved.

[msmitherdc]

Just an fyi on why i changed to using the arrow native geography type in the stac parquet files is running this query:

select sum(file_size) from read_parquet('{stac_path}') where ST_Intersects_Extent(geometry, ST_GeomFromWKB(?))

in duckdb 1.4.4:
for an incoming wkb geom about 10_000 sqkm

duck wkb       time: 0:00:15.171584
duck geo        time: 0:00:07.80105