|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "Apache Arrow nanoarrow 0.9.0 Release" |
| 4 | +date: "2026-08-14 00:00:00" |
| 5 | +author: pmc |
| 6 | +categories: [release] |
| 7 | +--- |
| 8 | +<!-- |
| 9 | +{% comment %} |
| 10 | +Licensed to the Apache Software Foundation (ASF) under one or more |
| 11 | +contributor license agreements. See the NOTICE file distributed with |
| 12 | +this work for additional information regarding copyright ownership. |
| 13 | +The ASF licenses this file to you under the Apache License, Version 2.0 |
| 14 | +(the "License"); you may not use this file except in compliance with |
| 15 | +the License. You may obtain a copy of the License at |
| 16 | +
|
| 17 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 18 | +
|
| 19 | +Unless required by applicable law or agreed to in writing, software |
| 20 | +distributed under the License is distributed on an "AS IS" BASIS, |
| 21 | +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 22 | +See the License for the specific language governing permissions and |
| 23 | +limitations under the License. |
| 24 | +{% endcomment %} |
| 25 | +--> |
| 26 | + |
| 27 | +The Apache Arrow team is pleased to announce the 0.9.0 release of |
| 28 | +Apache Arrow nanoarrow. This release consists of 38 resolved GitHub issues from |
| 29 | +5 contributors. |
| 30 | + |
| 31 | +## Release Highlights |
| 32 | + |
| 33 | +In addition to a number of bugfixes and minor build system improvements, we |
| 34 | +added several new features in nanoarrow 0.9.0. |
| 35 | + |
| 36 | +- Dictionary decoding support in IPC reader |
| 37 | +- Reference-counted array/buffer support |
| 38 | +- LZ4 decompression support in R and Python bindings |
| 39 | + |
| 40 | +See the |
| 41 | +[Changelog](https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.9.0/CHANGELOG.md) |
| 42 | +for a detailed list of contributions to this release. |
| 43 | + |
| 44 | +## Features |
| 45 | + |
| 46 | +### Dictionary decode support |
| 47 | + |
| 48 | +Whereas the nanoarrow IPC reader suppports most Arrow IPC features, dictionary support |
| 49 | +was a long requested gap in the reader functionality (mostly requested by users of |
| 50 | +the [DuckDB nanoarrow extension](https://github.com/paleolimbot/duckdb-nanoarrow), which uses nanoarrow's reader). Dictionary encoding is used to reduce the size of frequently |
| 51 | +repeated values and is the serialized equivalent of the "dictionary" data type that |
| 52 | +is exposed in most Arrow implementations. |
| 53 | + |
| 54 | +In nanoarrow 0.9.0 built with the IPC feature enabled, streams that include the most |
| 55 | +common forms of dictionary encoding (i.e., dictionary replacement) should now work |
| 56 | +out of the box. This includes nested/complex dictionary types and dictionary |
| 57 | +replacement but does not include "delta" dictionaries (i.e., dictionaires that |
| 58 | +grow larger as more values are encountered in the encoded values). |
| 59 | + |
| 60 | +In R this is accessible via `read_nanoarrow()`; in Python this is accessible via |
| 61 | +`nanoarrow.ArrayStream.from_readable()`; in C this is available via the |
| 62 | +higher level `ArrowIpcArrayStreamReader` API. Lower level users of the |
| 63 | +`ArrowIpcDecoder` will have to update existing usage to use the |
| 64 | +`ArrowIpcDecoder...WithDictionaries()` variants of some functions |
| 65 | +to support input with dictionary schemas or batches. |
| 66 | + |
| 67 | +### Reference-counted array/buffer support |
| 68 | + |
| 69 | +In previous versions (since the introduction of the IPC reader), the |
| 70 | +`ArrowIpcSharedBuffer` has supported reading IPC streams and sharing an |
| 71 | +underlying set of data buffers for a group of arrays; however, this wasn't |
| 72 | +quite sufficient for the more complex case of decoding a dictionary and |
| 73 | +attaching cheaply-cloned shared "values" arrays for potentially many batches. |
| 74 | +Version 0.9.0 moves this functionality to the `ArrowSharedBuffer` and expands it |
| 75 | +to support moving all of an array's buffers into a shared state that can be |
| 76 | +more cheaply cloned. |
| 77 | + |
| 78 | +### LZ4 decompression support in R and Python |
| 79 | + |
| 80 | +While LZ4 decompression support was has long been available via the pluggable decoder |
| 81 | +framework (and available since 0.8.0 as a built-in compile time option), reading |
| 82 | +IPC streams with LZ4 buffer compression was not possible in the R or Python bindings. |
| 83 | +In 0.9.0, the requisite configuration options were added such that the packages are |
| 84 | +built with LZ4 when it is available on the system. |
| 85 | + |
| 86 | +```python |
| 87 | +import io |
| 88 | +import nanoarrow as na |
| 89 | +import pyarrow as pa |
| 90 | + |
| 91 | +buf = io.BytesIO() |
| 92 | +batch = pa.record_batch({"x": range(1000)}) |
| 93 | +with pa.ipc.new_stream(buf, batch.schema, options=pa.ipc.IpcWriteOptions(compression="lz4")) as w: |
| 94 | + w.write_batch(batch) |
| 95 | + |
| 96 | +buf.seek(0) |
| 97 | +na.ArrayStream.from_readable(buf).read_all() |
| 98 | +# nanoarrow.Array<non-nullable struct<x: int64>>[1000] |
| 99 | +# {'x': 0} |
| 100 | +# {'x': 1} |
| 101 | +# {'x': 2} |
| 102 | +# {'x': 3} |
| 103 | +# {'x': 4} |
| 104 | +# {'x': 5} |
| 105 | +# {'x': 6} |
| 106 | +# {'x': 7} |
| 107 | +# {'x': 8} |
| 108 | +# {'x': 9} |
| 109 | +# ...and 990 more items |
| 110 | +``` |
| 111 | + |
| 112 | +```r |
| 113 | +library(nanoarrow) |
| 114 | +library(reticulate) |
| 115 | + |
| 116 | +# IPC Write with compression not available in arrow/R |
| 117 | +pa <- reticulate::import("pyarrow") |
| 118 | +io <- reticulate::import("io") |
| 119 | + |
| 120 | +buf <- io$BytesIO() |
| 121 | +batch <- arrow::record_batch(x = 1:1000) |
| 122 | +writer <- pa$ipc$new_stream(buf, batch$schema, options = pa$ipc$IpcWriteOptions(compression = "lz4")) |
| 123 | +writer$write_batch(batch) |
| 124 | +writer$close() |
| 125 | + |
| 126 | +nanoarrow::read_nanoarrow(as.raw(buf$getvalue())) |> |
| 127 | + tibble::as_tibble() |
| 128 | +#> # A tibble: 1,000 × 1 |
| 129 | +#> x |
| 130 | +#> <int> |
| 131 | +#> 1 1 |
| 132 | +#> 2 2 |
| 133 | +#> 3 3 |
| 134 | +#> 4 4 |
| 135 | +#> 5 5 |
| 136 | +#> 6 6 |
| 137 | +#> 7 7 |
| 138 | +#> 8 8 |
| 139 | +#> 9 9 |
| 140 | +#> 10 10 |
| 141 | +#> # ℹ 990 more rows |
| 142 | +``` |
| 143 | + |
| 144 | +## Contributors |
| 145 | + |
| 146 | +This release consists of contributions from 5 contributors in addition |
| 147 | +to the invaluable advice and support of the Apache Arrow community. |
| 148 | + |
| 149 | +```console |
| 150 | +$ git shortlog -sn apache-arrow-nanoarrow-0.9.0.dev..apache-arrow-nanoarrow-0.9.0 |
| 151 | + 36 Dewey Dunnington |
| 152 | + 2 Andrew Kane |
| 153 | + 2 Bryce Mecum |
| 154 | + 1 Michael Osipov |
| 155 | + 1 Oliver Borchert |
| 156 | +``` |
0 commit comments