Skip to content

Commit d7f1d31

Browse files
committed
docs: move avro upgrade note to 54 guide
1 parent b45ac7f commit d7f1d31

2 files changed

Lines changed: 36 additions & 41 deletions

File tree

docs/source/library-user-guide/upgrading/53.0.0.md

Lines changed: 0 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -472,42 +472,3 @@ Now:
472472
+-------+
473473
0 row(s) fetched.
474474
```
475-
476-
### Avro API, feature, and timestamp decoding changes
477-
478-
As part of the Avro reader migration (see [#17861]), DataFusion now delegates
479-
Avro-to-Arrow type decoding to `arrow-avro`, and several Avro-related
480-
APIs/feature wiring changed:
481-
482-
- `DataFusionError::AvroError` has been removed.
483-
- `From<apache_avro::Error> for DataFusionError` has been removed.
484-
- Avro crate re-export changed:
485-
- Before: `datafusion::apache_avro`
486-
- After: `datafusion::arrow_avro`
487-
- Cargo feature wiring changed:
488-
- `datafusion` crate `avro` feature no longer enables `datafusion-common/avro`
489-
- `datafusion-proto` crate `avro` feature no longer enables `datafusion-common/avro`
490-
- **Avro timestamp logical type interpretation changed.** Notable effects:
491-
- Avro `timestamp-*` logical types are read as UTC timezone-aware Arrow timestamps
492-
(`Timestamp(..., Some("+00:00"))`)
493-
- Avro `local-timestamp-*` logical types remain timezone-naive
494-
(`Timestamp(..., None)`)
495-
496-
**Who is affected:**
497-
498-
- Users matching on `DataFusionError::AvroError`
499-
- Users importing `datafusion::apache_avro`
500-
- Users depending on the old `datafusion-common/avro` feature wiring
501-
- Users relying on previous Avro timestamp logical type behavior
502-
503-
**Migration guide:**
504-
505-
- Replace `datafusion::apache_avro` imports with `datafusion::arrow_avro`.
506-
- Update error handling code that matches on `DataFusionError::AvroError` to use
507-
the current error surface.
508-
- If you depend on Avro feature propagation, update Cargo feature expectations:
509-
`datafusion`/`datafusion-proto` `avro` no longer enables `datafusion-common/avro`.
510-
- Validate timestamp handling where timezone semantics matter:
511-
`timestamp-*` is UTC timezone-aware, while `local-timestamp-*` is timezone-naive.
512-
513-
[#17861]: https://github.com/apache/datafusion/pull/17861

docs/source/library-user-guide/upgrading/54.0.0.md

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ fn apply_expressions(
7777

7878
**Node whose only expressions are in `output_ordering()` (e.g. a synthetic test node with no owned expression fields):**
7979

80-
````rust,ignore
80+
```rust,ignore
8181
fn apply_expressions(
8282
&self,
8383
f: &mut dyn FnMut(&dyn PhysicalExpr) -> Result<TreeNodeRecursion>,
@@ -90,6 +90,7 @@ fn apply_expressions(
9090
}
9191
Ok(tnr)
9292
}
93+
```
9394

9495
### `ExecutionPlan::partition_statistics` now returns `Arc<Statistics>`
9596

@@ -101,7 +102,7 @@ fn apply_expressions(
101102
fn partition_statistics(&self, partition: Option<usize>) -> Result<Statistics> {
102103
Ok(Statistics::new_unknown(&self.schema()))
103104
}
104-
````
105+
```
105106

106107
**After:**
107108

@@ -122,3 +123,36 @@ stats.column_statistics[0].min_value = ...;
122123
let mut stats = Arc::unwrap_or_clone(plan.partition_statistics(None)?);
123124
stats.column_statistics[0].min_value = ...;
124125
```
126+
127+
### Avro API and timestamp decoding changes
128+
129+
DataFusion has switched to use `arrow-avro` (see [#17861]) when reading Avro
130+
files which results in a few changes:
131+
132+
- `DataFusionError::AvroError` has been removed.
133+
- `From<apache_avro::Error> for DataFusionError` has been removed.
134+
- Avro crate re-export changed:
135+
- Before: `datafusion::apache_avro`
136+
- After: `datafusion::arrow_avro`
137+
- **Avro timestamp logical type interpretation changed.** Notable effects:
138+
- Avro `timestamp-*` logical types are read as UTC timezone-aware Arrow
139+
timestamps (`Timestamp(..., Some("+00:00"))`)
140+
- Avro `local-timestamp-*` logical types remain timezone-naive
141+
(`Timestamp(..., None)`)
142+
143+
**Who is affected:**
144+
145+
- Users matching on `DataFusionError::AvroError`
146+
- Users importing `datafusion::apache_avro`
147+
- Users relying on previous Avro timestamp logical type behavior
148+
149+
**Migration guide:**
150+
151+
- Replace `datafusion::apache_avro` imports with `datafusion::arrow_avro`.
152+
- Update error handling code that matches on `DataFusionError::AvroError` to use
153+
the current error surface.
154+
- Validate timestamp handling where timezone semantics matter:
155+
`timestamp-*` is UTC timezone-aware, while `local-timestamp-*` is
156+
timezone-naive.
157+
158+
[#17861]: https://github.com/apache/datafusion/pull/17861

0 commit comments

Comments
 (0)