From 6e4ff0b841a69b73e00f2ef72090d35d1b95dba0 Mon Sep 17 00:00:00 2001 From: Zach Schuermann Date: Fri, 19 Sep 2025 14:48:06 -0700 Subject: [PATCH 1/2] release 0.16.0 --- CHANGELOG.md | 124 ++++++++++++++++++++++++++++++++++++++++++++++ Cargo.toml | 2 +- README.md | 4 +- ffi/Cargo.toml | 2 +- kernel/Cargo.toml | 2 +- 5 files changed, 129 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e58bde92c..8b56e0cd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,129 @@ # Changelog +## [v0.16.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.16.0/) (2025-09-19) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.15.2...v0.16.0) + + +### ๐Ÿš€ Features / new APIs + +1. Add `UnaryExpression` and `ToJson` expression ([#1192]) +2. Feat!(catalog-managed): SnapshotBuilder ([#1189]) +3. Simplify the Expr::Transform API, add FFI support ([#1243]) +4. Add `numRecords` to `ADD_FILES_SCHEMA` ([#1235]) +5. Add `try_append_columns` to `EngineData` ([#1190]) +6. *(catalog-managed)* Add log_tail to list_log_files ([#1194]) +7. CommitInfo sets a txnId ([#1262]) +8. Add row tracking writer feature ([#1239]) +9. Allow LargeUTF8 -> String and LargeBinary -> Binary in arrow conversion ([#1294]) +10. Implement log compaction ([#1234]) +11. Disallow equal version in log compaction ([#1309]) +12. Introduce `Expression::Variadic` and `Coalesce` expressions ([#1198]) +13. Add `Iterable` to `StructType` ([#1287]) +14. New StructType constructors ([#1278]) +15. Introduce metadata column API ([#1266]) +16. ParsedLogPath for staged commits ([#1305]) +17. Default expression eval supports nested transforms ([#1247]) +18. Introduce row index metadata column ([#1272]) + +### ๐Ÿ› Bug Fixes + +1. Make ListedLogFiles::try_new internal-api (again) ([#1226]) +2. Pin comfy-table at 7.1.4 to restore kernel MSRV ([#1231]) +3. Arrow json decoder fix for breakage on long json string ([#1244]) +4. Make ColumnType pub(crate) ([#1258]) + +### ๐Ÿ“š Documentation + +1. Update README.md to enhance FFI documentation ([#1237]) + +### โšก Performance + +1. Make checkpoint visitor more efficient using short circuiting ([#1203]) + +### ๐Ÿšœ Refactor + +1. Factor out a method for LastCheckpointHint path generation ([#1228]) +2. Do not guess Vec size for checkpoints ([#1263]) +3. Introduce current_time_ms() helper ([#1256]) +4. Retention calculation into a new trait ([#1264]) +5. Migrate Snapshot::try_new_from into SnapshotBuilder ([#1289]) +6. Minor Refactoring in Log Compaction ([#1301]) +7. Rename SnapshotBuilder::new to new_for ([#1306]) +8. Move log replay into the action reconciliation module ([#1295]) +9. Introduce SnapshotRef type alias ([#1299]) +10. Row tracking write cleanup ([#1291]) + +### ๐Ÿงช Testing + +1. Update invalid-handle tests for rustc 1.90 ([#1321]) + +### โš™๏ธ Chores/CI + +1. Update changelog for 0.15.1 release ([#1227]) +2. Sync changelog for 0.15.2 ([#1251]) +3. Update data types test to validate full Arrow error message ([#1259]) +4. Add better panic message when not OK ([#1293]) +5. Add test for empty commits and clean up test error types ([#1252]) +6. Make StructType::into_fields return type consistent ([#1327]) +7. Update contributing.md ([#1206]) + +### Other + +1. Create expression benchmark for default engine ([#1220]) +2. Add CDvInfo struct ([#1286]) +3. Add numbers for `KernelError` enum variants ([#1313]) +4. Don't need a whole engine when converting into `ArrowEngineData` ([#1325]) + + +[#1192]: https://github.com/delta-io/delta-kernel-rs/pull/1192 +[#1189]: https://github.com/delta-io/delta-kernel-rs/pull/1189 +[#1226]: https://github.com/delta-io/delta-kernel-rs/pull/1226 +[#1227]: https://github.com/delta-io/delta-kernel-rs/pull/1227 +[#1231]: https://github.com/delta-io/delta-kernel-rs/pull/1231 +[#1228]: https://github.com/delta-io/delta-kernel-rs/pull/1228 +[#1203]: https://github.com/delta-io/delta-kernel-rs/pull/1203 +[#1244]: https://github.com/delta-io/delta-kernel-rs/pull/1244 +[#1243]: https://github.com/delta-io/delta-kernel-rs/pull/1243 +[#1251]: https://github.com/delta-io/delta-kernel-rs/pull/1251 +[#1235]: https://github.com/delta-io/delta-kernel-rs/pull/1235 +[#1190]: https://github.com/delta-io/delta-kernel-rs/pull/1190 +[#1194]: https://github.com/delta-io/delta-kernel-rs/pull/1194 +[#1258]: https://github.com/delta-io/delta-kernel-rs/pull/1258 +[#1259]: https://github.com/delta-io/delta-kernel-rs/pull/1259 +[#1263]: https://github.com/delta-io/delta-kernel-rs/pull/1263 +[#1256]: https://github.com/delta-io/delta-kernel-rs/pull/1256 +[#1262]: https://github.com/delta-io/delta-kernel-rs/pull/1262 +[#1264]: https://github.com/delta-io/delta-kernel-rs/pull/1264 +[#1239]: https://github.com/delta-io/delta-kernel-rs/pull/1239 +[#1237]: https://github.com/delta-io/delta-kernel-rs/pull/1237 +[#1294]: https://github.com/delta-io/delta-kernel-rs/pull/1294 +[#1234]: https://github.com/delta-io/delta-kernel-rs/pull/1234 +[#1220]: https://github.com/delta-io/delta-kernel-rs/pull/1220 +[#1289]: https://github.com/delta-io/delta-kernel-rs/pull/1289 +[#1301]: https://github.com/delta-io/delta-kernel-rs/pull/1301 +[#1293]: https://github.com/delta-io/delta-kernel-rs/pull/1293 +[#1306]: https://github.com/delta-io/delta-kernel-rs/pull/1306 +[#1295]: https://github.com/delta-io/delta-kernel-rs/pull/1295 +[#1286]: https://github.com/delta-io/delta-kernel-rs/pull/1286 +[#1313]: https://github.com/delta-io/delta-kernel-rs/pull/1313 +[#1299]: https://github.com/delta-io/delta-kernel-rs/pull/1299 +[#1309]: https://github.com/delta-io/delta-kernel-rs/pull/1309 +[#1198]: https://github.com/delta-io/delta-kernel-rs/pull/1198 +[#1287]: https://github.com/delta-io/delta-kernel-rs/pull/1287 +[#1321]: https://github.com/delta-io/delta-kernel-rs/pull/1321 +[#1278]: https://github.com/delta-io/delta-kernel-rs/pull/1278 +[#1252]: https://github.com/delta-io/delta-kernel-rs/pull/1252 +[#1291]: https://github.com/delta-io/delta-kernel-rs/pull/1291 +[#1266]: https://github.com/delta-io/delta-kernel-rs/pull/1266 +[#1305]: https://github.com/delta-io/delta-kernel-rs/pull/1305 +[#1325]: https://github.com/delta-io/delta-kernel-rs/pull/1325 +[#1247]: https://github.com/delta-io/delta-kernel-rs/pull/1247 +[#1327]: https://github.com/delta-io/delta-kernel-rs/pull/1327 +[#1206]: https://github.com/delta-io/delta-kernel-rs/pull/1206 +[#1272]: https://github.com/delta-io/delta-kernel-rs/pull/1272 + + ## [v0.15.2](https://github.com/delta-io/delta-kernel-rs/tree/v0.15.2/) (2025-09-03) [Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.15.1...v0.15.2) diff --git a/Cargo.toml b/Cargo.toml index 28510ed6b..b8a9c76d5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -25,4 +25,4 @@ license = "Apache-2.0" repository = "https://github.com/delta-io/delta-kernel-rs" readme = "README.md" rust-version = "1.84" -version = "0.15.2" +version = "0.16.0" diff --git a/README.md b/README.md index f57bc28df..9b808d849 100644 --- a/README.md +++ b/README.md @@ -52,10 +52,10 @@ consumer's own `Engine` trait, the kernel has a feature flag to enable a default ```toml # fewer dependencies, requires consumer to implement Engine trait. # allows consumers to implement their own in-memory format -delta_kernel = "0.15.2" +delta_kernel = "0.16.0" # or turn on the default engine, based on arrow -delta_kernel = { version = "0.15.2", features = ["default-engine", "arrow-56"] } +delta_kernel = { version = "0.16.0", features = ["default-engine", "arrow-56"] } ``` ### Feature flags diff --git a/ffi/Cargo.toml b/ffi/Cargo.toml index 61ef1e7e7..438bb0e80 100644 --- a/ffi/Cargo.toml +++ b/ffi/Cargo.toml @@ -25,7 +25,7 @@ url = "2" delta_kernel = { path = "../kernel", default-features = false, features = [ "internal-api", ] } -delta_kernel_ffi_macros = { path = "../ffi-proc-macros", version = "0.15.2" } +delta_kernel_ffi_macros = { path = "../ffi-proc-macros", version = "0.16.0" } [build-dependencies] cbindgen = "0.29" diff --git a/kernel/Cargo.toml b/kernel/Cargo.toml index 820f33bb5..eb21b3702 100644 --- a/kernel/Cargo.toml +++ b/kernel/Cargo.toml @@ -39,7 +39,7 @@ pre-release-hook = [ ] [dependencies] -delta_kernel_derive = { path = "../derive-macros", version = "0.15.2" } +delta_kernel_derive = { path = "../derive-macros", version = "0.16.0" } bytes = "1.10" chrono = "0.4.41" indexmap = "2.10.0" From 5a04c9b3b2ce553d16320603a6be9929c0168fee Mon Sep 17 00:00:00 2001 From: Zach Schuermann Date: Fri, 19 Sep 2025 15:19:46 -0700 Subject: [PATCH 2/2] upd changelog --- CHANGELOG.md | 81 ++++++++++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b56e0cd0..491a397fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,34 +4,40 @@ [Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.15.2...v0.16.0) - -### ๐Ÿš€ Features / new APIs - -1. Add `UnaryExpression` and `ToJson` expression ([#1192]) -2. Feat!(catalog-managed): SnapshotBuilder ([#1189]) -3. Simplify the Expr::Transform API, add FFI support ([#1243]) +### ๐Ÿ—๏ธ Breaking changes +1. New expression variants: `UnaryExpression` and `ToJson` expression ([#1192]) +2. New SnapshotBuilder API: `Snapshot::try_new(...)` replaced with `Snapshot::builder(...)` and its + associated methods. TLDR, you make a builder and call `build` to construct a `Snapshot`. ([#1189]) +3. Simplify the `Expr::Transform` API, add FFI support: + - Reworks the pub members of Transform used by Expr::Transform and introduce a new FieldTransform struct. + Also, rework Transform::new (constructor) and Transform::with_input_path (method) into a pair of + constructors, new_top_level and new_nested. + - Adds two new members to the FFI EngineExpressionVisitor struct -- visit_transform_expression and + visit_field_transform, which also changes the ordering of existing fields. ([#1243]) 4. Add `numRecords` to `ADD_FILES_SCHEMA` ([#1235]) -5. Add `try_append_columns` to `EngineData` ([#1190]) -6. *(catalog-managed)* Add log_tail to list_log_files ([#1194]) -7. CommitInfo sets a txnId ([#1262]) -8. Add row tracking writer feature ([#1239]) -9. Allow LargeUTF8 -> String and LargeBinary -> Binary in arrow conversion ([#1294]) -10. Implement log compaction ([#1234]) -11. Disallow equal version in log compaction ([#1309]) -12. Introduce `Expression::Variadic` and `Coalesce` expressions ([#1198]) -13. Add `Iterable` to `StructType` ([#1287]) -14. New StructType constructors ([#1278]) -15. Introduce metadata column API ([#1266]) -16. ParsedLogPath for staged commits ([#1305]) -17. Default expression eval supports nested transforms ([#1247]) -18. Introduce row index metadata column ([#1272]) +5. New `EngineData` trait required method: `try_append_columns` ([#1190]) +6. Make ColumnType private ([#1258]) +7. Add row tracking writer feature: updates `ADD_FILES_SCHEMA` (see PR for details) ([#1239]) +8. Migrate `Snapshot::try_new_from` into `SnapshotBuilder::new_from` ([#1289]) +9. (FFI) Add CDvInfo struct: The `CScanCallback` now takes a `&CDvInfo` and not a `&DvInfo`. ([#1286]) +10. (FFI) Add explicit numbers for each `KernelError` enum variants. (see PR for details) ([#1313]) +11. (more) new expression variants: `Expression::Variadic` and `Coalesce` expressions ([#1198]) +12. All new/modified `StructType` constructors, see PR for details ([#1278]) +13. Introduce metadata column API: `StructType` has new private field ([#1266]) +14. (FFI) `engine_data::get_engine_data` now takes an `AllocateErrorFn` instead of an engine. ([#1325]) +15. `StructType::into_fields` returns `DoubleEndedIterator + FusedIterator` ([#1327]) -### ๐Ÿ› Bug Fixes +### ๐Ÿš€ Features / new APIs -1. Make ListedLogFiles::try_new internal-api (again) ([#1226]) -2. Pin comfy-table at 7.1.4 to restore kernel MSRV ([#1231]) -3. Arrow json decoder fix for breakage on long json string ([#1244]) -4. Make ColumnType pub(crate) ([#1258]) +1. *(catalog-managed)* Add log_tail to list_log_files ([#1194]) +2. CommitInfo sets a txnId ([#1262]) +3. Allow LargeUTF8 -> String and LargeBinary -> Binary in arrow conversion ([#1294]) +4. Implement log compaction ([#1234]) +5. Disallow equal version in log compaction ([#1309]) +6. Add `Iterable` to `StructType` ([#1287]) +7. ParsedLogPath for staged commits ([#1305]) +8. Default expression eval supports nested transforms ([#1247]) +9. Introduce row index metadata column ([#1272]) ### ๐Ÿ“š Documentation @@ -47,16 +53,16 @@ 2. Do not guess Vec size for checkpoints ([#1263]) 3. Introduce current_time_ms() helper ([#1256]) 4. Retention calculation into a new trait ([#1264]) -5. Migrate Snapshot::try_new_from into SnapshotBuilder ([#1289]) -6. Minor Refactoring in Log Compaction ([#1301]) -7. Rename SnapshotBuilder::new to new_for ([#1306]) -8. Move log replay into the action reconciliation module ([#1295]) -9. Introduce SnapshotRef type alias ([#1299]) -10. Row tracking write cleanup ([#1291]) +5. Minor Refactoring in Log Compaction ([#1301]) +6. Rename SnapshotBuilder::new to new_for ([#1306]) +7. Move log replay into the action reconciliation module ([#1295]) +8. Introduce SnapshotRef type alias ([#1299]) +9. Row tracking write cleanup ([#1291]) ### ๐Ÿงช Testing 1. Update invalid-handle tests for rustc 1.90 ([#1321]) +2. Create expression benchmark for default engine ([#1220]) ### โš™๏ธ Chores/CI @@ -65,25 +71,14 @@ 3. Update data types test to validate full Arrow error message ([#1259]) 4. Add better panic message when not OK ([#1293]) 5. Add test for empty commits and clean up test error types ([#1252]) -6. Make StructType::into_fields return type consistent ([#1327]) -7. Update contributing.md ([#1206]) - -### Other - -1. Create expression benchmark for default engine ([#1220]) -2. Add CDvInfo struct ([#1286]) -3. Add numbers for `KernelError` enum variants ([#1313]) -4. Don't need a whole engine when converting into `ArrowEngineData` ([#1325]) +6. Update contributing.md ([#1206]) [#1192]: https://github.com/delta-io/delta-kernel-rs/pull/1192 [#1189]: https://github.com/delta-io/delta-kernel-rs/pull/1189 -[#1226]: https://github.com/delta-io/delta-kernel-rs/pull/1226 [#1227]: https://github.com/delta-io/delta-kernel-rs/pull/1227 -[#1231]: https://github.com/delta-io/delta-kernel-rs/pull/1231 [#1228]: https://github.com/delta-io/delta-kernel-rs/pull/1228 [#1203]: https://github.com/delta-io/delta-kernel-rs/pull/1203 -[#1244]: https://github.com/delta-io/delta-kernel-rs/pull/1244 [#1243]: https://github.com/delta-io/delta-kernel-rs/pull/1243 [#1251]: https://github.com/delta-io/delta-kernel-rs/pull/1251 [#1235]: https://github.com/delta-io/delta-kernel-rs/pull/1235