Release latest

Release latest

Release 1.0.0.Beta1

Release 1.0.0.Beta1

Based on the provided changelog for Release 1.0.0.Beta1, here are the notable changes:

Notable Changes

• Variant Support: iceberg consumer got a configuration option to store nested data to a variant field.
• Default Behavior Changes: The default consuming mode has been changed from upsert to append, and the default event format has been changed from json to connect.
• Parallel Processing: Concurrent parallel processing of the tables is now enabled, which should improve performance.
• Iceberg and Debezium Upgrades: The project has been upgraded to Iceberg 1.10.0 and Debezium 3.3.0.Beta1
• Documentation: Added migration guides for incompatible type changes.
• Build Environment: The release target has been set to Java 21.

NOTE:

• Please be aware that these modifications may affect backward compatibility. Thorough testing is strongly recommended before deployment.
• Switching from the json format to the connect format may introduce differences in how data types are handled. This could potentially lead to processing failures or unexpected data representations. We strongly recommend thoroughly testing your pipelines after making this change. Also please see migration guideline
• For new applications, we recommend starting with the connect format. It should be more performant and provide better data type support.

What's Changed

• Preserve required/optional field property when converting from source schema to Iceberg by @zilder in #575
• Fix IcebergSchemaHistory, dont delete history records by @ismailsimsek in #581
• Bump org.apache.iceberg:iceberg-bom from 1.8.1 to 1.9.1 by @dependabot[bot] in #579
• Bump io.debezium:debezium-build-parent from 3.1.1.Final to 3.1.2.Final by @dependabot[bot] in #578
• Bump version.debezium from 3.1.1.Final to 3.1.2.Final by @dependabot[bot] in #577
• Set release target to java 21 by @ismailsimsek in #582
• Bump org.testcontainers:testcontainers-bom from 1.21.0 to 1.21.1 by @dependabot[bot] in #576
• Connect, Use logicalTypeName to do precise value conversion by @ismailsimsek in #570
• Enforce non temporal(dynamic) precision mode by @ismailsimsek in #583
• Improve testing temporal type handling by @ismailsimsek in #585
• Fix JsonSchemaConverter dont increment filed ids for map key val by @ismailsimsek in #584
• Deprecate icebergevents consumer now with Variant support iceberg consumer is recommended instead by @ismailsimsek in #589
• Improvements documentation by @ismailsimsek in #590
• Add config to store nested data to variant field by @ismailsimsek in #587
• Fix deployment maven artifacts to github by @ismailsimsek in #588
• Change default consuming mode from upsert to append by @ismailsimsek in #591
• Change default event format from json to connect by @ismailsimsek in #592
• Add nessie catalog test as an example by @ismailsimsek in #593
• Improve namespace handling, don't fail if exists by @ismailsimsek in #595
• Testing, Replace jdbc catalog with nessie for better performance with testing by @ismailsimsek in #594
• Table destination mapper by @zilder in #599
• Fix class (org.apache.hadoop.mapreduce.lib.input.FileInputFormat) not found error with restart by @ismailsimsek in #606
• Enable concurent parallel processing by @ismailsimsek in #607
• docs: Add migration guide for incompatible type changes by @ismailsimsek in #612
• docs: Update migration guide for incompatible type changes by @ismailsimsek in #613
• Timestamp column in postgres is causing the debezium server to crash by @adasari in #621
• Consume nested fields to variant, And upgrade Iceberg to 1.10.0 by @ismailsimsek in #596
• Skip consuming heartbeat events by @ismailsimsek in #623
• Upgrade debezium to 3.3.x by @ismailsimsek in #622
• Bump org.codehaus.mojo:flatten-maven-plugin from 1.7.0 to 1.7.2 by @dependabot[bot] in #617
• Bump org.apache.iceberg:iceberg-nessie from 1.9.1 to 1.10.0 by @dependabot[bot] in #627

New Contributors

• @adasari made their first contribution in #621

Full Changelog: 1.0.0.Alpha1...1.0.0.Beta1

Release 1.0.0.Alpha1

Notable Changes

This release introduces the ability to consume records directly in the Debezium connect format. This bypasses the need for JSON serialization and deserialization, which is expected to improve performance. Users can now choose between the json and the newly added connect format.

Key tasks completed as part of this feature (#549):

• Implemented consumption of the EmbeddedEngineChangeEvent, which supports both ChangeEvent and RecordChangeEvent event types.
• Created EventConverter and SchemaConverter interfaces, establishing a flexible framework for handling format-specific conversions (JSON, Connect, etc.).
• Implement StructEventConverter, StructSchemaConverter classes for consuming Connect event format.

NOTE:

• Please be aware that these modifications may affect backward compatibility. Thorough testing is strongly recommended before deployment.
• Switching from the json format to the connect format may introduce differences in how data types are handled. This could potentially lead to processing failures or unexpected data representations. We strongly recommend thoroughly testing your pipelines after making this change. Also please see migration guideline
• For new applications, we recommend starting with the connect format. It should be more performant and provide better data type support.

What's Changed

• Improve reading config, isEventFlatteningEnabled by @ismailsimsek in #557
• Fix naming, Rename method cdcSourceTsMsValue to cdcSourceTsValue by @ismailsimsek in #558
• Improve record converter class and move deserializer variables to this class by @ismailsimsek in #559
• Consume events as EmbeddedEngineChangeEvent type, which supports both ChangeEvent and RecordChangeEvent by @ismailsimsek in #560
• EventConverter Interface Enables Dynamic Format Support (JSON, Connect, etc.) by @ismailsimsek in #553
• Test Json and Connect formats with github build actions by @ismailsimsek in #561
• Add kafka connect format support, add StructEventConverter by @ismailsimsek in #562
• Enable more tests for connect format by @ismailsimsek in #563
• Swap Schema import by @ismailsimsek in #565
• Add Connect format to the documentation by @ismailsimsek in #564
• Change Schema import by @ismailsimsek in #568
• Allow table columns to be excluded from the written iceberg table by @matthewmcnew in #569
• Start 1.X.X version, Set version to 1.0.0-SNAPSHOT by @ismailsimsek in #573

Full Changelog: 0.9.0.Final...1.0.0.Alpha1

Release 0.9.0.Final

Notable changes

NOTE: Please be aware that these modifications may affect backward compatibility. Thorough testing is strongly recommended before deployment.

This release includes several updates, including an upgrade to Apache Iceberg 1.8.1 and Debezium 3.1.1.Final, the addition of support for Iceberg decimals, and a change in how data de-duplication is handled in upsert mode.

• Upgraded Apache Iceberg to 1.8.1: This brings the latest features, performance improvements, and bug fixes from the Iceberg project.
• Upgraded Debezium to 3.1.1.Final: This incorporates the latest updates and improvements from the Debezium project.
• Added support for Iceberg decimals: Users can now correctly handle decimal data types when writing to Iceberg tables.
• Enabled usage as a Maven dependency: The project can now be easily integrated into other Maven projects using the flatten-maven-plugin.
• Switched to __source_ts_ns for upsert de-duplication: Data de-duplication in upsert mode now relies on the __source_ts_ns field. Be aware that this change may affect existing upsert configurations and data consistency if not accounted for.
• Improved Docker build process: The Docker build now correctly reflects the release version.
• Fixed deserializer usage: Corrected the deserializer used, resolving a potential issue.
• Published documentation to GitHub Pages: Project documentation is now available on GitHub Pages.

What's Changed

• Bump org.apache.iceberg:iceberg-bom from 1.7.1 to 1.8.1 by @dependabot in #529
• Bump io.debezium:debezium-build-parent from 3.1.0.Alpha2 to 3.1.0.Beta1 by @dependabot in #532
• Bump version.debezium from 3.1.0.Alpha2 to 3.1.0.Beta1 by @dependabot in #533
• Bump version.debezium from 3.1.0.Beta1 to 3.1.0.CR1 by @dependabot in #534
• Bump io.debezium:debezium-build-parent from 3.1.0.Beta1 to 3.1.0.CR1 by @dependabot in #535
• Add support for Iceberg decimals by @zilder in #537
• Utilize flatten-maven-plugin to allow debezium-server-iceberg to be used as a maven dependency by @matthewmcnew in #538
• Fix docker build should reflect release version by @ismailsimsek in #539
• Bump io.debezium:debezium-build-parent from 3.1.0.CR1 to 3.1.0.Final by @dependabot in #542
• Bump version.debezium from 3.1.0.CR1 to 3.1.0.Final by @dependabot in #541
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.5.2 to 3.5.3 by @dependabot in #536
• Bump org.testcontainers:testcontainers-bom from 1.20.5 to 1.20.6 by @dependabot in #530
• Bump io.smallrye:jandex-maven-plugin from 3.2.7 to 3.3.0 by @dependabot in #540
• Switch using __source_ts_ns for data de-duplicatinon, for upsert mode by @ismailsimsek in #546
• Revert "Bump io.smallrye:jandex-maven-plugin from 3.2.7 to 3.3.0" by @ismailsimsek in #547
• Publish coumentation to github pages by @ismailsimsek in #551
• Fix, Use correct deserializer by @ismailsimsek in #552
• Bump io.debezium:debezium-build-parent from 3.1.0.Final to 3.1.1.Final by @dependabot in #556
• Bump version.debezium from 3.1.0.Final to 3.1.1.Final by @dependabot in #555
• Bump org.testcontainers:testcontainers-bom from 1.20.6 to 1.21.0 by @dependabot in #554

New Contributors

• @zilder made their first contribution in #537
• @matthewmcnew made their first contribution in #538

Full Changelog: 0.9.0.Alpha...0.9.0.Final

Release 0.9.0.Alpha

Notable changes

This release introduces several enhancements and fixes across various areas.

• Enhanced Packaging and Licensing:
  • Maven packaging artifacts are now published to GitHub Packages, ensuring improved distribution and compliance.
  • Packaging has been enhanced with the inclusion of comprehensive third-party licenses.
• Dependency Upgrades and Removals:
  • Debezium has been upgraded to version 3.1.0.Alpha2.
  • Hadoop and PostgreSQL dependencies have been removed, streamlining the project.
• Simplified Configuration Management:
  • Configuration access and management have been centralized through a dedicated config class, utilizing Quarkus ConfigMapping.
• Improved Data Type Handling:
  • Enhanced data type handling now supports datetime, timestamp, and date formats. (When time.precision.mode is not dynamic) The connector accurately converts these data types to their Iceberg equivalents.
• Optimized Default Configuration for Iceberg Consumer:
  • Default configuration values have been optimized for the Iceberg consumer, incorporating recommended Debezium settings. These settings are now applied by default.
    • Default time.precision.mode changed to isostring.
    • Default decimal.handling.mode changed to double.
    • debezium.transforms is now set to unwrap with preconfigured unwrap settings. See default values for details. To disable it set debezium.transforms=,

NOTE: Please be aware that these modifications may affect backward compatibility. Thorough testing is strongly recommended before deployment.

What's Changed

• Add third party licenses to packaging by @ismailsimsek in #484
• Add action to Publishing Packages to GitHub Packages by @ismailsimsek in #486
• Fix github maven package deployment, Related to (#486) by @ismailsimsek in #487
• Update Create Release status badge by @ismailsimsek in #488
• Fix a typo in README.md by @cedricbatailler in #490
• Enable IcebergSchemaHistoryTest and add minor improvements to tests by @ismailsimsek in #492
• Set setDefaultTimeout and setDefaultPollInterval of Awaitility by @ismailsimsek in #493
• Disable releasing nexus-staging by @ismailsimsek in #495
• Simplify setting up spark settings, and add ResolvingFileIO to examples by @ismailsimsek in #496
• Remove hadoop dependencies and hadoop catalog by @ismailsimsek in #491
• Fix build warnins by @ismailsimsek in #497
• Improve naming and add code comments for IcebergSchemaInfo.java by @ismailsimsek in #503
• Move SchemaConverter to separate class by @ismailsimsek in #504
• Remove mongodb java driver from dependencies, use upstream driver by @ismailsimsek in #507
• Upgrade debezium to 3.1.0.Alpha2 by @ismailsimsek in #508
• Improve Handling Date fields and values by @ismailsimsek in #505
• Use config class, to centralize config access and management by @ismailsimsek in #510
• Improve the usage of test helper classes by @ismailsimsek in #511
• Use config values to handle data and schema conversion accordingly by @ismailsimsek in #509
• Improve handling Timestamp fields and values by @ismailsimsek in #513
• Simplify case when switch code by @ismailsimsek in #514
• Improve Time handling by @ismailsimsek in #515
• Create Config Class for BatchSizeWait by @ismailsimsek in #518
• Organize Config And Activate recommended default values for iceberg consumer by @ismailsimsek in #517
• Set default Decimal handling mode to double by @ismailsimsek in #519
• Document configuration and default values by @ismailsimsek in #520
• Move batch config to correct class by @ismailsimsek in #521
• Upgrade maven version to 3.9 for Dockerfile by @ismailsimsek in #523
• Remove postgresql test dependency by @ismailsimsek in #524
• Unify test injections by @ismailsimsek in #528
• Bump org.awaitility:awaitility from 4.2.2 to 4.3.0 by @dependabot in #525
• Bump org.testcontainers:testcontainers-bom from 1.20.4 to 1.20.5 by @dependabot in #526
• Bump io.smallrye:jandex-maven-plugin from 3.1.8 to 3.2.7 by @dependabot in #500

New Contributors

• @cedricbatailler made their first contribution in #490

Full Changelog: 0.8.2.Final...0.9.0.Alpha

Release 0.8.2.Final

Release 0.8.2.Final

Notable changes

• Debezium upgraded to 3.0.7.Final
• Quarkus and other dependency versions are aligned to upstream Debezium
• Various code improvements added around: python package, java testing and "schema change topic" handling

What's Changed

• Upgrade debezium to 3.x and Simplify dependencies by @ismailsimsek in #455
• Use iceberg-bom dependency and Fix jandex plugin version by @ismailsimsek in #456
• Cache maven packages by @ismailsimsek in #457
• Switch caching to actions/setup-java to cache maven by @ismailsimsek in #458
• Update dist pom, align with Debezium by @ismailsimsek in #461
• corrected directory path by @SiddiqueAhmad in #462
• Bump io.debezium:debezium-build-parent from 3.0.5.Final to 3.0.6.Final by @dependabot in #465
• Bump version.debezium from 3.0.5.Final to 3.0.6.Final by @dependabot in #464
• Update documentation and java version in Installation section by @ismailsimsek in #467
• Simplify Minio container test resource by @ismailsimsek in #468
• Add test using jdbc catalog by @ismailsimsek in #469
• Add test using rest catalog by @ismailsimsek in #470
• Add informative exception message for deduplication by @ismailsimsek in #471
• Improve python lib, add pyproject.toml by @ismailsimsek in #477
• Bump actions/stale from 9.0.0 to 9.1.0 by @dependabot in #478
• Bump io.minio:minio from 8.5.14 to 8.5.17 by @dependabot in #476
• Bump io.debezium:debezium-build-parent from 3.0.6.Final to 3.0.7.Final by @dependabot in #475
• Bump version.debezium from 3.0.6.Final to 3.0.7.Final by @dependabot in #474
• Improve schema change topic handling by @ismailsimsek in #480

New Contributors

• @SiddiqueAhmad made their first contribution in #462

Full Changelog: 0.8.1.Final...0.8.2.Final

Release 0.8.1.Final

Release 0.8.1.Final

What's Changed

• Bump version.debezium from 2.7.3.Final to 2.7.4.Final by @dependabot in #448
• Revert Quarkus to a Debezium compatible version by @panga in #447
• Align groovy dependency with Debezium. Update It to 4.0.17 and switch to new artifact org.apache.groovy by @ismailsimsek in #449
• Upgrade jackson from 2.14.2 to 2.16.2, align with Debezium by @ismailsimsek in #451
• Upgrade iceberg from 1.7.0 to 1.7.1 by @ismailsimsek in #452

New Contributors

• @panga made their first contribution in #447

Full Changelog: 0.8.0.Final...0.8.1.Final

Release 0.8.0.Final

Release 0.8.0.Final

Release 0.7.0.Final

• Debezium: 2.7.2.Final
• Apache Iceberg: 1.6.1
• Quarkus: 3.13.3

Breaking Changes

Equality deletes:

• Now Equality delete files are containing key fields only and they are written with record projection. Now delete files only includes key files, previously delete files were including full delete record.

Notable changes

• Various code improvement added. Now code base is more closely aligned with upstream iceberg-kafka-connect
• Now size of the files generated is defined write.target-file-size-bytes config, previously Long.MAX_VALUE was used
• Iceberg upgraded from 1.6.0 to 1.6.1
• Debezium upgraded from 2.7.0.Final to 2.7.2.Final
• Quarkus upgraded from 3.13.0 to 3.13.3

What's Changed

• Bump version.quarkus from 3.13.0 to 3.13.2 by @dependabot in #400
• Bump io.smallrye:jandex-maven-plugin from 3.2.1 to 3.2.2 by @dependabot in #399
• Bump version.debezium from 2.7.0.Final to 2.7.1.Final by @dependabot in #398
• Bump version.testcontainers from 1.20.0 to 1.20.1 by @dependabot in #396
• Add code improvements by @ismailsimsek in #409
• Improve code, Improve class and method namings by @ismailsimsek in #410
• Improve deduplication code by @ismailsimsek in #411
• Use write.target-file-size-bytes config, previously Long.MAX_VALUE was used by @ismailsimsek in #412
• Fix, Create GenericAppenderFactory respecting to identifier fields. And create it with table properties by @ismailsimsek in #413
• Explicitly set operationId as UUID when creating OutputFileFactory by @ismailsimsek in #414
• Use Key fields and Record Projection for deletes by @ismailsimsek in #408
• Rename variable equalityFieldIds to identifierFieldIds by @ismailsimsek in #415
• Bump org.awaitility:awaitility from 4.2.1 to 4.2.2 by @dependabot in #406
• Bump org.postgresql:postgresql from 42.7.3 to 42.7.4 by @dependabot in #407
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.1 to 3.5.0 by @dependabot in #405
• In case of exception, call writer.abort() to cleanup by @ismailsimsek in #417
• Upgrade debezium and Quarkus to latest minor version by @ismailsimsek in #418

Full Changelog: 0.6.0.Final...0.7.0.Final

Release 0.6.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.13.0

Notable changes

• new config added to control casing of iceberg table names. To change tables names uppercase or lowercase folloing configuration properties could be used, see documentation. debezium.sink.iceberg.destination-uppercase-table-names=true or debezium.sink.iceberg.destination-lowercase-table-names=true
• Quarkus upgraded from 3.12.x to 3.13.x, See Quarkus release notes here

What's Changed

• Add option to change case of destination/iceberg table names by @ismailsimsek in #389
• Bump version.quarkus from 3.12.3 to 3.13.0 by @dependabot in #392
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.5 to 3.3.1 by @dependabot in #395
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.0 to 3.3.1 by @dependabot in #394
• Bump io.smallrye:jandex-maven-plugin from 3.2.0 to 3.2.1 by @dependabot in #393

Full Changelog: 0.5.0.Final...0.6.0.Final