v3.4.2

What's changed

• chore(deps): bump com.google.cloud:google-cloud-nio from 0.127.36 to 0.128.5 (#540)
• chore(deps): bump com.google.cloud:google-cloud-storage from 2.52.3 to 2.58.0 (#541)
• chore(deps): bump com.azure:azure-storage-blob from 12.30.0 to 12.31.3 (#542)
• Apply the externalId to STS AssumeRoleRequests (#563)
• Fix missing decompression processing in AbstractSourceRecordIterator (#554)
• Reduce CPU when idle (#548)
• Update to readme document to show how to process any files with a regex (#551)
• Add 'native.start.key' configuration optoion to allow source connectors skip to specific native key on start up (#549)
• Add back off timer to polling code to stop constant polling when data is not available.
• Introduces a configuration object to allow s3-source users to select a starting object key
• Migrate gcs sink (#531)
• Migrate azure sink (#528)
• Move s3 source integration (#525)
• Move commons code to io.aiven.commons (#524)
• Migrate RingBuffer to aiven-commons directory and code (#532)
• Add config documentation for S3 sink (#518)

Co-authored by

• Aindriú Lavelle
• Claude Warren
• dependabot[bot]
• Ryan Skraba

Full Changelog

v3.4.1...v3.4.2

v3.4.1

What's Changed

• bump commons-io:commons-io from 2.18.0 to 2.19.0
• Fix NPE in STS client usage by @Claudenw in #510

Full Changelog: v3.4.0...v3.4.1

v3.4.0

What's Changed

• Implement Azure blob source
• Multiple version updates:
  • com.google.cloud:google-cloud-storage from 2.48.0 to 2.52.3
  • com.google.cloud:google-cloud-nio from 0.127.30 to 0.127.36
  • com.azure:azure-storage-blob from 12.29.0 to 12.30.0
  • Apache Parquet to version 1.15.1
  • org.mockito:mockito-junit-jupiter from 5.14.2 to 5.18.0
• Added automatic documentation generation by @Claudenw in #437
• Update FormatType with writer builder by @Claudenw in #467
• Update CompressionType with i/o methods and update associated tests by @Claudenw in #463
• Removed Outputformat from source config by @Claudenw in #474
• Added Fragment Setters. by @Claudenw in #480

New Contributors

• @davide-armand made their first contribution in #469

Full Changelog: v3.2.0...v3.4.0

Release v3.3.x does not exist.

v3.2.0

What's changed:

Common changes

• Various dependency upgrades
• All tests migrated AssertJ assertions
• Update README.md with information about semantic versioning. by @Claudenw in #320
• Update to CONTRIBUTING by @aindriu-aiven in #352
• Common source task assignment by @aindriu-aiven in #345
• Common config with fragments by @aindriu-aiven in #344
• Utility to extract variables from a rendered template by @RyanSkraba in #353
• Don't upload zip file from tar location by @afreysenacor in #371
• ensure timestamp is stable for a record by @mkeskells in #321
• Clarify javadocs for OjectDistributionStrategy by @RyanSkraba in #355

Aiven's S3 sink connector for Apache Kafka

• Removed unnessary checks in S3 sink by @Claudenw in #341
• Adding information for handling S3 Sink multipart upload aborted parts by @aindriu-aiven in #385

Aiven's GCS sink connector for Apache Kafka

• Fixed GcsSinkTask logger by @AnatolyPopov in #348
• gcs: Migration to java.time.Duration by @AnatolyPopov in #347

Aiven's Azure Blob Storage sink connector for Apache Kafka

• chore: fix Azure Blob sink connector artifacts upload by @AnatolyPopov in #310
• Implement KIP-898: Add Service Loader to Azure Blob Sink Connector by @aindriu-aiven in #377

Aiven's S3 source connector for Apache Kafka

• This release introduces new Aiven's S3 source connector for Apache Kafka

New Contributors

• @mkeskells made their first contribution in #322
• @afreysenacor made their first contribution in #371

Full Changelog: v3.1.0...v3.2.0

v3.1.0

What's changed:

Common changes

• Various dependency upgrades
• chore: fixed CodeQL GH action by @AnatolyPopov in #304
• Update README.md by @Claudenw in #307

Aiven's S3 sink connector for Apache Kafka

• No connector specific changes

Aiven's GCS sink connector for Apache Kafka

• No connector specific changes

Aiven's Azure Blob Storage sink connector for Apache Kafka

• This release introduces new Aiven's Azure Blob Storage sink connector for Apache Kafka

Full Changelog: v3.0.0...v3.1.0

v3.0.0

This release aggregates the the work previously done in separate repositories for AWS S3 Connector and GCS Connector. From now on this repository will contain both connectors as well as commons library. Old AWS S3 Connector and GCS Connector repositories are now archived.

What's changed:

Common changes

• Added metadata descriptor to enable Apache Kafka Connect use the plugin discovery mechanism #285
• Various dependency upgrades
• All Gradle scripts migrated to Kotlin

Aiven's S3 connector for Apache Kafka

• Added support for setting SSEA name Aiven-Open/s3-connector-for-apache-kafka#296
• Use default or custom provider chain for AWS S3 authentication Aiven-Open/s3-connector-for-apache-kafka#355
• The produced release artifact is now renamed to s3-sink-connector-for-apache-kafka as well as produced jar
  file is renamed to s3-sink-connector-for-apache-kafka

Aiven's GCS connector for Apache Kafka

• Added support for automatic discovery of default credentials Aiven-Open/gcs-connector-for-apache-kafka#292
• Added support for setting object metadata Content-Encoding Aiven-Open/gcs-connector-for-apache-kafka#359
• The produced release artifact is now renamed to gcs-sink-connector-for-apache-kafka as well as produced jar
  file is renamed to gcs-sink-connector-for-apache-kafka

v0.12.0

What's Changed

• feat: Add topic-partition-key record grouper by @jeqo in #200

Dependencies

• build(deps): bump org.xerial.snappy:snappy-java to 1.1.10.5 by @dependabot in #202
• build(deps): bump com.github.luben:zstd-jni to 1.5.5-6 by @dependabot in #203

Full Changelog: v0.11.0...v0.12.0

v0.11.0

What's Changed

• Add support for topic/partition in KeyRecordGrouper by @stephen-harris in #167
• feat: add key-based and max records validation by @jeqo in #193

Dependency bumps

• hadoopVersion from 3.3.5 to 3.3.6
• parquetVersion from 1.11.1 to 1.11.2
• org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.3
• com.github.luben:zstd-jni from 1.5.5-4 to 1.5.5-5

New Contributors

• @stephen-harris made their first contribution in #167

Full Changelog: v0.10.1...v0.11.0

v0.10.1

Fixes:

• fix(build): remove kafka-clients from distribution by @jeqo in #176

Dependency upgrades:

• build(deps): bump org.xerial.snappy:snappy-java from 1.1.10.0 to 1.1.10.1 by @dependabot in #169
• build(deps): bump com.github.luben:zstd-jni from 1.5.5-3 to 1.5.5-4 by @dependabot in #171

Full Changelog: v0.10.0...v0.10.1

v0.10.0

What's Changed

• fix: Moving from avro-converter to avro-data dependency.

Other improvements

• Multiple dependency upgrades

Full Changelog: v0.9.0...v0.10.0