v4.1.0-RC1

The highlight of this release is the consumer parallelization control contributed by @cmcmteixeira in #1493 (and #1501). The feature enables bounded parallelism using partition groups, effectively enabling bounded memory usage. This is particularly important when the number of assigned partitions is large (in the hundreds). The description in #1493 contains more details on the feature.

What's Changed

• Update sbt, scripted-plugin to 1.12.11 by @typelevel-steward[bot] in #1485
• Add KafkaConsumer#settings and KafkaProducer#settings by @vlovgr in #1487
• Refine result type of the withSerializers in the KafkaProducer.WithSettings by @iRevive in #1488
• Update kafka-avro-serializer, ... to 8.2.1 by @typelevel-steward[bot] in #1490
• Update scalafmt-core to 3.11.1 by @typelevel-steward[bot] in #1491
• Update slf4j-api to 2.0.18 by @typelevel-steward[bot] in #1492
• Update sbt-typelevel, sbt-typelevel-site to 0.8.6 by @typelevel-steward[bot] in #1495
• Update munit, munit-diff to 1.3.1 by @typelevel-steward[bot] in #1494
• Feat/add parallelization control to consumer by @cmcmteixeira in #1493
• Revert "Feat/add parallelization control to consumer" by @vlovgr in #1500
• Feat/add parallelization control to consumer by @cmcmteixeira in #1501
• Update scala3-library to 3.3.8 by @typelevel-steward[bot] in #1497
• Update sbt, scripted-plugin to 1.12.12 by @typelevel-steward[bot] in #1498
• Update munit, munit-diff to 1.3.3 by @typelevel-steward[bot] in #1499

New Contributors

• @iRevive made their first contribution in #1488

Full Changelog: v4.0.0...v4.1.0-RC1

v4.0.0

Both the fs2-kafka and vulcan projects have now moved to the Typelevel GitHub organization.
Thanks a lot to @armanbilge for all the help in getting these projects setup in their new home.

⚠️ For v4.x of fs2-kafka, the artifacts are now published under the org.typelevel organization.

libraryDependencies += "org.typelevel" %% "fs2-kafka" % "4.0.0"

Apart from the Java Kafka client upgrade from 3.x to 4.2, the biggest changes in fs2-kafka v4.x include the following.

• A redesign of CommittableOffset and CommittableOffsetBatch in #1471.
  • These are now safe by construction, supporting multiple consumers, and handle out-of-order offsets.
• A greatly simplified version of transactional producers by @cmcmteixeira in #1481.
  • See the v4.0.0-RC2 release notes for a detailed list of changes.
• Also thanks to @cmcmteixeira, the KafkaAdminClient has gotten a lot more methods in #1482.

What's Changed

• Prepare for 4.x release with updates by @vlovgr in #1469
• Drop sealed requirement on KafkaProducerConnection by @calvinlfer in #1202
• Change to merge versioned binary-compatibility traits by @vlovgr in #1470
• Change CommitTimeoutException to be a RetriableCommitFailedException by @vlovgr in #1472
• Update vulcan to 1.13.0 by @vlovgr in #1473
• Redesign CommittableOffset and CommittableOffsetBatch by @vlovgr in #1471
• Change Timestamp to provide value and type by @vlovgr in #1474
• Change to Typelevel organization by @vlovgr in #1475
• Update sbt, scripted-plugin to 1.12.9 by @typelevel-steward[bot] in #1476
• Update scalafmt-core to 3.10.7 by @typelevel-steward[bot] in #1477
• Merge KafkaProducer traits for binary compatibility by @vlovgr in #1480
• Feat/merge transactional regular producer by @cmcmteixeira in #1481
• Exposed new methods from the underlying kafka admin client through th… by @cmcmteixeira in #1482
• Update munit, munit-diff to 1.3.0 by @typelevel-steward[bot] in #1483
• Update scalafmt-core to 3.11.0 by @typelevel-steward[bot] in #1484

New Contributors

• @calvinlfer made their first contribution in #1202
• @typelevel-steward[bot] made their first contribution in #1476
• @cmcmteixeira made their first contribution in #1481

Full Changelog: v3.9.1...v4.0.0

v4.0.0-RC2

Thanks to @cmcmteixeira we've greatly simplified transactional producers. (#1481)
The transactions documentation has been updated and following are migration notes.

• TransactionalKafkaProducer, TransactionalProducerSettings, and KafkaProducerConnection have been removed.
  • The functionality from TransactionalKafkaProducer is now available on KafkaProducer instead.
    • The TransactionalKafkaProducer#produce method exists as KafkaProducer#produceAndCommitTransactionally.
    • The TransactionalKafkaProducer#produceWithoutOffsets method exists as KafkaProducer#produceTransactionally.
  • The following methods have been added to ProducerSettings.
    • The ProducerSettings#withTransactionId method sets the transactional.id property.
    • The ProducerSettings#withTransactionTimeout methods sets the transaction.timeout.ms property.
  • The KafkaProducerConnection#withSerializers method is available as KafkaProducer#withSerializers instead.
• Add KafkaProducer.transactional and transactionalStream to automatically initialize the producer for transactions.
• Add the following functions to KafkaProducer for more manual control of transactions.
  • KafkaProducer#initTransactions for initializing the producer for transactions.
    • This will be called automatically when using KafkaProducer.transactional or transactionalStream.
  • KafkaProducer#transaction for managing a transaction through a Resource.
    • Note the deadlock when opening a second transaction within a first transaction.
  • KafkaProducer#sendOffsetsToTransaction for including offset commits in a transaction.
    • Add KafkaConsumer#groupMetadata to support KafkaProducer#sendOffsetsToTransaction.
    • The method is also available in the consumer traits as KafkaSubscription#groupMetadata.
• The KafkaProducer.Metrics and KafkaProducer.PartitionsFor traits have merged into KafkaProducer. (#1480)
  • If you had references to these traits, then change these to KafkaProducer instead.

Also thanks to @cmcmteixeira, the KafkaAdminClient has gotten a lot more methods. (#1482)

⚠️ Please note this release does not maintain binary or source compatibility with v4.0.0-RC1.

There are no further changes planned for v4.0.0. Please give the release candidate a try and if no issues are encountered, it will become the final v4.0.0 release.

What's Changed

• Update sbt, scripted-plugin to 1.12.9 by @typelevel-steward[bot] in #1476
• Update scalafmt-core to 3.10.7 by @typelevel-steward[bot] in #1477
• Merge KafkaProducer traits for binary compatibility by @vlovgr in #1480
• Feat/merge transactional regular producer by @cmcmteixeira in #1481
• Exposed new methods from the underlying kafka admin client through th… by @cmcmteixeira in #1482

New Contributors

• @typelevel-steward[bot] made their first contribution in #1476
• @cmcmteixeira made their first contribution in #1481

Full Changelog: v4.0.0-RC1...v4.0.0-RC2

v4.0.0-RC1

Both the fs2-kafka and vulcan projects have now moved to the Typelevel GitHub organization.
Thanks a lot to @armanbilge for all the help in getting these projects setup in their new home.

⚠️ For v4.x of fs2-kafka, the artifacts are now published under the org.typelevel organization.

libraryDependencies += "org.typelevel" %% "fs2-kafka" % "4.0.0-RC1"

Apart from the Java Kafka client upgrade from 3.x to 4.2, the biggest change in fs2-kafka v4.x is a redesign of CommittableOffset and CommittableOffsetBatch in #1471. These are now safe by construction, supporting offsets from multiple consumers and are also handling out-of-order offsets.

Please give the release candidate a try and if no issues are encountered, it will become the final v4.0.0 release.

What's Changed

• Prepare for 4.x release with updates by @vlovgr in #1469
• Drop sealed requirement on KafkaProducerConnection by @calvinlfer in #1202
• Change to merge versioned binary-compatibility traits by @vlovgr in #1470
• Change CommitTimeoutException to be a RetriableCommitFailedException by @vlovgr in #1472
• Update vulcan to 1.13.0 by @vlovgr in #1473
• Redesign CommittableOffset and CommittableOffsetBatch by @vlovgr in #1471
• Change Timestamp to provide value and type by @vlovgr in #1474
• Change to Typelevel organization by @vlovgr in #1475

New Contributors

• @calvinlfer made their first contribution in #1202

Full Changelog: v3.9.1...v4.0.0-RC1

v3.9.1

What's Changed

• Update kafka-clients to 3.9.1 in series/3.x by @scala-steward in #1412
• Update sbt-typelevel, sbt-typelevel-site to 0.8.2 in series/3.x by @scala-steward in #1433
• Update kind-projector to 0.13.4 in series/3.x by @scala-steward in #1432
• Update sbt, scripted-plugin to 1.11.7 in series/3.x by @scala-steward in #1430
• Update fs2-core to 3.12.2 in series/3.x by @scala-steward in #1424
• Update cats-effect, cats-effect-laws, ... to 3.6.3 in series/3.x by @scala-steward in #1420
• Update kafka-avro-serializer to 7.9.4 in series/3.x by @scala-steward in #1428
• Update scalafmt-core to 3.9.10 in series/3.x by @scala-steward in #1427
• Update scala-library to 2.13.17 in series/3.x by @scala-steward in #1429
• Update scala3-library to 3.3.6 in series/3.x by @scala-steward in #1407

Full Changelog: v3.9.0...v3.9.1

v3.9.0

What's Changed

• feat: Implemented RebalanceRevokeMode by @wookievx in #1399

New Contributors

• @wookievx made their first contribution in #1399

Full Changelog: v3.8.0...v3.9.0

v3.8.0

What's Changed

• Update fs2-core to 3.12.0 in series/3.x by @scala-steward in #1400
• Update cats-effect, cats-effect-laws, ... to 3.6.1 in series/3.x by @scala-steward in #1402
• Update vulcan to 1.12.0 in series/3.x by @scala-steward in #1401
• Update scalafmt-core to 3.9.6 in series/3.x by @scala-steward in #1408
• removed retries value override by @Awethon in #1405

New Contributors

• @Awethon made their first contribution in #1405

Full Changelog: v3.7.0...v3.8.0

v3.7.0

There was an issue earlier today which prevented v3.7.0 from being released properly.
The issue has now been fixed. Please enjoy the original release notes by @aartigao.

---

This release includes internal changes to KafkaConsumer brought by @biochimia and @rodrigo-molina (thank you very much for your contributions! 🎉 ) as well as some bug fixes (welcome @ex-ratt and @eugkhp as new collaborators 🫂).

We've released two candidates that, after running some time in our production systems, we think we're in safe position for a new release 🥳 🚀

What's Changed

• Update cats-effect, cats-effect-laws, ... to 3.5.7 in series/3.x by @scala-steward in #1363
• Update sbt, scripted-plugin to 1.10.6 in series/3.x by @scala-steward in #1365
• Update kafka-avro-serializer to 7.7.2 in series/3.x by @scala-steward in #1361
• Update kafka-clients to 3.9.0 in series/3.x by @scala-steward in #1362
• chore: improve flaky test by @biochimia in #1370
• Update testcontainers-scala-kafka, ... to 0.41.5 in series/3.x by @scala-steward in #1371
• Update kafka-avro-serializer to 7.8.0 in series/3.x by @scala-steward in #1372
• Update sbt, scripted-plugin to 1.10.7 in series/3.x by @scala-steward in #1373
• Update sbt-typelevel, sbt-typelevel-site to 0.7.5 in series/3.x by @scala-steward in #1374
• chore: update copyright notices by @biochimia in #1377
• Instantiate underlying Kafka Consumer inside blocking context by @eugkhp in #1364
• Update scala-library to 2.13.16 in series/3.x by @scala-steward in #1382
• chore: make state transitions uncancelable by @biochimia in #1367
• Update scala3-library to 3.3.5 in series/3.x by @scala-steward in #1386
• Simply KafkaConsumer.commitAsync by @rodrigo-molina in #1375
• Update scalafmt-core to 3.8.6 in series/3.x by @scala-steward in #1388
• Update logback-classic to 1.3.15 in series/3.x by @scala-steward in #1379
• Update kafka-avro-serializer to 7.8.1 in series/3.x by @scala-steward in #1390
• Update testcontainers-scala-kafka, ... to 0.41.8 in series/3.x by @scala-steward in #1385
• Update sbt, scripted-plugin to 1.10.10 in series/3.x by @scala-steward in #1392
• Update sbt-typelevel, sbt-typelevel-site to 0.7.7 in series/3.x by @scala-steward in #1389
• Fix exeption of multiple promise completions (#1391) by @ex-ratt in #1393
• feat: drop use of FetchRequests by @biochimia in #1369
• chore: tweaks to handling of pending commits by @biochimia in #1368
• Update testcontainers-scala-kafka, ... to 0.43.0 in series/3.x by @scala-steward in #1394
• Update kafka-avro-serializer to 7.9.0 in series/3.x by @scala-steward in #1395
• Update sbt, scripted-plugin to 1.10.11 in series/3.x by @scala-steward in #1396
• Update scalafmt-core to 3.9.4 in series/3.x by @scala-steward in #1397
• fix: reset spillover after poll loop by @biochimia in #1398

New Contributors

• @eugkhp made their first contribution in #1364
• @ex-ratt made their first contribution in #1393

Full Changelog: v3.6.0...v3.7.0

fs2-kafka v3.7.0-RC2

What's Changed

• Update testcontainers-scala-kafka, ... to 0.43.0 in series/3.x by @scala-steward in #1394
• Update kafka-avro-serializer to 7.9.0 in series/3.x by @scala-steward in #1395
• Update sbt, scripted-plugin to 1.10.11 in series/3.x by @scala-steward in #1396
• Update scalafmt-core to 3.9.4 in series/3.x by @scala-steward in #1397
• fix: reset spillover after poll loop by @biochimia in #1398

Full Changelog: v3.7.0-RC1...v3.7.0-RC2

fs2-kafka v3.7.0-RC1

This is a pre-release for testing internal changes around KafkaConsumer introduced by @biochimia / @rodrigo-molina.

What's Changed

• Update cats-effect, cats-effect-laws, ... to 3.5.7 in series/3.x by @scala-steward in #1363
• Update sbt, scripted-plugin to 1.10.6 in series/3.x by @scala-steward in #1365
• Update kafka-avro-serializer to 7.7.2 in series/3.x by @scala-steward in #1361
• Update kafka-clients to 3.9.0 in series/3.x by @scala-steward in #1362
• chore: improve flaky test by @biochimia in #1370
• Update testcontainers-scala-kafka, ... to 0.41.5 in series/3.x by @scala-steward in #1371
• Update kafka-avro-serializer to 7.8.0 in series/3.x by @scala-steward in #1372
• Update sbt, scripted-plugin to 1.10.7 in series/3.x by @scala-steward in #1373
• Update sbt-typelevel, sbt-typelevel-site to 0.7.5 in series/3.x by @scala-steward in #1374
• chore: update copyright notices by @biochimia in #1377
• Instantiate underlying Kafka Consumer inside blocking context by @eugkhp in #1364
• Update scala-library to 2.13.16 in series/3.x by @scala-steward in #1382
• chore: make state transitions uncancelable by @biochimia in #1367
• Update scala3-library to 3.3.5 in series/3.x by @scala-steward in #1386
• Simply KafkaConsumer.commitAsync by @rodrigo-molina in #1375
• Update scalafmt-core to 3.8.6 in series/3.x by @scala-steward in #1388
• Update logback-classic to 1.3.15 in series/3.x by @scala-steward in #1379
• Update kafka-avro-serializer to 7.8.1 in series/3.x by @scala-steward in #1390
• Update testcontainers-scala-kafka, ... to 0.41.8 in series/3.x by @scala-steward in #1385
• Update sbt, scripted-plugin to 1.10.10 in series/3.x by @scala-steward in #1392
• Update sbt-typelevel, sbt-typelevel-site to 0.7.7 in series/3.x by @scala-steward in #1389
• Fix exeption of multiple promise completions (#1391) by @ex-ratt in #1393
• feat: drop use of FetchRequests by @biochimia in #1369
• chore: tweaks to handling of pending commits by @biochimia in #1368

New Contributors

• @eugkhp made their first contribution in #1364
• @ex-ratt made their first contribution in #1393

Full Changelog: v3.6.0...v3.7.0-RC1