fs2-kafka v0.20.0-M2

Changes

• Add fs2-kafka-vulcan for integration with vulcan. (#156)

Released on 2019-07-12.

fs2-kafka v0.20.0-M1

Overview

This release introduces several breaking changes to binary- and source-compatibility.
Note binary- and source-compatibility is not guaranteed between milestone releases.

Changes

• Add ConsumerSettings#withAllowAutoCreateTopics and withClientRack. (#153)
• Add producer with support for Kafka transactions. Thanks @danxmoran! (#128, #130, #144, #146)
• Add support for Scala 2.13. (#151, #154)
• Change Deserializer to support deserialization effects. (#120, #122, #131, #135)
  • Deserializer[A] is now Deserializer[F[_], A] with Sync[F].
  • ConsumerSettings[K, V] is now ConsumerSettings[F[_], K, V].
  • For Java Kafka Deserializer interoperability, refer to the documentation.
• Change KafkaAdminClient to use a blocking ExecutionContext. (#141)
  • Changed to Blocker as part of #150.
• Change Serializer to support serialization effects. (#118, #119, #122, #136)
  • Serializer[A] is now Serializer[F[_], A] with Sync[F].
  • ProducerSettings[K, V] is now ProducerSettings[F[_], K, V].
  • For Java Kafka Serializer interoperability, refer to the documentation.
• Change default auto.offset.reset to none in ConsumerSettings. (#121)
• Change to expose more Java Kafka type aliases. (#133)
• Change to parameterize AdminClientSettings on the effect type. (#125)
  • AdminClientSettings is now AdminClientSettings[F[_]] with Sync[F].
• Change to remove the concept of messages. (#144)
  • Rename ProducerMessage to ProducerRecords.
  • Rename CommittableMessage#committableOffset to offset.
  • Rename CommittableMessage to CommittableConsumerRecord.
• Change to simplify produce and commit with produce pipe. (#146)
  • Change KafkaProducer to no longer be sealed.
  • Remove KafkaProducer#producePassthrough.
  • Remove most batch commit pipes: commitBatch, commitBatchF, commitBatchOption, commitBatchOptionF, commitBatchChunk, commitBatchChunkF, commitBatchChunkOption, commitBatchChunkOptionF, commitBatchWithinF, and commitBatchOptionWithin, commitBatchOptionWithinF.
• Change to support creation effect for serializers. (#142)
• Change to use Blocker over ExecutionContext in settings. (#150)
  • ConsumerSettings#executionContext is now instead blocker.
  • ConsumerSettings#withExecutionContext is now withBlocker.
  • Similar changes to ProducerSettings and AdminClientSettings.
• Prevent mixing consumer group ids in the same CommittableOffsetBatch. (#148)
• Remove Header#headers. (#147)
• Remove functions for creating ExecutionContexts. (#145)
  • Removes the following functions: consumerExecutionContextResource, consumerExecutionContextStream, producerExecutionContextResource, producerExecutionContextStream, adminClientExecutionContextResource, and adminClientExecutionContextStream.
• Remove serialization support for org.apache.kafka.common.utils.Bytes. (#139)
• Remove the ProducerMessage[F].of syntax. (#134)
  • There is also no ProducerRecords[F].of after rename in #144.

Updates

• Update FS2 to 1.1.0-M1. (#149)
• Update Kafka to 2.3.0. (#153)

Documentation

• Add initial consumer documentation. (#138)
• Add initial producer documentation. (#152)
• Add initial transactions documentation. (#152)

Internals

• Change KafkaConsumerActor to not store records internally. (#129)
• Change to use internal WithConsumer construct. (#124)

Released on 2019-07-04.

fs2-kafka v0.19.9

Changes

• Change to handle commits during rebalancing. Thanks @CremboC! (#117)

Released on 2019-04-03.

fs2-kafka v0.19.8

Additions

• Add initial debug logging using SLF4J. Thanks @backuitist! (#108, #113)

Changes

• Change to only pause/resume partitions when necessary. (#112)
• Change to only start polling after streaming has started. Thanks @Krever! (#110, #114)
• Change to revoke previous duplicate fetch and issue warning log. Thanks @backuitist! (#107)
• Fix race condition which could cause duplicate records. (#111)

Released on 2019-04-02.

fs2-kafka v0.19.7

Changes

• Fix to include state changes during poll when handling records. (#109)

Released on 2019-03-29.

fs2-kafka v0.19.6

Changes

• Fix a race condition which could result in duplicate records. Thanks @backuitist! (#105, #106)

Released on 2019-03-28.

fs2-kafka v0.19.5

Changes

• Fix Acks#toString and AutoOffsetReset#toString. (#103)

Updates

• Update fs2 to 1.0.4. (#99)
• Update Kafka to 2.2.0. Thanks @CremboC! (#104)

Miscellaneous

• There is now a Gitter room for the library.

Released on 2019-03-27.

fs2-kafka v0.19.4

Additions

• Add improved support for unkeyed records. Thanks @ranjanibrickx! (#96, #97)
  • Add Deserializer#option, and Deserializer.option and unit.
  • Add HeaderDeserializer#option, and HeaderDeserializer.option and unit.
  • Add Serializer#option, and Serializer.option, asNull, empty and unit.
  • Add HeaderSerializer#option, and HeaderSerializer.option, asNull, empty and unit.

Released on 2019-03-01.

fs2-kafka v0.19.3

Additions

• Add functions for working with consumer offsets. Thanks @backuitist! (#92, #93)
  • Add KafkaConsumer#assignment.
  • Add KafkaConsumer#position.
  • Add KafkaConsumer#seekToBeginning.
  • Add KafkaConsumer#seekToEnd.
• Add Attempt[A] aliases for deserializers. (#95)
  • Add Deserializer.Attempt[A] = Deserializer[Either[Throwable, A]].
  • Add HeaderDeserializer.Attempt[A] = HeaderDeserializer[Either[Throwable, A]].

Released on 2019-02-27.

fs2-kafka v0.19.2

Additions

• Add describeCluster and createTopics to KafkaAdminClient. Thanks @danxmoran! (#88)
• Add maxPrefetchBatches to ConsumerSettings. (#83)
  • Controls prefetching behaviour before backpressure kicks in.
  • Use withMaxPrefetchBatches to change the default setting.
• Add several constructs for working with record headers. (#85)
  • Add HeaderDeserializer for deserialization of record header values.
  • Add HeaderSerializer for serializing values to use as header values.
  • Add Header.serialize for serializing a value and creating a Header.
  • Add Header#headers for creating a Headers with a single Header.
  • Add Header#as and attemptAs for deserializing header values.
  • Add Headers#withKey and alias apply for extracting a single Header.
  • Add Headers#concat for concatenating another Headers instance.
  • Add Headers#asJava for converting to Java Kafka-compatible headers.
  • Add Headers.fromIterable to create Headers from Iterable[Header].
  • Add Headers.fromSeq to create Headers from Seq[Header].
• Add several constructs for working with record serialization. (#85)
  • Add a custom Serializer to make it easier to create and compose serializers.
  • Add a custom Deserializer to make it easier to create and compose deserializers.
  • Add ProducerSettings.apply for using implicit Serializers for the key and value.
  • Add ConsumerSettings.apply for using implicit Deserializers for the key and value.

Changes

• Change to make fs2.kafka.Id public. Thanks @chenharryhua! (#86, #87)

Updates

• Update Kafka to 2.1.1. Thanks @sebastianvoss! (#90, #91)

Documentation

• Add a technical details section explaining backpressure. Thanks @backuitist! (#82, #84)

Released on 2019-02-22.