Delays in partition EOF being sent to additional instances

[foxesknow]

Description

I have an application that consume large amount of data from a topic, roundly around 1500 messages a second. The topic has 9 partitions.

When my application starts I join my consumer group and consume all messages until I encounter a partition EOF for all assigned partitions or until there are no messages available. This works without any problem. The reason we need to do this is that the messages contain market data used for trading, so we need to see all the updates that have happened since we last started, but also want to constantly receiving the latest messages as soon as they happen.

However, when I start a second instance of the application something strange happens. The partitions are rebalanced, so that typically Inst-1 has partitions 5-8 and the new instance (Inst-2) gets partitions 0-4. Inst-2 then begins in startup logic of consuming all messages until partition EOF or no more messages are available. The strange thing is that on some of the partitions it takes a long time to encounter a partition EOF (or no messages), sometimes up to 10 minutes! Also, 95% of the time this happens on partition #2.

Bizarrely, if I shut down Inst-1 during this phase then the partitions are all assigned to Inst-2 and it does get the partition EOF messages for each partition is a timely manner.

It feels like this is some sort of consumer group leader issue. However, I'm wondering if my expectations around partition EOF are wrong and that I shouldn't expect it to behave this way. If so then what is the best way to ensure you consume all messages up to the EOF (or equivilant)?

I'm using version 1.7 of the C# Kafka libraries. We're not able to use the 1.8 version due to issue around SSL authentication with the library.

[mhowlett]

what is the SSL authentication issue?

this may be an issue and may be resolved by changes in v1.9.0-RC10, i'd need to look over the debug logs to work it out. first thing to do is try the latest version.

[foxesknow]

what is the SSL authentication issue?

this may be an issue and may be resolved by changes in v1.9.0-RC10, i'd need to look over the debug logs to work it out. first thing to do is try the latest version.

We're using .P12 files for SSL authentication. It works fine with 1.7, but fails in 1.8 and the 1.9 RC.

I specify the path to the P12 file in the SslKeystoreLocation property on the consumer config. However, when I start my application using the 1.9.0-RC10 nuget package it says:

Error reading ssl.keystore.location PKCS#12 file: path-to-file.p12: error:0D06B08E:asn1 encoding routines:asn1_d2i_read_bio:not enough data

This isn't an issue with the 1.7 version of the library.

[edenhill]

What OS?
Can you try something like openssl pkcs12 -info path-to-file.p12 and see if there are any issues reported with that?

[foxesknow]

What OS?

It's Windows 10

Can you try something like openssl pkcs12 -info path-to-file.p12 and see if there are any issues reported with that?

I don't see any issues. It contains a private encrypted key and 5 certificates.

[mhowlett]

I think your SSL issue is likely this: confluentinc/librdkafka#3999 - it will be in the next release of librdkafka.

If you manage to collect some debug=all logs when this is occurring, that would be helpful. It seems likely this is a real issue to me, though it we should debug it on the latest version as potentially related things have changed in librdkafka.

[RedTail72]

@mhowlett when you say next release, are you referring to v1.9.3 or the version after that? If there is a road map with this info, I apologize as I didn't see it (or overlooked it).

[nhaq-confluent]

Can this thread be closed if the latest version has solved the problem?