Metadata on consumer can present UNKNOWN_TOPIC_OR_PART error

[sshanks-kx]

Running .kfk.Metadata[client]; from consumer can present callback msg as follows:

mtype                 topic client partition offset msgtime data                                                                        key      headers                 rcvtime                      
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UNKNOWN_TOPIC_OR_PART test1 0      0         -1001          "Subscribed topic not available: test1: Broker: Unknown topic or partition" `byte$() (`symbol$())!`symbol$() 2024.01.22D15:46:39.829536000

when doing this after doing a subscribe, it can make it look like the subscription had failed (when it hasn't).

Temp removing .kfk.Metadata[client]; after any subscription, the callback no longer fires with a UNKNOWN_TOPIC_OR_PART. It doesn't appear to stop the subscription from occurring, whether its there or not.

Will find out why & change this effect

[sshanks-kx]

Occurs every time .kfk.Metadata[client] called after subscription is made (e.g. call twice, get 2 UNKNOWN_TOPIC_OR_PART msgs).
Doesn't occur when called prior to any subscription.

[sshanks-kx]

Created librdkafka issue confluentinc/librdkafka#4589 (can see same occurrence without using kdb+).
Current workaround is that if you need to use .kfk.Metadata to see topics/etc, call it prior to subscription.

[sshanks-kx]

Update now on linked issue. May be issue with librdkafka. To try possible fix.

[sshanks-kx]

Tested with the pending librdkafka pull request - bug still appears to be there. Informed librdkafka dev. Current workaround is to do any metadata request prior to subscription.

[sshanks-kx]

bug doesnt occur with v2.0.2 of librdkafka. will await response from librdkafka devs.

[sshanks-kx]

looks like its working again on latest pr change to librdkafka (issue linked above). tested with kdb+. will await future librdkafka release to see if it makes it in.

[sshanks-kx]

Fixed in underlying librdkafka 2.4.0