.kfk.Metadata lists a topic even if it's deleted using .kfk.TopicDel

[rpoply1]

Deleting a topic using .kfk.TopicDel doesn't remove that from the list of topics returned by .kfk.Metadata

  client: .kfk.Consumer[`metadata.broker.list`group.id!`localhost:9092`0];
  .kfk.Metadata[client]`topics;
  topic: .kfk.Topic[client; `new_topic; ()!()];
  .kfk.Metadata[client]`topics;
  .kfk.TopicDel topic;
  .kfk.Metadata[client]`topics;
  ```

[cmccarthy1]

Hi @rpoply1

As a general point, This appears to be somewhat of a limitation on the broker side as alluded to here. Although that's not at the heart of what I think the behavioural mismatch is here.

We're not really controlling the removal of the topic from the brokers perspective which is the information returned from .kfk.Metadata rather the function .kfk.TopicDel is removing our control over the global list of topics that your q session has knowledge of.

For context .kfk.Topic creates a new topic associated with the client through invocation of the function rd_kafka_topic_new on librdkafka side which we map to a list of topics that your session has knowledge of, the following code snippet alludes to this

  rkt= rd_kafka_topic_new(rk, y->s, rd_topic_conf);
  js(&topics, (S) rkt); 
  return ki(topics->n - 1);

Essentially this is the reason you're doing the assignment of topic in the call to .kfk.Topic[client;`new_topic;()!()] and using this down the line

In essence the deletion function is destroying the reference mapping the topic handle you created and added to the kdb topics list in C. It does this by both destroying the client from the point of view of your client and removing any association with kafka to the kdb topic list

rd_kafka_topic_destroy(rkt);
kS(topics)[x->i]= (S) 0;

We'll look into if there is something we can invoke to remove the topics from broker side but this would be an addition to the interface rather than an overhaul of the above functions from what I'm aware at this juncture.

[mshimizu-kx]

This is done with a script:

./kafka-topics.sh --bootstrap-server localhost:9092 --topic test1 --delete

Note: that "if you have consumers up and running is that the topic will get auto-created if the cluster-wide property auto.create.topics.enable is true". Actually this property is necessary to create a topic with .kafka.newTopic