ESS Kafka infrastructure and schemas

[SimonHeybrock]

This goes beyond Beamlime. Things to coordinate, and document:

• Heartbeat and status mechanisms.
• Control topic message schema.
• da00: What to include in terms of metadata.
• Standards for (derived) source_name.

Where should this documentation live? In the documentation of a small helper package similar to streaming_data_types?

See also #288.

[ggoneiESS]

Although I am strongly in favour of using a rough schema to define these messages, it should be something we only specify rigidly in a streaming-data-type after hot commissioning, since there might be lots we learn in that process. I think a flat buffer is good, but we could already work within the framework of some (non-strict) JSON-defined schema and use json messages to communicate.

[SimonHeybrock]

I think I asked (you?) before: You are not running/using a schema registry, right? Are there plans for that?

[ggoneiESS]

the only thing I can think you asked in relation to that would be in the PR at ess-dmsc/python-streaming-data-types#102 which you were fine with closing, so if you think there's a use case for it now then perhaps we can reopen that?

[SimonHeybrock]

No, I am talking about running a schema registry for the Kafka infrastructure. I mentioned it on Slack to someone recently.

[ggoneiESS]

not to me - if you mean the Schema Registry from Confluent, it's not compatible with the flatbuffer format is it?

[hurvan]

We are not using the schema registry, and it's not in the plans for the foreseeable future.

But locking down the communication in a flatbuffer schema might be the best solution for standardizing the command (and maybe response) messages between NICOS and Beamlime.