[SWIP-3][DISCUSSION] Support rocketmq monitoring

[peachisai]

Motivation

RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications。Now that Skywalking has the capability to monitor OpenTelemetry metrics, I want to add RocketMQ monitoring via the OpenTelemetry Collector, which fetches metrics from the RocketMQ Exporter

Architecture Graph

There is no significant architecture-level change.

Proposed Changes

rocketmq-exporter collects metrics from rocketmq and transport the data to opentelemetry collector,using skyWalking openTelemetry receiver to receive these metrics。
Provide cluster ,broker and topic dimensions monitoring.

RocketMQ Cluster Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
Messages Produced Today	Count	meter_rocketmq_cluster_messages_produced_today	The number of the cluster messages produced today.	RocketMQ Exporter
Messages Consumed Today	Count	meter_rocketmq_cluster_messages_consumed_today	The number of the cluster messages consumed today.	RocketMQ Exporter
Total Producer Tps	Msg/sec	meter_rocketmq_cluster_total_producer_tps	The number of messages produced per second.	RocketMQ Exporter
Total Consume Tps	Msg/sec	meter_rocketmq_cluster_total_consumer_tps	The number of messages consumed per second.	RocketMQ Exporter
Producer Message Size	Bytes/sec	meter_rocketmq_cluster_producer_message_size	The max size of a message produced per second.	RocketMQ Exporter
Consumer Message Size	Bytes/sec	meter_rocketmq_cluster_consumer_message_size	The max size of the consumed message per second.	RocketMQ Exporter
Messages Produced Until Yesterday	Count	meter_rocketmq_cluster_messages_produced_until_yesterday	The total number of messages put until 12 o'clock last night.	RocketMQ Exporter
Messages Consumed Until Yesterday	Count	meter_rocketmq_cluster_messages_consumed_until_yesterday	The total number of messages read until 12 o'clock last night.	RocketMQ Exporter
Max Consumer Latency	ms	meter_rocketmq_cluster_max_consumer_latency	The max number of consumer latency.	RocketMQ Exporter
Max CommitLog Disk Ratio	%	meter_rocketmq_cluster_max_commitLog_disk_ratio	The max utilization ratio of the commit log disk.	RocketMQ Exporter
CommitLog Disk Ratio	%	meter_rocketmq_cluster_commitLog_disk_ratio	The utilization ratio of the commit log disk per brokerIp.	RocketMQ Exporter
Pull ThreadPool Queue Head Wait Time	ms	meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time	The wait time in milliseconds for pulling threadPool queue per brokerIp.	RocketMQ Exporter
Send ThreadPool Queue Head Wait Time	ms	meter_rocketmq_cluster_send_threadPool_queue_head_wait_time	The wait time in milliseconds for sending threadPool queue per brokerIp.	RocketMQ Exporter

RocketMQ Broker Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
Produce TPS	Msg/sec	meter_rocketmq_broker_produce_tps	The number of broker produces messages per second.	RocketMQ Exporter
Consume QPS	Msg/sec	meter_rocketmq_broker_consume_qps	The number of broker consumes messages per second.	RocketMQ Exporter
Producer Message Size	Bytes/sec	meter_rocketmq_broker_producer_message_size	The max size of the messages produced per second.	RocketMQ Exporter
Consumer Message Size	Bytes/sec	meter_rocketmq_broker_consumer_message_size	The max size of the messages consumed per second.	RocketMQ Exporter

RocketMQ Topic Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
Max Producer Message Size	Byte	meter_rocketmq_topic_max_producer_message_size	The maximum number of messages produced.	RocketMQ Exporter
Max Consumer Message Size	Byte	meter_rocketmq_topic_max_consumer_message_size	The maximum number of messages consumed.	RocketMQ Exporter
Consumer Latency	ms	meter_rocketmq_topic_consumer_latency	Consumption delay time of a consumer group.	RocketMQ Exporter
Producer Tps	Msg/sec	meter_rocketmq_topic_producer_tps	The number of messages produced per second.	RocketMQ Exporter
Consumer Group Tps	Msg/sec	meter_rocketmq_topic_consumer_group_tps	The number of messages consumed per second per consumer group.	RocketMQ Exporter
Producer Offset	Count	meter_rocketmq_topic_producer_offset	The max progress of a topic's production message.	RocketMQ Exporter
Consumer Group Offset	Count	meter_rocketmq_topic_consumer_group_offset	The max progress of a topic's consumption message per consumer group.	RocketMQ Exporter
Producer Message Size	Byte/sec	meter_rocketmq_topic_producer_message_size	The max size of messages produced per second.	RocketMQ Exporter
Consumer Message Size	Byte/sec	meter_rocketmq_topic_consumer_message_size	The max size of messages consumed per second.	RocketMQ Exporter

Imported Dependencies libs and their licenses.

No new dependency.

Compatibility

no breaking changes.

General usage docs

This feature is out of box.

[wu-sheng]

The general guide and dependencies should be empty, as this feature is out of box.

And you should list accepted metrics from OTEL, and metrics available for UI for various levels.

[peachisai]

Done

[wu-sheng]

You don't import new lib, I think. RocketMQ as a part of ASF is safe for sure, my point is, it's not a dependency.

[peachisai]

I got it

[wu-sheng]

👍 I am good for this proposal.

@wankai123 Please recheck.

[wankai123]

LGTM.

[wu-sheng]

@peachisai Could you polish this proposal? #11778 brings new metrics to this dashboard, right?

[peachisai]

@peachisai Could you polish this proposal? #11778 brings new metrics to this dashboard, right?

Okay, i will

[wu-sheng]

And this proposal should be landed in the repo docs now. Add the link to SWIP page and bump up the next SWIP ID.

[peachisai]

@wu-sheng I edited the descriptions for size and offset. Could you take a look and let me know if they are okay?

[wu-sheng]

It should be good. Just make sure the capital of unit is correct. Such as ms, TPS(represents for transactions per second), QPS... Right now, you seem to use first letter in upper case only and as always.

[peachisai]

@wu-sheng Ok,I did the re-Edited

[wu-sheng]

Final version could be found https://skywalking.apache.org/docs/main/next/en/swip/swip-3/