Custom Zabbix-Metrics not visible in Skywalking

[bangyogesh]

Hello There,

We are new to Skywalking however used Zabbix in past to monitor some custom metrics.
We are trying to see if those custom-metrics could be visible in Skywalking UI

The configuration done is as below

1. Zabbix-Agent in set Active mode and has following custom-metrics(a.k.a items in Zabbix terminology) defined

   #The linux command is get a random number from the system
   UserParameter=mymetrics.random, od -vAn -N2 -tu2 < /dev/urandom 

2. Zabbix-Agent is setup to point to SKywalking VM

   ServerActive= < IP of Skywalking-Machine >:10051

3. Skywalking application.yaml is setup for zabbix-receiver as below

receiver-zabbix:
selector: ${SW_RECEIVER_ZABBIX:default}
default:
port: ${SW_RECEIVER_ZABBIX_PORT:10051}
host: ${SW_RECEIVER_ZABBIX_HOST:0.0.0.0}
activeFiles: ${SW_RECEIVER_ZABBIX_ACTIVE_FILES:agent}

4. config/zabbix-rules/agent.yaml is configured to fetch the custom-metrics as below

   requiredZabbixItemKeys:

   • mymetrics.random

   metrics:

   • name: mymetrics_random
     exp: mymetrics_random

However the graph in Skywalking-UI shows Zero value where-as the mymetrics.random (the metrics key) has random values.

[wu-sheng]

Have you checked sw_metrics-all-* index(ElasticSearch) or mymetrics_random table? Is there any data in it?

[wu-sheng]

I can see metrics-all-20230203 has records, meaning you have data in the database.
Do you use the correct query range on UI? And do you set the Time Zone correctly?

Open the index, and check the time bucket, which represent the UTC+0(on docker or your service TZ) time for your metrics. BTW, UI uses your local browser TZ.

[bangyogesh]

@wu-sheng Thanks again for the steps.
I have selected date range from 5-Feb to 7-Feb 2023. However I still see the flat line with zero value.

[wu-sheng]

For days, this is downsampling. I am not sure which function you used. But anyway, pick range in precious 60 mins, you should have the peaks, if database says so.

[bangyogesh]

@wu-sheng ,
Here is the graph of the custom-metrics collected today

1. Random Number generated by Linux command using following configuration in zabbix-agent's conf file
   UserParameter=mymetrics.random, od -vAn -N2 -tu2 < /dev/urandom

2. Epoch calculated in seconds by Linux command using following configuration in zabbix-agent's conf file
   UserParameter=mymetrics.timestamp, data +%s
   

The flat-line of Random Number metrics is showing zero-value whereas its supposed to be changing randomly. Changing the timezone also do not have any impact on Random Number metrics.

[wu-sheng]

I can't tell from Zabbix part, but as I mentioned, you could check the database, to see whether there are values for your metrics. If 0 in the database, it means you really reports 0.

One more tip, we don't recommend double in the database, if you push numbers between 0 and 1 as double, it would be 0 eventually.