Confuse alert about service_cpm

[yueguiji]

In my case，We set a service_cpm alert rule to zookeeper, and 3 Oap node to read by zk.
like this:
rules: single_service_cpm_310_P4_rule: metrics-name: service_cpm op: '>' threshold: 20 count: 1 period: 1 silence-period: 1 include-names: - xxx:xxx message: '{name}' only-as-condition: true
We think if the cpm over 20 calls/min in xxx:xxx service, the Oap could send a alert to webhook.
but is can not work in this case, in the dashboard we could see the cpm always over 20 calls/min.

So,Why????

[wu-sheng]

Why period 1? That seems not working. You at least need 2 periods to make the last period in the window.

[yueguiji]

Thank for your help，I found some question like my case in the closed issue ，I will try to set 5 on the period.

[mumu123123]

We carefully read the tips you gave, and tried to set the statistical period to 2, 5, and 10 respectively, but none of them achieved the desired effect. Whenever the current statistical period ends, a new alarm will be generated immediately.
Like this: period=2

We checked the data stored in the sw_metrics-cpm index and found that the smallest unit of storage is per minute. At the same time, we made some verifications for this situation. We refreshed every 20 seconds within one minute and found that the last minute The cpm data is not consistent, including calling the API. Therefore, we guess that when the alarm rule is calculated, it will be calculated in real-world units of every minute. In the last minute, the wrong value will be calculated because the data has not been counted temporarily.
Enlarging the period is a good idea to improve the accuracy of the alarm, but it does not solve our pain point. We think whether it is necessary to exclude the last minute from the statistical period, and hope you can teach us how to implement the exclusion operation.
@wu-sheng Sincerely look forward to your reply and suggestions

[wu-sheng]

Last minute is changing, that is the reality. I don't understand your point.

[mumu123123]

I mean, the alarm rule calculates the minute that is changing during the calculation, and the calculated value must be less than the threshold of the configuration, thus triggering the alarm. So is there any configuration that can make the alarm rule not calculate the changing minute. When the minute is all over, it will be calculated according to the expression.

[wu-sheng]

That means your rule is wrong.
Why use cpm to trigger alert? Especially less than N cpm. This doesn't fit the SRE or operation practice.

[mumu123123]

I see. At present, we are just trying to configure the alarm rules of each indicator, without considering whether the configuration rules comply with the specifications. We will pay attention to this later.
Thank you again for your answer.

[mumu123123]

Regarding the alarm rule, we have a new question and would like to seek your help. We have configured an alarm rule for service_sla, set silence-period=0, and the actual index value has always been greater than the configured threshold. However, the sending frequency of the alarm information is not fixed every 1 minute, it will be sent every 2 to 2 minutes periodically, can you give us some advice on how to deal with this situation?
The following is my configuration information:
single_obverse_rule: metrics-name: service_sla op: '>' threshold: 9000 count: 1 period: 2 silence-period: 0 include-names-regex: .*(?:demo::demo).* message: '{name}'

status quo：

@wu-sheng Sincerely look forward to your reply and suggestions

[wu-sheng]

In this long discussion, I noticed, you never use and try to understand the system from a SRE/OPS team perspective, I can't see the point of all these questions.
Very strange feeling about your perspectives of asking these. I have to say.

[mumu123123]

We want to try to realize the recovery function of the alarm. When the same alarm is not received again within a few minutes after receiving the alarm information for the first time, we consider it has been recovered. Otherwise we will keep notifying the person in charge to deal with the incident...

[mumu123123]

In the end, sincerely thank you for your answer. ^_^
BR

[wu-sheng]

We want to try to realize the recovery function of the alarm. When the same alarm is not received again within a few minutes after receiving the alarm information for the first time, we consider it has been recovered. Otherwise we will keep notifying the person in charge to deal with the incident...

This is another thing wrong from SRE perspective. Alerting is for triggering auto or manual involvement, it is never just for recovering. Whether it is recovering should base on your SLI and testing methods. The metrics back to normal don't mean anything positive.

This is the whole point I want to say. SRE and Operation workflows are as same as the APM system capabilities.

[mumu123123]

Got it, thanks for your answer