Suppress consecutive identical logs

[jiemingwang]

Description

Currently POSIX (with klog) logs all consecutive identical events. This can generate huge amount of logs. Need to suppress such repeated logs (with count=N or something similar in the log).

Here is an example:
time=2026-06-16T10:30:18.237-04:00 level=WARN msg="handler error" origin=ctlog.example.com/2026h1 name=AddChain error="Too Many Requests"
time=2026-06-16T10:30:18.239-04:00 level=WARN msg="handler error" origin=ctlog.example.com/2026h1 name=AddChain error="Too Many Requests"

[jiemingwang]

I hit a different but similar issue, below is a log snip. Essentially it generated > 30k logs / second. This will ab a big problem for operation folks to handle (the log size is 17GB).

root@instance-template-20260525-20260525-210054:/var/log# tail -100000 tesseract.log.1| egrep 'time=2026-07-06T00:32:57'|wc -l
35615

root@instance-template-20260525-20260525-210054:/var/log# tail -100000 tesseract.log.1| egrep 'time=2026-07-06T00:32:57'|head
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.000Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"

root@instance-template-20260525-20260525-210054:/var/log# tail -100000 tesseract.log.1| egrep 'time=2026-07-06T00:32:57'|tail
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"
time=2026-07-06T00:32:57.999Z level=ERROR msg="Failed to commit antispam population tx" error="followFrom 835391652 > size 780631619"

[phbnf]

Hi @jiemingwang thanks for the report. I see that you're mentioning klog, and we've now moved to slog. I assume that you must be using (rightfully!) v0.1.1? We'll cut a release soon (see #896) and this matter should go away.

[jiemingwang]

Hi @phbnf , thanks for the update. This is good news. Will give a try once committed.

Yes I was using slog flag.

I observed 2 scenarios: the first one is related to excessive requests which should be addressed by #896, the second one is related to misconfiguration which was caused antispam size mismatch. Are you going to fix the 2nd one as well ?

[phbnf]

Hello! The second fix won't make it to the release unfortunately. Do you have more information about this though? Did it eventually recover?

[jiemingwang]

Thanks Phil. Fir the 2nd case, I was not able recover, and started over from scratch.