Bot traffic (Google Lighthouse) not filtered out

[LuizHAP]

Hi, I noticed a huge spike in views recorded by PostHog. After some digging I found it was the Google Lighthouse crawler crawling my website:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse

I open an issue because according to this question, this is where we should report missing bots.

Thanks!

[rafaeelaudibert]

Hey, Luiz!

We're already investigating this :)
#1779

We're slightly afraid of simply blocking this user agent because it might hide actual traffic from real users, so we're looking for some alternatives - such as reaching out to the people using the Lighthouse crawler and asking them to identify themselves
You can always block it for your own website using the custom_blocked_useragents config

[LuizHAP]

But just so I understand. Was this some recent configuration on Google's side? Is there a way to do some kind of filtering of these users in the current reports? I understand that I can change the configuration, but if there was some way to filter my Insights in Posthog to avoid this type of user, it would be good. It made a big change in our reports and we are suffering from it, because we think they are real users.

[rafaeelaudibert]

@LuizHAP First, thanks for sharing the arcjet team with us :)

Second:

• Google stopped reporting "Lighthouse" in the UA late last year, so that's why we're seeing more problems about this
• You can circumvent this on your dashboard by adding that specific user agent to the "Filter internal and tests users" config, and then turning that on in your settings, this will ignore all those events

[LuizHAP]

Thanks @rafaeelaudibert !