Increase Matomo Sampling to 100%, Eliminate load for bot request user-agents

[mekarpeles]

Feature Request

Problem / Opportunity

Right now we're forced to down-sample to 1% of of traffic for anonymized analytics.

We're losing out on the ability to measure how often searches are successful.

Drastically decreasing matomo/athena loads for bad bot traffic so we can turn matomo sampling to 100%

Proposal

1. First, we will need to update our nginx js rules to flag/label requests we want to exclude.
2. We will use this label + known user-agents to avoid loading matomo + athena.js
3. We will update matomo rates with @scottbarnes so 100% of traffic is samples.

Breakdown

Related files

Refer to this map of common Endpoints:

• https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/openlibrary/code.py#L1223-L1288
• https://github.com/internetarchive/openlibrary/pull/10840/changes#diff-d239b09d34ba084e956069ed3302fbb496c1fe0a864054c404c1dbb1be60d2d7R22 from PR Feature: add matomo #10840 i.e. ‎openlibrary/templates/site/head.html

is_bot from https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/openlibrary/code.py#L1321-L1348 is already a web.ctx global

We do something similar, i.e. $if not is_bot(): in

openlibrary/openlibrary/templates/site/alert.html

Lines 1 to 2 in 11caa2a

	$if not is_bot():
	$:render_template('site/donation_banner')

.

[Aayushdev18]

@mekarpeles This looks like a valuable improvement. I’d love to help with triage, implementation, or testing especially around bot detection and sampling changes.

[mekarpeles]

@Aayushdev18 are you on slack? If not, I can send you an invite you you send me your email to mek@archive.org. Let's coordinate there and we can find a good issue to collaborate on! This one may require some staff work and I have a PR already for part of it.

[Aayushdev18]

Thanks a lot! @mekarpeles I’m not on Slack yet. I just mailed you with my details

[padhle-yrr-adi]

@mekarpeles hyy, i m a new contributor here....can you plz assign this to me and help me out?