Add usage analytics tracking for stakeholder reporting

[gordonkoehn]

Problem

We need to demonstrate W-ASAP adoption to stakeholders (grant reports, justifying infrastructure). Current monitoring (Prometheus/Grafana) covers operational metrics but not business analytics like unique users, returning visitors, or query patterns.

Classical tracking tools (Google Analytics etc.) won't work - our user base is technical and heavy on ad blockers.

Requirements

• Track daily/weekly/monthly unique users
• Identify returning users (not just daily uniques)
• Track which LAPIS endpoints get used and query parameters (mutations, locations, date ranges)
• Long-term storage for quarterly/yearly reporting (Prometheus retention is 2 weeks)
• No cleartext PII storage
• Simple, maintainable, fits existing Ansible-managed infrastructure

Proposed Approach

Two-layer setup:

1. Operational monitoring (existing): Prometheus + nginx-exporter for real-time dashboards - request rates, latencies, errors. Keep as-is.

2. Business analytics (new):

   • Custom nginx log format capturing endpoint, query params, and a hashed user ID cookie
   • nginx sets a persistent cookie on first visit (no application changes needed)
   • Simple Python script (cron, hourly) parses logs → writes to SQLite
   • Grafana queries SQLite for reporting dashboards

Why this approach

• Server-side tracking bypasses ad blockers
• nginx cookie is lightweight, no LAPIS modifications needed
• SQLite is zero-ops (no separate DB to manage), sufficient for our scale
• Separation of concerns: Prometheus stays short-term operational, SQLite handles long-term analytics
• Ansible-native: all components (nginx config, Python script, cron, SQLite path) can be templated and version-controlled

Implementation outline

1. Add custom nginx log format to nginx role
2. Add cookie-setting logic in nginx for returning user tracking
3. Create new analytics role: Python script + cron + SQLite setup
4. Add Grafana dashboard for usage reporting

[gordonkoehn]

@DrYak, about tracking - my rough idea.

[gordonkoehn]

Consider Plausible.