Capture a defensible pre-redesign analytics baseline

[lambduhh]

Capture a defensible pre-redesign analytics baseline

Problem

The 2026 vision commits to redesigning ClojureDocs. Evaluating whether that redesign actually improves things requires a before/after comparison — and a comparison is only as good as the baseline captured beforehand. Today we cannot produce a defensible baseline, for four reasons:

1. Front-end behavioral data only accrues forward. Search terms, result-selection, and event data are not recorded until instrumentation ships (see #84). Anything not captured before the redesign is permanently unrecoverable — the baseline window is closing every week.
2. Core metrics aren't segmentable. Matomo has no custom dimension for authentication state, so "logged-in vs. anonymous" — the split most relevant to a contribution-focused redesign — cannot be measured before or compared after.
3. No snapshot or retention guarantee. Matomo Cloud purges raw data after a retention window (exact setting on idSite=15 unconfirmed). If the redesign ships months out, today's baseline reports may not survive to be compared against.
4. Traffic counts are soft. ClojureDocs' audience skews heavily toward ad-blocker users, so Matomo materially — and probably inconsistently — undercounts. A "before" number that a skeptic can dismiss undermines the whole comparison.

This issue tracks capturing a baseline that is comparable, segmentable, durable, and cross-checked before the redesign lands. It is the measurement companion to the usage investigation in #24; no schema changes.

Current state

Capability	Status
Page views, entry/exit, referrers, session depth, visit trends	Available (Matomo idSite=15, ≥1 year of history)
Content & contribution history (examples, notes, contributors, timestamps)	Available (MongoDB; server-side, ad-blocker-immune, historical)
Site search / result-selection / full-search events	Written, pending merge + runtime verification (#84)
Segment by logged-in vs. anonymous	Not measurable — no auth_state custom dimension
Segment by page type / library	Not measurable — no page_type / library dimension
Conversion goals (contributed, logged in, search → var page)	Not configured
Timeline annotations (ship dates, conferences, redesign launch)	Not used
Baseline durability / export	Unconfirmed — retention window unknown, no snapshot taken
Ad-blocker undercount factor	Unquantified — no server-log cross-check

What's missing to make before/after defensible

Framed as capabilities, for whoever picks this up to design:

• A coarse authentication-state segment so every metric can be split logged-in vs. anonymous (no PII).
• A small, fixed set of conversion goals mapping to stakeholder-legible value, with frozen definitions so before/after is apples-to-apples.
• A baseline snapshot/export taken before the redesign, plus confirmation of the Matomo retention window.
• A server-log cross-check (nginx) to quantify the Matomo undercount, contingent on those logs being retained.
• A discipline of annotating the Matomo timeline at each significant event, so the eventual before/after chart is self-documenting.

Open questions

• Attribution. A big-bang redesign on a single site yields correlation, not clean causation — seasonality, conferences, Clojure releases, or a novelty bump can move the same numbers. What is the most credible design available: a long enough baseline to model seasonality, segment-level lift, a staged/canary rollout, or a brief parallel run of old and new? Which of these the new architecture even permits is unknown.
• Scope discipline. This architecture is slated for replacement (per project note); instrumentation added now is throwaway code on a soon-retired site. Which capabilities are worth building for the baseline alone, and where does added tracking stop earning its keep?
• Retention. What is the actual data-retention setting on idSite=15, and does it force an export deadline?

References

• #24 — the usage-analytics baseline investigation this measures
• #84 — search/event tracking (starts the front-end baseline clock)
• #43 — entity-attribute model (the redesign track)
• 2026 vision — the redesign this baseline will be measured against