Dashboard for User & Project Metrics

[KastanDay]

• 1. North start metric: documents per project (or even tokens per project).
• 2. North start metric: conversations per project.
• 3. Total number of projects (over time)
• 4. ✅ ⭐️ Number of unique users per project.
• 5. [TODO] ⭐️ Number of projects per user.
• 6. ⭐️ Per project, number of days since last interaction?
• 7. Avg session duration, or number of chat messages per session?
• 8. Pageview per user.
• 9. Per user, number of projects interacted with.Stretch goal:* What was the last user interaction (or screen recording) before they dropoff for >30 days?

[KastanDay]

New metric: Admins vs non-admins.

• Per conversation, was it initiated by a User or Admin?
• Per course, what percentage of conversations were initiated by a User or Admin?

Average number of "turns" per conversation.
Example 4 turn conversation:

1. User: "What is this HW?"
2. Bot: "It's this... xyz."
3. User: "No I meant this"
4. Bot: "Oh ok it's this".

[96feilin96]

New Metrics:

1. Active admin vs active users

2. Bounce rate = (Single-page visits / Total visits) × 100%

3. Cumulative graphs for everything

4. Accuracy of response, user satisfaction --- add satisfaction

5. Admin side of uiuc chat – ask for access

6. Usage patterns= what are the peak hours or peak times

7. Customer segmentation 2- admins and users

8. Market the product ? survey of professors/ student who already use it

[KastanDay]

⭐️ == WORK FOR YOU ALL
👉 == WORK FOR KASTAN

⭐️ Retention per URL? Per project, number of days since last interaction?

✅ Per user email, number of days since last interaction? -- good enough with retention chart. Retention per user.
✅ Active user - track every time a user logs in to a specific course/project.
e.g. Find users who viewed course within 7 days.
✅ bounce rate - session duration after going directly to a page...
✅ Number of unique users per project. - unique user who has pageview on pagename...

👉 /chat feedback from user - thumbs up / report problem
👉 Project created (Properites: Project_id, person_id)
👉 Admins updated (Properties: number of admins, list of admin person_ids, and project_id)
👉 Users updated (Properties: Number of users, list of user IDs, project_id, is_public_project)

[KastanDay]

Finishing work

A/B test the new landing page:

• Bounce rate // Conversion (new users sign up // returning users click on ANYTHING).
• 7 day user retention (return rate)

Clean Dashboard of metrics on Posthog

• Lots of metrics in "Product analytics" https://us.posthog.com/project/46448/insights

[KastanDay]

@KastanDay to do: New metric I want to track: e2e latency of time to first token. From sending a "Chat" to "first token response".