Sprint - Nov 25 - December 6

[Phanatic]

Global Sprint Planning

3 things that might take us down

1. EOY on call schedule
2. Django thrashing (@benjackwhite assemble djangovengers)

Urgent incident follow-ups

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red


### High priority

-

### Low priority / side quests

-

[Phanatic]

Team Surveys

Retro

✅ Adaptive sampling for collecting survey responses.
✅ Surveys bug fixes
✅ Feature preview release of no-code experiments
✅ Documentation for no-code experiments.

---

OKRs

1. Implement Enterprise-ready features for Surveys
2. Increase adoption of Surveys.

High priority

Features:

1. Surveys: Show progress of survey input #25432
2. Surveys: Add a new interaction to the survey to re-order a list #25477

Investigation

• Surveys via emails.

[pauldambra]

Team Really Excellent People Loving Angular (Yikes!)

most in the whole sprint

Support hero: @ioannisj

Retro

• @pauldambra very happy with progress on tricky bugs, switching from sprint goal feels the right decision
• sad that @richard-better left but sending 💖 since he'll see this notification
• @marandaneto support hero full on, plus on-call, plus 10-15 bug fixes... lots of context switching
• @ioannisj giving love to bugs

Mobile

• 🟡 iOS network instrumentation and logs Refactor network logs instrumentation for session replay posthog-ios#204 Network bodies for network logs posthog-ios#159 @ioannisj
• 🟡 flutter replay alpha and getting feedback @marandaneto
  • parked
• 🟡 react native person profiles @marandaneto
  • android, ios and flutter are done already
  • RN draft PR
  • need to figure out correct behaviour - there are inconsistencies in how web does it

Replay

• 😵 a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra
• ✅ finish replay URL denylist and then trigger replay on event @richard-better

Low priority / side quests

• ✅ get deadclicks into heatmaps @pauldambra
• ✅ start the UX refresh @pauldambra
• ✅ Noticed that some unit tests are flaky when run multiple times. This may suggest some bugs with concurrency (in either the testing instrumentation or SDK itself. Will try to look more into this @ioannisj
• 🔄 long sessions on mobile - Session Id didn't rotate after 23 hours posthog-ios#227. @ioannisj
• ✅ Deal with this wireframe issue Wireframe Session Recording appears mangled posthog-ios#231 @ioannisj
• ✅ android crashing on compose (specific version and user) @marandaneto

OKRs 🟡

1. OKR, status (red/yellow/green) and action points if yellow/red

🟡📱Goal 1: People think of PostHog as a mobile solution
🟡 📹 Goal 2: Replay 4000
- this is now so excellent it's Replay 4000
- #26098
🟡 🎤 Goal3: Ship the things users keep asking for

• queryable network data
• start replay on trigger e.g. event
• magic playlists

High priority

Mobile

• Capture network headers and body (assuming new instrumentation is in place by then) Network bodies for network logs posthog-ios#159 @ioannisj
• 🪲 long sessions on mobile - Session Id didn't rotate after 23 hours posthog-ios#227. @ioannisj
  • still not sure what's causing this but quite a few reports
• get people using iOS autocapture @ioannisj and @marandaneto
  • ask for help from marketing? blog post? socials?
  • find customers that asked for it in the past?
• flutter replay alpha and getting feedback @marandaneto
• react native person profiles @marandaneto

Replay

• replay 4000 low hanging fruit @pauldambra
  • we want to switch back to more complicated backend things soonest

Low priority / side quests

• Capture iOS logs for recordings Capture logs for recordings posthog-ios#258 @ioannisj
• Enforce max session duration Enforce max session duration  posthog-ios#245 @ioannisj
• a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra
• (can we|do we) measure watch time and buffer time? @pauldambra
• go through support and find people who wanted block list and event trigger @pauldambra
• android min session duration (cost control) @marandaneto
  • waiting on external contributor

[daibhin]

Team Stack Trace Wrangler Demanglers

@neilkakkar off for 3 weeks

Retro

✅ Deploy cymbal end-to-end @oliverb123
Cymbal now serving all non-Sentry exceptions traffic
Fixed most obvious papercuts. Carrying over work on data quality

🟢 In-app stack trace @daibhin
Redesigned stack trace view. Uploading source maps in-app is on track
Investigated adding and built code context to each frame

OKR

🟢 Human readable stack traces for Javascript
🟢 Ensure scalability of query time issue grouping
⚪️ Alerting via CDP
⚪️ Launch

High priority

Cymbal stack frame data quality @oliverb123 @daibhin

• Missing function names, stack cache misses, lines without function declaration / bugs
• Delete / Replace symbol sets

Cymbal Python support @oliverb123

• Add raw frame type
• Tag frame with language in SDK @daibhin
• Processing logic (using line context from SDK)

Update frontend querying and UI @daibhin

• Update queries to rely on new $exception_fingerprint and $exception_issue_id properties
• Surfacing unresolved frame errors to debug symbol set resolution
• Issue merging via overrides table

Low priority / side quests

Migrate to Argo for deployments and external secret management (ft. infra) @oliverb123
First instance alerting via Hog functions @daibhin
Investigate build time source map uploading @daibhin

[jurajmajerik]

Team Experiments 🧪

Retro

• 🟡 Implemented exposures chart for Trends experiments, parked it due to pending migration (enabled for PostHog team only) @danielbachhuber
• 🟢 Shipped holdouts to all customers + added documentation @danielbachhuber
• 🟢 Added credible intervals for secondary metrics @danielbachhuber
• 🟡 Started data warehouse integration @danielbachhuber
  • Need to join data warehouse rows with feature flag called events to get variants for the breakdown
  • Blocked by some bugs
  • Many thanks to Tom and Eric for their help with debugging
• 🟢 Completed the HogQL migration @jurajmajerik
  • Refactored the experiments logic
  • Finished the code migration of the frontend
  • Migrated the US and EU records
• 🟢 Minute level date picker for Insights - important for exploring experiment results as an Insight @neilkakkar

Sprint goals

• Ship an MVP of the data warehouse integration and validate with a customer (maybe DW team knows relevant customers?) @danielbachhuber
• Plug improved statistical calculations for Trends (simple counts) + write docs - @danielbachhuber, @jurajmajerik
• Implement a stacked delta chart @jurajmajerik
• Improve metrics - allow multiple, make them shareable/reusable @jurajmajerik

[mariusandra]

Team Cringe Dinosaur Podcast

• Support: Marius

Retro

• CDP focus on ad conversion platforms integrations
  • @mariusandra client side destinations will be ready by end of week
  • @benjackwhite support issues and distracted by work in the US
  • @oliverb123 error tracking (and incidents)
  • @MarconLP built functions for most server side ads destinations
  • @annaszell first growth review

OKRs

• Widespread usage of the new CDP
• Messaging V1
• Ingestion is rock solid

Sprint focus

High priority

• @oliverb123 team error tracking
• @benjackwhite client->posthog->destination event mapping system for ad destinations + infra
• @mariusandra support + mapping
• @MarconLP building and testing client side destinations, E2E tests
• @annaszell action items from growth review (better metrics, churn analysis), user interviews

[EDsCODE]

Team Data Storehouse

OKR Q4 2024

Objective

Query 3000

• Key Results:
  • Autocomplete
  • Increase general BI experience/product BI TODOs meta#157
  • Declutter the data warehouse UI and make the features intuitive to find

Data Modeling MVP

• Key Results:
  • Infrastructure decided and implemented
  • Integrating external data with feature flags
  • External data everywhere in insights/persons/cohorts
  • Get billing team to use modeling in posthog for their invoices_with_annual table

Retro

• Release dashboard insight variables @Gilbert09
• BigQuery source release to everyone @Gilbert09
• Reduce temporal-data-warehouse pod size (memory) @Gilbert09
  • Became main focus + starting a rebuild of import pipeline that’s much simpler
• SQL editor 3000 MVP @EDsCODE
  • most of this will be good for internal use by friday
• SPMC Batch Exports @tomasfarias
  • backed up by incident
• Splitting persons batch exports @tomasfarias
  • Not going to do. Initially wanted to split these queries but Tim’s query performance improvement of persons exports works well
• HogQL ambiguous query error when joining tables with the same field names @rossgray
• dataviz tests @rossgray
• view validation @rossgray
• One of: Backfill UI, test connection button, or sessions/groups batch exports @rossgray

Priority

• Data import rebuild @Gilbert09
• SQL editor 3000 finish MVP section + move to new priorities @EDsCODE
• Single producer multiple consumer Batch Exports @tomasfarias
• Query cancellation @rossgray
• Autoscaling based on metrics @rossgray @tomasfarias

[dmarticus]

Team Flags

Support

Starting support hero rotation, Dylan will go 11-18 to 11-25, Haven will take 11-25 to 12-2 (dylan supporting as needed).

Retro

Things we got done

• Breakage from lack of type safety in List logic
  • shipped customer-facing issue that our types didn't really help us catch
  • fixed issue, fixed types around this domain so that we don't get bitten by this thing going forward
• Made changes to metrics (tracking creation_context) and Sentry dashboard + issue assigning
• RFC proposed on Feature Management changes sidebar UI

---

Sprint

High priority

• testing/productionizing new /flags service @dmarticus
  • functional testing via django-compat integration tests
  • "real" testing via mirroring traffic to new service (meeting with ingestion team 11-20 to go through this)
  • generally ironing out service (centralizing error handling, reducing memory usage overhead with smarter copying/cloning)
• Start work on feature management UI @havenbarnes
• land v0 of flag staleness feature @havenbarnes
• Track feature flag called events in usage reports so we can track up who set up flags successfully @dmarticus

Side Quests

• Continue to make sentry more actionable and useful @havenbarnes
• Figure out how to cut off the flags bit of /decide (and, in future /flags) when folks go over – right now we don't stop serving flags when overage happens @dmarticus
• Continue to look at growth metrics and see what other properties we can add to better understand our growth @dmarticus

Things to chew on

• Can we know when people are starting to make new features?

OKRs

1. Ship new feature flags services (/flags) in production and accomplish the 10x scale + 2x speed boost performance targets (ideally decreases infra spend by N% as well, will try to figure out a more specific target going forward)
2. Maintain current /decide service and make improvements where necessary.

[fuziontech]

Team ClickHouse

OKR Q2 2024

Objective

Goals for Q4:

• Improve elasticity and flexibility of our data store by putting all our data in Iceberg
  • Work with Altinity effectively to ship read path for Iceberg on ClickHouse - Brett Hoerner
  • Setup infrastructure to ship all of our data to Iceberg on S3 - James Greenhill
  • Shipping Query logs to Iceberg - James Greenhill
• Continue improving CH operational expertise
  • Upgrade to a later version of Clickhouse - James Greenhill
  • Capacity planning - James Greenhill
  • Automation - Daniel Escribano
  • Put some of the basic mitigation operations in runbooks - Daniel Escribano
• Schema management
  • Tool for schema migration (coordinator schemas) - Daniel Escribano
  • Tool for long running mutations - Daniel Escribano
• Continued investment in performance
  • Tooling for other teams to understand which queries are slow and why - Ted Kaemming
  • Investigate variability of queries - Ted Kaemming
  • Per-team limits on queries/query complexity (needs product work) - Ted Kaemming

Board

https://github.com/orgs/PostHog/projects/85/views/2

Retro

🔥 Incidents 🔥

• #inc-2024-11-14-clickhouse-eu-replication-issues
• #inc-2024-11-11-batch-exports-backlog-in-us

Last Sprint's Goals

• Continue 24.8 upgrade and test fix @fuziontech ✅
• Ongoing Delete tasks (support) @fuziontech ✅
• Remove unused Materialized Columns @tkaemming ⌛
  • Dropping indexes is timing out, need to figure out why that's happening 🤔
• Reshard US into 8 shards 3 replicas w/ coordinators @Daesgar ⏳
  • Already in 2024 for partitions - might be complete this week! Definitely next week.
  • Only 1 replica currently
• Snapshot querying of Iceberg tables proof of concept @bretthoerner @fuziontech ❌
  • Working on _timestamp table for now which meets the needs of BE today
• Fix maintenance tasks such as snapshot pruning @bretthoerner ✅
  • Was a bunch of spark tunables that are in TF and have been working for a week
• Document pyspark/EMR runbook stuff in general before 🛫 @bretthoerner ✅

This Sprint's Goals

🦃 day plans

@Daesgar Will cover Wed, 🦃-day, Friday
@tkaemming & @fuziontech will be in the desert frying a 🦃
@bretthoerner - Enjoying Funemployment (Last day is Wednesday)

TL;DR

• Validate cluster data parity between old and new clusters @fuziontech
• Upgrade clusters to 24.8 (US, EU, new-US, Dev) @fuziontech @Daesgar
• Anticipating new 24.8 cluster w/ real lightweight deletes - fix deletes @fuziontech
• Migrate to new 8 shard lower cost cluster w/ coordinator nodes @Daesgar
• Investigate how to speed up index dropping for Mat Cols @tkaemming
• Create new users for different workloads and isolate resources based on users @fuziontech
• (low priority) Expose Iceberg table for testing @fuziontech
  • Current version this will be quite slow
  • New 24.8 version will be quite fast
• Better caching of object store bits for Iceberg (Working in CH codebase) @bretthoerner
  • Code does a HEAD request for every file even if it's in cache for files that don't change

[aspicer]

Team Product Analytics

🎉 Peter Joining Dec 9th!
Support hero: @thmsobrmlr @anirudhpillai

Q4 2024 objectives

Legacy 0x - Thomas owning

• All insights are query-based
• rm -rf posthog/queries/
• Insight query is stored in the URL for instant sharing of unsaved insights

First-class environments - Michael owning

• Environments are rolled out to everyone

Threshold-based alerts - Anirudh owning

• All users get to set up alerts on PostHog metrics
• Alerts are launched with Marketing, and positioned into paid plans
• We're using alerts ourselves for key trends and funnels

Flexible funnels - Sandy owning

• UDF funnels rolled out to everyone, unlocking arbitrary complexity
• Allowing parts of funnels to be unordered. No more separation between regular, unordered, and strict funnels
• Optional funnel steps

10x onboarding and ease of discovering features - Anirudh and Sandy owning

• We run an exercise in implementing PostHog in a hobby project. What stands out as painful or unclear?

User delight 2x - support rotation owning

• Support is 1 hero + 1 sidekick
• Every week support folks ship those small requested features (see list of requests so far)
• Goal: 90% of tickets fulfill the SLA

B2B analytics research - Anna owning

• Understand the market and user needs better in order to improve our current Group analytics product next quarter

Retro

Anirudh (Support Nov 11 - 15)

• 🟢 Alerts currently rolled out to 20% of users, sprint goal to roll out to 100%
  • Stability Fixes
• 🟢 RFC for Saved Filters / Saved HogQL Snippets
  • PR up for experimenting with different approaches
• 🟢 (Unplanned) Added trends with breakdowns to alerts
• (Reprioritized) Add slack messaging alerts
• (Side Quest - Repriotized) Prometheus push gateway for gathering metrics from Celery

Thomas

• 🟢 Filter Removal (Dashboard templates)
• 🟢 Custom colors on funnels / trends
• 🟢 Chart reloading when disable breakdowns

Sandy (Out Nov 11, Support Nov 18 - 22)

• 🟢 Password protection (for third party shares) Password protection for Shared Insights/Dashboards #10869
• (On Hold - Thomas investigating if it overlaps with his work on event metadata) Investigate getting date first seen onto a person Track and allow graphing users/groups by "date first seen" #11952 Allow filtering by "Date first seen" #3031

Upcoming Sprint

Anirudh (Support second week)

• Alert Comms / Launch
• Support for launch / fast-follow features for alerts
• Cached dashboards (Cached dashboards #26309)

Thomas (Support First Week)

• Migrate api endpoints to use hogql queries
• Look into filters removal broadly
• Fix lifecycle charts with personless events

Sandy

• Finish Cohort Migration
• Spec out funnel improvements for December Optional funnel steps #17313
• Plan Peter's Onboarding, find tasks

Anna

• Group Analytics Research, generate plan for Q1

[benjackwhite]

Team Infrastructure

Support hero: @danielxnj

Retro

• 🟢 Fine tune our instance types to match our deployment workloads (cost optimisation to utilise the most of a node) @danielxnj
  • Most of the heavy stuff has been moved to dedicated nodepools, capture-replay coming up
• 🔴 100% do the reservation changes ASAP - @benjackwhite
  • This was blocked by the previous
• 🟡 Optimize the LB/envoy deployments to remove all cross AZ traffic (cost optimization) @MichaelKutsch-ph
  • Lots of slow moving parts to delicately get this deployed and running with AZ specific deployments
  • Tested out the ALB changes but didnt have the cost change we expected, investigating further why that is
• 🟢 Tune the Warpstream dedicated node pools @danielxnj
• 🟡 ArgoCD - move lowest priority posthog services over (cdp, temporal-workers, cymbal) @bciaraldi
  • Getting Cymbal up and running worked well
  • Property defs deployed with Argo
  • A bunch of stuff missing on our internal cluster to make this stuff super robust
• (Will potentially update some outcomes of the offsite later)

OKR

1. 🏃 Continuous Delivery
2. 💰 Cost visualization
3. 🧪 Continuous Integration

High priority

• Get our plan together reservations @danielxnj
• Lockdown our AWS pricing @MichaelKutsch-ph
• Follow up with the contour cross AZ cost work @MichaelKutsch-ph
• Lock in WS @benjackwhite
• Get internal cluster into the same state as our prod clusters (monitoring stack, SSO access etc.) @danielxnj
• ArgoCD @bciaraldi
  • Fix up the cluster networking
  • Get PostHog web (and workers) onto ArgoCD

Low priority / side quests

[zlwaterfield]

Team Growth

Retro

Retro items

• @zlwaterfield
  • RBAC
    • Get access models shipped
    • Get base middleware implemented
    • Pick first product and get the base RBAC setup for it
  • Get Teams Trials fully live
  • Ship lowering of self serve credits requirements
  • Finish and ship project limits API checks
  • Onboarding for Surbhi
  • if time:
    • (from last time) Make a few improvements to the technical exercise (based on using it for 2 weeks)
    • (from last time) Get the in-app credit card entry experiment shipped
    • Review existing billing metrics, hookup to billing on call, see if we should add others
    • customer io transaction api
    • More billing metrics for on call
• @patricio-posthog
  • Continue with forecasting bugs, trying to understand what's happening and how to solve it.
  • (From last time): Turn off daily invoice sync to rely 100% on webhooks.
  • Try to optimize /billing endpoint, maybe separating forecasting into its own endpoint (after fixing it and understand how it works)
    Extra things if there is time:
  • Start to analyze forecasting and see its accuracy after each period.
• @raquelmsmith
  • Get product analytics dashboard templates out again
  • follow up on replay templates / reengagement campaign
  • Daniel in-person onboarding
  • Multi-product-select in onboarding (with Daniel)
• @surbhi-posthog
  • onboarding
  • first 3 prs - sign up page bugs

Q4 Goals

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

1. 🟡 Get to a 4/10 in Enterprise platform support (Zach)
   • 🟡 Release Enterprise addon and iterate with sales to find a pricing model that makes sense
   • 🟡 Build RBAC
2. 🟡 Make revenue & forecasting more reliable (Patricio)
   • 🟡 Rejig how we do sync_invoices so we don’t see big drops - respond to webhooks instead of large syncs
   • ⚪ Evaluate temporal vs celery for usage reports and billing tasks so we can adhere to rate limits better
   • ⚪ Side hustle, if time permits: Rejig how anonymous / identified events are billed, as the current implementation is confusing.
3. 🟡 Activation: reverse trials, analytics onboarding flow, understand CC activation (Raquel)
   • ⚪ Dig into the data, watch replays to figure out why CC activation rate has been falling
   • ⚪ Add reverse trial to onboarding
   • ✅ Add reverse trial to billing page for teams

This sprint

Time off: @raquelmsmith, @zlwaterfield and @surbhi-posthog in Palm Springs for offsite.

• @zlwaterfield
  • RBAC
    • Get middleware PR tested and shipped
    • Get RBAC UI shipped for 1 product
  • Improve how we show value to teams addon customers when unsub'd or cancelling trial (think about how to do this for all products)
  • Talk to DW about materialzed views
  • Talk to Mine/Joe about better startup plan labels
  • Continue talking to Stripe PM about billing problems we are running into
  • Continue move to c.io transactional emails
• @surbhi-posthog
  • Review exiting ICP research
  • Connect with Mine about ICP data being sync'd into PostHog
  • Do some analysis on our metrics through signup, onboarding, converting to paid based on our ICP scoring data
  • Suggest (and maybe implement) changes to the onboarding product experience based on learnings from the analysis.
• @patricio-posthog
  • Forecasting: 1 - Pato: 0. Still fighting this battle.
  • New pricing for queries (API) and mobile

[robbie-c]

Team web analytics

Support hero: @robbie-c + SDK support hero

Retro

Mike --> Rafael

Created SDK Support Hero: rota RFC

Tasks

🔴 Finish cookieless prototype
🔴 Make a start on page reports / core web vitals
🟢 Custom channel types in queries
🟢 Fancy UI for custom channel types

OKR

• Continue to improve performance
• Build a small amount of more advanced marketing features, to measure interest
• Cookieless tracking
• Improve the NextJS experience
• More page-level insights
• Improve the documentation, and link to it more from the app

High priority

Robbie

• Onboard Rafael
• Support hero
• Finish cookieless prototype

Stretch goals

Mike

• Web Analytics: Change period to make onboarding easier #26003

[Twixes]

Team AI PM

Retro

High priority

• 🟢 Results summarization
• 🟢 Generalize Session Replay's OpenAI data processing opt-in (to be determined with @fraserhopper) @Twixes
• 🟢 Stream feedback on the agent's current task in conversation @Twixes – initial experience is released
• PostHog dashboard for AI product manager use @Twixes – deprioritized in favor of quality
• 🟢 Tune Trends generation for more relevant results (smarter application of breakdowns, better property value examples) @skoob13 – work in progress, but good progress
• 🟡 Handle errors (agent recursion limit + graceful handling of e.g. ClickHouse errors) @skoob13

Low priority / side quests

• 🟢 Results evaluation framework - a benchmark @skoob13 – prioritized due to quality focus
• Environments out as a feature preview @Twixes – deprioritized in favor of AI PM quality

OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

High priority

• Fix response streaming in production @skoob13 with @PostHog/team-infra
• Make summarizer self-aware when we couldn't fully answer the question, and why @Twixes
• All other classic insights types (order of importance: retention, paths, stickiness, lifecycle) @Twixes
• Max AI rolled out as feature preview @PostHog/team-ai-product-manager
• Memory, prewarmed by analyzing existing data and definitions @skoob13
• SQL generation MVP @skoob13

Low priority / side quests

• In your product's website, out a complete description of the product for the agent's context
• Environments out as a feature preview @Twixes