Sprint - Nov 11 to Nov 22

[marandaneto]

Global Sprint Planning

3 things that might take us down

Urgent incident follow-ups

https://github.com/orgs/PostHog/projects/103/views/2

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red


### High priority

-

### Low priority / side quests

-

[mariusandra]

Team Can Direct Priorities

• Support: Ben

Retro

• @mariusandra Get messaging v0 ready for dogfooding.
  • ⚪ out for testing, but see below (on hold)
• @benjackwhite CDP UI/UX + coming back from holiday (odds/ends)
  • 🔴 catching up took time, no work on CDP, that's for next week
• @oliverb123 Olly 80% on error tracking. Improve live debugging of CDP.
  • 🟢 merged PR for better error handling
  • 🟢 error tracking --> being built
• @MarconLP Support sprint + new destinations
  • 🟢 coming along, being built

OKRs

• Widespread usage of the new CDP
• Messaging V1
• Ingestion is rock solid

Sprint focus

High priority

• CDP focus on ad conversion platforms integrations

[pauldambra]

Team Claire will leave me if I travel in December as well

most in the whole sprint

Support hero: @marandaneto

Retro

• yiannis - Autocapture has been at “almost ready” for too long. 98% of the work was done early and fast. Then some minor refactoring, some slow feedback loops and some indecision from my part for minor details. Also didn’t take documentation into account
• yiannis - Still in a “need-to-ask-permission” mindset that I meed to grow out of. Working on this :)
• yiannis - Been getting more comfortable and faster with community responses and support
• paul - support feels pretty draining at the moment. i went through a bunch of tickets and questions / setup advice is a big chunk of load, will go back through those specifically and see if there's anything we can fix here
• manoel - medical leave after surgery and offsite, just worked on fixing bugs the last and this sprint

---

Mobile

• 🔄 mobile replay billing clarity @pauldambra
  • a little more clarity, RFC written, there's not an obvious route yet
• iOS Autocapture should be out this week 🙌
• lots of small wins @ioannisj can see. Some potential issues:
  • 🔄 $feature_flag_called should be triggered again for an already trigged flag if there are new values posthog-ios#211
  • ✅ Option to add custom timestamp posthog-ios#210
  • 🔄 Document all public declarations posthog-ios#220
• refactor net logs instrumentation @ioannisj
  • Refactor network logs instrumentation for session replay posthog-ios#204
• 🚫 flutter replay alpha and getting feedback @marandaneto
  • no network logs for now
  • medical leave and offsite + fixing bugs across all sdks, did not finish testing/fixing android yet

Replay

• replay URL denylist @richard-better
  • also interrupted by travel
  • feature is working and we're making sure it's releasable
• a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra
  • lots of user support meant i made very little progress on this 😵

Low priority / side quests

• ✅ nobody is lost or arrested in Valencia
• 🔄 trigger to start recordings on event @richard-better
• 🔄 lazy loading replay in the SDK @pauldambra

OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

🟡📱Goal 1: People think of PostHog as a mobile solution
🟡 📹 Goal 2: Replay playback 3000
- delayed-ish till Nov and Cory time in Valencia
🟡 🎤 Goal3: Ship the things users keep asking for

High priority

Mobile

• iOS network instrumentation and logs Refactor network logs instrumentation for session replay posthog-ios#204 Network bodies for network logs posthog-ios#159 @ioannisj
• flutter replay alpha and getting feedback @marandaneto
  • no network logs for now
• react native person profiles @marandaneto
  • android, ios and flutter are done already

Replay

• a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra
• finish replay URL denylist and then trigger replay on event @richard-better

Low priority / side quests

• get deadclicks into heatmaps @pauldambra
• start the UX refresh @richard-better
• Noticed that some unit tests are flaky when run multiple times. This may suggest some bugs with concurrency (in either the testing instrumentation or SDK itself. Will try to look more into this @ioannisj
• Try to identify the root cause of this Session Id didn't rotate after 23 hours posthog-ios#227. @ioannisj
• Deal with this wireframe issue Wireframe Session Recording appears mangled posthog-ios#231 @ioannisj
• android min session duration (cost control) @marandaneto
• android crashing on compose (specific version and user) @marandaneto

[Phanatic]

Team Surveys

Retro

✅ Beta testing no-code experiments with customers
✅ Surveys bug fixes
✅ No-code experiemnts bug fixes
✅ Investigate adaptive sampling for collecting survey responses.
✅ Support queues were split
⌛ Feature preview release of no-code experiments
⌛ Documentation for no-code experiments.

Side Quests

---

OKRs

1. Implement Enterprise-ready features for Surveys
2. Increase adoption of Surveys.

High priority

• Adaptive sampling for collecting survey responses.
• Support no-code experiments.

[jurajmajerik]

Team Experiments

Retro

• 🟢 @danielbachhuber joined the team! 🥳

  • @raquelmsmith is onboarding him in person
  • First PRs in: UI improvements to Experiment goal, onboarding flow, up to 10 secondary metrics
  • Running first experiment - select multiple products in the product screen

• 🟢 [priority_customer]

  • Holdout groups implemented @neilkakkar @jurajmajerik
  • Added support for 10 secondary metrics @danielbachhuber
  • Backend for reusable metrics - 90% done @neilkakkar
  • We have issue where we keep track of any new requests: https://github.com/PostHog/product-internal/issues/667
  • The plan is to over communicate with them in their Slack channel to keep them engaged/gather feedback/show we're proactive.

• 🟡 Continued on HogQL for Experiments @jurajmajerik (Experiments HogQL rewrite #24914)

  • Opted for a large refactor of the Experiments logic
  • Shared components (selectors) with Product Analytics, but tightly coupled with their logic. Convoluted state flows, hard to reason about.
  • Forked these components - we want them to work with our logic
  • We want to be in control of these components - make sure the statistics always work correctly for any goal metrics.
  • This is taking extra time/delaying the HogQL completion, but worthwhile, will enable us to move faster.

• 🟢 Review statistics with a contractor @jurajmajerik

  • Latest feedback incorporated, seem to be done for now - all main metric types supported.
  • Next step: plug these methods into our APIs and write tests
  • Might need some tweak later

• 🟢 Support hero

  • Experiments queue is set up now
  • Workload manageable

Sprint goals

• Complete the onboarding list (mostly UI improvements) @danielbachhuber
• Data Warehouse integration for [priority-customer] - pending information from them - start collecting requirements @danielbachhuber
• HogQL rewrite @jurajmajerik
  • Finish Experiments logic refactor
  • Migrate Experiment records
  • Test behind feature flag
• Support hero @jurajmajerik

[robbie-c]

Team web analytics

Support hero: @robbie-c

Retro

Onboarding Mike, currently doing a hackathon. Cookieless prototype is pretty close but dealing with a couple of edge cases.

Tasks

🟠 Finish cookieless prototype
🟢 Onboard Mike / have a successful offsite
🟢 Finish up the v1 sessions table PR

OKR

• Continue to improve performance
• Build a small amount of more advanced marketing features, to measure interest
• Cookieless tracking
• Improve the NextJS experience
• More page-level insights
• Improve the documentation, and link to it more from the app

High priority

Mike

• Finish Allow customers to customize the Channel Type calculation #21195
• UX improvements to the graphs, Web Analytics: "previous" is unclear #26000 and Web Analytics: tooltips to see Persons on the graphs and world map are hard to click #25999
• Improve goals query UX, Web Analytics: Add better UI for choosing which actions / events are shown in the Goals query #26004 and Bug: Deleted Actions continue to appear in Web analytics Goals #25978

Robbie

• Finish cookieless prototype Feature request: Improve cookieless tracking #25117
• Make a start on page reports / core web vitals

Stretch goals

Mike

• Web Analytics: Change period to make onboarding easier #26003

[fuziontech]

Team 🧊 Data

OKR Q2 2024

Objective

Goals for Q4:

• Improve elasticity and flexibility of our data store by putting all our data in Iceberg
  • Work with Altinity effectively to ship read path for Iceberg on ClickHouse - Brett Hoerner
  • Setup infrastructure to ship all of our data to Iceberg on S3 - James Greenhill
  • Shipping Query logs to Iceberg - James Greenhill
• Continue improving CH operational expertise
  • Upgrade to a later version of Clickhouse - James Greenhill
  • Capacity planning - James Greenhill
  • Automation - Daniel Escribano
  • Put some of the basic mitigation operations in runbooks - Daniel Escribano
• Schema management
  • Tool for schema migration (coordinator schemas) - Daniel Escribano
  • Tool for long running mutations - Daniel Escribano
• Continued investment in performance
  • Tooling for other teams to understand which queries are slow and why - Ted Kaemming
  • Investigate variability of queries - Ted Kaemming
  • Per-team limits on queries/query complexity (needs product work) - Ted Kaemming

Board

https://github.com/orgs/PostHog/projects/85/views/2

Retro

🔥 Incidents 🔥

• #inc-2024-11-03-eu-ingestion-lag
• #inc-2024-10-29-clickhouse-us-lag
• #inc-2024-10-29-clickhouse-eu-lag

Last Sprint's Goals

• Continue 24.8 upgrade and test fix @fuziontech @tkaemming help ❌ ♻️
• Work with DW/BE for pulling fresh data from Iceberg table OR Kafka for Batch Exports @bretthoerner ⚠️
• Ongoing Delete tasks (support) @fuziontech @Daesgar ✅
• Plan work with DW to start writing Iceberg @fuziontech ⚠️
• Nullable performance improvements @tkaemming ✅
• Query log to Iceberg @fuziontech ❌
• Setup TTL for events for 48 months to gp3 disk @fuziontech ❌ ♻️
• EU to single AZ in eu-central-1c @Daesgar ✅
• Snapshot querying of Iceberg tables PR to CH @bretthoerner @fuziontech ⚠️

📣 Most work has been re-prioritized mid week in order to support snapshot diffing and re-clustering US
Tasks impacted marked with ❌ ♻️

This Sprint's Goals

TL;DR

• Continue 24.8 upgrade and test fix @fuziontech @tkaemming
• Ongoing Delete tasks (support) @fuziontech @Daesgar
• Remove unused Materialized Columns @tkaemming
• Reshard US into 8 shards 3 replicas w/ coordinators @Daesgar
• Snapshot querying of Iceberg tables proof of concept @bretthoerner @fuziontech
• Fix maintenance tasks such as snapshot pruning, for some reason it's broken when it used to work @bretthoerner
• Document pyspark/EMR runbook stuff in general before 🛫 @bretthoerner

[EDsCODE]

Team Data Storehouse

OKR Q4 2024

Objective

Query 3000

• Key Results:
  • Autocomplete
  • Increase general BI experience/product BI TODOs meta#157
  • Declutter the data warehouse UI and make the features intuitive to find

Data Modeling MVP

• Key Results:
  • Infrastructure decided and implemented
  • Integrating external data with feature flags
  • External data everywhere in insights/persons/cohorts
  • Get billing team to use modeling in posthog for their invoices_with_annual table

Retro

• Using warehouse person properties in experiment metrics
• SQL editor @EDsCODE
• release dashboard insight variables to everyone @Gilbert09
• BigQuery source release to everyone @Gilbert09
• Investigate reducing temporal-data-warehouse pod size @Gilbert09
• Resolving batch exports replication lag - some solution, working with team clickhouse @tomasfarias
• More onboarding tasks @rossgray
  • Release chargebee source
  • Select series colors in BI
  • HogQL ambiguous query error when joining tables with the same field names
  • BI breakdowns

Priority

• Release dashboard insight variables @Gilbert09
• BigQuery source release to everyone @Gilbert09
• Reduce temporal-data-warehouse pod size (memory) @Gilbert09
• SQL editor 3000 MVP @EDsCODE
• SPMC Batch Exports @tomasfarias
• Splitting persons batch exports @tomasfarias
• HogQL ambiguous query error when joining tables with the same field names @rossgray
• One of: Backfill UI, test connection button, or sessions/groups batch exports @rossgray

[daibhin]

Team Offsite Gate Crashers

@neilkakkar working on Experiments for 1 more week (apparently)

Retro

🟢 Stack trace demangling enabled for PostHog team
More work than expected here and lost time to modelling churn
Strong progress at offsite. End-to-end resolution, in-memory caching and saving for frames & source maps
Still need to put all the pieces together and deploy to test with live events

🔴 In-app source map uploading @daibhin
Lost some time to the rrweb upgrade
Good progress but has dependency on demangling goal. Carried over

✅ Frontend query caching to improve page load speed @daibhin

OKR

🟢 Human readable stack traces for Javascript
🟢 Ensure scalability of query time issue grouping
⚪️ Alerting via CDP
🟢 Launch

High priority

Deploy cymbal end-to-end @oliverb123

• Configure S3 env variables for deployment
• Writing events to ClickHouse in cymbal rather than plugin server (for team 2)
• Expected long tail of edge case handling / papercuts
  • Exceptions without a stack trace
  • Liveness checks
  • Evaluation of scaling / perf

In-app stack trace @daibhin

• Upload source maps on unresolved frames
• Redesign stack trace display component
• Investigate adding adjacent lines as context to an individual frame

Investigate build time source map uploading @neilkakkar (?)

[zlwaterfield]

Team Growth

Retro

• @zlwaterfield
  • RBAC
    • Finish RFC for billing and get feedback on it
    • Start working on it - get at least 1 PR merged for it
  • Get the in-app credit card entry experiment shipped
  • Make a few improvements to the technical exercise (based on using it for 2 weeks)
  • extra
    • picked up teams trials (live)
    • billing on call
    • credits eligibility lowered
    • sso enforced invite team member flow
• @raquelmsmith
  • Slowly roll out personless default change, deal with any issues - holding off for now
  • Fix bugs with product analytics templates onboarding flow
  • Ship session replay "templates" screen
  • Write data warehouse activation query
  • Think about how to redo onboarding so people can pick multiple products at once
  • Respond to customer about projects-on-invites
• @patricio-posthog
  • Billing limit: Get this bug fixed once and for all so we can move on
  • Forecasting bug: Fix forecasting and if it's a good idea, reimplement it with a new version
  • Improve current billing API endpoints (optimize or spit current ones into separate endpoints)
  • Check if the webhooks for invoices are working fine (check events at Posthog) / turn off the daily sync to test that.
  • extra
    • admin improvements for viewing json and permissions
    • running usage reports via admin (no more sshing needed)

Q4 Goals

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

1. 🟡 Get to a 4/10 in Enterprise platform support (Zach)
   • 🟡 Release Enterprise addon and iterate with sales to find a pricing model that makes sense
   • 🟡 Build RBAC
2. 🟡 Make revenue & forecasting more reliable (Patricio)
   • 🟡 Rejig how we do sync_invoices so we don’t see big drops - respond to webhooks instead of large syncs
   • ⚪ Evaluate temporal vs celery for usage reports and billing tasks so we can adhere to rate limits better
   • ⚪ Side hustle, if time permits: Rejig how anonymous / identified events are billed, as the current implementation is confusing.
3. 🟡 Activation: reverse trials, analytics onboarding flow, understand CC activation (Raquel)
   • ⚪ Dig into the data, watch replays to figure out why CC activation rate has been falling
   • ⚪ Add reverse trial to onboarding
   • ✅ Add reverse trial to billing page for teams

This sprint

Time off: Zach(1/2 day), Raquel (1 day)

• @zlwaterfield
  • RBAC
    • Get access models shipped
    • Get base middleware implements
    • Pick first product and get the base RBAC setup for it
  • Get Teams Trials fully live
  • Ship lowering of self serve credits requirements
  • Finish and ship project limits API checks
  • Onboarding for Surbhi
  • if time:
    • (from last time) Make a few improvements to the technical exercise (based on using it for 2 weeks)
    • (from last time) Get the in-app credit card entry experiment shipped
    • Review existing billing metrics, hookup to billing on call, see if we should add others
    • customer io transaction api
    • More billing metrics for on call
• @patricio-posthog
  • Continue with forecasting bugs, trying to understand what's happening and how to solve it.
  • (From last time): Turn off daily invoice sync to rely 100% on webhooks.
  • Try to optimize /billing endpoint, maybe separating forecasting into its own endpoint (after fixing it and understand how it works)
    Extra things if there is time:
  • Start to analyze forecasting and see its accuracy after each period.
• @raquelmsmith
  • Get product analytics dashboard templates out again
  • follow up on replay templates / reengagement campaign
  • Daniel in-person onboarding
  • Multi-product-select in onboarding (with Daniel)

[aspicer]

Team Product Analytics

Support hero: @aspicer and @anirudhpillai 

Q4 2024 objectives

Legacy 0x - Thomas owning

• All insights are query-based
• rm -rf posthog/queries/
• Insight query is stored in the URL for instant sharing of unsaved insights

First-class environments - Michael owning

• Environments are rolled out to everyone

Threshold-based alerts - Anirudh owning

• All users get to set up alerts on PostHog metrics
• Alerts are launched with Marketing, and positioned into paid plans
• We're using alerts ourselves for key trends and funnels

Flexible funnels - Sandy owning

• UDF funnels rolled out to everyone, unlocking arbitrary complexity
• Allowing parts of funnels to be unordered. No more separation between regular, unordered, and strict funnels
• Optional funnel steps

10x onboarding and ease of discovering features - Anirudh and Sandy owning

• We run an exercise in implementing PostHog in a hobby project. What stands out as painful or unclear?

User delight 2x - support rotation owning

• Support is 1 hero + 1 sidekick
• Every week support folks ship those small requested features (see list of requests so far)
• Goal: 90% of tickets fulfill the SLA

B2B analytics research - Anna owning

• Understand the market and user needs better in order to improve our current Group analytics product next quarter

Retro

Sandy

• ✅ Support (5 days)
• 🟢 24.8 query porting
• 🟡 Continue porting cohorts
• ✅ Funnel exclusion fixes

Thomas (Returns on Nov 4th!)

• ✅ Support

Anirudh (OOO through Nov 6th)

• 🟢 Resume alerting roll-out

Upcoming Sprint

Anirudh (Support Nov 11 - 15)

• Alerts currently rolled out to 20% of users, sprint goal to roll out to 100%
• Add slack messaging alerts
• RFC for Saved Filters / Saved HogQL Snippets Saved filters #20314 Saved filters (or creating actions out of series) #2895
• (Side Quest) Prometheus push gateway for gathering metrics from Celery

Thomas

• Filter Removal (Dashboard templates)
• Feature Request: Ability to change color and order of series on bar chart #23093 - Look into colors
• Improved filtering on results in trends #25609 - Chart reloading when disable breakdowns

Sandy (Support Nov 18 - 22)

• Investigate getting date first seen onto a person Track and allow graphing users/groups by "date first seen" #11952 Allow filtering by "Date first seen" #3031 
• Password protection (for third party shares) Password protection for Shared Insights/Dashboards #10869

[Twixes]

Team AI Product Manager

Retro

High priority

• 🟢 Funnels support @skoob13
• 🟢🔁 Tune agent/prompt for more relevant results (primary: excluding most $-prefixed events/properties) @skoob13 @Twixes
• ➡️ Stream feedback on the agent's current task in conversation @Twixes
• ➡️ PostHog dashboard for AI product manager use @Twixes
• ➡️ Generalize Session Replay's OpenAI data processing opt-in (to be determined with @fraserhopper) @Twixes

Low priority / side quests

• ➡️ Results evaluation framework - a benchmark @skoob13
• 🟡 Environments out as a feature preview @Twixes

Hang over items from previous sprint

• Generalize Session Replay's OpenAI data processing opt-in (to be determined with @fraserhopper)
• Stream feedback on the agent's current task in conversation
• PostHog dashboard for AI product manager use
• Results evaluation framework - a benchmark

Objective

LLM-based product manager as a service

High priority

• Results summarization
• Generalize Session Replay's OpenAI data processing opt-in (to be determined with @fraserhopper) @Twixes
• Stream feedback on the agent's current task in conversation @Twixes
• PostHog dashboard for AI product manager use @Twixes
• Tune Trends generation for more relevant results (smarter application of breakdowns, better property value examples) @skoob13
• Handle errors (agent recursion limit + graceful handling of e.g. ClickHouse errors) @skoob13

Low priority / side quests

• Results evaluation framework - a benchmark @skoob13
• Environments out as a feature preview @Twixes

[benjackwhite]

Team Infrastructure

Support hero: @danielxnj

Retro

• 🟡 Finish ArgoCD migration @bciaraldi
  • https://github.com/PostHog/charts/issues/1836
  • Almost all off-the-shelf stuff is migrated
• 🟢 Finish spot instances (at least for Decide and events, which are most expensive and can handle random load the best) @xneyder
  • Decide/events/monitoring all moved over
  • Tuned a bunch of limits and instance types to match a spot workload better
  • Final more sensitive services like recordings and ingestion need some more tweaking
• 🔴 Reserve instances @xneyder
  • Purposefully delayed until spot work is fully done as we are making big savings there already that affect the calculation
• 🔴 Move most web/workers to same AZ as Clickhouse + easy failover.
  • (Will follow up after)
• (Half the time taken up by onboarding/offsite)

OKR

1. 🏃 Continuous Delivery
2. 💰 Cost visualization
3. 🧪 Continuous Integration

High priority

• 100% do the reservation changes ASAP - @benjackwhite
• Fine tune our instance types to match our deployment workloads (cost optimisation to utilise the most of a node) @danielxnj
• Optimize the LB/envoy deployments to remove all cross AZ traffic (cost optimization) @MichaelKutsch-ph
• Tune the Warpstream dedicated node pools @danielxnj
• ArgoCD - move lowest priority posthog services over (cdp, temporal-workers, cymbal) @bciaraldi
• (Will potentially update some outcomes of the offsite later)

Low priority / side quests

• Docker cache (working with Tom as warehouse still uses minio) @danielxnj

[dmarticus]

Team Flags

Support

Now that we've split the queues, I'm the permanent support person for all things, but as Haven ramps up on support we will start to do a support hero rotation between the two of us. Will have an update on that timeline by next sprint.

Retro

Seeing some non-deterministic behavior from the maxmindDB; seems like it starts to return different values from the IP address we expect sometimes. Still figuring out exactly what the resolution is here (see thread: https://posthog.slack.com/archives/C06GG249PR6/p1730842770193459), but wanted to raise here since it's a cross-team dependency.

Things we got done

• Finish static and dynamic cohort matching for new feature flag service
  • support full subset of cohort match operations (not_in, etc) at the service/DB level
• Prepare for Haven's onboarding in Seattle

Things we moved over

• Port over existing feature flag integration tests from the Django to rust repo (lots of valuable alpha in our existing /decide tests, but this doesn't need to happen this sprint) - @dmarticus

---

OKRs

1. Ship new feature flags in production and accomplish the 10x scale + 2x speed boost performance targets (ideally decreases infra spend by N% as well, will try to figure out a more specific target going forward)
2. Maintain current /decide service and make optimizations.

Sprint

High priority

(all new flags work is tracked in this mega-ticket: #22131)

• Implement holdout group targeting in new /flags - @dmarticus
• Port over existing feature flag integration tests from the Django to rust repo (lots of valuable alpha in our existing /decide tests, but this doesn't need to happen this sprint) - @dmarticus
• Continue to onboard and get up to speed - @havenbarnes
  • Flags Django app (flags UI)
  • Flags decide service (old django service, new Rust service)
• Feature flag UI is capped at showing/searching up to 300 flags #25214 - @havenbarnes