Sprint - Dec 9 - Dec 20

[pauldambra]

Global Sprint Planning

3 things that might take us down

Urgent incident follow-ups

https://github.com/orgs/PostHog/projects/103/views/2

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red


### High priority

-

### Low priority / side quests

-

[pauldambra]

Team And They All Rolled Over And One Fell Out

xmas party sprint

Support hero: @pauldambra

Retro

• @pauldambra safari bug was horrible, but at least fixed, good but tiring to make progress as well as fight bugs
• @ioannisj progress on understanding long sessions but requires a bit of refactoring on how we rotate session and manage timestamps. Number of small fixes and improvements
• @marandaneto fun conference, trying to sell to huge companies while he's there
• changing teams, and new people - sad but exciting

## what we shipped

Mobile

• 🔄 Capture network headers and body (assuming new instrumentation is in place by then) Network bodies for network logs posthog-ios#159 @ioannisj
• 🔄 long sessions on mobile - Session Id didn't rotate after 23 hours posthog-ios#227. @ioannisj
• 🔄 get people using iOS autocapture @ioannisj and @marandaneto
  • people already finding and using it
• ✅ flutter replay alpha and getting feedback @marandaneto
• react native person profiles @marandaneto

Replay

• ✅ replay 4000 low hanging fruit @pauldambra
  • got a chunk of this done
  • meeting adam in london to talk ux

Low priority / side quests

• Capture iOS logs for recordings Capture logs for recordings posthog-ios#258 @ioannisj
  • no progress here
• Enforce max session duration Enforce max session duration  posthog-ios#245 @ioannisj
  • part of new refactor for session manager
• 🔄 a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra
• 🔄 (can we|do we) measure watch time and buffer time? @pauldambra
• 🔄 go through support and find people who wanted block list and event trigger @pauldambra
• android min session duration (cost control) @marandaneto
  • waiting on external contributor

OKRs 🟡

1. OKR, status (red/yellow/green) and action points if yellow/red

🟡📱Goal 1: People think of PostHog as a mobile solution
🟡 📹 Goal 2: Replay 4000
- this is now so excellent it's Replay 4000
- #26098
🟡 🎤 Goal3: Ship the things users keep asking for

• queryable network data
• start replay on trigger e.g. event

High priority

Mobile

• 🍾 big golden bottle of champagne in london to celebrate the final platform being released
• size of mobile replay @marandaneto @ioannisj
  • smaller snapshot images for all four mobile platforms instead of jpeg
    • need to vendor or add a dependency for iOS probably
  • can we avoid sending duplicates to reduce traffic
• manual API for starting and stopping all 4 platforms @marandaneto @ioannisj
• 🔄 Capture network headers and body (assuming new instrumentation is in place by then) Network bodies for network logs posthog-ios#159 @ioannisj
• 🔄 long sessions on mobile - Session Id didn't rotate after 23 hours posthog-ios#227. @ioannisj
• handover from @Phanatic

Replay

• replay 4000 UI
  • improve overview tab using sessions data
• a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra

Low priority / side quests

• 🔄 get people using iOS autocapture @ioannisj and @marandaneto
• (can we|do we) measure watch time and buffer time? @pauldambra
• go through support and find people who wanted block list and event trigger @pauldambra

[jurajmajerik]

Team Experiments 🧪

Retro

• 🟢 Anders' onboarding in Amsterdam went great. Lots of great product discussions and did a deep dive into statistics. @andehen
  • First PRs in: feat: bypass insight refresh limit if is_staff or is_dev #26611, chore: add mprocs as an alternative to bin/start #26624
• 🟡 Ship an MVP of the data warehouse integration @danielbachhuber
  • The trickiest part done: figured out the JOIN with exposure events to get the experiment variants on the DW data.
  • Currently testing on our own DW data, next up: testing with 2 customers
• 🟢 Implement a stacked delta chart @jurajmajerik
• 🟡 Support multiple experiment metrics @jurajmajerik
  • Will continue for the rest of the sprint
• 🔴 Plug improved statistical calculations for Trends (simple counts) + write docs - @danielbachhuber, @jurajmajerik
  • Didn't start. We did a deep dive into statistics during the onboarding though.

Sprint goals

Support hero: @danielbachhuber

• Continue with onboarding tasks and get familiar with the codebase. @andehen
  • Low hanging UI improvements.
  • Improve the error response from the results calculation for better diagnostics of "no results".
  • Familiarize himself with our statistical methods.
• Test the Data Warehouse integration with 1-2 customers (Freepik, potentially ElevenLabs). @danielbachhuber
• Finish support for multiple experiment metrics. @jurajmajerik
  • Add UI to support this
• Plug improved statistical calculation for Trends + add docs @danielbachhuber, @andehen, @jurajmajerik
  • Clean up our current 3 statistical methods and run everything by our statistician contractor once more.
  • Implement the method for Trend counts.
  • Enable it for newly created experiments.

[daibhin]

Team Error Tracking

@neilkakkar off (1 week)
@oliverb123 CDP support rotation (2 weeks)

Retro

✅ Cymbal stack frame data quality @oliverb123 @daibhin
Stack trace resolution bugs solved
Symbol set management (deleting / uploading source maps)

✅ Cymbal Python support @oliverb123
Redesigned stack trace view. Uploading source maps in-app is on track
Investigated adding and built code context to each frame

🟢 Update frontend querying and UI @daibhin
Writing fingerprint overrides to ClickHouse
Aggregating exceptions based on overrides WIP but will be finished this week

✅ Migrate to Argo for deployments and external secret management (ft. infra) @oliverb123

🔴 First instance alerting via Hog functions @daibhin
PR with one possible approach ready. Need to figure out how this fits in with threshold based alerting / CDP

🟠 Investigate build time source map uploading @daibhin
Understand how it's done by Sentry. Holding off taking things further until a customer needs it

OKR

✅ Human readable stack traces for Javascript
🟢 Ensure scalability of query time issue grouping
🟢 Alerting via CDP
🟢 Launch

High priority

Launch closed beta @daibhin

• Calls with 5 customers and make sure they get setup
• Implement quick win feedback (e.g. clientside fingerprinting) @daibhin
• Reimplement error tracking in billing repo

Cymbal improvements @oliverb123

• Performance / reliability for scaling load (e.g. event batching, column materialisation)
• Alerting & monitoring for core metrics @daibhin
• Fix EU deployment (e.g. cleanup symbol sets)
• Investigate potential memory leak

Alerting @oliverb123

• Implement "first instance" alerting to unblock beta testers
• RFC on possible long term approaches (e.g. integrating with threshold based alerting)

Low priority / side quests

• Implement panels based UI for issue page @daibhin
• Improve issue creation (remove foreign key or use deterministic uuid)

[Phanatic]

Team Surveys

@Phanatic offboarding December 23rd.

Retro

✅ Shipped Adaptive sampling for survey responses.
✅ Bug fixes for NPS recurring surveys.
✅ Customer support

Sprint goals

• Fixing bugs reported by Annika and a few customers that we're getting more survey responses than allowed when using Adaptive Sampling.
• Fixing bugs reported by customers of NPS recurring surveys who are confused about when the survey is shown or hidden.
• Handing off no-code experiments to the Experiments team.
• Handing off Surveys to @marandaneto and @ioannisj
• Customer support
• Q1 Planning

[MichaelKutsch-ph]

Team Infrastructure

(WIP)

Retro

• 🔴 100% do the reservation changes ASAP - @danielxnj
  • we want to wait until the clickhouse migration is done so we see the actual feasible spend
  • discussions about reseller / PPA with AWS has a huge impact on this decision, we could do some wrong decisions here if this is not finalized first
• 🟢 Optimize the LB/envoy deployments to remove all cross AZ traffic (cost optimization) @MichaelKutsch-ph
  • has positive effects on the bill, but not as impactful as expected -> loads of work on every tiny part, therefore better focus on pricing agreements
• 🟡 ArgoCD - move lowest priority posthog services over (cdp, temporal-workers, cymbal) @bciaraldi
  • Cymbal is deployed 👍
  • we had some other issues that were more important to resolve, therefore prioritized down
  • A bunch of stuff missing on our internal cluster to make this stuff super robust
• updating VM to latest to resolve alerting issues
  • challenging and time consuming task - but we prioritized this as we have issues with alerting
  • we have to talk to VM to figure out some dependencies
• initial discussions with nOps, Pump and DoIT

OKR

1. 🏃 Next level ArgoCD - Continuous Delivery
2. ☁️ 🕸️ Scalable networking
3. 💰 Mr. Bezos, bring down those costs!
4. 🎮 A stable pre-prod environment

High priority

• install nOps agent to detect x-az traffic issues @danielxnj
• dive into details and decide about AWS Pricing @MichaelKutsch-ph
• Lock in WS @timgl
• Continue w/ internal cluster into the same state as our prod clusters (monitoring stack, SSO access etc.) @danielxnj
• ArgoCD @bciaraldi
  • some prep stuff to do for the cluster
  • start moving apps to argo in parallel @MichaelKutsch-ph
    • Get PostHog web (and workers) onto ArgoCD
• when Frank is back: @benjackwhite
  • plan move to Warpstream for capture/events
• exhaustion of IPs on prod: looking into it @bciaraldi

Low priority / side quests

[robbie-c]

Team web analytics

Support hero: @rafaeelaudibert

Retro

Robbie
🟢 Onboard Rafael
🟢 SDK Support hero exists
🟠 Finish cookieless prototype
🟢 Wrote RFC about marketing analytics: https://github.com/PostHog/product-internal/pull/684

Rafael
🟢 Was onboarded (and updated some onboarding docs)
🟢 Languages & Timezones queries
🟢 Comparison to previous period on all queries
🟡 Conversion goal

OKR

🟠 Continue to improve performance
🟢 Build a small amount of more advanced marketing features, to measure interest
🟡 Cookieless tracking prototype
🔴 Improve the NextJS experience
🔴 More page-level insights
🇬🇦 Improve the documentation, and link to it more from the app

High priority

Rafael

• Get comfortable doing web analytics support
• Conversion goals
• Add more options for previous period comparison (use product analytics UI)

Robbie

• Finish cookieless prototype
• Finish RFC about marketing center brain: https://github.com/PostHog/product-internal/pull/684

Stretch goals

Rafael

• Help out SDK Support with next/react in mind

Robbie

• Page level insights

[benjackwhite]

Team Conversion Drivers & Planners (CDP)

• Support: @oliverb123

Retro

• 🔴 @benjackwhite / @mariusandra client->posthog->destination event mapping system for ad destinations + infra
  • Swapped to Marius' todo - not a huge amount of progress due to sickness + incident work
• 🟢 @mariusandra plumbing for allowing custom JS (GTM replacement anybody??)
  • Pretty much there although waiting on the Mappings + stateless asset delivery to be production ready
• 🟡 @MarconLP building and testing client side destinations, E2E tests
  • E2E testing was a bust due to captchas etc so something to come back to in the future
  • Client side destinations weren't ready but major improvements to the Google Ads integration that is now basically there
• @benjackwhite also distracted with continous onboarding of Infra team as well as planning ahead to new hires

OKRs

• 🟢 Widespread usage of the new CDP
• 🔴 Messaging V1
• 🟡 Ingestion is rock solid

High priority

• @benjackwhite gradually roll out RemoteConfigs to everyone (providing the code for site-functions)
• @mariusandra finish Mappings work with focus on TikTok Ads
• @MarconLP build out and test TikTok Ads client side integration

Side quests

• Ideas for growth hacking of destinations (big list of all known SaaS we could work with that people can search for to upvote etc.
• Continue investigations into Warpstream events capture (utilize return of Frank perhaps)

[Twixes]

Team AI PM

Retro

High priority

• 🟢 Fix response streaming in production @skoob13 with @PostHog/team-infra
• 🔴 Make summarizer self-aware when we couldn't fully answer the question, and why @Twixes
• 🔴 All other classic insights types (order of importance: retention, paths, stickiness, lifecycle) @Twixes
• 🔴 Max AI rolled out as feature preview @PostHog/team-ai-product-manager
• 🍋‍🟩 Memory, prewarmed by analyzing existing data and definitions @skoob13
• 🔴 SQL generation MVP @skoob13

Low priority / side quests

• 🔴 In your product's website, out a complete description of the product for the agent's context
• 🟢 Environments out as a feature preview @Twixes

Hang over items from previous sprint

• Make summarizer self-aware when we couldn't fully answer the question, and why @Twixes
• All other classic insights types (order of importance: retention, paths, stickiness, lifecycle) @Twixes
• Max AI rolled out as feature preview @PostHog/team-ai-product-manager

OKR

High priority

• Make summarizer self-aware when we couldn't fully answer the question, and why @Twixes
• All other classic insights types (order of importance: retention, paths, stickiness, lifecycle) @Twixes
• Max AI rolled out as feature preview @PostHog/team-ai-product-manager
• Human-in-the-loop for clarifying taxonomy @skoob13

[timgl]

Team Movers of Data

Support hero: Tomas/Ross

Retro

• We managed to get to the bottom of missing events in batch exports
• Could have better alerts

Hang over items from previous sprint

• Item 1. prioritised/deprioritise

OKR

Coming soon!

High priority

• Check batch export queries are cancelled #26317 - @rossgray
• Batch exports: Add created_at to all persons exports #26644 - @rossgray
• Use encrypted fields - @tomasfarias
• https://github.com/PostHog/posthog-cloud-infra/issues/3305 - @tomasfarias

[fuziontech]

Team ClickHouse

OKR Q2 2024

Objective

Goals for Q4:

• Improve elasticity and flexibility of our data store by putting all our data in Iceberg
  • Work with Altinity effectively to ship read path for Iceberg on ClickHouse - Brett Hoerner
  • Setup infrastructure to ship all of our data to Iceberg on S3 - James Greenhill
  • Shipping Query logs to Iceberg - James Greenhill
• Continue improving CH operational expertise
  • Upgrade to a later version of Clickhouse - James Greenhill
  • Capacity planning - James Greenhill
  • Automation - Daniel Escribano
  • Put some of the basic mitigation operations in runbooks - Daniel Escribano
• Schema management
  • Tool for schema migration (coordinator schemas) - Daniel Escribano
  • Tool for long running mutations - Daniel Escribano
• Continued investment in performance
  • Tooling for other teams to understand which queries are slow and why - Ted Kaemming
  • Investigate variability of queries - Ted Kaemming
  • Per-team limits on queries/query complexity (needs product work) - Ted Kaemming

Board

https://github.com/orgs/PostHog/projects/85/views/2

Retro

🔥 Incidents 🔥

• #inc-2024-11-25-clickhouse-24-8-upgrade
• #inc-2024-11-29-clickhouse-eu-heavy-load

Last Sprint's Goals

• Validate cluster data parity between old and new clusters @fuziontech ⏳
• Upgrade clusters to 24.8 (US, EU, new-US, Dev) @fuziontech @Daesgar ✅ 🎉
• Anticipating new 24.8 cluster w/ real lightweight deletes - fix deletes @fuziontech ⏳
• Migrate to new 8 shard lower cost cluster w/ coordinator nodes @Daesgar ⏳
• New mechanism for managing Mat Cols @tkaemming ✅
• Create new users for different workloads and isolate resources based on users @fuziontech ⏳
• (low priority) Expose Iceberg table for testing @fuziontech
  • Current version this will be quite slow
  • New 24.8 version will be quite fast
• Better caching of object store bits for Iceberg (Working in CH codebase) @bretthoerner ❌
  • Code does a HEAD request for every file even if it's in cache for files that don't change

This Sprint's Goals

Holiday plans

@Daesgar 💒 out from Dec 5 -> Dec 18

TL;DR

- Review increasing cost on all AWS @fuziontech 
~~~ Validate and move over to new cluster quickly ~~~
- Validate cluster data parity between old and new clusters @fuziontech 
- Fix deletes @fuziontech 
- Benchmark to new 8 shard lower cost cluster w/ coordinator nodes @fuziontech  
~~~ Mat Col optimizations ~~~
- Investigate how to speed up index dropping for Mat Cols @tkaemming 
- Yeet unused mat cols @tkaemming 
~~~ Below the fold ~~~
- Investigate what needs to be done for queue metering / QoS @fuziontech 
- Get query logs into iceberg. @fuziontech 
- (low priority) Expose Iceberg table for testing @fuziontech 
	- Current version this will be quite slow
	- New 24.8 version will be quite fast

[dmarticus]

Team Flags

Support

• 12-2 to 12-9 @dmarticus
• 12-9 to 12-16 @havenbarnes

Retro

Things we got done

• Track feature flag called events in usage reports so we can track up who set up flags successfully @dmarticus
• generally ironing out service (centralizing error handling, reducing memory usage overhead with smarter copying/cloning)
  • found a bug in the following two paths (1) feature flag payloads, and (2) group targeting without group overrides. PR to come later today.
• land v0 of flag staleness feature @havenbarnes
• Fixed issue with feature flag payloads SDK methods on the server-side (Node, Python) not correctly sending feature_flag_response payloads @dmarticus
• Show feature flag payload as official PostHog property @havenbarnes
• Adding some debugging event properties to $feature_flag_called to help indicate when SDKs are using bootstrapping

Things we didn't get done and why

• testing/productionizing new /flags service @dmarticus
  • functional testing via django-compat integration tests
  • "real" testing via mirroring traffic to new service (meeting with ingestion team 11-20 to go through this.

As I was writing more integration tests, I came across some bugs, and needed to fix them.

• Start work on feature management UI @havenbarnes
  • needed realignment on the correct UI abstractions for this feature; made an initial PR to start that conversation, and are iterating on it.

---

Sprint

High priority

• testing/productionizing new /flags service @dmarticus
  • fixing P0 bugs along /flags hotpath
  • functional testing via django-compat integration tests
  • "real" testing via mirroring traffic to new service

Side Quests

• Continue to make sentry more actionable and useful @havenbarnes
• Figure out how to cut off the flags bit of /decide (and, in future /flags) when folks go over – right now we don't stop serving flags when overage happens @dmarticus
• Continue to look at growth metrics and see what other properties we can add to better understand our growth @dmarticus

Things to chew on

• Can we know when people are starting to make new features?

OKRs

1. Ship new feature flags in production and accomplish the 10x scale + 2x speed boost performance targets (ideally decreases infra spend by N% as well, will try to figure out a more specific target going forward)
2. Maintain current /decide service and make optimizations.

[zlwaterfield]

Team Growth

Retro

Retro items

• @zlwaterfield
  • RBAC
    • Get middleware PR tested and shipped
    • Get RBAC UI shipped for 1 product
  • Improve how we show value to teams addon customers when unsub'd or cancelling trial (think about how to do this for all products)
  • Talk to DW about materialzed views
  • Talk to Mine/Joe about better startup plan labels
  • Continue talking to Stripe PM about billing problems we are running into
  • Continue move to c.io transactional emails
  • extra
    • Personal API @ current endpoint for Swiftly
    • 2FA - move into our UI and get backup codes working (WIP)
    • Increased billing postgres size + adding in pgbouncer and read replica (WIP)
• @surbhi-posthog
  • Review exiting ICP research
  • Connect with Mine about ICP data being sync'd into PostHog
  • Do some analysis on our metrics through signup, onboarding, converting to paid based on our ICP scoring data
  • Suggest (and maybe implement) changes to the onboarding product experience based on learnings from the analysis.
• @patricio-posthog
  • Forecasting: 1 - Pato: 0. Still fighting this battle.
  • New pricing for queries (API) and mobile

Q4 Goals

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

1. 🟡 Get to a 4/10 in Enterprise platform support (Zach)
   • 🟡 Release Enterprise addon and iterate with sales to find a pricing model that makes sense
   • 🟡 Build RBAC
2. 🟡 Make revenue & forecasting more reliable (Patricio)
   • ✅ Rejig how we do sync_invoices so we don’t see big drops - respond to webhooks instead of large syncs
   • ⚪ Evaluate temporal vs celery for usage reports and billing tasks so we can adhere to rate limits better
   • ⚪ Side hustle, if time permits: Rejig how anonymous / identified events are billed, as the current implementation is confusing.
3. 🟡 Activation: reverse trials, analytics onboarding flow, understand CC activation (Raquel)
   • ⚪ Dig into the data, watch replays to figure out why CC activation rate has been falling
   • ⚪ Add reverse trial to onboarding
   • ✅ Add reverse trial to billing page for teams

Stretch goals:

• 🟡 API query pricing (Raquel & Patricio)
• ⚪ Move the annual credits flow out of Zapier

This sprint

Time off: @zlwaterfield off 2 days and working 1/2 days for 2 days.

• @zlwaterfield

  • Support hero week 1
  • Misc fixes from RBAC middleware changes
  • RBAC UI behind a feature flag
  • Finish the 2FA improvements
  • Block usage reporting when on a trial (request from sales)
  • Finish billing postgres db improvements - pgbouncer and read replica

• @surbhi-posthog

  • Support her week 2
  • custom sql analysis on ICP data
  • load icp properties into Posthog?
  • Add events for Team's features in the backend

• @patricio-posthog

  • Keep monitoring the new forecasting, check that everything runs smoothly
  • Update /billing endpoint to include the new forecasting for customers
  • API query pricing
  • Start working on the new pricing for mobile RFC

[EDsCODE]

Team Data Storehouse

OKR Q4 2024

Objective

Query 3000

• Key Results:
  • Autocomplete
  • Increase general BI experience/product BI TODOs meta#157
  • Declutter the data warehouse UI and make the features intuitive to find

Data Modeling MVP

• Key Results:
  • Infrastructure decided and implemented
  • Integrating external data with feature flags
  • External data everywhere in insights/persons/cohorts
  • Get billing team to use modeling in posthog for their invoices_with_annual table

Retro

Team split in 2

• Data import rebuild @Gilbert09
  • release internally by end of week
• SQL editor 3000 finish MVP section + move to new priorities @EDsCODE
  • release internally be end of week

Priority

Tom off for most of sprint

• Draft queries + reorg sidebar (Epic: Editor & Modeling 3001 #26193) @EDsCODE
• Materialization Tab (Epic: Editor & Modeling 3001 #26193) @EDsCODE
• Permanent search bar
• Release data import to everyone @Gilbert09