fix: omit None fields from click inserts to stop bucket churn - #332
fix: omit None fields from click inserts to stop bucket churn#332Zingzy wants to merge 1 commit into
Conversation
Explicit nulls hard-close time-series buckets on every null-to-string type flip (schemaIncompatible rollover): prod averages 4 clicks per bucket across 16.2M buckets, and the bucket index alone is 3.6GB. Verified on PSMDB 8.0: 200 alternating clicks made 200 buckets with nulls present and 1 bucket with the fields omitted. Readers are unaffected: group-bys use $ifNull and sentinel filters already carry an $exists:false arm. Also aligns bootstrap granularity with prod (hours, collMod'd from seconds) so fresh deploys bucket correctly.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review. 📝 WalkthroughWalkthroughThe clicks time-series collection now uses hour granularity. ChangesClick time-series alignment
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The PR omits null-valued optional click fields and uses hourly bucketing for newly created collections while preserving required identity fields and existing collection behavior. No actionable merge-blocking risk remains beyond normal checks. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Problem
The clicks time-series collection averages 4.07 docs per bucket (66M docs across 16.2M buckets). The bucket index alone is 3.6GB, about half the database's total index footprint, and the resulting memory pressure is behind the recent Mongo incidents.
Root cause: click documents store empty analytics fields as explicit nulls (
referrer: null,utm_source: null,bot_name: null). A time-series bucket cannot hold two BSON types in one column, so every alternation between a direct click (null referrer) and a referred click (string referrer) hard-closes the bucket with a schema-change rollover. Measured on prod: 1,035 of 10,254 inserts closed a bucket this way in one hour; the busiest link created 695 buckets in a single day from 1,106 null/string flips.Fix
ClickDoc.to_mongo()now serialises withexclude_none=True, so empty fields are absent instead of null. Scoped to clicks only; the sharedMongoBaseModel.to_mongo()is unchanged.granularity: "hours", matching prod (collMod'd from "seconds" earlier). Fresh deploys previously got the churn-prone setting.Why reads are safe
$ifNullsentinels (Direct / unknown / (none)).{$exists: false}arm because clicks recorded before feat(analytics): add device and UTM click dimensions #259 lack the device/utm fields entirely; mixed shapes are already the norm in this collection.ClickDocfield has a default, sofrom_mongohandles absent keys.Verification
StatsServiceover a collection holding 120 old-shape and 120 new-shape docs: 15 exact-number checks (totals, every dimension group-by, sentinel filters includingreferrer=Direct, non-sentinel filters, per-link path) all match independently computed values.Existing buckets keep their shape (time-series buckets are immutable in place); this stops new churn, and a history repack can reclaim the index separately.
Summary by CodeRabbit
Bug Fixes
nullvalues.Tests