Fix: Remove cartesian product from monthly loan aggregations#7
Open
jjoyce0510 wants to merge 1 commit intomainfrom
Open
Fix: Remove cartesian product from monthly loan aggregations#7jjoyce0510 wants to merge 1 commit intomainfrom
jjoyce0510 wants to merge 1 commit intomainfrom
Conversation
Remove the problematic LEFT JOIN to loans table that was causing a cartesian product. The join condition (loan_type_name) was too broad, causing each monthly aggregation row to multiply by the number of individual loans of that type. Issue: The assertion expecting 10-30 rows was failing because the table was producing hundreds of rows due to the fan-out from joining aggregated data to granular loan records. Root cause: customer_id doesn't belong in a monthly aggregation table as each month represents multiple customers. The table grain should remain one row per month per loan type. Fixes the 10 consecutive assertion failures (100% failure rate). Related to: PR #3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🐛 Bug Fix: Remove Cartesian Product from Monthly Aggregations
This PR fixes the critical SQL bug introduced in PR #3 that caused the
agg_monthly_loanstable to produce hundreds of rows instead of the expected 10-30 rows.Problem Identified
The
LEFT JOINto theloanstable was creating a cartesian product:Why this breaks:
monthly_originationscontains aggregated data (1 row per month/loan_type)loans(fct_loan_details) contains individual loan records (100s of rows)loan_type_namealone causes each aggregated row to multiply by the number of loans of that typeWhat This PR Does
✅ Removes the problematic
LEFT JOIN loans✅ Removes the
customer_idcolumn (doesn't make sense in monthly aggregations)✅ Restores table to correct grain: one row per month per loan type
Impact
Root Cause Analysis
The original intent in PR #3 was to "track which customers are associated with each month's loan activity," but this is conceptually incompatible with monthly aggregations:
Testing
After this fix, the table should return to expected behavior:
row_count_between_10_and_30will passRelated Issues
f9d33f8c8e54a1d46de01ea7a6410cb6072e40caFiles Changed
models/marts/agg_monthly_loans.sql- Removed cartesian join and customer_id columnReady for review! 🚀