Fix client provider dist acc aggregation crashing#65
Conversation
kacperzuk-neti
force-pushed
the
fix-client-provider-dist-acc-perf
branch
2 times, most recently
from
d913270 to
94454b0
Compare
| "jest": "^29.7.0", | ||
| "prettier": "^3.3.3", | ||
| "prisma": "^5.22.0", | ||
| "prisma": "^6.0.1", |
There was a problem hiding this comment.
Makes it consistent with @prisma/client
| provider = "prisma-client-js" | ||
| output = "./generated/client" | ||
| previewFeatures = ["typedSql", "omitApi", "metrics"] | ||
| previewFeatures = ["typedSql", "metrics"] |
There was a problem hiding this comment.
It's no longer a preview feature in prisma 6
| //const latestStored = | ||
| // await prismaService.client_provider_distribution_weekly_acc.findFirst({ | ||
| // select: { | ||
| // week: true, | ||
| // }, | ||
| // orderBy: { | ||
| // week: 'desc', | ||
| // }, | ||
| // }); | ||
| //let nextWeek = latestStored | ||
| // ? DateTime.fromJSDate(latestStored.date, { zone: 'UTC' }) // we want to reprocess the last stored week, as it might've been incomplete | ||
| // : DateTime.fromSeconds(3847920 * 30 + 1598306400).startOf('week'); // nv22 start week - a.k.a. reprocess everything |
There was a problem hiding this comment.
This commented code is for incremental processing - it only recalculates since last processed week (inclusive, as last processed week could be incomplete or could be current ongoing week).
My plan is to first do the uncommented version that simply reprocesses everything every time. That's because we actually need to recalculate current data - it's wrong, data uniqueness was only calculated "per week" and not "up to a week". So a client could store that same data with the same SP on week 1 and week 2 and it wouldn't detect it as duplicates.
After it runs at least once, we can switch to the incremental version for performance.
kacperzuk-neti
force-pushed
the
fix-client-provider-dist-acc-perf
branch
2 times, most recently
from
eb3e3e7 to
2a3cb95
Compare
kacperzuk-neti
temporarily deployed
to
production-fidl
GitHub Actions
Inactive
kacperzuk-neti
changed the title
kacperzuk-neti
force-pushed
the
fix-client-provider-dist-acc-perf
branch
from
2a3cb95 to
27512c1
Compare
kacperzuk-neti
temporarily deployed
to
production-fidl
GitHub Actions
Inactive
2 participants
This PR:
truncatetodelete allfor data consistency (though this will need to be monitored closely by @lukasz-wal to make sure the DB can handle vacuuming this)The biggest issue I see that this new aggregation on it's own requires ~2h. But that's a temporary until we switch to the incremental version, which will take <5 minutes.