-
Notifications
You must be signed in to change notification settings - Fork 1.8k
feat(server-core, query-orchestrator): Use the fastest xxhash for cache keys and model versions #9397
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
64626e7
to
bf7a59a
Compare
const key = crypto.createHash('md5') | ||
.update(args.map((v) => JSON.stringify(v)).join(',')) | ||
.digest('hex'); | ||
const key = xxh3.xxh64(args.map((v) => JSON.stringify(v)).join(',')).toString(16); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's make a function somewhere in @cubejs-backend-shared,
with a name like defaultHasherHex16
(or defaultHasher
and a tiny interface over it) and use it everywhere in the code instead of defining and requiring @node-rs/xxhash
in each package.
This approach is how the Rust language does it: DefaultHasher
. It will allow us to change it on the fly for testing via envs or replace the implementation whenever necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Absolutely agree! Great point!
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9397 +/- ##
===========================================
- Coverage 76.40% 47.88% -28.53%
===========================================
Files 400 171 -229
Lines 104055 21306 -82749
Branches 3685 3685
===========================================
- Hits 79508 10203 -69305
+ Misses 24112 10668 -13444
Partials 435 435
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Switch to using faster
xxhash
for cache keys and versions of data models instead of slowcrypto.createHash('md5')
.Use the fastest node xxhash implementation, written in ... of course rust :)
Here are some more info for those who is interested:
Changes: