Skip to content

feat: implement store query chunking #2511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
weboko merged 5 commits into from
Jul 18, 2025
Merged

feat: implement store query chunking #2511

weboko merged 5 commits into from
Jul 18, 2025

Conversation

@weboko
Copy link
Collaborator

@weboko weboko commented Jul 18, 2025 •
edited
Loading

Problem / Description

nwaku will have limitation of having 24 hour maximum range for Store queries from 0.37

Solution

implement query chunking.

Notes

Dogfooding

Performed with lab.waku.org/dogfooding, queries are split just fine, and served.
No possibility to check with nwaku that implements this limitation as only next version (0.37) will have this limitation.

Regular scenario works fine.

image

Checklist

@weboko weboko requested a review from a team as a code owner July 18, 2025 22:29
@github-actions
Copy link

github-actions bot commented Jul 18, 2025 •
edited
Loading

size-limit report 📦

Path Size Loading time (3g) Running time (snapdragon) Total time
Waku node 70.21 KB (+0.34% 🔺) 1.5 s (+0.34% 🔺) 1.2 s (-8.79% 🔽) 2.6 s
Waku Simple Light Node 122.16 KB (+0.2% 🔺) 2.5 s (+0.2% 🔺) 1.5 s (-2.6% 🔽) 3.9 s
ECIES encryption 23.06 KB (0%) 462 ms (0%) 471 ms (-5.97% 🔽) 932 ms
Symmetric encryption 22.45 KB (0%) 449 ms (0%) 358 ms (-52.83% 🔽) 806 ms
DNS discovery 51.95 KB (0%) 1.1 s (0%) 807 ms (-24.99% 🔽) 1.9 s
Peer Exchange discovery 52.4 KB (0%) 1.1 s (0%) 1.1 s (+23.17% 🔺) 2.1 s
Local Peer Cache Discovery 46.1 KB (0%) 922 ms (0%) 837 ms (-12.79% 🔽) 1.8 s
Privacy preserving protocols 53.54 KB (+0.11% 🔺) 1.1 s (+0.11% 🔺) 997 ms (+21.07% 🔺) 2.1 s
Waku Filter 55.25 KB (-0.1% 🔽) 1.2 s (-0.1% 🔽) 651 ms (-20.11% 🔽) 1.8 s
Waku LightPush 52.9 KB (-0.02% 🔽) 1.1 s (-0.02% 🔽) 624 ms (-23.44% 🔽) 1.7 s
History retrieval protocols 59.27 KB (+0.4% 🔺) 1.2 s (+0.4% 🔺) 778 ms (-49.24% 🔽) 2 s
Deterministic Message Hashing 28.39 KB (0%) 568 ms (0%) 892 ms (+68.74% 🔺) 1.5 s

weboko
weboko commented Jul 18, 2025
View reviewed changes
packages/core/src/lib/store/store.ts
const timeDiff =
queryOpts.timeEnd.getTime() - queryOpts.timeStart.getTime();
if (timeDiff > MAX_TIME_RANGE) {
throw new Error("Time range bigger than 24h");
Copy link
Collaborator Author

@weboko weboko Jul 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replicate limitation that will be in nwaku in Store core implementation
chucking is done in SDK

@weboko weboko merged commit 36f6884 into master Jul 18, 2025
17 of 18 checks passed
@weboko weboko deleted the weboko/store-query-chunck branch July 18, 2025 23:37
@weboko weboko mentioned this pull request Jul 18, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adklempner adklempner adklempner approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: split Store queries into max 24 hours range

3 participants

@weboko @adklempner