statement-store: make E2E benchmarks more realistic. #10850
Description
Lessons and results from running statement_store_many_nodes_bench on versi.
Setup
- Statement store nodes run in versi cluster and pass the
polkadot-parachain benchmark machine, they were configured with the following resources:
resources:
requests:
cpu: "8"
memory: "12Gi"
limits:
cpu: "8"
memory: "12Gi"
- Statements are submitted from my local machine, with each
Participantbeing its own tokio task. - Polkadot binary is running the mega-pr with all optimisations in it and subscription api.
Results
10 nodes
Summary min avg max
total, s 8 15 18
250k, 3 10 13
submit 1 5 8
retries 0 0 0
15 nodes
Summary min avg max
total, s 7 16 19
250k, 1 11 14
submit 1 5 8
retries 0 0 0
Lessons
- Our optimisations are impactful and help us scale.
- Submitting all the statements from a single machine becomes the bottleneck and becomes the most significant contributor of the times we measure, just submitting 300k expired statements(no signature checks or runtime calls) ends up taking ~8s from the total time.
- At least from a single machine the best results are obtained when some
Participantsshare the RPC connection, having an RPC connection per participant or sharing the same RPC connections for all participants doubles the submit time, not really important lesson because in the real world each participant comes from a different device. - With 15 nodes the time spent in the networking system becomes relevant as well statement-store: add parallel processing of statements that came from network #10814 (comment), so that should be a place we should try to optimise things as well.
What is needed
We should make our benchmarks more realistic, so for that we should build a setup in versi, where we can configure participants to run on different machines.
At a minimum I think what we need is the ability to run multiple instances of statement_store_bench from different containers, all at the same time.
Metadata
Metadata
Assignees
Labels
No labels