Add datasource benchmark tests - #635
Conversation
Tests run on PR are regression tests - each datasource's performance is compared with base reference. Because memory datasource operates on memory only, datasources run was limited to only smallest file (261M), excluding the 7.5G one. To support that list of read/write files was moved to fixture. Manually workflow gives more freedom. There is an option to run stream/memory datasource against our default one to see how different datasources performance compares.
yngve793
left a comment
yngve793
left a comment
There was a problem hiding this comment.
Compiles and run tests locally without any problems. Nice touch to allow the user to select test options when running tests on Github. If it is possible it would be nice with a fast option, like 1-2 minutes. Fastest option I fond was 13 minutes with memory only.
achaikou
left a comment
achaikou
left a comment
There was a problem hiding this comment.
Thanks, good point! 👍
So looks like best thing I can do now is not to create large file (40s) when datasource path is running and file is not needed.
Problem is that memory tests themselves (which run on smallest file with small subset of functions) take only around 17s.
Everything else is a build time.
Most of the slowness is caused by this thing discovered by Alex.
To be able to fully benefit from this commit we would have to bump base references to it, once it is merged.
Should be fine with memory/stream datasource as those are new anyway. It can bring memory run down to 2m then.
But not sure about bumping commit id for the default flow. It points to our latest release, so we always compare performance to latest version users have.
I see that running "old" version on manylinux2014_x86_64 and "new" version on manylinux_2_28_x86_64 causes performance differences, so they better be aligned.
Maybe we can run memory/stream on new setup and everything related to default datasource still purely on old one?
Probably this will have to wait until Alex's PR is merged.
|
As this is 2.0 functionality, we decided it is better to be added after 2.0 release. |
2 participants
Agreed to try running regression for all types of datasources on each PR and then see how troublesome it is.