Description
Motivation
Benchmarks are currently design in such a way that they require the developer to think about every change if it will introduce breaking changes in a way that requires code changes in the benchmark code.
In #447 it can be seen that there's a need to upload some build artifacts that will later be used to execute tests and compare timing.
The problem can be seen here too #979.
Design Alternatives
Make sure we don't break any thing every time?
Not a very good option.
Design
In general, the way I see it is that, the maplibre-gl file of each version is used to run the tests without bundeling it with all kind of files.
Assuming the public API doesn't change much this will allow testing different version using the same style/sources etc.
If the public API changes then probably a small wrapper can be used in the latest code to mitigate for the breaking changes. These breaking changes will also require a major version change so this would make sense to think about it too at that point.
Any ideas are welcome :-)
Acceptance criteria
- Benchmarks are running as part of the CI
- It's easy to test previous versions without the need to generate code per version and store it somewhere
- It doesn't have to be as elaborate as the manual benchmark, it can be simpler and can co-exist.
- Before implementation starts a design is presented and accepted