Add accurate benchmarks

[davidmorgan]

The current _benchmark tool is very rough as it was intended to measure very large changes to performance.

To start making continuous smaller improvements we need something much more accurate.

Do that :)

[amrgetment]

I think you need a sample repo with a lot of generators, like freezed,auto_route, json ,etc
with a fork to test a real scenario with a lot of classes
A fork so you point to your updated commit hash or local path in the forked generator

[davidmorgan]

The _benchmark folder has some code to create test apps of various shapes/sizes using built_value, json_serializable or freezed. It can certainly be expanded to cover more ground.

But, what I intend by this issue is a bit different: we need a way to collect precise+repeatable timings from those, such that we can say things like "this is now 5% faster in some cases and not slower in other cases".

The _benchmark code currently runs lots of tests in parallel, which is convenient for rough measurement but probably can't be accurate enough :) ... that's one issue, the other is where the benchmarks run, ideally we want some dedicated hardware + covering Win + Linux + Mac.

[gmpassos]

IMHO, we need to be able to control what affects a benchmark through build parameters. For example:

• Ignore caches
• Ignore build scripts
• Do not rebuild scripts
• Force rebuild scripts
• Force use of cached version

Since these steps account for a significant portion of the total build time, it’s hard to compare scenarios if we can’t specify the exact build configuration through command parameters.

[davidmorgan]

Benchmarking is always hard for this reason :)

The most important benchmark is reasonably easy to get: do an initial build, make a small modification, do another build. By the time of the second build everything that needs setting up has happened, so the incremental result is reasonably realistic.

Timing watch mode is another complication, I didn't write any code for that yet :)

For manually timing things I have a simple script edit-benchmark

#!/bin/bash --

random="$RANDOM$RANDOM$RANDOM"
sed -i -e "s#CACHEBUSTER.*#CACHEBUSTER$random#" lib/app.dart

which replaces CACHEBUSTER in lib/app.dart with a randomized version, so that when I run

edit-benchmark; dart run build_runner build

this is a useful incremental build.

But, yeah. Lots to do here :)