[Feature] Test against log size regressions

[vicsn]

🚀 Feature

Add tests in .ci scripts that assert the logfiles are below a certain size. Use --verbosity 4. Each testfile and nodetype can have its own expected size bound.

[ljedrz]

Suggestions for reducing the size of logs:

• downgrade the logs that aren't essential to ad-hoc debugging to at least DEBUG, and never run TRACE outside of local debugging sessions

• the date+time is 28B per log, amounting to ~2.67MiB on its own per 100'000 lines; if the log aggregators provide their own timestamps (which they routinely do), we could disable that field completely, or at least remove the date (saving 10B per log)

• the module names can take 50B or more, with most being at least ~28B; I personally don't find them particularly important for debugging, but even if we want to ensure that we're able to find the exact source, we could use a fast 8B hash to compress them, and use a trivial tool to map it to the module name if needed for debugging; it would roughly look like this:

  INFO snarkos_node_bft::helpers::storage: Starting round 3...
  INFO 49f1b40dc00a58ef: Starting round 3...
  

  we could even have an exhaustive module dictionary that would map module names to short aliases, like:

  "snarkos_cli::commands::start" => "start",
  "snarkos_node_bft::gateway" => "gateway"
  "snarkos_node_bft::bft" => "bft",
  (...)
  

  which would be more readable and perhaps even shorter on average, but would require a bit more maintenance.

[vicsn]

Update: let's expect our validators to just run with --verbosity 1. We don't have to put effort on lowering log frequencies.

For the regression testing, we should test against both verbosity 1 and verbosity 4. Maybe we can run half the scripts with one verbosity level, and the other half of the scripts with the other verbosity.