WIP second attempt

Author: nfrisby

scripts/estimating-validation-bitrate/README

@@ -0,0 +1,17 @@
+- `db-analyser --benchmark-ledger-ops` lists the fixed stats, including validation times and the full blockSize.
+
+- `MsgBlock` has an overhead of 2 bytes (ie the list length and the word tag, since both are <24).
+
+- I _think_ `network-mux`'s SDU overhead is 8 bytes, from https://github.com/IntersectMBO/ouroboros-network/blob/db61131c2f375842f0930a8a9cf7f83b0cb80992/network-mux/src/Network/Mux/Codec.hs#L28-L40.
+  However, I don't know how many SDUs each byte requires.
+  So I'll omit this.
+ 
+Thus the number of bytes on-the-wire is sufficiently dominated by blockSize.
+
+-----
+
+The `nix-shell -p gawk gnuplot --run 'source stream-then-plot.sh'` command renders images that help roughly answer the question: will a full buffer containing B blocks be able to refill before its entire contents is validated?
+(As of slot 134028831, it runs for about about 3.5 minutes on my laptop.)
+
+The image width is as great as Cairo would allow.
+If you open them in Firefox and then left-click, it will zoom to full height; then you can scroll along the x-axis.

scripts/estimating-validation-bitrate/README.md

@@ -0,0 +1,10 @@
+BEGIN { CONVFMT = "%.18g"; }
+
+{
+  SlotNo = $1; Microseconds = $12 + $13; Bytes = $14;
+
+  CumuBytes        = CumuBytes        + Bytes       ;
+  CumuMicroseconds = CumuMicroseconds + Microseconds;
+
+  print SlotNo, CumuMicroseconds, CumuBytes;
+}

scripts/estimating-validation-bitrate/cumuboth.awk

@@ -0,0 +1,17 @@
+if (!exists('prefix')) prefix = 'catch'
+if (!exists('suffix')) suffix = ''
+
+set terminal pngcairo transparent enhanced size 32767, 1024
+set output 'plot'.suffix.'.png'
+
+set grid ytics
+set xtics 500
+
+set xlabel 'total duration of validation (s)'
+set ylabel 'megabits per second'
+
+sizes = '10 100 1000'
+
+# FYI: words() gives length and word(,) extracts one word
+
+plot for [i=1:words(sizes)] prefix.'-'.word(sizes, i) using ($1/1000000):($2*8 < 100 ? $2*8 : 100) title word(sizes, i).' block buffer'

scripts/estimating-validation-bitrate/gp.scr

@@ -0,0 +1,13 @@
+# Crunch all the data.
+for i in 10 100 1000; do B=$i . streaming.sh & done; wait
+
+# Split the x-axis in half, so the plots are more legible.
+#
+# 125000 seconds is _currently_ the end of the x-axis if I plot the whole data set in one image.
+for i in 10 100 1000; do cat catch-$i | awk '($1/1000000 <  125000/2) {print $0}' > catch1-$i & done
+for i in 10 100 1000; do cat catch-$i | awk '($1/1000000 >= 125000/2) {print $0}' > catch2-$i & done
+wait
+
+# Render plot-1.png and plot-2.png.
+for i in 1 2; do gnuplot -e "prefix='catch$i'" -e "suffix='-$i'" plot.gp & done
+wait

scripts/estimating-validation-bitrate/gp.scr2

@@ -0,0 +1,22 @@
+### Will a full buffer of size B refill before it empties?
+
+out=catch-$B
+
+# The resulting file has 3 columns: SlotNo, CumuMicroseconds, CumuBytes.
+tail -n+2 benchmark-ledger-ops.csv | awk -f cumuboth.awk >$out
+
+# Discard Byron.
+# cat $out | awk '($1 >= 4492800) { print $0 }' >$out.tmp; mv $out.tmp $out
+
+# Time and space sizes of windows of B-blocks
+#
+# ChainSel and BlockFetch clients use a buffer of 10 blocks. On top of that,
+# BlockFetch itself is buffered according to the low/high watermark, which are
+# at least 192 kibibytes and 384 kibibytes, respectively. This logic here only
+# considers the block-counted buffer, not the bytes in-flight.
+paste $out <(tail -n+$((B + 1)) $out) | awk '(NF > 3) {print $2, $5 - $2, $6 - $3}' >$out.tmp; mv $out.tmp $out
+
+# The scatter plot of this data informs the question: assuming the buffer is
+# currently full, what bit rate would be necessary in order to completely
+# refill the buffer before it empties.
+paste $out <(tail -n+2 $out) | awk '(NF > 3) {print ($1 + $4) / 2, $6 / $2}' >$out.tmp; mv $out.tmp $out