Sample pooling

[xichen2020]

I have experimented with sampling pooling by using

• a pool of fixed-size (64) list of samples, in which case the p99 write latency has become higher and spikier)
• a pool of fixed-size (64) full array of samples and a pool of fixed-size empty array of samples (this is to pool the array after the samples are used), in which case the cpu used to be allocated for creating samples is no longer seen in the profile, but the p99 write latency/ flush interval / cpu load remained the same.

Punting this for now, may pick this back up when there's more time.

[jeromefroe]

On a side note, when we do revisit this in the future it may be worth trying the sync.Pool in the standard library to see if it offers any benefits for our particular workload.