[WIP] Optimize encoding and decoding float64 slices - #117
Conversation
jeromefroe
left a comment
There was a problem hiding this comment.
Any idea why the packed format is so much more performant than the native one?
| } | ||
| decoded, pool := it.getFloat64SliceFor(numValues) | ||
| for i := 0; i < numValues; i++ { | ||
| decoded = append(decoded, it.decodeFloat64()) |
There was a problem hiding this comment.
Seems in decodeFloat64SlicePacked we check the error after decoding each value, should we do the same here?
| largeFloat64s []float64 | ||
| ) | ||
|
|
||
| func BenchmarkEncodeFloat64NativeSmall(b *testing.B) { |
There was a problem hiding this comment.
nit: take it or leave it, but could combine these into a single benchmark with a table, for example:
func BenchmarkFloat64(b *testing.B) {
// initialize float slice
benchmarks := []struct {
name string
vals []float64
encoding encodingType
} {
{
name: "EncodeNativeSmall"
vals: smallFloat64a,
encoding: nonPackedEncoding,
},
...
}
for _, bm := range benchmarks {
b.Run(bm.name, func(b *testing.B) {
for i := 0; i < b.N; i++ {
AppendFloat(dst[:0], bm.float, bm.fmt, bm.prec, bm.bitSize)
}
})
}
}
| } | ||
| if numValues <= it.largeFloatsSize { | ||
| newCapcity := int(math.Max(float64(numValues), float64(cap(it.cachedFloat64s)*2))) | ||
| if newCapcity > it.largeFloatsSize { |
There was a problem hiding this comment.
Why not just make the capacity of it.cachedFloat64s to it.largeFloatSize when we initialize the iterator? It seems like after enough time it will reach that value, no?
| } | ||
| for i := 0; i < numValues; i++ { | ||
| byteOrder.PutUint64(enc.tmpBuf, math.Float64bits(values[i])) | ||
| _, enc.encodeErr = enc.bufEncoder.Buffer().Write(enc.tmpBuf) |
There was a problem hiding this comment.
Do we want to write the bytes on at a time or would it perhaps be beneficial to write them only into the buffer in the loop and then write the entire buffer once outside the loop?
cc @cw9 @jeromefroe
Need to fix the tests that have been commented out so marking it as work in progress, but the core implementation is complete. Benchmarks have shown promising results with the optimizations in this PR for encoding and decoding float64 slices, which is a significant portion of the overall encoding and decoding work.
Benchmark results (native is the current implementation, packed is the new implementation)
As shown above, with the packed encoding/decoding, for medium sized
[]float64with 1120 values, the new encoder is ~1.9x faster than the current encoder and the new decoder is ~5.4x faster than the current decoder.