Skip to content

Commit 9e55ec4

Browse files
albertlockettalbertlockett
authored
Update size test metrics stream implementation for otel-arrow encoding (#214)
An issue was identified where otel-arrow would perform worse than OTLP in the `TestMetricsMultipart` size test. This PR resolves the issue by changing how the results are compressed. otel-arrow can apply compression to telemetry batches in two places: - the record within the Arrow IPC stream - the protobuf serialized `BatchArrowRecord` message This test was originally compressing just the IPC stream, but otel-arrow actually gets better compression ratio when compressing the proto messages. This PR also upgrades otel-arrow to the latest version. Results from the original `TestMetricsMultipart` size test: ``` go test -timeout 90s -count=1 -v -run ^TestMetricsMultipart$ github.com/splunk/stef/benchmarks === RUN TestMetricsMultipart hostandcollector-otelmetrics Comp Bytes Ratio OTLP none 22219873 x 1.00 STEF none 1493558 x 14.88 STEFU none 1559921 x 14.24 Otel ARROW none 13571951 x 1.64 astronomy-otelmetrics Comp Bytes Ratio OTLP none 145844039 x 1.00 STEF none 10343709 x 14.10 STEFU none 10793486 x 13.51 Otel ARROW none 92128187 x 1.58 hostandcollector-otelmetrics Comp Bytes Ratio OTLP zstd 2675305 x 1.00 STEF zstd 246609 x 10.85 STEFU zstd 371536 x 7.20 Otel ARROW zstd 1316311 x 2.03 astronomy-otelmetrics Comp Bytes Ratio OTLP zstd 19596366 x 1.00 STEF zstd 3381914 x 5.79 STEFU zstd 4171492 x 4.70 Otel ARROW zstd 11690516 x 1.68 --- PASS: TestMetricsMultipart (10.18s) PASS ok github.com/splunk/stef/benchmarks 10.562s ``` **Interpretation of the results** The fact that STEF systematically achieves better compression than OTAP is expected. The tradeoffs between OTAP and STEF are radically different. OTAP seeks to optimize across multiple dimensions: zero deserialization, low memory allocations, data processing speed (SIMD support), compression rate, and better impedance with modern telemetry backends, all of which are columnar-oriented. By contrast, STEF is mainly optimized for compression rate (inter data center use case). The other dimensions mentioned above are not optimized. We are considering a second round of optimization on the compression rate for OTAP (there are still some ideas left to explore), which should reduce the gap with STEF. However, for the reasons mentioned earlier, it is unlikely that OTAP will achieve a better compression rate than STEF, except perhaps for very large batches. These two protocols have complementary use cases.
1 parent d2ec70e commit 9e55ec4

File tree

4 files changed

+187
-148
lines changed

4 files changed

+187
-148
lines changed

‎benchmarks/encodings/otelarrow/otelarrow.go‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,16 @@
11
package otelarrow
22

33
import (
4-
v1 "github.com/open-telemetry/otel-arrow/api/experimental/arrow/v1"
5-
"github.com/open-telemetry/otel-arrow/pkg/config"
6-
"github.com/open-telemetry/otel-arrow/pkg/otel/arrow_record"
7-
"github.com/open-telemetry/otel-arrow/pkg/otel/metrics/otlp"
8-
"github.com/open-telemetry/otel-arrow/pkg/werror"
4+
v1 "github.com/open-telemetry/otel-arrow/go/api/experimental/arrow/v1"
5+
"github.com/open-telemetry/otel-arrow/go/pkg/config"
6+
"github.com/open-telemetry/otel-arrow/go/pkg/otel/arrow_record"
7+
"github.com/open-telemetry/otel-arrow/go/pkg/otel/metrics/otlp"
8+
"github.com/open-telemetry/otel-arrow/go/pkg/werror"
99
"go.opentelemetry.io/collector/pdata/pmetric"
1010
"google.golang.org/protobuf/proto"
1111

1212
"github.com/splunk/stef/benchmarks/encodings"
13+
"github.com/splunk/stef/benchmarks/testutils"
1314
)
1415

1516
type OtelArrowEncoding struct {
@@ -106,18 +107,19 @@ func (*OtelArrowEncoding) Name() string {
106107

107108
func (e *OtelArrowEncoding) StartMultipart(compression string) (encodings.MetricMultipartStream, error) {
108109
opts := []config.Option{}
109-
if compression == "zstd" {
110-
opts = append(opts, config.WithZstd())
111-
} else {
112-
opts = append(opts, config.WithNoZstd())
113-
}
110+
opts = append(opts, config.WithNoZstd())
114111
arrowProducer := arrow_record.NewProducerWithOptions(opts...)
115-
return &multipart{producer: arrowProducer}, nil
112+
113+
return &multipart{
114+
producer: arrowProducer,
115+
compression: compression,
116+
}, nil
116117
}
117118

118119
type multipart struct {
119-
producer *arrow_record.Producer
120-
bytes []byte
120+
producer *arrow_record.Producer
121+
compression string
122+
bytes []byte
121123
}
122124

123125
func (m *multipart) AppendPart(part pmetric.Metrics) error {
@@ -129,7 +131,11 @@ func (m *multipart) AppendPart(part pmetric.Metrics) error {
129131
if err != nil {
130132
return err
131133
}
134+
if m.compression == "zstd" {
135+
bytes = testutils.CompressZstd(bytes)
136+
}
132137
m.bytes = append(m.bytes, bytes...)
138+
133139
return nil
134140
}
135141

‎benchmarks/go.mod‎

Lines changed: 26 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -5,55 +5,58 @@ go 1.24.0
55
require (
66
github.com/go-echarts/go-echarts/v2 v2.6.3
77
github.com/klauspost/compress v1.18.0
8-
github.com/open-telemetry/otel-arrow v0.31.0
8+
github.com/open-telemetry/otel-arrow/go v0.43.0
99
github.com/parquet-go/parquet-go v0.25.1
1010
github.com/splunk/stef/go/otel v0.0.8
1111
github.com/splunk/stef/go/pdata v0.0.0
1212
github.com/splunk/stef/go/pkg v0.0.8
1313
github.com/stretchr/testify v1.11.1
14-
go.opentelemetry.io/collector/pdata v1.19.0
14+
go.opentelemetry.io/collector/pdata v1.42.0
1515
golang.org/x/text v0.29.0
1616
google.golang.org/protobuf v1.36.9
1717
modernc.org/b/v2 v2.1.9
1818
)
1919

2020
require (
2121
github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect
22-
github.com/andybalholm/brotli v1.1.0 // indirect
23-
github.com/apache/arrow/go/v17 v17.0.0 // indirect
24-
github.com/axiomhq/hyperloglog v0.0.0-20230201085229-3ddf4bad03dc // indirect
25-
github.com/davecgh/go-spew v1.1.1 // indirect
22+
github.com/andybalholm/brotli v1.1.1 // indirect
23+
github.com/apache/arrow-go/v18 v18.2.0 // indirect
24+
github.com/axiomhq/hyperloglog v0.2.5 // indirect
25+
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
2626
github.com/dgryski/go-metro v0.0.0-20180109044635-280f6062b5bc // indirect
27-
github.com/fxamacker/cbor/v2 v2.4.0 // indirect
28-
github.com/go-logr/logr v1.4.2 // indirect
27+
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
28+
github.com/go-logr/logr v1.4.3 // indirect
2929
github.com/go-logr/stdr v1.2.2 // indirect
30-
github.com/goccy/go-json v0.10.3 // indirect
30+
github.com/goccy/go-json v0.10.5 // indirect
3131
github.com/gogo/protobuf v1.3.2 // indirect
32-
github.com/google/flatbuffers v24.3.25+incompatible // indirect
33-
github.com/google/go-cmp v0.6.0 // indirect
32+
github.com/google/flatbuffers v25.2.10+incompatible // indirect
33+
github.com/google/go-cmp v0.7.0 // indirect
3434
github.com/google/uuid v1.6.0 // indirect
35+
github.com/hashicorp/go-version v1.7.0 // indirect
3536
github.com/json-iterator/go v1.1.12 // indirect
36-
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
37+
github.com/kamstrup/intmap v0.5.1 // indirect
38+
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
3739
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
38-
github.com/modern-go/reflect2 v1.0.2 // indirect
39-
github.com/pierrec/lz4/v4 v4.1.21 // indirect
40-
github.com/pmezard/go-difflib v1.0.0 // indirect
40+
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
41+
github.com/pierrec/lz4/v4 v4.1.22 // indirect
42+
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
4143
github.com/x448/float16 v0.8.4 // indirect
4244
github.com/zeebo/xxh3 v1.0.2 // indirect
43-
go.opentelemetry.io/collector/config/configtelemetry v0.114.0 // indirect
44-
go.opentelemetry.io/otel v1.31.0 // indirect
45-
go.opentelemetry.io/otel/metric v1.31.0 // indirect
46-
go.opentelemetry.io/otel/trace v1.31.0 // indirect
45+
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
46+
go.opentelemetry.io/collector/featuregate v1.42.0 // indirect
47+
go.opentelemetry.io/otel v1.37.0 // indirect
48+
go.opentelemetry.io/otel/metric v1.37.0 // indirect
49+
go.opentelemetry.io/otel/trace v1.37.0 // indirect
4750
go.uber.org/multierr v1.11.0 // indirect
48-
golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect
51+
golang.org/x/exp v0.0.0-20250718183923-645b1fa84792 // indirect
4952
golang.org/x/mod v0.27.0 // indirect
5053
golang.org/x/net v0.43.0 // indirect
5154
golang.org/x/sync v0.17.0 // indirect
5255
golang.org/x/sys v0.35.0 // indirect
5356
golang.org/x/tools v0.36.0 // indirect
54-
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
55-
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
56-
google.golang.org/grpc v1.68.0 // indirect
57+
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
58+
google.golang.org/genproto/googleapis/rpc v0.0.0-20250707201910-8d1bb00bc6a7 // indirect
59+
google.golang.org/grpc v1.75.1 // indirect
5760
gopkg.in/yaml.v3 v3.0.1 // indirect
5861
)
5962

0 commit comments

Comments
 (0)