Skip to content

Conversation

@abhisheksplunk
Copy link
Collaborator

No description provided.

Abhishek Chatterjee added 6 commits December 9, 2025 23:32
- Implement seed-based testing in ReadWriteTest.java
- Add 3 Java-specific seeds from issues #256, #267, #268
- Create seeds directory with documentation
- Tests now record and replay failing seeds for regression prevention
- Updated package imports from oteltef to otelstef after main branch merge
- Verified seed-based regression testing implementation matches PR #263
- All tests pass successfully
- 3 Java-specific seeds loaded and tested correctly
@github-actions
Copy link

Benchmark Result

Benchmark diff with base branch
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                                                 │ bench-main.txt │           bench-new.txt            │
                                                 │     sec/op     │    sec/op     vs base              │
SerializeNative/STEF/serialize-4                     10.25m ±  9%   10.39m ± 13%       ~ (p=0.699 n=6)
SerializeNative/STEFU/serialize-4                    30.92m ±  1%   31.28m ±  2%  +1.16% (p=0.009 n=6)
DeserializeNative/STEF/deser-4                       2.539m ±  2%   2.553m ±  4%       ~ (p=0.093 n=6)
DeserializeNative/STEFU/deser-4                      7.171m ±  1%   7.183m ±  1%       ~ (p=0.310 n=6)
SerializeFromPdata/STEF/serialize-4                  124.8m ±  6%   127.6m ±  3%       ~ (p=0.065 n=6)
SerializeFromPdata/STEFU/serialize-4                 31.25m ±  1%   31.63m ±  2%  +1.22% (p=0.009 n=6)
DeserializeToPdata/STEF/deserialize-4                47.84m ±  1%   48.10m ±  3%       ~ (p=0.093 n=6)
DeserializeToPdata/STEFU/deserialize-4               64.07m ±  1%   64.25m ±  2%       ~ (p=0.699 n=6)
STEFReaderRead-4                                     2.614m ±  1%   2.646m ±  1%       ~ (p=0.065 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4        3.325 ± 25%    3.283 ± 25%       ~ (p=0.818 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     75.91m ± 13%   78.46m ± 11%       ~ (p=0.485 n=6)
ReadSTEF-4                                           2.685m ±  1%   2.663m ±  1%       ~ (p=0.240 n=6)
ReadSTEFZ-4                                          3.662m ±  5%   3.832m ±  7%       ~ (p=0.132 n=6)
ReadSTEFZWriteSTEF-4                                 8.607m ±  2%   8.259m ±  5%       ~ (p=0.065 n=6)
geomean                                              21.26m         21.42m        +0.74%

                                                 │ bench-main.txt │           bench-new.txt            │
                                                 │   sec/point    │  sec/point    vs base              │
SerializeNative/STEF/serialize-4                     153.2n ±  9%   155.3n ± 13%       ~ (p=0.699 n=6)
SerializeNative/STEFU/serialize-4                    462.4n ±  1%   467.7n ±  2%  +1.15% (p=0.009 n=6)
DeserializeNative/STEF/deser-4                       37.97n ±  2%   38.18n ±  4%       ~ (p=0.093 n=6)
DeserializeNative/STEFU/deser-4                      107.2n ±  1%   107.4n ±  1%       ~ (p=0.379 n=6)
SerializeFromPdata/STEF/serialize-4                  1.867µ ±  6%   1.907µ ±  3%       ~ (p=0.061 n=6)
SerializeFromPdata/STEFU/serialize-4                 467.3n ±  1%   473.0n ±  2%  +1.21% (p=0.009 n=6)
DeserializeToPdata/STEF/deserialize-4                715.6n ±  1%   719.5n ±  3%       ~ (p=0.093 n=6)
DeserializeToPdata/STEFU/deserialize-4               958.3n ±  1%   961.1n ±  2%       ~ (p=0.619 n=6)
STEFReaderRead-4                                     39.09n ±  1%   39.57n ±  1%       ~ (p=0.065 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       4.226µ ± 25%   4.173µ ± 25%       ~ (p=0.818 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     96.50n ± 13%   99.70n ± 11%       ~ (p=0.461 n=6)
ReadSTEF-4                                           40.19n ±  1%   39.85n ±  1%       ~ (p=0.290 n=6)
ReadSTEFZ-4                                          54.81n ±  5%   57.34n ±  7%       ~ (p=0.132 n=6)
ReadSTEFZWriteSTEF-4                                 128.8n ±  2%   123.6n ±  5%       ~ (p=0.065 n=6)
geomean                                              223.6n         225.2n        +0.73%

                                                 │ bench-main.txt │            bench-new.txt             │
                                                 │      B/op      │     B/op      vs base                │
SerializeNative/STEF/serialize-4                     3.344Mi ± 0%   3.346Mi ± 0%       ~ (p=0.074 n=6)
SerializeNative/STEFU/serialize-4                    7.530Mi ± 0%   7.530Mi ± 0%       ~ (p=0.515 n=6)
DeserializeNative/STEF/deser-4                       934.4Ki ± 0%   934.4Ki ± 0%       ~ (p=1.000 n=6) ¹
DeserializeNative/STEFU/deser-4                      1.471Mi ± 0%   1.471Mi ± 0%       ~ (p=1.000 n=6) ¹
SerializeFromPdata/STEF/serialize-4                  74.82Mi ± 0%   74.82Mi ± 0%       ~ (p=0.983 n=6)
SerializeFromPdata/STEFU/serialize-4                 7.530Mi ± 0%   7.530Mi ± 0%  -0.00% (p=0.030 n=6)
DeserializeToPdata/STEF/deserialize-4                31.97Mi ± 0%   31.97Mi ± 0%       ~ (p=0.929 n=6)
DeserializeToPdata/STEFU/deserialize-4               38.64Mi ± 0%   38.64Mi ± 0%       ~ (p=0.364 n=6)
STEFReaderRead-4                                     936.2Ki ± 0%   936.2Ki ± 0%       ~ (p=1.000 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       3.363Gi ± 0%   3.363Gi ± 0%       ~ (p=0.699 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     20.41Mi ± 0%   20.41Mi ± 0%       ~ (p=0.571 n=6)
ReadSTEF-4                                           936.2Ki ± 0%   936.2Ki ± 0%       ~ (p=0.697 n=6)
ReadSTEFZ-4                                          10.27Mi ± 0%   10.27Mi ± 0%       ~ (p=0.565 n=6)
ReadSTEFZWriteSTEF-4                                 13.43Mi ± 0%   13.43Mi ± 0%       ~ (p=0.219 n=6)
geomean                                              10.47Mi        10.47Mi       +0.01%
¹ all samples are equal

                                                 │ bench-main.txt │            bench-new.txt            │
                                                 │   allocs/op    │  allocs/op   vs base                │
SerializeNative/STEF/serialize-4                      2.655k ± 0%   2.659k ± 0%       ~ (p=0.091 n=6)
SerializeNative/STEFU/serialize-4                      884.0 ± 0%    884.0 ± 0%       ~ (p=1.000 n=6)
DeserializeNative/STEF/deser-4                         465.0 ± 0%    465.0 ± 0%       ~ (p=1.000 n=6) ¹
DeserializeNative/STEFU/deser-4                        469.0 ± 0%    469.0 ± 0%       ~ (p=1.000 n=6) ¹
SerializeFromPdata/STEF/serialize-4                   134.7k ± 0%   134.7k ± 0%       ~ (p=1.000 n=6)
SerializeFromPdata/STEFU/serialize-4                   886.0 ± 0%    885.0 ± 0%       ~ (p=0.242 n=6)
DeserializeToPdata/STEF/deserialize-4                 756.2k ± 0%   756.2k ± 0%       ~ (p=1.000 n=6)
DeserializeToPdata/STEFU/deserialize-4                944.9k ± 0%   944.9k ± 0%       ~ (p=1.000 n=6) ¹
STEFReaderRead-4                                       465.0 ± 0%    465.0 ± 0%       ~ (p=1.000 n=6) ¹
STEFSerializeMultipart/astronomy-otelmetrics-4        13.15M ± 0%   13.15M ± 0%       ~ (p=0.818 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4      2.294k ± 0%   2.294k ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEF-4                                             466.0 ± 0%    466.0 ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEFZ-4                                            503.0 ± 0%    503.0 ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEFZWriteSTEF-4                                  1.233k ± 0%   1.233k ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               6.360k        6.360k       +0.00%
¹ all samples are equal
Benchmark result
benchstat bench-new.txt
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                                                 │ bench-new.txt │
                                                 │    sec/op     │
SerializeNative/STEF/serialize-4                    10.39m ± 13%
SerializeNative/STEFU/serialize-4                   31.28m ±  2%
DeserializeNative/STEF/deser-4                      2.553m ±  4%
DeserializeNative/STEFU/deser-4                     7.183m ±  1%
SerializeFromPdata/STEF/serialize-4                 127.6m ±  3%
SerializeFromPdata/STEFU/serialize-4                31.63m ±  2%
DeserializeToPdata/STEF/deserialize-4               48.10m ±  3%
DeserializeToPdata/STEFU/deserialize-4              64.25m ±  2%
STEFReaderRead-4                                    2.646m ±  1%
STEFSerializeMultipart/astronomy-otelmetrics-4       3.283 ± 25%
STEFDeserializeMultipart/astronomy-otelmetrics-4    78.46m ± 11%
ReadSTEF-4                                          2.663m ±  1%
ReadSTEFZ-4                                         3.832m ±  7%
ReadSTEFZWriteSTEF-4                                8.259m ±  5%
geomean                                             21.42m

                                                 │ bench-new.txt │
                                                 │   sec/point   │
SerializeNative/STEF/serialize-4                    155.3n ± 13%
SerializeNative/STEFU/serialize-4                   467.7n ±  2%
DeserializeNative/STEF/deser-4                      38.18n ±  4%
DeserializeNative/STEFU/deser-4                     107.4n ±  1%
SerializeFromPdata/STEF/serialize-4                 1.907µ ±  3%
SerializeFromPdata/STEFU/serialize-4                473.0n ±  2%
DeserializeToPdata/STEF/deserialize-4               719.5n ±  3%
DeserializeToPdata/STEFU/deserialize-4              961.1n ±  2%
STEFReaderRead-4                                    39.57n ±  1%
STEFSerializeMultipart/astronomy-otelmetrics-4      4.173µ ± 25%
STEFDeserializeMultipart/astronomy-otelmetrics-4    99.70n ± 11%
ReadSTEF-4                                          39.85n ±  1%
ReadSTEFZ-4                                         57.34n ±  7%
ReadSTEFZWriteSTEF-4                                123.6n ±  5%
geomean                                             225.2n

                                                 │ bench-new.txt │
                                                 │     B/op      │
SerializeNative/STEF/serialize-4                    3.346Mi ± 0%
SerializeNative/STEFU/serialize-4                   7.530Mi ± 0%
DeserializeNative/STEF/deser-4                      934.4Ki ± 0%
DeserializeNative/STEFU/deser-4                     1.471Mi ± 0%
SerializeFromPdata/STEF/serialize-4                 74.82Mi ± 0%
SerializeFromPdata/STEFU/serialize-4                7.530Mi ± 0%
DeserializeToPdata/STEF/deserialize-4               31.97Mi ± 0%
DeserializeToPdata/STEFU/deserialize-4              38.64Mi ± 0%
STEFReaderRead-4                                    936.2Ki ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4      3.363Gi ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4    20.41Mi ± 0%
ReadSTEF-4                                          936.2Ki ± 0%
ReadSTEFZ-4                                         10.27Mi ± 0%
ReadSTEFZWriteSTEF-4                                13.43Mi ± 0%
geomean                                             10.47Mi

                                                 │ bench-new.txt │
                                                 │   allocs/op   │
SerializeNative/STEF/serialize-4                     2.659k ± 0%
SerializeNative/STEFU/serialize-4                     884.0 ± 0%
DeserializeNative/STEF/deser-4                        465.0 ± 0%
DeserializeNative/STEFU/deser-4                       469.0 ± 0%
SerializeFromPdata/STEF/serialize-4                  134.7k ± 0%
SerializeFromPdata/STEFU/serialize-4                  885.0 ± 0%
DeserializeToPdata/STEF/deserialize-4                756.2k ± 0%
DeserializeToPdata/STEFU/deserialize-4               944.9k ± 0%
STEFReaderRead-4                                      465.0 ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4       13.15M ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4     2.294k ± 0%
ReadSTEF-4                                            466.0 ± 0%
ReadSTEFZ-4                                           503.0 ± 0%
ReadSTEFZWriteSTEF-4                                 1.233k ± 0%
geomean                                              6.360k

class ReadWriteTest {
@Test
public void testCopy() throws IOException {
private boolean testCopySeed(long seed) throws IOException {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the seed used in this test? I can't find any usage?

I don't think we need it here. We want seed saving and load tests generated via writerTest.java.tmpl. This should be equivalent of what we do for Go in this PR: https://github.com/splunk/stef/pull/263/files#diff-6cea806d1350c5a7621b9e3b780184637490a125dbc98f0296224c0fdf2e7eeb

Once writerTest.java.tmpl is modified, you can run make all in various directories (e.g. in go/otel and stef) to generate Java code and then run ./gradlew test to see how the tests work.

When you are happy with writerTest.java.tmpl, running make all in root directory will help confirm everything works correctly.

@tigrannajaryan
Copy link
Collaborator

Nit: please use more descriptive commit message, PR title and description. Feel free to borrow from #263 which does similar things for Go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants