perf: add perf nim libp2p #658

p-shahi · 2025-06-05T18:38:39Z

No description provided.

p-shahi · 2025-06-05T18:39:29Z

@vladopajic Here's the perf run: https://github.com/libp2p/test-plans/actions/runs/15475924179/job/43571160935
~~https://github.com/libp2p/test-plans/actions/runs/15474725262~~ failed due to incorrect folder name

vladopajic · 2025-06-05T19:54:59Z

@p-shahi thanks man. if it passes i am thinking of changing version to v1.10 (since it uses nim-libp2p v1.10).

p-shahi · 2025-06-05T20:32:31Z

@vladopajic I already changed it to match the versionsInput.json (without this change, the runner was trying to access a nonexistent v0.1 folder)

Right now the run is failing for this reason (linked to the run in the above comment)

=== Starting client nim-libp2p/v1.1/tcp
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
Command failed: ssh -o StrictHostKeyChecking=no [email protected] 'for i in {1..10}; do timeout 20s ./impl/nim-libp2p/v1.1/perf --server-address 35.167.168.83:4001 --transport tcp --upload-bytes 9007199254740991 --download-bytes 0 || [ $? -eq 124 ]; done'

vladopajic · 2025-06-05T20:37:32Z

@p-shahi thanks sir! can you check with e6e38b9 commit?

p-shahi · 2025-06-05T21:31:26Z

@vladopajic I lifted this change: 7eb409d from your new branch.
It's failing due to the same error

=== Starting client nim-libp2p/v1.1/tcp
dialer.nim(360)          dial
???                      dial
perf.nim(75)             runClient
perf.nim(96)             main
asyncfutures.nim(649)    waitFor
Error: unhandled exception: Unable to establish outgoing link [DialFailedError]
Command failed: ssh -o StrictHostKeyChecking=no [email protected] 'for i in {1..10}; do timeout 20s ./impl/nim-libp2p/v1.1/perf --server-address 35.87.207.248:4001 --transport tcp --upload-bytes 9007199254740991 --download-bytes 0 || [ $? -eq 124 ]; done'

vladopajic · 2025-06-06T10:33:12Z

@p-shahi sir, my apologies for back-and-forth on this one. i have added 3 new commits to #659. could you please try now?

could you entirely reset this branch to match forked branch?

p-shahi · 2025-06-06T17:03:49Z

@p-shahi sir, my apologies for back-and-forth on this one. i have added 3 new commits to #659. could you please try now?

could you entirely reset this branch to match forked branch?

NP, done. Here's the run I triggered: https://github.com/libp2p/test-plans/actions/runs/15495806737/job/43632128611

vladopajic · 2025-06-06T17:50:15Z

thank you! i see, it failed again.

make: Nothing to be done for 'nim-libp2p'.

why?

 impl/nim-libp2p/
impl/nim-libp2p/v0.1/
impl/nim-libp2p/v0.1/perf

         32,768   0%   31.25MB/s    0:00:00  
      6,541,664 100%   48.74MB/s    0:00:00 (xfr#20, to-chk=11/43)
impl/nim-libp2p/v1.10/
impl/nim-libp2p/v1.10/.gitignore

             80 100%    0.61kB/s    0:00:00  
             80 100%    0.61kB/s    0:00:00 (xfr#21, to-chk=10/43)
impl/nim-libp2p/v1.10/Makefile

          1,075 100%    8.20kB/s    0:00:00  
          1,075 100%    8.20kB/s    0:00:00 (xfr#22, to-chk=9/43)
impl/nim-libp2p/v1.10/config.nims

            139 100%    1.06kB/s    0:00:00  
            139 100%    1.06kB/s    0:00:00 (xfr#23, to-chk=8/43)
impl/nim-libp2p/v1.10/perf.nim

          2,922 100%   22.29kB/s    0:00:00  
          2,922 100%   22.29kB/s    0:00:00 (xfr#24, to-chk=7/43)
impl/nim-libp2p/v1.10/pref.nimble

            241 100%    1.84kB/s    0:00:00  
            241 100%    1.84kB/s    0:00:00 (xfr#25, to-chk=6/43)

why does it show v0.1?

== Version nim-libp2p/v1.10
=== Starting server nim-libp2p/v1.10


=== Starting client nim-libp2p/v1.10/tcp
timeout: failed to run command ‘./impl/nim-libp2p/v1.10/perf’: No such file or directory

i think server started, i'll add log for this. but client failed to start?

p-shahi · 2025-06-06T17:54:05Z

@vladopajic it's because the perf executable is in the v0.1 directory when instead it should be inside v1.10

I can move it and run it again

p-shahi · 2025-06-06T17:55:32Z

Latest run https://github.com/libp2p/test-plans/actions/runs/15496684245/job/43634962368

vladopajic · 2025-06-06T17:58:09Z

@p-shahi hey i pushed one more commit. arguments have format --name value and not --name=value. could you please use this commit as well :)

edit: please wait; args parsing will not work with spaces this way, i'll need to change it. i'll let you know

vladopajic · 2025-06-06T18:40:58Z

@p-shahi now it's fixed. tested args parsing locally. code should build and run now. could you try with latest code from same branch? :)

p-shahi · 2025-06-06T19:09:17Z

triggered run after adding this change 3b0a465 and 2bf1f1a https://github.com/libp2p/test-plans/actions/runs/15498770750/job/43641664863

@vladopajic still some failures

=== Starting client nim-libp2p/v1.10/tcp
...
running client
using flags: (runServer: false, serverIpAddress: 34.221.157.20:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
/home/runner/work/test-plans/test-plans/perf/runner/src/index.ts:211
        const result = JSON.parse(line) as ResultValue;
                            ^
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at runClient (/home/runner/work/test-plans/test-plans/perf/runner/src/index.ts:211:29)
    at runBenchmarkAcrossVersions (/home/runner/work/test-plans/test-plans/perf/runner/src/index.ts:150:28)
    at main (/home/runner/work/test-plans/test-plans/perf/runner/src/index.ts:23:9)
    at Object.<anonymous> (/home/runner/work/test-plans/test-plans/perf/runner/src/index.ts:271:1)
    at Module._compile (node:internal/modules/cjs/loader:1529:14)
    at Module.m._compile (/home/runner/work/test-plans/test-plans/perf/runner/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1613:10)
    at Object.require.extensions.<computed> [as .ts] (/home/runner/work/test-plans/test-plans/perf/runner/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1275:32)

Is the client reporting intermediary progress every second per https://github.com/libp2p/test-plans/blob/master/perf/README.md?plain=1#L92 ?

vladopajic · 2025-06-06T22:05:04Z

thanks man! iam very pleased that it is finally running.

no, currently it's not reporting that. i wanted to make it run first (setting up). i'll add intermediate reports soon and report back once iam done.

vladopajic · 2025-06-09T16:30:16Z

hey, could you check again? i have added two commits: 249e368 and 81b7403

p-shahi · 2025-06-09T16:33:41Z

@vladopajic triggered run here: https://github.com/libp2p/test-plans/actions/runs/15539543975/job/43746772694

@vladopajic similar failure as before. Maybe a serialization issue?

vladopajic · 2025-06-10T10:37:21Z

@vladopajic similar failure as before. Maybe a serialization issue?

not really sure, when i run locally i get this output:

nim-libp2p:

{"type":"intermediary","timeSeconds":1.000315726,"uploadBytes":10000,"downloadBytes":33554432}
{"type":"intermediary","timeSeconds":2.000926548,"uploadBytes":10000,"downloadBytes":78118912}
{"type":"final","timeSeconds":2.491974892,"uploadBytes":10000,"downloadBytes":100000000}

go-libp2p:

2025/06/10 12:30:33 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
{"type":"intermediary","timeSeconds":1.001955393,"uploadBytes":0,"downloadBytes":22806528}
{"type":"intermediary","timeSeconds":1.001234978,"uploadBytes":0,"downloadBytes":26943488}
{"type":"intermediary","timeSeconds":1.001145969,"uploadBytes":0,"downloadBytes":32833536}
{"type":"final","timeSeconds":3.581006198,"uploadBytes":10000,"downloadBytes":100000000}

i have made #661 to help developers out in this situation, could you please check that one?
manually compared outputs, there are the same?

…d-perf-nim-libp2p

p-shahi · 2025-06-10T22:12:46Z

@vladopajic I merged that PR into master and this branch, then triggered a run. Here's what the logs show:

== Version nim-libp2p/v1.10
=== Starting server nim-libp2p/v1.10
=== Starting client nim-libp2p/v1.10/tcp
Could not parse ResultValue from line: 
== Version js-libp2p/v2.8
=== Starting server js-libp2p/v2.8
=== Starting client js-libp2p/v2.8/tcp
= Benchmark throughput/download on versions quic-go, rust-libp2p, https, go-libp2p, nim-libp2p, js-libp2p
== Version quic-go/v0.45
=== Starting server quic-go/v0.45
=== Starting client quic-go/v0.45/quic-v1

could be that the data isn't being written properly since the ResultValue is empty

github-actions · 2025-06-10T22:44:33Z

See new metrics at https://observablehq.com/@libp2p-workspace/performance-dashboard?branch=55c81bf180c17ba8fc3eb3454a60c4b4e96096ba

vladopajic · 2025-06-11T10:40:50Z

@p-shahi first successful run, good job for us!

could be that the data isn't being written properly since the ResultValue is empty

yeah i have imagined that there is some empty line, that's being written like maybe last report should not be with line brake. but, thankfully with #661 we don't have to think about that anymore.

questions:

is nim-libp2p perf implementation missing anything now?
in report there is no nim-libp2p report in Throughput section?

vladopajic · 2025-06-11T15:15:19Z

also i have added one more commit 485ad5a.

if with this commit workflow succeeds may we merge current pr?

p-shahi · 2025-06-11T17:00:16Z

@vladopajic I added some more logging, and found that the intermediate values are reported but not the final. So the error is there.

using flags: (runServer: false, serverIpAddress: 54.212.82.213:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.119355802,"uploadBytes":69992448,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.068374717,"uploadBytes":104267776,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.015831531,"uploadBytes":139264000,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.975211002,"uploadBytes":174325760,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.9377315,"uploadBytes":209453056,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.897812215,"uploadBytes":244514816,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.857765447,"uploadBytes":279576576,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.818475514,"uploadBytes":314441728,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.783617167999999,"uploadBytes":349372416,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.743338368,"uploadBytes":384434176,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.183787162,"uploadBytes":436862976,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.143014454,"uploadBytes":471924736,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.105352471,"uploadBytes":507052032,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.064573321,"uploadBytes":541982720,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.02722892,"uploadBytes":577044480,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.984099572,"uploadBytes":611909632,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.944024003,"uploadBytes":646905856,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.90498[7101](https://github.com/libp2p/test-plans/actions/runs/15590060951/job/43906557856#step:3:7113),"uploadBytes":681902080,"downloadBytes":0}
Could not parse ResultValue from line:

Does the isFinal block ever run? couldn't find where stats.isFinal is set

    if stats.isFinal:
      let result =
        %*{
          "type": "final",
          "timeSeconds": stats.duration.nanoseconds.float / 1_000_000_000.0,
          "uploadBytes": stats.uploadBytes,
          "downloadBytes": stats.downloadBytes,
        }
      stderr.writeLine($result)
      stdout.writeLine($result)
      done.complete()
      return

Here's the run: https://github.com/libp2p/test-plans/actions/runs/15590060951/job/43906557856

Added another log line eb265dd to see whats going on https://github.com/libp2p/test-plans/actions/runs/15591661636/job/43911987318.

Seems like isFinal is never hit:

== Version nim-libp2p/v1.10
=== Starting server nim-libp2p/v1.10
=== Starting client nim-libp2p/v1.10/tcp
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.056887277,"uploadBytes":96927744,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.706938076,"uploadBytes":121110528,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.021641816,"uploadBytes":169541632,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.678482576,"uploadBytes":193658880,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.995480059,"uploadBytes":242155520,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.651222931,"uploadBytes":266272768,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.966023475,"uploadBytes":314638336,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.624172756,"uploadBytes":338821120,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.937084765,"uploadBytes":387121152,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.592774024000001,"uploadBytes":411238400,"downloadBytes":0}
{"type":"intermediary","timeSeconds":10.909825,"uploadBytes":459669504,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.564585039,"uploadBytes":483786752,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.876668686,"uploadBytes":532086784,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.191129752,"uploadBytes":580452352,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.847630814,"uploadBytes":604569600,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.163632925,"uploadBytes":653000704,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.820034123,"uploadBytes":677183488,"downloadBytes":0}
{"type":"intermediary","timeSeconds":18.13726434,"uploadBytes":725483520,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":0.851147108,"uploadBytes":43515904,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.967309031,"uploadBytes":78446592,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.919766926,"uploadBytes":113770496,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.873556862,"uploadBytes":149225472,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.833923003,"uploadBytes":185204736,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.835881999,"uploadBytes":222101504,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.834918035,"uploadBytes":258867200,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.831809353,"uploadBytes":295567360,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.833797184,"uploadBytes":332529664,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.832998521,"uploadBytes":369229824,"downloadBytes":0}
{"type":"intermediary","timeSeconds":10.831159971,"uploadBytes":405995520,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.835344284,"uploadBytes":442826752,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.83894905,"uploadBytes":479723520,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.841162902,"uploadBytes":516620288,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.841919976,"uploadBytes":553451520,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.84628221,"uploadBytes":590479360,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.845667954,"uploadBytes":627245056,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.849477676,"uploadBytes":664141824,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":0.903077839,"uploadBytes":59375616,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.744714135,"uploadBytes":90308608,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.00695851,"uploadBytes":135856128,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.852837736,"uploadBytes":166199296,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.695507457,"uploadBytes":196542464,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.960018451,"uploadBytes":242024448,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.799535769,"uploadBytes":272302080,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.065370004,"uploadBytes":317915136,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.918539887,"uploadBytes":348323840,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.765119649000001,"uploadBytes":378601472,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.040532075,"uploadBytes":424214528,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.887664832,"uploadBytes":454492160,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.734331166,"uploadBytes":484769792,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.007569807,"uploadBytes":530186240,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.854884239,"uploadBytes":560726016,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.123392749,"uploadBytes":606404608,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.965717124,"uploadBytes":636682240,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.808621065,"uploadBytes":667090944,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.120261462,"uploadBytes":87359488,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.699944529,"uploadBytes":108527616,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.854482612,"uploadBytes":150929408,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.015484161,"uploadBytes":193331200,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.176500732,"uploadBytes":235798528,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.758167437,"uploadBytes":257097728,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.921551228,"uploadBytes":299630592,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.085642104,"uploadBytes":342032384,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.247882745,"uploadBytes":384499712,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.828380063000001,"uploadBytes":405667840,"downloadBytes":0}
{"type":"intermediary","timeSeconds":10.990289338,"uploadBytes":448135168,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.152722985,"uploadBytes":490471424,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.313503651,"uploadBytes":532873216,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.892791866,"uploadBytes":554106880,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.05570039,"uploadBytes":596639744,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.226302834,"uploadBytes":639041536,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.386471722,"uploadBytes":681508864,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.96921199,"uploadBytes":702742528,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.008739279,"uploadBytes":84869120,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.564997268,"uploadBytes":104923136,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.666006368,"uploadBytes":145620992,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.790878332,"uploadBytes":186777600,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.914741813,"uploadBytes":227803136,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.041089421,"uploadBytes":268959744,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.603612075,"uploadBytes":289538048,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.730633436,"uploadBytes":330760192,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.855599296999999,"uploadBytes":371851264,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.984230601,"uploadBytes":413007872,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.112488621,"uploadBytes":454098944,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.678164856,"uploadBytes":474677248,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.807524918,"uploadBytes":515768320,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.936293722,"uploadBytes":556924928,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.06148337,"uploadBytes":598016000,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.18802817,"uploadBytes":639107072,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.749630804,"uploadBytes":659619840,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.875103189,"uploadBytes":700776448,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.172825266,"uploadBytes":87293952,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.801493788,"uploadBytes":110034944,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.062749641,"uploadBytes":155451392,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.694719919,"uploadBytes":178126848,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.954599777,"uploadBytes":223412224,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.223976532,"uploadBytes":268697600,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.855937165,"uploadBytes":291241984,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.111876823999999,"uploadBytes":336527360,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.74392302,"uploadBytes":359202816,"downloadBytes":0}
{"type":"intermediary","timeSeconds":10.012261001,"uploadBytes":404422656,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.279858826,"uploadBytes":449708032,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.914401469,"uploadBytes":472383488,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.169875352,"uploadBytes":517537792,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.799910744,"uploadBytes":540147712,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.06400559,"uploadBytes":585302016,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.327879116,"uploadBytes":630456320,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.960971701,"uploadBytes":653000704,"downloadBytes":0}
{"type":"intermediary","timeSeconds":18.227014742,"uploadBytes":698220544,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":0.985410721,"uploadBytes":65798144,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.091655911,"uploadBytes":100335616,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.074432128,"uploadBytes":135987200,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.080379224,"uploadBytes":172163072,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.08285155,"uploadBytes":208076800,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.097894923,"uploadBytes":244056064,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.10703474,"uploadBytes":280100864,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.115134349,"uploadBytes":316014592,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.124497595999999,"uploadBytes":352059392,"downloadBytes":0}
{"type":"intermediary","timeSeconds":10.13715626,"uploadBytes":388038656,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.150830851,"uploadBytes":424083456,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.161507935,"uploadBytes":460062720,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.171808647,"uploadBytes":496173056,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.18237811,"uploadBytes":532086784,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.195189013,"uploadBytes":568197120,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.206516096,"uploadBytes":604110848,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.217285387,"uploadBytes":640221184,"downloadBytes":0}
{"type":"intermediary","timeSeconds":18.226565604,"uploadBytes":676134912,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.018577183,"uploadBytes":79757312,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.082914159,"uploadBytes":118030336,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.167584777,"uploadBytes":156827648,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.257972316,"uploadBytes":195756032,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.346662878,"uploadBytes":234618880,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.422363111,"uploadBytes":273416192,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.505939856,"uploadBytes":312475648,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.589074404,"uploadBytes":351338496,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.670806172000001,"uploadBytes":390332416,"downloadBytes":0}
{"type":"intermediary","timeSeconds":10.761882638,"uploadBytes":4[2919](https://github.com/libp2p/test-plans/actions/runs/15591661636/job/43911987318#step:3:2931)5264,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.859015999,"uploadBytes":468123648,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.950057978,"uploadBytes":507052032,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.03613685,"uploadBytes":545783808,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.113513351,"uploadBytes":584515584,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.202489294,"uploadBytes":623443968,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.291601465,"uploadBytes":662306816,"downloadBytes":0}
{"type":"intermediary","timeSeconds":18.381800221,"uploadBytes":701235200,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.083595554,"uploadBytes":54722560,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.832566269,"uploadBytes":108265472,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.987387335,"uploadBytes":150667264,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.140509473,"uploadBytes":193134592,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.717644716,"uploadBytes":214302720,"downloadBytes":0}
{"type":"intermediary","timeSeconds":5.878111114,"uploadBytes":256966656,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.034952632,"uploadBytes":299433984,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.189723296,"uploadBytes":341835776,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.768032290000001,"uploadBytes":363003904,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.924521449,"uploadBytes":405471232,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.081146763,"uploadBytes":447938560,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.237626213,"uploadBytes":490405888,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.815731187,"uploadBytes":511639552,"downloadBytes":0}
{"type":"intermediary","timeSeconds":13.96834323,"uploadBytes":553975808,"downloadBytes":0}
{"type":"intermediary","timeSeconds":15.120636048,"uploadBytes":596312064,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.274090417,"uploadBytes":638648320,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.851438801,"uploadBytes":659816448,"downloadBytes":0}
{"type":"intermediary","timeSeconds":18.00560049,"uploadBytes":702087168,"downloadBytes":0}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 9007199254740991, downloadBytes: 0)
running client
{"type":"intermediary","timeSeconds":1.089873482,"uploadBytes":99221504,"downloadBytes":0}
{"type":"intermediary","timeSeconds":1.796679985,"uploadBytes":124911616,"downloadBytes":0}
{"type":"intermediary","timeSeconds":2.51207314,"uploadBytes":150601728,"downloadBytes":0}
{"type":"intermediary","timeSeconds":3.949967407,"uploadBytes":202113024,"downloadBytes":0}
{"type":"intermediary","timeSeconds":4.66677586,"uploadBytes":227803136,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.112522004,"uploadBytes":279445504,"downloadBytes":0}
{"type":"intermediary","timeSeconds":6.833940465,"uploadBytes":305201152,"downloadBytes":0}
{"type":"intermediary","timeSeconds":7.552348268,"uploadBytes":330825728,"downloadBytes":0}
{"type":"intermediary","timeSeconds":8.993174828000001,"uploadBytes":382271488,"downloadBytes":0}
{"type":"intermediary","timeSeconds":9.713701741,"uploadBytes":408027136,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.165887229,"uploadBytes":459866112,"downloadBytes":0}
{"type":"intermediary","timeSeconds":11.896900394,"uploadBytes":485818368,"downloadBytes":0}
{"type":"intermediary","timeSeconds":12.62309976,"uploadBytes":511705088,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.07177046,"uploadBytes":563347456,"downloadBytes":0}
{"type":"intermediary","timeSeconds":14.793793591,"uploadBytes":589103104,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.245413884,"uploadBytes":640614400,"downloadBytes":0}
{"type":"intermediary","timeSeconds":16.965074471,"uploadBytes":666304512,"downloadBytes":0}
{"type":"intermediary","timeSeconds":17.683104298,"uploadBytes":692060160,"downloadBytes":0}
Could not parse ResultValue from line: 
= Benchmark throughput/download on versions go-libp2p, nim-libp2p
== Version go-libp2p/v0.41
=== Starting server go-libp2p/v0.41
=== Starting client go-libp2p/v0.41/tcp

similarly, doesn't seem to report Final value for download benchmark

using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 0, downloadBytes: 9007199254740991)
running client
{"type":"intermediary","timeSeconds":0.99895335,"uploadBytes":0,"downloadBytes":14352384}
{"type":"intermediary","timeSeconds":1.999336403,"uploadBytes":0,"downloadBytes":50528256}
{"type":"intermediary","timeSeconds":3.000994993,"uploadBytes":0,"downloadBytes":86769664}
{"type":"intermediary","timeSeconds":4.001642421,"uploadBytes":0,"downloadBytes":122945536}
{"type":"intermediary","timeSeconds":5.001696635,"uploadBytes":0,"downloadBytes":159252480}
{"type":"intermediary","timeSeconds":6.001160205,"uploadBytes":0,"downloadBytes":195493888}
{"type":"intermediary","timeSeconds":7.002725074,"uploadBytes":0,"downloadBytes":231800832}
{"type":"intermediary","timeSeconds":8.002716317000001,"uploadBytes":0,"downloadBytes":268042240}
{"type":"intermediary","timeSeconds":9.003642915,"uploadBytes":0,"downloadBytes":304611328}
{"type":"intermediary","timeSeconds":10.003249901,"uploadBytes":0,"downloadBytes":340983808}
{"type":"intermediary","timeSeconds":11.003458603,"uploadBytes":0,"downloadBytes":377290752}
{"type":"intermediary","timeSeconds":12.00525521,"uploadBytes":0,"downloadBytes":413597696}
{"type":"intermediary","timeSeconds":13.005730453,"uploadBytes":0,"downloadBytes":449839104}
{"type":"intermediary","timeSeconds":14.005217379,"uploadBytes":0,"downloadBytes":485949440}
{"type":"intermediary","timeSeconds":15.005966239,"uploadBytes":0,"downloadBytes":522256384}
{"type":"intermediary","timeSeconds":16.006391164,"uploadBytes":0,"downloadBytes":558563328}
{"type":"intermediary","timeSeconds":17.006760325,"uploadBytes":0,"downloadBytes":594870272}
{"type":"intermediary","timeSeconds":18.007373849,"uploadBytes":0,"downloadBytes":631177216}
Could not parse ResultValue from line: 
= Benchmark Connection establishment + 1 byte round trip latencies on versions go-libp2p, nim-libp2p

But when uploading and downloading 1 byte i.e. connection establishment test, it seems to always report the final value

== Version nim-libp2p/v1.10
=== Starting server nim-libp2p/v1.10
=== Starting client nim-libp2p/v1.10/tcp
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 1, downloadBytes: 1)
running client
isFinal
{"type":"final","timeSeconds":0.17179998,"uploadBytes":1,"downloadBytes":1}
using flags: (runServer: false, serverIpAddress: 44.245.5.235:4001, transport: "tcp", uploadBytes: 1, downloadBytes: 1)
running client
isFinal
{"type":"final","timeSeconds":0.171615057,"uploadBytes":1,"downloadBytes":1}

github-actions · 2025-06-11T18:08:34Z

See new metrics at https://observablehq.com/@libp2p-workspace/performance-dashboard?branch=c3ea796c2387f5ca5dbcc277825d07377de80488

vladopajic · 2025-06-12T09:38:50Z

@p-shahi hey, thanks!

i think i know what's going on. i'am talking days off, but i should be able to push this update.

vladopajic · 2025-06-13T10:31:31Z

@p-shahi hey man!

could you check again with 2e95f97?

Does the isFinal block ever run? couldn't find where stats.isFinal is set

here. note pr was just merged (if you searched master branch), up until today we used nim-libp2p version here in perf from that branch.

p-shahi · 2025-06-13T17:08:30Z

gotcha, triggered here: https://github.com/libp2p/test-plans/actions/runs/15639940398/job/44064636827
🤞

Same issue on this run as well.

github-actions · 2025-06-13T17:36:55Z

See new metrics at https://observablehq.com/@libp2p-workspace/performance-dashboard?branch=fb7cd6d616b53eab7e0cfe0be216910e8b3876fa

vladopajic · 2025-06-24T10:57:45Z

Same issue on this run as well.

hey, thank you! just did some investigation and it appears that nim perf is hitting timeout (20sec) - it would be cool if this message is actually shown in logs.
for some reason nim perf takes much longer then expected. i'll investigate this.

vladopajic · 2025-06-24T13:04:12Z

@p-shahi

question, intermediary report should include data since last report?
that is time should be elapsed time since last report and upload/download bytes are diff since last report as well, or they are current total?

p-shahi mentioned this pull request Jun 5, 2025

feat(perf): add nim-libp2p #657

Closed

p-shahi force-pushed the add-perf-nim-libp2p branch from 476a7a8 to 4b14d12 Compare June 6, 2025 17:02

vladopajic added 8 commits June 6, 2025 12:00

add pref nim libp2p

aaba2a1

fixed flags parsing

353b8e0

fix

2a6065a

perf dependencies reduced to min

d8bc6c6

fix args parsing

7d1a54f

rm v0.1

8ba1575

fix arg variable in log

b0cc3b8

use duration from perf client

4b328dd

p-shahi force-pushed the add-perf-nim-libp2p branch from 29f41fe to 4b328dd Compare June 6, 2025 19:01

p-shahi added 2 commits June 6, 2025 12:22

add nim-libp2p to runner test-filter

3b0a465

fix Makefile

2bf1f1a

vladopajic added 2 commits June 9, 2025 09:32

improve build

1dccca1

add intermediateReport and yamux

ce5b40b

vladopajic mentioned this pull request Jun 10, 2025

perf: gracefully parse ResultValue from output #661

Merged

Merge branch 'master' of https://github.com/libp2p/test-plans into ad…

10ebafa

…d-perf-nim-libp2p

perf: update benchmark results

55c81bf

vladopajic mentioned this pull request Jun 11, 2025

feat(perf): add stats vacp2p/nim-libp2p#1452

Merged

vladopajic and others added 2 commits June 11, 2025 09:07

various improvements perf

13c1844

log result

a84dcd4

p-shahi and others added 3 commits June 11, 2025 10:31

add isFinal log

eb265dd

limit to go and nim for speedier results

81e223a

perf: update benchmark results

c3ea796

bump nim-libp2p lib

0122876

perf: update benchmark results

fb7cd6d

vladopajic mentioned this pull request Jun 25, 2025

perf: protocol is very slow comapred to other implementations vacp2p/nim-libp2p#1484

Open

dhuseby added the perf Related to perf tests label Nov 6, 2025

perf: add perf nim libp2p #658

Are you sure you want to change the base?

perf: add perf nim libp2p #658

Uh oh!

Conversation

p-shahi commented Jun 5, 2025

Uh oh!

p-shahi commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladopajic commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

p-shahi commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladopajic commented Jun 5, 2025

Uh oh!

p-shahi commented Jun 5, 2025

Uh oh!

vladopajic commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

p-shahi commented Jun 6, 2025

Uh oh!

vladopajic commented Jun 6, 2025

Uh oh!

p-shahi commented Jun 6, 2025

Uh oh!

p-shahi commented Jun 6, 2025

Uh oh!

vladopajic commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladopajic commented Jun 6, 2025

Uh oh!

p-shahi commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladopajic commented Jun 6, 2025

Uh oh!

vladopajic commented Jun 9, 2025

Uh oh!

p-shahi commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladopajic commented Jun 10, 2025

Uh oh!

p-shahi commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 10, 2025

Uh oh!

vladopajic commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladopajic commented Jun 11, 2025

Uh oh!

p-shahi commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

vladopajic commented Jun 12, 2025

Uh oh!

vladopajic commented Jun 13, 2025

Uh oh!

p-shahi commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 13, 2025

Uh oh!

vladopajic commented Jun 24, 2025

Uh oh!

vladopajic commented Jun 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

p-shahi commented Jun 5, 2025 •

edited

Loading

vladopajic commented Jun 5, 2025 •

edited

Loading

p-shahi commented Jun 5, 2025 •

edited

Loading

vladopajic commented Jun 6, 2025 •

edited

Loading

vladopajic commented Jun 6, 2025 •

edited

Loading

p-shahi commented Jun 6, 2025 •

edited

Loading

p-shahi commented Jun 9, 2025 •

edited

Loading

p-shahi commented Jun 10, 2025 •

edited

Loading

vladopajic commented Jun 11, 2025 •

edited

Loading

p-shahi commented Jun 11, 2025 •

edited

Loading

p-shahi commented Jun 13, 2025 •

edited

Loading