RPCs broken at higher pings

[TheYellowArchitect]

Version Number and Operating System(s):

Windows 7 x64
Unity 2018.1.1f1
Forge Version 3/10/2018 - 19.53.38,04 (I downloaded it in 2021, it is the latest version of master branch)

Expected behavior:

Ping increases, packets are delayed for half the ping distance
So, for a connection of 100 ping, latest packet sent by an RPC, has been received after ~50 ms

Actual behavior:

Ping increases, packets are delayed almost exponentially
On a 110 ping connection, packets are delayed for even 2 seconds.

Ping.increasing.bytes.received.mp4

Look.at.packet.delay.of.latest.snapshot.on.100.ping.mp4

Steps to reproduce:

1. Download https://www.mediafire.com/file/5t32ydu7ex8b48o/ForgeRPCBug.zip/file (my project, but deleted Library folder)
2. Open it in Unity 2018.1.1f1
3. Open ActiveGameScene
4. Press Play
5. Press PageUp if you have no controller, to activate keyboard emulation

^I use parrelsync to get 2 builds playing at the same time. If you don't use ParrelSync, see Update() in MultiplayerMenu.cs to Host/Connect.
Remove the #if definition if you make a build

Important Notes:

I thought this was a bug with the program clumsy. A mistake in emulation.
After a month, I tested in a real network to confirm. I uploaded my game on Steam, gave some keys to my friend, put steam host/invite functionality, and tested with my friend.
100 ping gameplay on steam, was exactly the same as local clumsy emulation.

I pinpointed the bug in a single line, so as to make this post (it is instantly reproducible this way)
It is in the code where I send my position snapshot. It is the only RPC I send constantly, instead of user input or some action in-game (e.g. took damage)
It is on NetworkPositionInterpolationController.cs line 84. Comment this line, and you will find no issues.

For context, every 50 ms, host sends his position snapshot on client, and client interpolates properly, after having 2 snapshots. The classic. It works perfectly on 0 ping, and even a bit above, up to ~30 there isn't any serious issue. But damn, the bandwidth increases like crazy.
The packet I send is serialized by Forge, from my NCW, the RPC arguments, are a Vector2, and a float.

I also tested out of curiosity, if, other RPC calls, bug the bytes received/sent. I put 800 ping, and just sent inputs (Vector2), and the bytes received/sent went to insane numbers. So it really seems like a Forge bug, and not a bug with my code. I use this to get bytes sent/received

I cannot release my game on Steam like this, it's literally exclusively multiplayer (local multiplayer it works perfectly), and so, even though the game is literally finished, it's unplayable... On 100 ping, position packets take like 1.5 seconds to arrive, wtf...
The alternative if I fail to solve this, is to convert the entire netcoding to lockstep (rip 2 more months of development). But then, how do I know this packet bug won't happen again?!

[Optional] Discord Username:

Balroth
Thorlar#6572

[TheYellowArchitect]

I found out what causes this. All my RPCs are reliable.
So, sending the position every 50 ms, causes the client to send back confirmation it received it, in whatever process it has. I cannot confirm what exactly caused the delay (100 ms delay having 1 second delay -_-) but it definitely has to do with RPC reliable/unreliable, simply because changing 1 line to unreliable, fixes the above issue.

As for the bytes received, it's still a bug I think, either in the logic of packets, or in the NetworkerStatistics class itself (more likely, than Forge being broken ;)
I opened an issue exclusively for the bytes received

tl;dr: When you send position via RPC, just make it unreliable.

[TheYellowArchitect]

Reopening because it has been confirmed by 2 more developers, who stumbled onto this bug (packets delaying even up to 30 seconds)