How to fix RR server errors during high load?

[codercms]

My RR version is: 2.10.4 (build time: 2022-06-10T23:29:27+0000, go1.18.3)
I'm running a Laravel app in RR with following config:

version: "2.7"

server:
  command: "php -d opcache.enable-cli=1 ./vendor/bin/rr-worker start --relay-dsn tcp://127.0.0.1:10000"
  relay: tcp://127.0.0.1:10000
  env:
    - IS_ROADRUNNER: "1"

http:
  address: 127.0.0.1:9000

  # override http error code for the internal RR errors (Default: 500)
  internal_error_code: 505

  # HTTP access logs (Default: false)
  access_logs: false

  # Maximal incoming request size in megabytes. Zero means no limit. (Default: 0)
  max_request_size: 100

  middleware: [ ]
  pool:
    max_jobs: 5000
    num_workers: 8

And doing workload test using hey utility:

/hey -n 10000 -c 100 http://127.0.0.1:9004/endpoint

Summary:
  Total:        62.9507 secs
  Slowest:      1.8121 secs
  Fastest:      0.0148 secs
  Average:      0.6251 secs
  Requests/sec: 158.8545


Response time histogram:
  0.015 [1]     |
  0.195 [40]    |■
  0.374 [1892]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.554 [1864]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.734 [2812]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.913 [2262]  |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  1.093 [911]   |■■■■■■■■■■■■■
  1.273 [146]   |■■
  1.453 [30]    |
  1.632 [33]    |
  1.812 [9]     |


Latency distribution:
  10% in 0.2893 secs
  25% in 0.4601 secs
  50% in 0.6294 secs
  75% in 0.7801 secs
  90% in 0.9267 secs
  95% in 0.9810 secs
  99% in 1.1939 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0148 secs, 1.8121 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0136 secs
  resp wait:    0.6250 secs, 0.0138 secs, 1.8120 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0246 secs

Status code distribution:
  [200] 9984 responses
  [505] 16 responses

There is 16 505 responses. Then I check container errors and see 16 errors:

network error, worker will be restarted {"reason": "network", "pid": 7417, "internal_event_name": "EventWorkerError", "error": "sync_worker_exec: Network:\n\tsync_worker_exec_payload: EOF"}
1.6559827890104344e+09  error   http    execute {"start": 1655982787.8992217, "elapsed": 1.111210197, "error": "sync_worker_exec: Network:\n\tsync_worker_exec_payload: EOF"}
worker stopped  {"internal_event_name": "EventWorkerWaitExit", "error": "signal: killed; process_wait: signal: killed", "errorCauses": [{"error": "signal: killed"}, {"error": "process_wait: signal: killed"}]}

UPD: Is this somehow related to #1008 ?
UPD2: Maybe RR doesn't implement graceful worker reload?

[rustatian]

Hey @codercms 👋🏻
max_jobs: 5000 this option means the following: after the 5000 executions reached per worker, RR should restart the worker.
You may switch to pipes for the server instead of TCP. Because when the worker stopped, there is no way to correctly answer via tcp relay. Thus you see these errors.

[codercms]

@rustatian I updated config to:

server:
  command: "php -d opcache.enable-cli=1 ./vendor/bin/rr-worker start"
  relay: pipes
  env:
    - IS_ROADRUNNER: "1"

But unfortunately it doesn't help:

network error, worker will be restarted {"reason": "network", "pid": 128, "internal_event_name": "EventWorkerError", "error": "sync_worker_exec: Network:\n\tsync_worker_exec_payload: EOF"}
1.6559875608195963e+09  error   http    execute {"start": 1655987560.2741997, "elapsed": 0.545394998, "error": "sync_worker_exec: Network:\n\tsync_worker_exec_payload: EOF"}
worker stopped  {"internal_event_name": "EventWorkerWaitExit", "error": "exit status 255; process_wait: exit status 255", "errorCauses": [{"error": "exit status 255"}, {"error": "process_wait: exit status 255"}]}

[rustatian]

Try to use wrk instead of hey. Or remove max_jobs and use supervisor.

[codercms]

@rustatian I dug deeper for debugging and found a memory leak in the PHP worker process, the problem is in the laravel/sanctum package (how it deals with AuthManager).
I added SanctumServiceProvider to providers reset list and boom! The problem is fixed! Thanks a lot for help

[codercms]

P.S. Will try to make it work without resetting service provider