Terrible performance after switching from nginx-unit to granian

[dxks]

Hi,

with NetBox 4.5.2, it's necessary to use NetBox Docker version 4.0.0 and switch to Granian as the application/web server.

After the switch, we've observed performance issues with NetBox 4.5.2 and 4.5.3 as soon as the system is under load (30+ clients). The time required for a GraphQL query can increase tenfold (from 2 to 20 seconds).

Using the Docker container without further configuration doesn't seem to be straightforward.

We're currently wondering if other users have encountered similar performance problems and if there are any solutions to restore the previous nginx performance, as it appears Granian requires significantly more configuration than nginx-unit.

Kind Regards,
Kay

[cr-labs]

Hi. I've been wrestling with the same problem as I have been testing 4.5.5 in staging as an upgrade from 4.4.7. There is an easy fix to get your app unjammed.

We use the community Helm chart with Kubernetes. Your situation may be a little different.

Netbox is launched by this script. Notice that if GRANIAN_WORKERS is not set, we get 4 workers. We could provision 50 CPUs and we'd still get 4 workers.

$ cat /opt/netbox/launch-netbox.sh
#!/bin/bash
exec granian \
  --host "::" \
  --port "8080" \
  --interface "wsgi" \
  --no-ws \
  --workers "${GRANIAN_WORKERS:-4}" \
...

Here are all the Granian config options: https://github.com/emmett-framework/granian?tab=readme-ov-file#options
There are many, many, many ways to tune performance BUT if you read the detailed discussion of "Workers and threads" you will find this little gift that says it's ok to start small and get fancy later:

In general, Granian will try its best to automatically pick proper values for the threading configuration, leaving to you the responsibility to choose the number of workers you need.

So we need to set GRANIAN_WORKERS to something > 4.

Granian config options can be set as switches or ENV vars. Since netbox is launched by a script that can't change, we need to set an ENV var.

IIRC the off the shelf guideline for gunicorn was something like "2 * CPUs + 1".

I did a little bit of perf testing and found on our machines, I could get to around 100% CPU utilization on one CPU by driving 3 to 4 cores.. so there's a limit. But the workers need time to recover and we already know that "4" isn't enough. For starters, I tested with 10 workers but decided to roll out (6 * CPUs + 1). For one CPU, that's 7 workers; 2 CPUs = 13 workers. etc. If loads are constant and high, that may be too many. If loads are spotty but hit hard (like terraform plan) then it may be fine. This is a new development, so I will be watching the charts and working on pushing for max performance.

Since we're using Kubernetes, this is the change to the values file to create 7 workers. See performance measures below.

netbox:
  extraEnvs:
    - name: "GRANIAN_WORKERS"
      value: "7"

Performance

Before

Some example timings from tofu plan, three runs, each one right after the other. You can see performance fall apart.

tofu plan  1.02s user 0.41s system 57% cpu 2.479 total
tofu plan  0.75s user 0.26s system 3% cpu 29.118 total
tofu plan  0.67s user 0.23s system 3% cpu 29.352 total

After

Even with wild parallelism (we aren't configuring 200 APIs worth of Netbox stuff, but there are a few dozen API endpoints hit at once), no problem now vs. huge problem before

tofu plan -parallelism=200  0.64s user 0.21s system 44% cpu 1.909 total
tofu plan -parallelism=200  0.62s user 0.26s system 41% cpu 2.139 total
tofu plan  0.71s user 0.22s system 42% cpu 2.201 total
tofu plan  0.73s user 0.29s system 31% cpu 3.257 total