Netbox query performance degradation (v4.5.4)

[weakcamel]

First of all, thanks for Netbox - I've started using it relatively lately but already loving it and can't believe we survived for years without it!

We have indeed upgraded recently to netbox 4.5.4 (unfortunately I'm not sure what was the previous version, likely an early 4.x), our deployment is running on Kubernetes (Azure cluster). Unfortunately, we're noticing a performance degradation rather than improvement. Since the upgrade, our lookups (from Ansible, using netbox.netbox collection 3.22.0) have started failing when the number of parallel queries reaches ~10; in the past this just didn't happen at all, no matter the parallelism.

Details of the deployment:

• netbox and pod: 3 CPU, 6 GB RAM (but monitoring doesn't ever show getting even to half of that)
• netbox-worker pod: same as netbox

Observations from running a simple AI-generated script:

$ ./netbox_perf.py --api-key "$API_KEY" --threads 10 --duration 10
Running test against https://netbox.example.com
Target hostname: gpu002.mgmt.example.com
Duration: 10s, Threads: 10

=== NetBox Performance Summary ===
req=201 ok=201 (100.0%) rps=19.3
latency[s]: avg=0.5070 min=0.2845 max=1.4392 (p50=0.4366 p95=0.9352 p99=1.2315)

$ ./netbox_perf.py --api-key "$API_KEY" --threads 20 --duration 10
Running test against https://netbox.example.com
Target hostname: gpu002.mgmt.example.com
Duration: 10s, Threads: 20

=== NetBox Performance Summary ===
req=250 ok=239 (95.6%) rps=18.5
latency[s]: avg=0.8636 min=0.2835 max=5.0042 (p50=0.5764 p95=1.9896 p99=5.0039)

$ ./netbox_perf.py --api-key "$API_KEY" --threads 50 --duration 10
Running test against https://netbox.example.com
Target hostname: gpu002.mgmt.example.com
Duration: 10s, Threads: 50

=== NetBox Performance Summary ===
req=301 ok=234 (77.7%) rps=21.9
latency[s]: avg=1.7822 min=0.2799 max=5.1338 (p50=0.8995 p95=5.1039 p99=5.1293)

$ ./netbox_perf.py --api-key "$API_KEY" --threads 100 --duration 10
Running test against https://netbox.example.com
Target hostname: gpu002.mgmt.example.com
Duration: 10s, Threads: 100

=== NetBox Performance Summary ===
req=429 ok=259 (60.4%) rps=29.8
latency[s]: avg=2.5274 min=0.2826 max=5.2704 (p50=1.1716 p95=5.2287 p99=5.2611)

10 parallel queries for a specific IP address in IPAM (Edit: device ) in Netbox are doing fine. 20 of them hare a 95% success rate and if gets worse and worse with the number of parallel queries.

Any ideas how to improve that? is it just me or has 4.5.2 indeed make Netbox slower rather than faster?

[Markethh]

I may be seeing something similar after upgrading my dev environment from 4.4.10 to 4.5.9. Requests to 4.5 are frequently timing out, whereas the requests to 4.4 succeed quickly.

Similarly lowering the number of parallel requests succeeds more often (though is still significantly slower than the same requests on 4.4, with occasional timeouts).