gunicorn spends 100% of each request

[freedomwarrior]

Hello.
I got fresh install of netbox 3.4.7 + ubuntu 22.04
Some pages opens to long, average ~5s, and in same time gunicorn uses 100% of CPU.

[abhi1693]

How many objects are you opening per page? The greater the value, the longer it takes to open the page.

[candlerb]

It looks like the specific page you showed a waterfall for is /ipam/ip-addresses/ which is a listing of IP addresses.

What have you set the page length to? (See "Per Page" at the bottom of the table). If the page length is 1000 then 6 seconds doesn't sound too unreasonable - and I'd expect gunicorn to be chomping CPU while it does this.

Try setting the page length to 25 and see if it helps. If not, then further investigation will be required. Please provide a list of the specific URLs where it takes a long time to render.

[freedomwarrior]

Login page:

Providers page(only 2 providers there):

Pages are different and random.

[candlerb]

How sure are you that gunicorn is spinning at 100% for this period? If it wasn't for that, I'd say this is very likely an issue with reverse DNS lookups timing out after 5 seconds.

Is gunicorn actually spinning at 100% of the time, all the time? Is there other load on the system, e.g. API calls?

If the system is mostly idle, and gunicorn is only spinning at 100% while you're fetching the login page, then it might be instructive to attach strace to it. You can then see if it's doing lots of system call activity, or is just calculations.

[freedomwarrior]

I monitoring it with top, in idle state I did not see gunicorn in top, at the same time I load random page and this 5 sec I see gunicorn in top with 100% CPU, when the page is loaded - cpu dropped.
But I will check DNS theory and will give feedback about it.

[candlerb]

I monitoring it with top, in idle state I did not see gunicorn in top, at the same time I load random page and this 5 sec I see gunicorn in top with 100% CPU, when the page is loaded - cpu dropped.

Ok, that's good. So try running this at the shell (as root), before hitting the page:

strace -ftt -p <pid-of-gunicorn> 2>/var/tmp/strace.out

Then hit the page, then when it's finished loading, go back and hit ctrl-C on the strace command. Have a look in the file. Find the timestamps representing the start and end of that 5 second period. What syscalls do you see within that range? In particular, if you see a gap of 5 seconds, then can you show 20 lines either side of that gap?