[Bug]: nb_inventory is taking several minutes to run due to the "limit&offset" behaviour. Can we turn that off?

[alice-rc]

Ansible NetBox Collection version

v3.20.0

Ansible version

ansible [core 2.17.7]
  config file = <REDACTED>/ansible.cfg
  configured module search path = ['<REDACTED>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = <REDACTED>/lib64/python3.11/site-packages/ansible
  ansible collection location = /<REDACTED>/collections
  executable location = <REDACTED>/bin/ansible
  python version = 3.11.9 (main, Sep 11 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2.0.1)] (<REDACTED>/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True

NetBox version

NetBox Community v4.1.11

Python version

3.11

Steps to Reproduce

$ date; ansible-inventory -i inventory/nb_test_vms.yml --list -vvv 2>&1 >testout; date

Content of inventory config file:

---
plugin: netbox.netbox.nb_inventory
api_endpoint: "https://netbox.\<REDACTED>"
validate_certs: false
flatten_custom_fields: false
plurals: false
group_by:
  - tenant
group_names_raw: true
query_filters:
  - tenant__n: \<REDACTED\>
device_query_filters:
  - id: -1
prefixes: false
services: false
fetch_all: true
racks: false
rename_variables:
  - pattern: custom_fields
    repl: nb_inventory_custom_fields
  - pattern: is_virtual
    repl: nb_inventory_is_virtual

Expected Behavior

The plugin should not be taking several minutes to run. Need an option to turn off the "limit" behavior.

Using just a single limit, takes 14 seconds

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?limit=0&tenant__n=<REDACTED>&exclude=config_context" -H "Authorization: Token $NETBOX_TOKEN" > /dev/null; date
Tue 14 Jan 16:33:58 UTC 2025
Tue 14 Jan 16:34:12 UTC 2025

Using no limit, takes 2 seconds

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?tenant__n=<REDACTED>&exclude=config_context" -H "Authorization: Token $NETBOX_TOKEN" > /dev/null; date
Tue 14 Jan 16:35:54 UTC 2025
Tue 14 Jan 16:35:56 UTC 2025

Observed Behavior

Output from the above inventory command:
Inventory plugin takes almost 7 minutes

Tue 14 Jan 01:14:36 UTC 2025
Tue 14 Jan 01:21:27 UTC 2025

Beginning of the output file:

...
Fetching: https://netbox.<REDACTED>/api/status
Fetching: https://netbox.<REDACTED>/api/dcim/devices/?limit=0&tenant__n=<REDACTED>&id=-1&exclude=config_context
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?limit=0&tenant__n=<REDACTED>&exclude=config_context
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=1000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=2000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=3000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=4000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=5000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=6000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=7000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=8000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=9000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=10000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=11000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=12000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=13000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=14000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=15000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=16000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=17000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=18000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=19000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=20000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=21000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=22000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=23000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=24000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=25000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=26000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=27000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=28000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=29000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=30000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=31000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/dcim/sites/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/regions/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/site-groups/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/locations/?limit=0
Fetching: https://netbox.<REDACTED>/api/tenancy/tenants/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/device-roles/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/platforms/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/device-types/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/manufacturers/?limit=0
Fetching: https://netbox.<REDACTED>/api/virtualization/clusters/?limit=0
Parsed <REDACTED>/inventory/nb_test_vms.yml inventory source with auto plugin
...

[sc68cal]

Can you also do a test where you do the API call without the limit, then with the limit to just make sure it's not the case where the first API call warmed up a cache?

[alice-rc]

Curl doesn't cache anything, but incase there could be caching on netbox itself....

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?tenant__n=<REDACTED>&exclude=config_context&foo=$RANDOM" -H "Authorization: Token $NETBOX_TOKEN" -H "Cache-Control: no-cache" > test_nolimit; date
Wed 15 Jan 18:24:31 UTC 2025
Wed 15 Jan 18:24:33 UTC 2025

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?limit=0&tenant__n=<REDACTED>&exclude=config_context&foo=$RANDOM" -H "Authorization: Token $NETBOX_TOKEN" -H "Cache-Control: no-cache" > test_limit; date
Wed 15 Jan 18:24:54 UTC 2025
Wed 15 Jan 18:25:07 UTC 2025

$ cat test_nolimit | jq . | grep \"count\"
  "count": 30978,
$ cat test_limit | jq . | grep \"count\"
  "count": 30978,

[alice-rc]

But I'm not so concerned with the performance of the raw curl, but more with the performance of the curl commands vs the nb_inventory plugin. 2 seconds -> 13 seconds is far less concerning than 2/13 seconds to almost 7 minutes

[sc68cal]

I understand. However looking at your logs it appears that you have over 31k virtual machines.

Thank you for checking that we are getting the same amount of machines with the limit vs non limit query because that was going to be my next suspicion as well.

We need to investigate why the inventory plugin is pulling in chunks, perhaps at one point it was needed in order to not hammer NetBox but it appears that forcing NetBox to do pagination is making performance worse now.

[alice-rc]

So doing more investigation on this it seems that there are some NetBox configuration settings that come into play here. The big one is that, by default, MAX_PAGE_SIZE is set to 1000 which is what seems to be the underlying reason for the limit&offset. That being the case I'm not sure how we can get around the limit&offset restriction other than to use the GraphQL API instead of the regular API as it does not have that page limit restriction. That introduces a whole other complexity as there seems to be current limitations on filtering when using the the GraphQL API (netbox-community/netbox#17688).
Our biggest issue is that the nb_inventory plugin is unusable given that it takes 7 minutes to run.

We are currently investigating how we could write a custom inventory plugin using the GraphQL API as we don't have a broad range of filter or field retrieval requirements.

[sc68cal]

Have you enabled caching?