Risk of inconsistent API response data

[TribuneX]

Problem

We store devices in NetBox with 1000+ interfaces and periodically query them via GraphQL, including related data. These queries retrieve large datasets, including all interfaces of a device.

We observed that responses sometimes contain inconsistent data. For example, the description of a recently updated interface may be missing or outdated within the same response.

Potential Root Cause

Write operations in NetBox are executed within database transactions, while read operations (via REST API or GraphQL) are not explicitly wrapped in transactions.

Even if read queries were executed within a transaction, PostgreSQL’s default isolation level, Read Committed, allows a query to see data committed by other transactions during its execution.

By analyzing the SQL generated by our GraphQL queries, we found that fields such as the interface description are fetched via multiple separate SELECT statements within a single GraphQL request. This likely leads to inconsistent snapshots of the data being returned.

Workaround

As a workaround, we enabled ATOMIC_REQUESTS and set the transaction isolation level to REPEATABLE READ (see also django docs).

With this setup, each request runs within a single transaction and sees a consistent snapshot of the database, eliminating the inconsistencies observed in read queries.

Request for Feedback

We are interested in how others handle this issue and whether there are known drawbacks to running NetBox with ATOMIC_REQUESTS enabled and the isolation level set to REPEATABLE READ. Since the django docs include a warning for this level:

Under higher isolation levels, your application should be prepared to handle exceptions raised on serialization failures. This option is designed for advanced uses.

One issue we have encountered is that seemingly read-only requests may still perform write operations (e.g., updating the last-used timestamp of an API token). Under higher isolation, this can lead to errors such as:

could not serialize access due to concurrent update

We appreciate any insights or best practices regarding this setup.