Add covering indexes for Machine Tracker searches

[adobloug]

Scope and purpose

Adds covering database indexes on arp and cam to speed up Machine Tracker
IP address and MAC address searches.

The Machine Tracker views (nav.web.machinetracker) query arp/cam
filtering on an ip- or mac-range together with end_time, ordered by that
same leading column:

• IP search: WHERE ip BETWEEN .. AND end_time > .. ORDER BY ip, mac, start_time
• MAC search (arp): WHERE mac BETWEEN .. AND end_time > .. ORDER BY mac, ip, start_time
• MAC search (cam): WHERE mac BETWEEN .. AND end_time > .. ORDER BY mac, sysname, module, port

The stock single-column indexes (arp_ip_btree, arp_mac_btree,
cam_mac_btree) cover only the leading column, so end_time is applied as a
post-scan Filter — every historical row in the ip/mac range is read and then
discarded — and for the IP search the planner instead BitmapAnds the ip and
end_time indexes, scanning the whole end_time index.

This PR adds three indexes that fold end_time into the index key so it becomes
part of the Index Cond, and let the leading column supply the ORDER BY
prefix for free:

• arp_ip_end_time_btree (ip, end_time) INCLUDE (mac, start_time)
• arp_mac_end_time_btree (mac, end_time)
• cam_mac_end_time_btree (mac, end_time) INCLUDE (netboxid, port, sysname)

Verified on a production database with EXPLAIN (ANALYZE, BUFFERS): the new
indexes are selected, end_time moves from a post-scan Filter into the
Index Cond, and the leading column supplies the sort prefix. The figures cited
below (e.g. cam MAC search 10 197 → 293 buffers; arp IP search BitmapAnd
replaced by a single index scan, 6 598 → 1 593 buffers) come from one production
database and illustrate the potential gain only — absolute numbers depend on
table size, data distribution and cache state. The INCLUDEd columns additionally let ad-hoc
ip/mac-over-time reporting queries — those selecting only the included columns —
run as near index-only scans; the Machine Tracker views themselves fetch full
rows, so they gain from the Index Cond and sort-prefix improvements rather
than from covering.

This pull request

• changes the database

Contributor Checklist

• Added a changelog fragment for towncrier
• Added/amended tests for new/changed code — N/A: schema-only change (new indexes), no code paths added/changed
• Added/changed documentation — N/A
• Linted/formatted the code with ruff — N/A: SQL only
• Wrote the commit message following https://cbea.ms/git-commit/
• Based this pull request on the correct upstream branch (master, this is an addition)
• If applicable: Created new issues — N/A
• Described how a reviewer can observe the effect (see below)
• If this results in changes in the UI — N/A
• If this adds a new Python source code file — N/A

How to observe

Machine Tracker's IP/MAC searches (nav.web.machinetracker) filter an
arp/cam row on an ip- or mac-range together with end_time, ordered by
that same leading column, all via BETWEEN (the cam and arp-MAC searches also
join netbox through select_related). The stock single-column indexes cover
only the leading column, so end_time is applied as a post-scan Filter. The
new indexes fold end_time into the index key — it becomes part of the
Index Cond — and the leading column supplies the ORDER BY prefix for free.

Note

The figures below are from one production database and illustrate the
potential performance gain only. Absolute numbers vary with table size, data
distribution, cache state and hardware.

Run EXPLAIN (ANALYZE, BUFFERS) before and after navsyncdb (substitute a real
3-octet OUI for <oui>, e.g. aa:bb:cc). The cam MAC search, measured on a
production database over one OUI range — 191 active rows out of ~10 000
historical rows in that range (the netbox join these views add is unaffected
by the new indexes and is omitted here for clarity):

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM cam
WHERE end_time > (CURRENT_DATE - 7)
  AND mac BETWEEN '<oui>:00:00:00' AND '<oui>:ff:ff:ff'
ORDER BY mac, sysname, module, port, start_time DESC;

Before — end_time is a post-scan Filter on cam_mac_btree, so all
10 424 rows in the mac range are read and 10 233 are thrown away:

 Index Scan using cam_mac_btree on cam  (actual time=4.109..76.068 rows=191 loops=1)
   Index Cond: ((mac >= '98:90:96:00:00:00') AND (mac <= '98:90:96:ff:ff:ff'))
   Filter: (end_time > (CURRENT_DATE - 7))
   Rows Removed by Filter: 10233
   Buffers: shared hit=1493 read=8704
 Execution Time: 76.680 ms

After — end_time folds into Index Cond on cam_mac_end_time_btree;
only the 191 matching rows are visited:

 Index Scan using cam_mac_end_time_btree on cam  (actual time=0.086..0.783 rows=191 loops=1)
   Index Cond: ((mac >= '98:90:96:00:00:00') AND (mac <= '98:90:96:ff:ff:ff') AND (end_time > (CURRENT_DATE - 7)))
   Buffers: shared hit=293
 Execution Time: 1.444 ms

Buffers drop from 10 197 to 293 (~35×), execution time from 77 ms to 1.4 ms
(~53×), and the post-scan Filter disappears (no more Rows Removed by Filter).

The arp IP search benefits even more, because without the combined index the
planner resorts to a BitmapAnd of arp_ip_btree and arp_end_time_btree —
and the end_time side scans hundreds of thousands of entries. Measured over a
/22 (1 003 active rows):

Before — two-index BitmapAnd (the end_time bitmap alone scans 639 019
rows), followed by a full Sort:

 Sort  (actual time=320.809..320.887 rows=1003 loops=1)
   Sort Key: ip, mac, start_time DESC
   Buffers: shared hit=5694 read=904
   ->  Bitmap Heap Scan on arp  (actual time=319.156..320.157 rows=1003 loops=1)
         Recheck Cond: ((ip >= '..') AND (ip <= '..') AND (end_time > (CURRENT_DATE - 7)))
         ->  BitmapAnd
               ->  Bitmap Index Scan on arp_ip_btree        (rows=97812)
               ->  Bitmap Index Scan on arp_end_time_btree  (rows=639019)
 Execution Time: 320.957 ms

After — a single arp_ip_end_time_btree scan with end_time in the
Index Cond and ip supplying the sort prefix (Incremental Sort):

 Incremental Sort  (actual time=0.270..9.075 rows=1003 loops=1)
   Sort Key: ip, mac, start_time DESC
   Presorted Key: ip
   Buffers: shared hit=1593
   ->  Index Scan using arp_ip_end_time_btree on arp  (actual time=0.022..8.264 rows=1003 loops=1)
         Index Cond: ((ip >= '..') AND (ip <= '..') AND (end_time > (CURRENT_DATE - 7)))
         Buffers: shared hit=1593
 Execution Time: 9.129 ms

Buffers drop from 6 598 to 1 593 and execution time from 321 ms to 9 ms (~35×).
The arp MAC search improves the same way via arp_mac_end_time_btree.

INCLUDE columns — ad-hoc reporting

The Machine Tracker views fetch full rows (and join netbox), so they always
visit the heap; the INCLUDEd columns do not make those queries index-only.
Where INCLUDE pays off is ad-hoc "locate hosts by ip/mac over time" reporting
that selects only the indexed/included columns — such a query runs as a near
Index Only Scan. For example, a subnet lookup expressed with the <<
network-contains operator (which Postgres rewrites into the same btree range
bounds, applying exact containment as a recheck Filter):

EXPLAIN (ANALYZE, BUFFERS)
SELECT ip, end_time, mac, start_time FROM arp
WHERE end_time > (CURRENT_DATE - 7)
  AND ip << '10.0.60.0/22'
ORDER BY ip;

 Index Only Scan using arp_ip_end_time_btree on arp  (actual time=0.023..7.437 rows=1003 loops=1)
   Index Cond: ((ip > '10.0.60.0/22') AND (ip <= '10.0.63.255') AND (end_time > (CURRENT_DATE - 7)))
   Filter: (ip << '10.0.60.0/22')
   Heap Fetches: 370
   Buffers: shared hit=1509

(The residual Heap Fetches are rows on heap pages not yet marked all-visible
in the visibility map, not columns missing from the index.)

Note

Creating these indexes may take a while during navsyncdb, as the arp and
cam tables are typically large.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud