Commit 800565b
committed
erts: optimize CA tree node cache-line layout for ordered_set
The CA tree (contention-adapting tree) backs ETS ordered_set tables.
Profiling the struct layout on 64-byte cache lines reveals two
performance problems that this commit fixes by reordering struct fields
— no code changes, no memory overhead.
Problem 1: Base node false sharing
In the old layout, lock_statistics (written by contending threads
WITHOUT the lock held) sits on the same cache line (CL1) as lock tail
fields and is_valid (read/written by the lock holder). When a contender
bumps lock_statistics, the hardware invalidates the entire 64-byte cache
line on the lock holder's core, forcing a re-fetch of unrelated lock
internals.
Fix: Move root and is_valid up (CL1, co-located with lock tail — all
accessed under the lock, no conflict). Move lock_statistics down to CL2,
isolated with only the cold free_item field. Contenders now dirty a
cache line the lock holder never touches.
Before CL1: [lock.q | lock.type | lock.rq_end | lock.tdata | lock_statistics | is_valid]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^
lock holder reads/writes these contenders write this
→ FALSE SHARING
After CL1: [lock.q | lock.type | lock.rq_end | lock.tdata | root | is_valid]
all accessed under the lock — no cross-core conflict
After CL2: [lock_statistics | free_item]
contenders write here — isolated from lock holder
Problem 2: Route node poor cache locality
During lock-free traversal (find_base_node), each route node requires
reading: is_base_node (CL0), key.term (CL2), then left or right (CL1) —
3 cache-line fetches per node. For a tree of depth D, that is 3D
cache-line loads to reach a base node.
Fix: Move left and right to the front of the route node struct, placing
them on CL0 alongside is_base_node. The traversal loop now reads
is_base_node (CL0), key.term (CL2), then left/right (CL0, already
cached) — 2 cache-line fetches per node, a 33% reduction in cache misses
during traversal.
Before: is_base_node[CL0] → key.term[CL2] → left/right[CL1] = 3 CLs
After: is_base_node[CL0] → key.term[CL2] → left/right[CL0] = 2 CLs
Both changes are pure field reordering in erl_db_catree.h. All field
accesses in erl_db_catree.c use named fields (->u.base.root,
->u.route.left, etc.), and allocation size macros use offsetof(), so
they auto-adjust. No .c file changes required.1 parent daf66ec commit 800565b
1 file changed
Lines changed: 6 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
48 | | - | |
49 | 47 | | |
| 48 | + | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
56 | 59 | | |
57 | 60 | | |
58 | 61 | | |
59 | | - | |
60 | | - | |
61 | 62 | | |
62 | | - | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| |||
0 commit comments