Skip to content

Commit aca9146

Browse files
committed
erts: optimize CA tree node cache-line layout for ordered_set
The CA tree (contention-adapting tree) backs ETS ordered_set tables. Profiling the struct layout on 64-byte cache lines reveals two performance problems that this commit fixes by reordering struct fields — no code changes, no memory overhead. Problem 1: Base node false sharing In the old layout, lock_statistics (written by contending threads WITHOUT the lock held) sits on the same cache line (CL1) as lock tail fields and is_valid (read/written by the lock holder). When a contender bumps lock_statistics, the hardware invalidates the entire 64-byte cache line on the lock holder's core, forcing a re-fetch of unrelated lock internals. Fix: Move root and is_valid up (CL1, co-located with lock tail — all accessed under the lock, no conflict). Move lock_statistics down to CL2, isolated with only the cold free_item field. Contenders now dirty a cache line the lock holder never touches. Before CL1: [lock.q | lock.type | lock.rq_end | lock.tdata | lock_statistics | is_valid] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^ lock holder reads/writes these contenders write this → FALSE SHARING After CL1: [lock.q | lock.type | lock.rq_end | lock.tdata | root | is_valid] all accessed under the lock — no cross-core conflict After CL2: [lock_statistics | free_item] contenders write here — isolated from lock holder Problem 2: Route node poor cache locality During lock-free traversal (find_base_node), each route node requires reading: is_base_node (CL0), key.term (CL2), then left or right (CL1) — 3 cache-line fetches per node. For a tree of depth D, that is 3D cache-line loads to reach a base node. Fix: Move left and right to the front of the route node struct, placing them on CL0 alongside is_base_node. The traversal loop now reads is_base_node (CL0), key.term (CL2), then left/right (CL0, already cached) — 2 cache-line fetches per node, a 33% reduction in cache misses during traversal. Before: is_base_node[CL0] → key.term[CL2] → left/right[CL1] = 3 CLs After: is_base_node[CL0] → key.term[CL2] → left/right[CL0] = 2 CLs Both changes are pure field reordering in erl_db_catree.h. All field accesses in erl_db_catree.c use named fields (->u.base.root, ->u.route.left, etc.), and allocation size macros use offsetof(), so they auto-adjust. No .c file changes required.
1 parent 8c0b3d8 commit aca9146

1 file changed

Lines changed: 6 additions & 6 deletions

File tree

erts/emulator/beam/erl_db_catree.h

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,23 +44,23 @@ typedef struct {
4444

4545
typedef struct {
4646
erts_rwmtx_t lock; /* The lock for this base node */
47-
erts_atomic_t lock_statistics;
48-
bool is_valid; /* If this base node is still valid */
4947
TreeDbTerm *root; /* The root of the sequential tree */
48+
bool is_valid; /* If this base node is still valid */
49+
erts_atomic_t lock_statistics;
5050
ErtsThrPrgrLaterOp free_item; /* Used when freeing using thread progress */
5151

5252
char end_of_struct__;
5353
} DbTableCATreeBaseNode;
5454

5555
typedef struct {
56+
erts_atomic_t left;
57+
erts_atomic_t right;
58+
erts_mtx_t lock; /* Used when joining route nodes */
5659
#ifdef ERTS_ENABLE_LOCK_CHECK
5760
Sint lc_order;
5861
#endif
59-
ErtsThrPrgrLaterOp free_item; /* Used when freeing using thread progress */
60-
erts_mtx_t lock; /* Used when joining route nodes */
6162
bool is_valid; /* If this route node is still valid */
62-
erts_atomic_t left;
63-
erts_atomic_t right;
63+
ErtsThrPrgrLaterOp free_item; /* Used when freeing using thread progress */
6464
DbRouteKey key;
6565
} DbTableCATreeRouteNode;
6666

0 commit comments

Comments
 (0)