Skip to content

Latest commit

 

History

History
420 lines (330 loc) · 20.1 KB

File metadata and controls

420 lines (330 loc) · 20.1 KB

Benchmark record

This document states what was measured, how it was measured, and what it does not prove. Generated regions are checked against committed JSON in CI.

Environment

Measurements were recorded on 2026-08-19.

Property Recorded value
Runtime .NET 8.0.30, Release, Server GC
OS / architecture Ubuntu 24.04.3 LTS, X64
Logical processors 8
Monotonic clock 1000 MHz
CRC-32C SSE4.2 (hardware)

The host is virtualized and shared. CPU frequency, core isolation, interrupt routing, NUMA placement, and NIC hardware were not controlled. Results are useful for regression and design comparison on this host; they are not hardware-independent capacity claims.

Method

  • Harnesses perform untimed warm-up and report the median of at least five trials. Min and max stay in JSON to expose noise.
  • Matching cases are state-preserving two-command cycles. Setup and generated scripts sit outside the timed region; resting population does not drift.
  • Book sweeps discard one full pass before recording, allowing tiered JIT promotion and shared interface call sites to stabilize across implementations.
  • Allocation uses GC.GetAllocatedBytesForCurrentThread on single-threaded paths and GC.GetTotalAllocatedBytes on concurrent queue paths.
  • Packet bytes and checksums are consumed so dead-code elimination cannot remove protocol work.
  • WAL timings stop at the policy's acknowledgement point; final disposal sync is outside the timed region. Range and recovery trials are warm-cache filesystem measurements.
  • Transport load is open-loop. Source timestamps precede dissemination, so backlog increases measured latency instead of reducing offered load.
  • CI runs smoke-sized benchmarks for rot detection; it does not gate performance on shared runners.

The harness is dependency-free rather than BenchmarkDotNet-based. It does not provide process isolation, CPU affinity, frequency stabilization, hardware counters, or overhead subtraction. Raw ranges are therefore part of the result.

Protocol v2

Artifact: bench/results/protocol-v2.json. One million base iterations, seven trials.

Case Packet Median Min–max Rate Allocation
seal incremental 50 B 27.7 ns 26.2–72.7 ns 36.15 M/s 0 B/op
validate incremental 50 B 18.6 ns 18.4–21.2 ns 53.65 M/s 0 B/op
validate max batch 1,394 B 579.8 ns 572.9–822.7 ns 1.73 M/s 0 B/op
validate max snapshot 1,395 B 204.1 ns 203.2–230.0 ns 4.90 M/s 0 B/op
encode + decode + apply 50 B 121.8 ns 121.1–354.3 ns 8.21 M/s 0 B/op

Batch validation scans 97 message boundaries; snapshot validation scans one large message. The end-to-end case includes seal, CRC validation, decoder locking, bounded reorder and A/B identity retention, sequencing, and in-place depth application. CRC correctness is checked against the standard 123456789 vector. Corruption tests assert that state and sequence do not advance.

Durable publication and recovery

Artifact: bench/results/durability-v2.json. Five trials; 5,000 append acknowledgements per trial, 50,000-message recovery log, and 100 ten-message range requests. The JSON embeds runtime and host metadata.

Append contract Policy Payload Median Min–max Rate Syncs/trial Allocation
OS page cache OsBuffered 64 B 976.7 ns 958.0–2,088.2 ns 1,023,815/s 0 0 B/op
periodic 1 ms SyncPeriodic 64 B 2,318.3 ns 2,149.5–3,507.9 ns 431,343/s 4 0 B/op
group commit 64 OsBuffered 64 B 27,301.7 ns 17,837.9–42,412.7 ns 36,628/s 79 0 B/op
fsync each SyncEachRecord 64 B 972,646.2 ns 898,242.4–1,196,500.7 ns 1,028/s 5,000 0 B/op
seal + packet WAL OsBuffered 50 B 813.7 ns 755.1–1,676.6 ns 1,228,894/s 0 0 B/op
Messages Checkpoint Full replay Checkpoint + tail Speed-up
50,000 47,500 24.68 ms (21.00–40.83) 2.92 ms (2.64–3.84) 8.45×
10-message range Queries Index entries Median Min–max Allocation
sparse index 100 196 73.6 µs 70.2–81.7 µs 1,736 B/request
segment scan 100 0 1,946.8 µs 927.1–2,530.3 µs 5,848 B/request

OS page cache includes framing, CRC-32C, and one unbuffered managed write into the kernel cache; it is not power-loss durability. seal + packet WAL also seals and validates the 50-byte feed packet. The 1 ms periodic case stresses group sync; the configurable server default is 200 ms. fsync each measures this virtual disk, not a portable storage latency.

Recovery trials alternate full-first and checkpoint-first order. The checkpoint is at sequence 47,500; complete segments before it are skipped. The sparse range index stores one entry per 256 records and incrementally follows the live tail. Both range cases copy the same ten payloads; the table isolates lookup strategy.

Matching engine

Artifact: bench/results/matching-v2.json. Each value is the median over 200,000 state-preserving cycles and five trials.

Resting orders Add + cancel Cancel + add Match + replenish Execution risk + book Policy + execution + book
100 96.3 ns 123.7 ns 101.3 ns 326.6 ns 590.2 ns
1,000 120.0 ns 79.9 ns 98.3 ns 334.6 ns 578.9 ns
10,000 78.6 ns 84.8 ns 107.9 ns 300.9 ns 625.1 ns
100,000 99.9 ns 120.9 ns 103.0 ns 364.5 ns 560.9 ns

Each cycle contains two engine commands. Match-plus-replenish uses exact 50-share removals and same-price replacements. Execution risk adds directional reservation accounting and ownership. Full policy also runs participant lookup, entitlement, kill, quantity/notional, position, token bucket, and credit checks. Both risk paths have separate steady-state zero-allocation tests. These values are not isolated-command latency. Growth with population can reflect cache and TLB pressure; hardware counters would be required to attribute it.

Aggregated books

Artifact: bench/results/books-v2.json. Every implementation sees the same seeded 200,000-operation stream. Setup is outside the timed region.

Depth Implementation Mixed ns/op Touch ns/op Top-10 ns/op Clear ns/op Publish B/op
10 SortedArrayBook 33.1 2.6 8.2 29.9 0
10 VectorizedBook 25.2 6.1 19.3 39.2 0
10 LadderBook 25.6 7.2 52.7 134.5 0
10 TreeBook 54.6 12.8 353.2 63.3 104
100 SortedArrayBook 47.8 2.7 10.2 30.2 0
100 VectorizedBook 37.2 6.1 20.3 50.2 0
100 LadderBook 27.6 7.3 52.8 1149.5 0
100 TreeBook 89.5 15.1 224.3 157.8 152
1,000 SortedArrayBook 64.5 2.5 7.4 32.6 0
1,000 VectorizedBook 53.8 6.0 19.6 227.6 0
1,000 LadderBook 35.9 7.5 53.0 12052.4 0
1,000 TreeBook 123.9 18.4 304.9 983.4 200

The ranking changes with depth: shifting contiguous arrays is competitive shallow, while direct price indexing wins deeper update workloads. Tree enumeration allocates a traversal stack and is kept as a contrasting structure, not the zero-allocation production publish path.

Queue hand-off

Artifact: bench/results/queue-v2.json. One producer, one consumer, one million items, capacity 8,192, seven trials. Concurrent cases busy-spin.

Queue Median Min–max Throughput Allocation
RingBuffer (single thread) 3.5 ns/item 3.5–4.1 285.7 M item/s 0.066 B/item
Channel (single thread) 69.5 ns/item 61.8–109.7 14.4 M item/s 0.001 B/item
RingBuffer (producer + consumer) 14.3 ns/item 11.3–27.2 69.8 M item/s 0.066 B/item
RingBuffer batched (prod + cons) 6.3 ns/item 5.8–7.2 158.5 M item/s 0.066 B/item
Channel (producer + consumer) 163.8 ns/item 100.3–273.9 6.1 M item/s 0.133 B/item

The small ring allocation is fixed task/harness setup divided by the item count; ring operations are allocation-free. Busy-spin throughput is not an energy, fairness, or end-to-end latency result.

Real market data

Artifacts: replay-sample-v2-AMZN.json and replay-sample-v2-MSFT.json. Five trials; throughput is the median. Committed gzip files are checksum-verified before tests.

Symbol Depth Book Transitions Exact Accuracy Msg/s
AMZN 10 SortedArray 19,999 19,999 100.0000% 589,186
AMZN 10 Vectorized 19,999 19,999 100.0000% 479,542
AMZN 10 Ladder 19,999 19,999 100.0000% 127,975
AMZN 10 Tree 19,999 19,999 100.0000% 115,348
MSFT 5 SortedArray 19,999 19,999 100.0000% 1,089,271
MSFT 5 Vectorized 19,999 19,999 100.0000% 1,326,013
MSFT 5 Ladder 19,999 19,999 100.0000% 581,543
MSFT 5 Tree 19,999 19,999 100.0000% 343,020

Transition replay seeds from the published book, applies one event, and compares the determined prefix with the exchange's next row. This is exact where a finite-depth source is observable. Cumulative replay is a separate observability experiment: hidden liquidity below depth N cannot be reconstructed from a depth-N history.

Full-session artifacts cover 269,747 AMZN, 112,672 GOOG, and 595,799 MSFT transitions: 978,218 exact transitions per implementation. Those artifacts predate transport v2 but exercise the same book/replay layer.

Protocol-v2 multicast

Artifact: bench/results/protocolv2-summary.json. Two instruments target 1,000 aggregate updates/s, batch limit 16, 1 ms partial-batch flush, 4 s warm-up, and 10 s measurement.

Subscribers Delivered msg/s Per subscriber Mean p50 p99 Gaps / CRC / divergence / stale
100 97,028 970 0.620 ms 0.366 ms 6.942 ms 0 / 0 / 0 / 0
500 485,393 971 1.663 ms 1.070 ms 12.150 ms 0 / 0 / 0 / 0

Loopback still copies each datagram into every local socket. A switched multicast network performs replication elsewhere; these runs do not model switch queues, NIC rings, packet loss, or propagation. The capacity run uses one line. Integration smoke tests exercise A/B publication and duplicate arbitration with zero sequence gaps.

Transport scaling (pre-v2 generation)

These runs predate protocol v2 and use a separate 4-vCPU host. Compare rows within this section; do not compare them with v2 results.

CPU Intel(R) Xeon(R) Processor @ 2.80GHz, 4 vCPU
CPU features avx2, avx512f, avx512bw, avx512dq, avx512vl, bmi1, bmi2, popcnt
Memory 15.7 GB
OS Ubuntu 24.04.4 LTS, kernel 6.18.5-fc-v20
Runtime .NET 8.0.30, all projects target net8.0
Build Release, Server GC
Topology server and load generator as separate processes on the same host

Unicast fan-out

TCP fan-out performs one write per subscriber per update. These runs have similar delivered rates but different subscriber counts:

Subscribers Feed rate Fan-out Mean latency
100 100 upd/s 9,490 msg/s 1.61 ms
1,000 10 upd/s 10,100 msg/s 18.63 ms

At approximately 10,000 messages/s, increasing the audience from 100 to 1,000 subscribers raised mean latency from 1.61 ms to 18.63 ms.

The full sweep, feed rate held at 100 updates/s aggregate:

Subscribers Fan-out (msg/s) Mean (ms) p50 p99 p99.9 Max Delivered Gen. rate Server CPU Host CPU Sustained
100 9,490 1.61 1.38 5.8 26.1 28.4 99.1% 96% 52.8% 18.2% yes
200 19,319 4.19 4.11 8.2 18.9 26.1 99.4% 97% 77.8% 92.6% yes
300 29,253 4.13 3.76 10.3 18.4 49.8 99.6% 98% 107.7% 141.2% yes
400 38,294 7.35 7.44 15.4 28.9 41.9 99.0% 97% 131.4% 186.4% yes
500 48,117 8.54 8.04 28.4 62.6 97.4 99.2% 97% 160.9% 242.4% yes
600 57,798 9.71 9.28 25.1 51.1 55.1 99.4% 97% 188.4% 286.9% yes
700 67,385 16.35 14.05 80.5 103.0 131.9 99.6% 97% 216.3% 344.0% yes
800 77,760 18.73 15.05 83.8 133.2 181.2 99.5% 98% 216.3% 355.6% yes
900 86,621 46.15 23.75 216.7 254.2 322.8 99.2% 97% 229.4% 378.8% yes

Every point sustained; 900 subscribers is the top of the sweep, not a measured limit. Server CPU rose from 52.8% to 229.4%; p99 reached 216.7 ms at 900 subscribers.

A second sweep holds the message rate constant on a lighter feed:

Subscribers Fan-out (msg/s) Mean (ms) p50 p99 p99.9 Max Delivered Gen. rate Server CPU Host CPU Sustained
1,000 10,100 18.63 18.05 50.0 61.1 61.6 100.0% 101% 42.8% 65.1% yes
2,000 19,868 34.97 35.15 77.3 137.2 137.9 103.5% 96% 68.7% 121.1% yes
3,000 30,100 55.35 49.95 223.2 264.4 291.9 100.3% 100% 108.9% 198.3% yes
4,000 39,998 69.22 65.25 180.2 289.6 357.2 100.0% 100% 134.1% 253.9% yes
5,000 50,334 96.83 87.95 321.1 387.1 422.2 100.7% 100% 157.4% 304.2% yes

Multicast fan-out

The publisher encodes each update once and sends a single datagram; the network performs the replication.

Subscribers Fan-out (msg/s) Mean (ms) p50 p99 Max Delivered Server pkt/s Gaps Stale Server CPU Host CPU Sustained
100 9,906 0.31 0.28 0.9 6.0 99.4% 99.7 0 0 13.8% 12.7% yes
250 24,762 0.54 0.50 1.3 23.1 99.5% 99.5 0 0 16.5% 16.0% yes
500 49,524 0.85 0.81 2.1 5.3 99.4% 99.6 0 0 19.0% 15.8% yes
1,000 97,499 1.68 1.63 4.0 9.6 98.9% 98.6 0 0 25.8% 47.4% yes
2,000 199,895 3.22 2.99 8.5 24.0 99.6% 100.3 0 0 30.9% 166.3% yes
4,000 392,451 11.13 9.95 35.6 127.9 99.2% 98.9 0 0 49.0% 358.7% yes
6,000 594,067 34.39 23.75 184.7 470.3 99.9% 99.1 0 0 73.2% 372.5% yes
8,000 594,357 744.52 350.45 10365.0 24032.8 91.2% 81.5 594 0 91.6% 373.9% NO

Server packet rate stayed between 98.6 and 100.3/s through 6,000 subscribers. The 8,000-subscriber run was not sustained and recorded 594 sequence gaps.

Subscribers Unicast mean Multicast mean Improvement
100 1.61 ms 0.31 ms 5.3×
500 8.54 ms 0.85 ms 10.1×

Cost per delivered message, at each transport's highest sustained point:

Transport Highest sustained subscribers Messages/s Server CPU Server CPU per message
Unicast gRPC 900 86,621 229.4% 26.48 µs
Multicast 6,000 594,067 73.2% 1.23 µs

The next multicast point failed; no unicast failure point was measured.

Batching

The following runs use 1,000 subscribers and 1,000 aggregate updates/s while varying packet batch size.

Max batch Fan-out (msg/s) Mean (ms) p99 Server pkt/s Server CPU Host CPU
1 866,035 3.32 14.3 890.0 78.1% 377.7%
4 971,437 2.25 5.6 299.6 51.2% 244.5%
16 972,472 2.18 5.2 199.9 46.7% 172.3%
64 971,792 2.09 4.7 201.0 45.0% 170.9%

Batch 64 versus batch 1 reduced packet rate 4.4×, mean latency 1.6×, p99 3.1×, and host CPU 2.2×; delivered throughput increased 12%.

Repeatability

Point Runs Median Min Max Spread
4,000 subscribers, 10 upd/s 3 47.18 ms 43.25 60.35 1.40×
500 subscribers, 100 upd/s 3 8.64 ms 7.73 8.66 1.12×

The 4,000-subscriber configuration varied 1.4× across three runs. Sweep rows are single runs.

Market realism

Exact book reconstruction does not make generated order flow realistic. Mid-price returns sampled every 20 updates provide a blunt distributional check:

Series Observations Excess kurtosis |r| ac(1) |r| ac(10) >3σ >5σ
AMZN (real) 13,487 23.49 0.1793 0.1149 1.690% 0.289%
GOOG (real) 5,633 24.20 0.2262 0.0915 1.207% 0.373%
MSFT (real) 29,789 14.53 0.0253 0.0391 0.940% 0.940%
simulator 19,999 7,969.82 0.2244 0.2808 0.620% 0.055%

The real sessions show fat tails and persistent absolute-return autocorrelation. The simulator's extreme kurtosis with lower tail counts describes a nearly static price punctuated by rare jumps, not a market-like return distribution. The simulator is therefore a systems load generator, not a market model. Distribution-dependent studies use real data.

Artifact boundary

Files carrying v2 or protocolv2 are the current protocol/recovery record. Unsuffixed JSON holds the earlier transport sweeps, microstructure study, regenerated controls, and repeatability runs, presented under Transport scaling.

The generations use different hosts and are not compared. docgen.py rejects mixed kernel instances within either generation.

Reproduce

dotnet build MarketDataSimulator.sln -c Release

dotnet run --project Bench -c Release --no-build -- \
  protocol --iterations 1000000 --trials 7 --out bench/results/protocol-v2.json

dotnet run --project Bench -c Release --no-build -- \
  durability --records 5000 --payload 64 --trials 5 --range-queries 100 \
  --out bench/results/durability-v2.json

dotnet run --project Bench -c Release --no-build -- \
  queue --items 1000000 --capacity 8192 --trials 7 --out bench/results/queue-v2.json

dotnet run --project Bench -c Release --no-build -- \
  matching --sizes 100,1000,10000,100000 --out bench/results/matching-v2.json

dotnet run --project Bench -c Release --no-build -- \
  replay --data data/sample --trials 5 --out bench/results/replay-sample-v2.json

python3 bench/run_multicast.py --subscribers 100 500 --rates 500 \
  --instruments 2 --warmup 4 --duration 10 --max-batch 16 \
  --flush-interval-ms 1 --tag protocolv2

python3 bench/docgen.py --write
python3 bench/docgen.py --check

Refresh the complete pre-v2 generation together; mixed-host partial refreshes are rejected.

python3 bench/environment.py

python3 bench/run.py --tag scale50 --rates 50 --subscribers 100 200 300 400 500 600 700 800 900
python3 bench/run.py --tag scale5  --rates 5  --subscribers 1000 2000 3000 4000 5000

python3 bench/run_multicast.py --tag mcast --rates 50 \
  --subscribers 100 250 500 1000 2000 4000 6000 8000

for b in 1 4 16 64; do
  python3 bench/run_multicast.py --tag batch$b --max-batch $b --flush-interval-ms 1 \
    --rates 500 --subscribers 1000
done

for i in 1 2 3; do
  python3 bench/run.py --tag qchannel$i --rates 50 --subscribers 500
  python3 bench/run.py --tag qring$i --ring --rates 50 --subscribers 500
  python3 bench/run.py --tag rep$i --rates 5 --subscribers 4000
done

A capacity study requires reserved hosts, process and interrupt affinity, frequency and thermal telemetry, separate publishers and consumers, controlled loss/reordering, hardware counters, and x64/Arm64 runs.