TaoBench: Warmup May Not Be Working as Intended

[SeungsuBaek]

Hi!

I have a question about TaoBench warmup and test.

Environments and Commands

# Environments
# Two machines with identical specifications 

Xeon 5128 Gold (16cores) * 2
Memory 32GB * 8
10Gbit/s network

DCPerf v1.0

# server machine command, use only single socket
numactl --cpunodebind=0 --membind=0 ./benchpress_cli.py run tao_bench_custom -r server -i '{"interface_name": "eno1np0", "memsize": 64, "warmup_time": 3600, "test_time": 1800}'

# client machine command, use all socket
./benchpress_cli.py run tao_bench_custom -r client -i '{"server_hostname": "server", "server_memsize": 64, "warmup_time": 3600, "test_time": 1800, "clients_per_thread": 380}'

Problem

Based on my current understanding, under the default settings the warmup may have limited effect. Because the key distribution is R:R and each process seeds its RNG from the current time (--randomize), the warmup and test phases run in separate processes with different seeds; consequently, the test likely generates a fresh key stream and ends up re-warming the cache.

I understand that warmup is intended to pre-populate the cache, and the README references an approximate 0.88 hit-ratio target. In my runs, even after I fully warm up to a 0.89 hit ratio and then launch the test, the test phase behaves as if cold: it climbs back from a low hit ratio, and the time it takes to reach ~0.89 again is essentially the same as during warmup.

Question

Could you please let me know if I’ve misunderstood anything above? Also, to avoid a seed mismatch between processes, would it be acceptable to disable --randomize (or set a fixed seed) so that warmup and test use the same key sequence?

[excelle08]

This is interesting, in our experience the test phase should have 0.89 hit ratio at the beginning. While there's --randomize flag to seed the random generator differently every time the client starts, the range of the keys should be fixed and is calculated based on the server memsize. Could you try the job tao_bench_autoscale to see if the behavior is the same, and it would be helpful if you can share the benchmark_metrics folder on your server side and also the benchpress.log on the client side.

Also, to avoid a seed mismatch between processes, would it be acceptable to disable --randomize (or set a fixed seed) so that warmup and test use the same key sequence?

Yes it's OK to disable --randomize for better reproducibility