NEST as a SPEC CPU benchmark

[heshpdx]

Hello friends,

I’m a CPU architect at Ampere Computing where I do performance analysis and workload characterization. I also serve on the SPEC CPU committee, working on benchmarks for the next version of SPEC CPU, CPUv8 . We try to find computationally intensive workloads in diverse fields, to help measure performance across a wide variety of behaviors and application domains. Based on the longevity of nest, its large active community in biology and its use in education, I have proposed the nest neural network model be included in the next set of marquee benchmarks in SPEC CPU.

As part of the effort, we have ported and integrated the nest mainline code into the SPEC CPU harness so that it can be tested on a wide variety of systems in a controlled environment to produce reproducible results. We have even built it on native Windows using MSVC and the Intel compiler for Windows – we are happy to share the changes if someone is interested in testing and integrating it back into the upstream mainline for the benefit of the community.

The piece we need help with is an understanding of the multithreaded workloads. Right now, we have single-threaded nest command lines which run and produce verifiable output across many compilers (llvm, gcc, icc, aocc, nvhpc, cray), ISAs (aarch64, x86, power) and operating systems (linux, windows, android). We verify the run via checking the .dat files which come out of the simulation runs to make sure that there are no differences in the resulting output. A problem arises when we run with multiple threads, since there are a different number of files produced, and I am unfamiliar with how to coalesce them to verify.

First some fundamental questions: Does a nest invocation with 8 threads perform the same amount of work as a run with 16 threads? Or is it that the problem being solved is larger? If it is the same, how can we verify that? Does this answer change based on the .sli script used?

In the example below, if I run examples/nest/brunel-2000_newconnect_dc.sli (with a small edit to make it run longer)... I tried with 8 threads and 16. It looks like I am simulating the same number of Neurons and Synapses. The 8-thread version outputs 16 files and the 16-threaded version outputs 24 files. The total lines in the files are close. Do they fundamentally contain the same information, just at different sample points?

$ ./nest_s_base.O3-64 --userargs=threads=8 brunel-2000_newconnect_dc_LONG.sli
NEST 3.5.0-post0.dev0 (C) 2004 The NEST Initiative
Configuring neuron parameters.
Creating the network.
Configuring neuron parameters.
Creating excitatory spike recorder.
Creating inhibitory spike recorder.
Connecting excitatory neurons.
Connecting inhibitory population.
Connecting spike recorders.
Starting simulation.

Brunel Network Simulation
Number of Threads : 8
Number of Neurons : 95000
Number of Synapses: 902500100
       Excitatory : 722000000
       Inhibitory : 180500000
Excitatory rate   : 6.99 Hz
Inhibitory rate   : 6.825 Hz
Building time     : 23.76 s
Simulation time   : 448.07 s

$ ls -l brunel-2-in-threaded-95002-* | wc -l
16

$ wc -l brunel-2-in-threaded-95002-* | tail -1
 2811 total

$ ./nest_s_base.O3-64 --userargs=threads=16 brunel-2000_newconnect_dc_LONG.sli
NEST 3.5.0-post0.dev0 (C) 2004 The NEST Initiative
Configuring neuron parameters.
Creating the network.
Configuring neuron parameters.
Creating excitatory spike recorder.
Creating inhibitory spike recorder.
Connecting excitatory neurons.
Connecting inhibitory population.
Connecting spike recorders.
Starting simulation.

Brunel Network Simulation
Number of Threads : 16
Number of Neurons : 95000
Number of Synapses: 902500100
       Excitatory : 722000000
       Inhibitory : 180500000
Excitatory rate   : 6.985 Hz
Inhibitory rate   : 7.36 Hz
Building time     : 35.39 s
Simulation time   : 332.74 s

$ ls -l brunel-2-in-threaded-95002-* | wc -l
24

$ wc -l brunel-2-in-threaded-95002-* | tail -1
 2909 total

Overall, the goal is to be able to verify that the same amount of work was completed between these two command lines, and verify that they calculated the same result. This allows a benchmark to run on systems with a varying number of hardware cores, so we can measure CPU performance between them. We are allowed to provide some tolerance, in case there is floating point rounding error.

For the multithreaded benchmark, I am exercising the scripts below. The goal is to showcase scalable threading performance, as well as cover a variety of behaviors in the nest simulator.

brunel-2000_newconnect_dc_LONG.sli
brunel_ps_LONG.sli
hpc_benchmark.sli
microcircuit.sli

If you have feedback on which are more or less useful as multithreaded benchmarks, please share your thoughts!

Thank you!

[heplesser]

Hello Mahesh!

We are very excited that you have proposed NEST for inclusion in the SPEC CPUv8 benchmark suite! We would be happy to work with you to make this happen. I will answer your more specific questions below.

One important point about NEST (and other neuronal network simulators) is that you can run a wide range of neuronal network models on the NEST simulator, constructing the networks either through SLI or through Python scripts. And if networks are defined in the PyNN specification language, they can be executed on a range of neuronal network simulators, including some neuromorphic systems. Therefore, there isn't "the nest neural network model". The advantage of this is that one can configure networks that are suitable for benchmarking.

In our own work, we have mainly used the hpc_benchmark.sli and the microcircuit.sli benchmarks. The latter has become a reference benchmark also for neuromorphic and GPU-based simulators (I will post references later).

Concerning the specific Brunel-benchmark you used: Increasing the number of threads will distribute the same workload across more threads, i.e., strong scaling.

I am a bit confused by the number of output files you report. When running with eight threads, I would expect eight brunel-2-ex-threaded-... and eight brunel-2-ex-threaded-... files, and with 16 threads correspodingly 16 ex and 16 in files. Could you double check that?

I noticed that you used a NEST 3.5.0-post0.dev0 version from Github. We made some substantial improvements in threaded scaling in NEST 3.6, so I would strongly suggest that you update to at least NEST 3.6, ideally to NEST 3.7. With NEST 3.6, we see excellent scaling for the "microcircuit" benchmark up to 128 threads on dual AMD Epyc Rome systems.

Best, Hans Ekkehard

[heshpdx]

Thanks for responding, Hans!

I had a feeling you would ask me to rebase, so I attempted that three-way merge right after I posted above. The SPEC CPU harness builds applications in a totally different manner, so the process of adding a benchmark requires taking humpty-dumpty apart and putting him back together. Something may have gotten lost in that process, because after my merge, I get this runtime error immedately:

$ ./nest_r_base.O3-64 

sli-init Fatal []: 
    While executing module initializer: {(nest-init) run}

load Error []: UndefinedName

Start Error []: 
    Something went wrong during initialization of NEST or one of its modules. 
    Probably there is a bug in the startup scripts. Please report the output 
    of NEST at https://github.com/nest/nest-simulator/issues . You can try to 
    find the bug by starting NEST with the option --debug

Start Error []: 
    Something went wrong during initialization of NEST or one of its modules. 
    Probably there is a bug in the startup scripts. Please report the output 
    of NEST at https://github.com/nest/nest-simulator/issues . You can try to 
    find the bug by starting NEST with the option --debug

Here is the output from ./nest --debug: debug.log

Can you provide some hints as to what I should look at? I am building all the modules as well as models/models.cpp and models/modelsmodule.cpp. Is there a config or other flag I should be aware of?

[heshpdx]

On the topic of workloads, thank you for the feedback on the benchmark scripts.

When running with eight threads, I would expect eight brunel-2-ex-threaded-... and eight brunel-2-ex-threaded-... files, and with 16 threads correspodingly 16 ex and 16 in files. Could you double check that?

Indeed, you are correct! I counted wrong, sorry about that. So the question is, how can I use those files to prove the same work occurred between the two runs, and check that they both came to the "correct answer"? SPEC CPU benchmarks need to scale from 1 thread to N threads, and verify the results are the same using any N. This is the challenge in front of us.

For the single-threaded benchmark, I am using these scripts which seem to work well. Are they ok?

cuba_stdp.sli
structural_plasticity_benchmark.sli
ArtificialSynchrony.sli

[heshpdx]

Ahoy! Sometimes crafting a detailed message to the author is all that is required to solve one's problem. In making the debug.log and looking at the first line of the output, I realized that I had forgotten to update the nest/sli/sli-init.sli and other runtime files to your latest mainline. I did that and the program was able to run again! I will work on testing all my cmdlines with my 3.7 rebase. Thanks!

[heshpdx]

Regarding microcircuit.sli, we noticed that there is a bug if we try to run with more than 63 threads. It has something to do with a file not being able to be created. We got around that issue by capping the max thread count in that script. Is this a known limitation?

diff -u Potjans_2014/microcircuit.sli ~/867.nest_s/data/refspeed/input/microcircuit.sli
--- Potjans_2014/microcircuit.sli     2024-06-07 16:31:55.464111605 +0000
+++ ~/867.nest_s/data/refspeed/input/microcircuit.sli   2024-05-10 16:28:22.329645268 +0000
@@ -137,6 +137,14 @@
 
 } def
 
+% This model only supports up to 63-way concurrency
+/maxthreads 63 def
+/Truncate63
+{
+  dup maxthreads gt {
+     pop maxthreads
+  } if
+} def
 
 % PrepareSimulation - Set kernel parameters and RNG seeds based on the
 % settings in sim_params.sli.
@@ -148,9 +156,9 @@
     % set global kernel parameters
     <<
        /resolution dt
-       /total_num_virtual_procs n_vp
+       /local_num_threads local_num_threads Truncate63
+       /total_num_virtual_procs n_vp Truncate63
        /overwrite_files overwrite_existing_files
        /rng_seed rng_seed
        output_path (.) neq {
            /data_path output_path
        } if

[heplesser]

@heshpdx NEST creates a separate spike recorder instance for each thread to allow non-blocking recording and each of these instances opens a separate file. Thus, the brunel....sli script running on 64 threads would open 128 output files. Could it be that you hit an operating system limit on the number of open files per process?

I presume that with SPEC CPU you want to test CPU performance rather than I/O system performance. If this is indeed the case, I would suggest to drop the spike time recording to file. This would solve the problem with limited numbers of files. If SPEC rules should allow you to do one run with some form of recording of spikes and another run without it, one could use the following approach: Perform a run with spike recorders in which the spikes are recorded to memory. After then simulation time is up, extract spike data from the spike recorder in the SLI script and write it to file. This only requires one file per process instead of one per thread. Also, read out the spike counter in NEST. Then turn off spike recording completely and re-run the simulation. Read out the spike counter (it is always active) and you should get exactly the same number of spikes as when recording. I can send you code to do this.

When simulating the same network with different numbers of threads, you will obtain results that differ in detail, since the random number sequences in NEST are thread-specific. We have developed several measures to verify that simulations do produce statistically consistent results, see the paper by Gutzen et al below. If you positively need identical results independent of the number of threads used, we can develop a work-around for that.

For strong-scaling experiments, I would encourage you to use the microcircuit model, as it is currently the most widely used network model used for benchmarking, see the paper by Kurth et al below. That paper also gives a rather recent description of the state of the art, although NEST 3.6 and later now outperform NEST 2.14 that was used in that paper.

Concerning a single-thread benchmark, what are the constraints on running a single benchmark? On my MacBook with Core i5, hpc_benchmark.sli takes in total about two minutes for a single run. If that is acceptable, I would suggest using hpc_benchmark.

[heplesser]

And here the promised references:

• Gutzen, R., Von Papen, M., Trensch, G., Quaglio, P., Grün, S., & Denker, M. (2018). Reproducible Neural Network Simulations: Statistical Methods for Model Validation on the Level of Network Activity Data. Frontiers in Neuroinformatics, 12, 90. https://doi.org/10.3389/fninf.2018.00090
• Kurth, A. C., Senk, J., Terhorst, D., Finnerty, J., & Diesmann, M. (2022). Sub-realtime simulation of a neuronal network of natural density. Neuromorphic Computing and Engineering, 2(2), 021001. https://doi.org/10.1088/2634-4386/ac55fc

[heshpdx]

You are correct we want to be CPU bound and not IO bound. But even with the current code, outputting a file per thread, we are quite CPU bound.

Your spike recorder in memory idea is very good. If you can coalesce that using sli directives at the end of the simulation, that would solve our problem. The one limitation we have is that the run should fit within 64 GB of virtual memory (for the parallel runs). I imagine that is big enough.

Please do work on that when you get a chance. I recognize that the NEST conference is coming up and you will be busy with that!

The thread count limitation was only seen in microcircuit.sli. I'll see if I can dig up why. Most OSs have a default open file size limit of 4096 so I think the issue may be more esoteric.

I'll take a look at hpc benchmark for single threaded runs. I wanted to have a wide variety of code coverage and CPU behavior in three or four workloads. The time limit for single threaded is 3-5 minutes on a "modern CPU" and it needs to fit within 1.8 GB of memory.

[heshpdx]

Regarding exact answers - we do allow some tolerance based on floating point rounding. If there is a lot of randomness, we can try to reduce that via more deterministic randomness. I've seen that in many benchmark candidates, less (or zero) randomness still provides valid answers and doesn't detract from creating a benchmark representative of the application behavior in the field.

With my latest rebase I still get issues with running Potjans_2014/microcircuit.sli with more than 63 threads (in the SPEC CPU harness):

$ ./nest_s_base.O3-64 --userargs=threads=80 microcircuit.sli
NEST 3.7.0-post0.dev0 (C) 2004 The NEST Initiative

SimulationManager::set_status Info []: 
    Temporal resolution changed from 0.1 to 0.1 ms.

NodeManager::prepare_nodes Info []: 
    Preparing 79729 nodes for simulation.

RecordingBackendASCII::prepare() Error []: 
    I/O error while opening file 'spikes_3_0-74246-63.dat'.

Simulate_d Error []: IOError

I have enough file descriptors:

$ ulimit -Sn
1024
$ ulimit -Hn
1048576

I played around with the mainline nest build, and realized the issue is not with threads, but with /total_num_virtual_procs. I was able to recreate the problem on the mainline by increasing Potjans_2014/microcircuit.sli's /total_num_virtual_procs to be 80 and it fails the same way as above. Can you share the correct way to scale up the parallelism in microcircuit.sli? What is your cmdline and how would should I set the parallelism to 32, and to 80?

[heplesser]

I had another look at microcircuit.sli and it creates a lot of files. The model consists of four cortical layers with two populations each, i.e., a total of eight populations. For each population, the script creates one spike recorder and one voltmeter, thus a total of 16 recording devices (implemented like this because it makes analysis easier later as one does not need to split data into populations later). Each of these devices is replicated once per thread, so for 64 threads you would get 1024 files, for 80 threads 1280 and thus hit your soft limit for file descriptors. To turn this off, open sim_param.sli and set the three variables /save_... to false (around line 92). Spikes will then be stored in memory and no files should be opened.

Concerning setting the number of threads, I am surprised that --userargs=threads=80 had an effect. Did you make some changes to microcircuit.sli to make this work? Since NEST combines MPI- and thread-based parallelization, we have

• the number of MPI ranks M
• the number of threads T per rank
• the number of virtual processes N_VP = M * T

M is defined via mpirun. Inside a NEST script you can either set T through the parameter local_num_threads or N_VP through the parameter total_num_virtual_procs, which must be divisible by M. Since you work with M=1 only, both approaches are equivalent and you can stick with setting total_num_virtual_procs as is used microcircuit.sli. NEST guarantees that one gets identical results for fixed N_VP, no matter how it is split between M and T.

The set of files in Potjans_2014 assumes that parameters are modified in sim_params.sli and network_params.sli, respectively, but I assume that for your purposes, you prefer to set parameters on the command line?

One more question: We are mostly using Python scripts in our benchmarking now. Would that be feasible for your benchmarks as well or does that introduce to many confounding factors so that you would want to avoid Python?