Parallel SpatialPooler, Connections

[breznak]

Third (the middle) level of proposed parallelization.

This one will make the core algorithm (SP, Connections, TM) run suitable methods in parallel.

PROs:

• all code benefits (SP, TM), not only if using SPRegion etc
  • bonus if Connection can be optimized this way, as all code: SP, TM, CP uses Conn!
• even single-task workloads benefit (such as MNIST example #242 MNIST which uses only SP)

CONs:

• more complex to implement

TODO:

• find suitable methods in Connections, SP for parallel execution
• may want to use atomic, lock-less data structures (see project folly)
• benchmark on MNIST SP MNIST example #242

Naively, I think we could get the most bang/buck by parallelizing the methods:

Connections::computeActivity and NOT using mutex locks. This method is fault tolerant and does not need to be exact.
Connections::adaptSegment which will need mutex lock on conenctions object only when a synapse changes its connected/potential state, so that it can modify the presynaptic-maps. This method would not be parallel, but rather would be thread safe and assume that the caller is parallel. Alternatively if this method were parallel it might be safe to eschew the mutex locks, but I think this alternative would have less performance gain.

from #214 (comment)

[marty1885]

I'm looking into what parallelizing framework will be the most suitable for NuPIC. I use OpenMP in tiny-htm. But that's just because it is the most convenient one, not sure being the best one. Here are some of them:

1. OpenMP

   • Pros:
     • Lest amount of code change
     • Easy to use/tune
     • Support on all major compilers, clang, gcc, msvc
     • Even SIMD support to a degree
   • Cons:
     • Requires compiler support
     • Not in C++ standard

2. TBB

   • Pros:
     • Standard C++ code, no extension needed
     • As fast as OpenMP
   • Cons:
     • Messier code
     • Harder to tune

3. C++11 threads/C++17 Parallel STL

   • Pros:
     • C++ standard, not changing any time soon
   • Cons:
     • Difficult to work with. They are not designed for this kind of computing
     • Extra overhead as we need to hand-craft some control functions (for loop scheduling).

Any suggestions?

[breznak]

@marty1885 I just found this thread! Would it be possible to bring some parallel goodness from tiny-htm to this repo?

• or share some tools, benchmarks, examples..that can serve for validation and benchmarking?

[breznak]

FTR, CUDA on c++17
https://developer.nvidia.com/blog/accelerating-standard-c-with-gpus-using-stdpar/

My past attempts (almost exactly 1y ago) did not yield any gains, but let's revisit the issue.
Overall, I'm sad that htmcore is not built upon any "large" framework like (py)torch, tf, ..