diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..b542a101d7 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,39 @@ +# Agent rules — NEURON CPU network SoA adoption + +Full handoff and starting prompt: **`GROK-NETWORK-SOA.md`** (read on new sessions). + +## Workspace + +- Repo worktree: `~/neuron/cpu_net_soa` (branch `local/cpu-network-soa`, tracks `origin/master`). +- Primary git object store: `~/neuron/nrngpu` — commit from **this** worktree cwd. +- Sibling GPU track (paused network buffers): `~/neuron/nrngpu` @ `local/gpu-native-qualification`. + +## Build + +```bash +source ~/neuron/bin/nrnenv nrngpu build-cpu-net-soa # create on first session if missing +# or: mkdir -p ~/neuron/cpu_net_soa/build && cd build && cmake .. -DNRN_ENABLE_GPU=OFF ... +``` + +Prefer CPU-only or default GPU-off builds until integration explicitly needs GPU mirrors. + +## Scope (this branch) + +- Network SoA: `Point_process`, `NetCon`, `PreSyn`, `weights`, `SelfEvent` in `neuron::container` style. +- HOC wrappers as permutation-stable handles over backing store — **not** a second pointer graph. +- **Out of scope:** Stage 2/3 GPU `net_buf_receive`, ringtest GPU network buffers (resume after SoA merges to master). + +## Execute, don’t delegate + +Run builds and tests yourself (`ctest`, ringtest CPU spike parity). Do not tell the user what to run unless blocked. + +## Key references + +| Topic | Path | +|-------|------| +| Handoff | `GROK-NETWORK-SOA.md` | +| Phase 0 scaffold | `doc/network-soa-phase0.md` | +| Node/mechanism SoA pattern | `src/neuron/container/soa_container.hpp`, `data_handle.hpp` | +| PreSyn `thvar_` handle (prototype) | `src/nrncvode/netcon.h` | +| CoreNEURON layout reference | `src/coreneuron/sim/multicore.hpp`, `network/netcon.hpp` | +| nrncore export (prior art) | `src/nrniv/nrncore_write/` | \ No newline at end of file diff --git a/GROK-NETWORK-SOA.md b/GROK-NETWORK-SOA.md new file mode 100644 index 0000000000..8df713bed6 --- /dev/null +++ b/GROK-NETWORK-SOA.md @@ -0,0 +1,178 @@ +# Grok handoff: NEURON CPU network SoA (`cpu_net_soa`) + +Use this file when starting a **new** Grok session rooted in `~/neuron/cpu_net_soa`. + +--- + +## Status (2026-07) + +| Phase | Status | Notes | +|-------|--------|--------| +| **0** Spec | **Done** | `doc/network-soa-phase0.md` + layered `doc/network-soa/` | +| **1** PointProcess + Weight SoA dual-write | **Done** | Handles + create/destroy | +| **2** NetCon SoA; HOC weight → Weight SoA | **Done** | SoA HOC-primary; heap MOD scratch | +| **3** PreSyn SoA + fanout `NcIndex`/`NcCount` | **Done** | Rebuild at sort / `init_events` + lazy on spike | +| **4** SelfEvent `weight_index` + receive-by-index | **Done** | Still `pnt_receive(..., double*, flag)` (nocmodl) | +| **Sort wiring** | **Done** | Network in `nrn_ensure_model_data_are_sorted` | +| **Heap policy** | **Settled (no full free yet)** | Long-lived `weight_` required for `net_send(_w)` / FOR_NETCONS / INITIAL | +| **SaveState / BBSaveState dual-write** | **Done** | SoA values + NetCon-index SelfEvent identity | +| **5** GPU net buffers | **Out of scope** | After SoA is reviewable / on master | + +**Developer docs (preferred reading order):** `doc/network-soa/README.md` → topology → dual-write → sort → phase0. + +### Known gaps + +- Full free of `NetCon::weight_`: needs FOR_NETCONS + INITIAL + SelfEvent never keying on temp `double*` (see `doc/network-soa/dual-write-and-heap.md`). +- Default codegen **nocmodl**; CoreNEURON-style `weight_index` receive is a separate ABI project. +- Fanout: SoA ranges preferred when sorted; `dil_` still filled and used as rebuild source / fallback (**fanout authority** not fully flipped — see topology doc). +- `InputPreSyn` thin gid→fanout deferred (phase0 §5.5.1). + +### Default codegen + +**nocmodl** (not NMODL). SoA weights work via materialize-around-`pnt_receive`. Native `weight_index` in MOD requires a separate nocmodl/ABI project. + +--- + +## Why a separate worktree + +Network SoA is a focused CPU/infrastructure PR. It should **not** carry the GPU-native qualification commit stack (`local/gpu-native-qualification`). + +| Worktree | Branch | Purpose | +|----------|--------|---------| +| `~/neuron/cpu_net_soa` | `local/cpu-network-soa` | Network SoA → PR to **master** | +| `~/neuron/nrngpu` | `local/gpu-native-qualification` | Mechanism GPU, Stage 1 buffer plumbing; Stages 2–3 **paused** until SoA lands | + +--- + +## North star + +One CPU backing store for integration-hot network data: + +- `std::vector` columns + permutation indices +- `data_handle` / `owning_handle` stable across permute +- `NrnThread` = lightweight slice (`offset` + count) — **no duplicate CPU copies** +- HOC extras in sidecars keyed by handle + +CoreNEURON layout is the **integration reference**. + +--- + +## Architecture (current dual-write) + +```text +HOC / Python + │ + ▼ +Legacy shells (Point_process*, NetCon, PreSyn) + owning_handle _soa + │ + ▼ +neuron::container::network::{PointProcess,Weight,NetCon,PreSyn} + │ + ▼ +Hot path: fanout order + weight_index → materialize → pnt_receive(double*) +``` + +Key paths: + +| Path | File / API | +|------|------------| +| Containers | `src/neuron/container/network/*.hpp` | +| Model ownership | `src/neuron/model_data.hpp` | +| PP dual-write | `Point_process::_soa`, `nrn_point_process_soa_sync` | +| NetCon / weights | `NetCon::_soa`, `weight_soa_`, `soa_sync` | +| Fanout | `PreSyn::ensure_fanout_order`, global NetCon* order | +| Sort / repack | `network_soa_sort.cpp`, `sort_network_data` in ensure_sorted | +| Receive by index | `nrn_pnt_receive_by_weight_index` | + +--- + +## Sort wiring (implemented) + +`nrn_ensure_model_data_are_sorted()` now freezes and sorts network containers after nodes + mechanisms: + +1. **PointProcess** — partition by `ThreadId`; set `cache.thread[i].point_process_offset` +2. **Weight** — repack contiguous per-NetCon blocks ordered by target thread; set `weight_offset` +3. **NetCon** — same order as weight packing; set `netcon_offset`; refresh dual-write indices +4. **PreSyn** — partition by thread; rebuild `NcIndex`/`NcCount` fanout; set `presyn_offset` + +Implementation: `src/nrncvode/network_soa_sort.cpp`, `neuron/container/network/sort.hpp`. + +## Heap-drop policy (decision) + +| Path | Weight source | Notes | +|------|---------------|--------| +| NetCon deliver | SoA → long-lived `weight_` → `pnt_receive` → SoA | Heap is MOD scratch; **required** so `net_send(..., _w)` keeps SelfEvent identity | +| NetCon deliver (FOR_NETCONS) | Same + sync all NetCons on target | MOD walks other NetCon `weight_` pointers | +| `pnt_receive_init` / HOC INITIAL | heap buffer + SoA sync | Keep until INITIAL uses index | +| SaveState | NetCon obj index + SoA weights | Dual-write restore sync done; `weight2netcon` remains live-queue helper | +| BBSaveState | DEList ncindex + SoA weights | SelfEvent match heap or weight_index; bind both on restore | +| HOC `weight[i]` | SoA `data_handle` | Already SoA-primary | + +**Do not free `weight_` until** FOR_NETCONS, SaveState, and INITIAL no longer need stable heap bases. Prefer short-lived materialize everywhere else (current default for simple deliver). + +Native nocmodl `weight_index` ABI is **out of scope** for this branch; keep materialize around generated `pnt_receive(double*)`. + +## Recommended next work + +1. ~~Sort + SaveState + BBSaveState dual-write~~ (done). +2. **Decide PR posture:** draft/review PR of dual-write+sort (no claim “land for perf yet”) vs private until heap-free / GPU (see `doc/network-soa/dual-write-and-heap.md` §5). +3. If pursuing heap-free: FOR_NETCONS index view + INITIAL by index + SelfEvent identity without heap pointer (doc L2 roadmap). +4. If pursuing authority cleanup: make fanout SoA sole hot-path truth; stop using `dil_` on spike (topology L1). +5. GPU Phase 5 only after SoA shape is stable enough to upload the same columns. + +--- + +## Build + +```bash +cd ~/neuron/cpu_net_soa +mkdir -p build && cd build +cmake .. -DCMAKE_INSTALL_PREFIX=$PWD/install \ + -DNRN_ENABLE_MPI=ON -DNRN_ENABLE_TESTS=ON \ + -DNRN_ENABLE_CORENEURON=OFF -DNRN_ENABLE_NMODL=OFF +cmake --build . --parallel +# Prefer install for integration tests that need share/hoc +cmake --build . --target install +``` + +--- + +## Tests (CPU gates) + +```bash +cd build + +# Unit: SoA containers + dual-write helpers +./bin/test/testneuron '[network]' --reporter compact +./bin/test/testneuron '[data_structures]' --reporter compact + +# Integration delivery gate (also under ctest pytest group) +# Requires PYTHONPATH / build env for neuron +python -m pytest ../test/pytest/test_network_soa_delivery.py -v + +# Broader (env-sensitive) +ctest -j 4 -R 'unit_tests::testneuron|network_soa|pytest' --output-on-failure +# Full ctest: complete install first; CoreNEURON jobs skipped when disabled +``` + +--- + +## Starting prompt (new session) + +``` +Read ~/neuron/cpu_net_soa/GROK-NETWORK-SOA.md, AGENTS.md, and doc/network-soa-phase0.md. + +Repo: ~/neuron/cpu_net_soa, branch local/cpu-network-soa. +Sibling: ~/neuron/nrngpu @ local/gpu-native-qualification (GPU network buffers paused). + +Phases 0–4 dual-write, sort wiring, SaveState/BBSaveState dual-write done. +Read doc/network-soa/README.md (topology, heap policy, sort). weight_ remains +MOD scratch under nocmodl. Next: PR posture, or FOR_NETCONS/fanout-authority +work toward heap-free — not GPU on this branch unless rebasing after SoA. +``` + +--- + +## Old GPU-native context + +Stage 1 `NetReceiveBuffer` on gpu-native remains a valid **pattern**; indexing aligns with SoA after merge. See `~/neuron/nrngpu/GROK-GPU-NATIVE.md`. diff --git a/cmake/NeuronFileLists.cmake b/cmake/NeuronFileLists.cmake index ca60833d2e..e506417a23 100644 --- a/cmake/NeuronFileLists.cmake +++ b/cmake/NeuronFileLists.cmake @@ -262,6 +262,7 @@ set(NRNCVODE_FILE_LIST cvtrset.cpp htlist.cpp netcvode.cpp + network_soa_sort.cpp nrndaspk.cpp occvode.cpp tqueue.cpp) diff --git a/doc/network-soa-phase0.md b/doc/network-soa-phase0.md new file mode 100644 index 0000000000..46062947c6 --- /dev/null +++ b/doc/network-soa-phase0.md @@ -0,0 +1,611 @@ +# Network SoA — Phase 0 design spec + +**Status:** Authoritative for `local/cpu-network-soa`. Phases 0–4 dual-write + network sort wiring implemented; see `GROK-NETWORK-SOA.md` for status and gaps. +**Branch:** `local/cpu-network-soa` (from `origin/master`) +**Sibling (paused):** `~/neuron/nrngpu` @ `local/gpu-native-qualification` — GPU network buffers Stages 2–3 +**Handoff:** `GROK-NETWORK-SOA.md` + +--- + +## 1. Problem + +NEURON network objects (`Point_process`, `NetCon`, `PreSyn`, `SelfEvent`) use pointer-heavy layouts suited to HOC interactivity: + +| Entity | Hot-path problem | +|--------|------------------| +| `Point_process` | `sec`, `node`, `prop`, `ob`, `presyn_`, `_vnt` pointers | +| `NetCon` | per-NetCon `double* weight_` heap allocation | +| `PreSyn` | `NetConPList dil_` + fat HOC fields (`Object*`, recording, …) | +| `SelfEvent` | `double* weight_` into NetCon heap | + +CoreNEURON GPU / multicore integration assumes **flat per-thread arrays** with integer indices (`pntprocs`, `weights`, `netcons`, `presyns`). See `src/coreneuron/sim/multicore.hpp`, `src/coreneuron/network/netcon.hpp`. + +Bridging with per-event shims on the legacy layout does not scale. **One SoA backing store** with HOC wrappers as permutation-stable handles matches the existing node/mechanism container model (`src/neuron/container/soa_container.hpp`, `data_handle.hpp`). + +**Existing network prototype:** `PreSyn::thvar_` is already `neuron::container::data_handle` (`src/nrncvode/netcon.h`). Threshold evaluation and GPU threshold use stable handles / row offsets relative to `NrnThread::_node_data_offset`. + +--- + +## 2. Goals + +1. Single CPU copy of integration-hot network data in `std::vector` columns. +2. Permutation-stable `data_handle` / `owning_handle` (same lifetime model as `Node`). +3. `NrnThread` (and `neuron::cache::Thread`) hold **offsets + counts** into contiguous global slices — no per-thread duplicate heap graphs. +4. HOC / Python API unchanged at the interpreter boundary (`weight()`, `loc()`, `NetCon.active`, …). +5. CPU `psolve` hot path becomes index-based (CoreNEURON-shaped), not `double* weight_` / `dil_` walks. +6. After SoA merges to master, the GPU track can upload the same columns (out of scope here). + +## 3. Non-goals (Phases 0–4 on this branch) + +| Out of scope | Resume when | +|--------------|-------------| +| GPU `net_buf_receive` codegen / Stage 3 wire-up | SoA on master; rebase `gpu-native-qualification` | +| Remote input / `InputPreSyn` split; MPI multisend redesign | After cpu-net-soa (see §5.5.1) | +| Removing HOC `Object*` identity | Never — sidecars remain | +| Changing public HOC/Python method signatures | Not required | + +--- + +## 4. Reference layouts + +### 4.1 CoreNEURON (integration hot) + +From `coreneuron::NrnThread` and network headers: + +| Entity | Key fields | +|--------|------------| +| `Point_process` | `_i_instance`, `_type`, `_tid` | +| `weights` | flat `double[n_weight]`; `NetCon.u.weight_index_` base | +| `NetCon` | `target_`, `u.weight_index_`, `delay_`, `active_` | +| `PreSyn` | `nc_index_`, `nc_cnt_`, `thvar_index_`, `threshold_`, `gid_` | +| `SelfEvent` | `target_`, `weight_index_`, `flag_` | + +Weights for one NetCon are **contiguous** at `[weight_index, weight_index + pnt_receive_size)`. + +### 4.2 NEURON node/mechanism precedent + +```text +owning_handle → owns a row; destructor frees row +handle → non-owning; stable across permute; dies when owner dies +data_handle → erases field identity; used for pointers into columns +``` + +Storage lives under `neuron::model()` (`Model::node_data()`, `Model::mechanism_data(type)`). +Thread slice: `NrnThread::_node_data_offset` + `neuron::cache::Thread::{node,mechanism}_offset`. +Sort gate: `nrn_ensure_model_data_are_sorted()` freezes tokens, permutes, rebuilds `neuron::cache::model`. + +### 4.3 PreSyn `thvar_` prototype + +- Construction takes `data_handle` for the watched voltage (or related variable). +- `notify_when_handle_dies` disconnects when the underlying node dies. +- Do **not** re-introduce raw `double*` for threshold sources in new code. + +--- + +## 5. Containers and field tags + +### 5.1 Ownership and file layout + +**Decision:** Network storages are members of `neuron::Model`, analogous to `m_node_data`. + +``` +src/neuron/container/network/ + point_process.hpp # Phase 1 — tags, storage, handle API + weights.hpp # Phase 1 + netcon.hpp # Phase 2 + presyn.hpp # Phase 3 + self_event.hpp # Phase 4 (if queued structurally) +``` + +```cpp +// neuron::Model (illustrative) +container::network::PointProcess::storage& point_processes(); +container::network::Weight::storage& weights(); +container::network::NetCon::storage& netcons(); // Phase 2 +container::network::PreSyn::storage& presyns(); // Phase 3 +``` + +Namespace root: `neuron::container::network`. +Each entity is a nested namespace with `field::*` tags, `storage`, `handle`, `owning_handle` — same shape as `neuron::container::Node`. + +### 5.2 Point_process (Phase 1) + +**Integration columns** (CoreNEURON-compatible): + +| Tag | C++ type | Default | Meaning | +|-----|----------|---------|---------| +| `field::Instance` | `int` | `-1` | Mechanism SoA row (`_i_instance`) | +| `field::MechType` | `int` | `-1` | Mechanism type (`_type`) | +| `field::ThreadId` | `int` | `-1` | Owning `NrnThread` id (`_tid`) | + +**Handle typedefs:** + +| Name | Type | +|------|------| +| `PointProcess::handle` | `handle_interface>` | +| `PointProcess::owning_handle` | `handle_interface>` | + +**Accessors on `handle_interface`:** `instance()`, `mech_type()`, `thread_id()`, plus `*_handle()` returning `data_handle` where useful for debugging / HOC bridges. + +**Not in SoA (sidecars / dual-write legacy fields):** see §7. + +### 5.3 Weights (Phase 1) + +**Integration columns:** + +| Tag | C++ type | Default | Meaning | +|-----|----------|---------|---------| +| `field::Value` | `double` | `0.0` | One weight scalar | + +One **row** = one scalar weight entry. A NetCon with `cnt = pnt_receive_size[type]` owns a **contiguous block** of `cnt` rows. + +| Name | Type | +|------|------| +| `Weight::handle` / `Weight::owning_handle` | same pattern as Node | + +**Contiguity invariant (CoreNEURON-compatible):** + +```text +weights[weight_index + k] for k in [0, weight_count) +``` + +- Allocation API (Phase 1→2) must allocate **blocks** of size `weight_count` as consecutive rows. +- Permutation of the weight container must either: + 1. **Block-permute** (move whole NetCon groups and rewrite `NetCon::WeightIndex`), or + 2. **Repack** at sort time into contiguous groups (preferred at `nrn_ensure_model_data_are_sorted`). +- Freeing a NetCon frees its entire block (swap-with-last **block** or hole + repack). + +**Why not one SoA row per NetCon with runtime array dim?** Weight counts differ by target mechanism type (`pnt_receive_size`). A flat pool matches CoreNEURON export (`nrncore_write`) and GPU upload. + +**Hot-path access:** integer `weight_index` base (current row of first weight in the group), **not** a stable identifier, so delivery stays CoreNEURON-shaped. After any weight permute/repack, all `NetCon` weight indices are remapped. HOC `weight(i)` may use `data_handle` obtained from the base row + array shift once the group is a single logical array, or `weights.get(weight_index + i)`. + +### 5.4 NetCon (Phase 2 — tags fixed now) + +| Tag | C++ type | Default | Meaning | +|-----|----------|---------|---------| +| `field::Target` | `int` | `-1` | Row in `PointProcess` storage (or sentinel) | +| `field::WeightIndex` | `int` | `-1` | Base row in `Weight` storage | +| `field::WeightCount` | `int` | `0` | `pnt_receive_size` for target type | +| `field::Delay` | `double` | `1.0` | Delivery delay (ms) | +| `field::Active` | `int` | `1` | `bool` as `int` (GPU-friendly, CoreNEURON style) | +| `field::SrcPreSyn` | `int` | `-1` | Source PreSyn row, or `-1` if none (management / reverse edge; see §5.4.1) | + +**Optional later:** `field::ObjectId` only if HOC identity must be recovered from a row without a sidecar map. + +**Event-queue identity:** `NetCon` remains a `DiscreteEvent` subclass for queue compatibility through Phase 2–3. The SoA row is the data plane; the C++ object is the control/HOC plane during dual-write. Long term the queued payload may shrink to `(type, row)` — not required for Phase 2 gate. + +#### 5.4.1 NetCon linkage: `target_` and `src_` (pointers vs handles vs indices) + +Today NEURON keeps a **mutual pointer graph**: + +```text +PreSyn ──dil_──► NetCon* ──src_──► PreSyn + │ + └──target_──► Point_process* +``` + +CoreNEURON’s integration layout does **not** retain that graph: `NetCon` has `target_` (struct pointer into `pntprocs`) and `weight_index_`; source is implicit because the NetCon sits in a PreSyn’s fanout range (`nc_index_` / `nc_cnt_`). There is no `src_` on the hot path. + +**Decision for NEURON SoA — three layers, not one type for all uses:** + +| Concern | Representation | Why | +|---------|----------------|-----| +| **Spike / delivery hot path** | Integer **SoA row indices** (`Target`, `WeightIndex`; fanout via PreSyn `NcIndex`/`NcCount`) | CoreNEURON-shaped, GPU-uploadable, no pointer chasing, contiguous scans | +| **Stable identity across permute / dual-write / HOC** | `PointProcess::handle` / `PreSyn::handle` (or `non_owning_identifier`) on the wrapper object | Same model as `Node::handle` — tracks row through permute without remapping by hand on every HOC touch | +| **Legacy shell (transitional)** | Keep `Point_process* target_` and `PreSyn* src_` on the C++ `NetCon` during dual-write | Existing disconnect / HOC / queue code keeps working until dual-read is complete | + +**Do not use `data_handle` for `target_` / `src_`.** +`data_handle` is a stable reference to a **scalar field in a column** (e.g. `PreSyn::thvar_` → a `double` voltage). A NetCon’s target is a **row of another entity**, not a double. The correct SoA tools are: + +- **row index `int`** after sort (integration), +- **`handle` / `owning_handle` / `non_owning_identifier`** when a stable cross-reference is needed outside the frozen sorted window. + +**Directionality of the assembly (important):** + +```text +Hot path (spike → targets): + PreSyn ──NcIndex/NcCount──► NetCon rows ──Target index──► PointProcess row + │ + └── WeightIndex ──► weight block + +Reverse edge (management / HOC only): + NetCon ──SrcPreSyn index or PreSyn::handle──► PreSyn +``` + +- **Forward fanout is authoritative** for delivery (replaces walking `dil_` as a pointer list). +- **`src_` / `SrcPreSyn` is a reverse edge**: disconnect, `replace_src`, mindelay queries, interpreter inspection. It is **not** required for CoreNEURON-shaped deliver. It may remain on the HOC wrapper / dual-write shell longer than `target_`, or stay as a SoA `int` column for convenience when destroying a PreSyn and sweeping its NetCons (until ranges alone suffice). + +**Why not keep pointer mutuality as the long-term design?** + +- Pointer graphs fight permutation, thread packing, and GPU mirrors. +- Two-way raw pointers duplicate topology already expressed by fanout ranges + `Target`. +- Mutuality is valuable for **interactive** NEURON; express it with **handles + indices**, not with a second heap of cross-pointers that own the topology. + +**Phase timing:** Phase 2 dual-writes `Target` / weights while leaving `target_`/`src_` pointers valid; Phase 3 builds fanout ranges; hot path drops `dil_` walks and pointer fanout; pointer fields become optional wrappers over handles/indices and can shrink later without another layout break. + +### 5.5 PreSyn (Phase 3 — tags fixed now) + +| Tag | C++ type | Default | Meaning | +|-----|----------|---------|---------| +| `field::Threshold` | `double` | `10.0` | Spike threshold | +| `field::Gid` | `int` | `-1` | Output gid or −1 | +| `field::NcIndex` | `int` | `-1` | Start of fanout range in NetCon order array | +| `field::NcCount` | `int` | `0` | Fanout count | +| `field::OutputIndex` | `int` | `0` | CoreNEURON `output_index_` | +| `field::ThVarRow` | `int` | `-1` | Optional denormalized node-voltage row for threshold scan; **canonical source remains `data_handle` in sidecar / dual-write `thvar_`** | +| `field::ThreadId` | `int` | `-1` | Owning thread | + +**Fanout model (replaces `dil_`):** + +```text +Global NetCon order array (or NetCon SoA already sorted by source): + netcons[NcIndex .. NcIndex+NcCount) + +On spike: for i in [0, NcCount): deliver netcons[NcIndex+i] +``` + +Matches CoreNEURON `nc_index_` / `nc_cnt_` into `netcon_in_presyn_order_`. At sort time, NetCons are (re)ordered by source PreSyn so ranges are contiguous. + +**Threshold source:** keep `data_handle` (today’s `thvar_`) in the dual-write / sidecar plane. Optionally cache `ThVarRow` when the handle refers to node voltage for vectorized threshold checks. + +#### 5.5.1 PreSyn vs remote input (`InputPreSyn`) + +Two **roles** appear on a rank; CoreNEURON splits them into types, NEURON historically does not. + +| Role | Job on this rank | Threshold / `thvar` / HOC recording? | Fanout (`NcIndex`/`NcCount`)? | +|------|------------------|--------------------------------------|-------------------------------| +| **Local / output PreSyn** | Detect spike, optional gid send, fan out local NetCons | Yes | Yes | +| **Remote / input source** | Map received `(gid, t)` to local NetCons | No | Yes | + +CoreNEURON’s `InputPreSyn` is the second role reduced to `nc_index_` / `nc_cnt_` (plus optional multisend bookkeeping). NEURON today stores **full `PreSyn*`** in both `gid2out_` and `gid2in_` (`netpar.cpp`), so remote sources pay for `ConditionEvent`, threshold fields, and sidecars they never use. That cost matters when connectivity is dense and `gid2in` is large — the common case where **MPI_Allgather** (or compressed collectives) remains near-optimal because almost every rank needs almost every spike. Point-to-point / multisend sophistication (BlueGene-era) is orthogonal: it changes the **wire** path, not whether a remote source needs threshold state. + +**Is the role still useful?** Yes. Remote delivery only needs: + +```text +gid → (NcIndex, NcCount) → NetCon rows → targets / weights +``` + +**Must NEURON clone CoreNEURON’s dual class tree?** No. Prefer a **data-plane** split when the time comes: + +1. **Local PreSyn SoA** — threshold columns + gid/output + fanout range + HOC sidecars. +2. **Remote inputs** — thin **`gid → fanout range`** (dedicated small SoA or map), not a fat PreSyn and not a HOC-facing type. +3. Spike exchange resolves gid → range → enqueue NetCons (or one lightweight input event holding the range + time). + +That preserves CoreNEURON’s memory and export shape without a second interactive pointer graph. Optional fields on one “spike source” SoA are a weaker fit (mixed sort keys, threshold code special-cases). + +**Timing (explicitly out of scope for this branch’s Phase 0–4):** + +| When | What | +|------|------| +| Phases 1–2 | Ignore remote split | +| Phase 3 | Local fanout on existing PreSyn; `gid2in_` may stay fat PreSyn | +| **After** cpu-net-soa finishes (post–Phase 3/4 MPI milestone) | Thin remote representation; align `nrncore` / CoreNEURON `gid2in` | +| GPU net buffers | Upload the same fanout ranges; role split reduces payload | + +**Design constraint for Phase 3:** build fanout so a later remote-input table can own the same style of `(NcIndex, NcCount)` ranges **without** restating NetCon layout. Do not implement `InputPreSyn` work on `local/cpu-network-soa`. + +### 5.6 SelfEvent (Phase 4) + +SelfEvents are short-lived queue items. Prefer **not** a long-lived SoA of all SelfEvents; instead: + +| Field on event / pool object | Type | Meaning | +|------------------------------|------|---------| +| `target_pnt` | `int` | PointProcess row | +| `weight_index` | `int` | Into weights (same base as NetCon or NULL weights) | +| `flag` | `double` | NET_RECEIVE flag | + +If a structural pool is useful later, tags match the above. Gate is `pnt_receive(weight_index)` migration (§9). + +### 5.7 Tag summary by phase + +| Phase | Container | Gate | +|-------|-----------|------| +| **0** | Spec (this doc) | Spec reviewed | +| **1** | `PointProcess` + `Weight` | Handles survive permute; dual-write on create/destroy | +| **2** | `NetCon` SoA; HOC `weight()` → flat Weight SoA | dual-write; handles survive permute | +| **3** | `PreSyn` SoA + fanout `NcIndex`/`NcCount` | dual-write; hot path uses fanout order | +| **4** | SelfEvent weight_index; `nrn_pnt_receive_by_weight_index` | dual-write M2 (double* still for MOD) | +| **5** | (gpu-native track) net buffers | GPU parity — **not this branch** | + +--- + +## 6. Thread slicing + +### 6.1 Layout + +Mirror nodes: + +```text +Global PointProcess SoA: [ thread0 rows | thread1 rows | ... ] +Global Weight SoA: [ thread0 block | thread1 block | ... ] +Global NetCon SoA: [ thread0 rows | thread1 rows | ... ] +Global PreSyn SoA: [ thread0 rows | thread1 rows | ... ] +``` + +### 6.2 Fields + +**On `NrnThread` (or `neuron::cache::Thread` — prefer cache for offsets that only matter when sorted):** + +| Field | Meaning | +|-------|---------| +| `_pntproc_offset` / `pntproc_count` | Slice of PointProcess storage | +| `_weight_offset` / `weight_count` | Slice of Weight storage | +| `_netcon_offset` / `netcon_count` | Slice of NetCon storage | +| `_presyn_offset` / `presyn_count` | Slice of PreSyn storage | + +**Decision (open Q1 resolved):** Weights are **partitioned per thread** (CoreNEURON `nt.weights` / `nt.n_weight`). Cross-thread NetCon is already restricted; weight blocks live on the **target** thread of the Point_process. + +**Hot-path address:** + +```cpp +double* w = weight_storage.get_field_data() + nt.weight_offset; +// delivery uses absolute weight_index into global storage, or local index +// local = absolute - nt.weight_offset +``` + +Absolute global indices simplify dual-write; local indices match CoreNEURON. Prefer **absolute** during dual-write, document conversion at nrncore export. + +### 6.3 When slices are computed + +During `nrn_ensure_model_data_are_sorted()` / network sort pass: + +1. Assign each PointProcess to a thread (from cell partition / existing `_vnt`). +2. Permute PointProcess so thread regions are contiguous; set offsets. +3. Order NetCons by `(src_presyn, …)`; place on target thread of `Target`. +4. Pack weight blocks per NetCon into the target thread’s weight region; rewrite `WeightIndex`. +5. Order PreSyn by thread; build `NcIndex`/`NcCount` into the NetCon order. + +--- + +## 7. HOC sidecar policy + +### 7.1 Principle + +Integration columns are the **only** data the hot path and (future) GPU need. +Interpreter / recording / identity live in **sidecars** keyed by stable id, **not** a second pointer graph that owns network topology. + +### 7.2 Keying + +| Key | Use | +|-----|-----| +| `owning_handle` / `non_owning_identifier_without_container` | Primary: survives permute; dies with row | +| HOC `Object*` | External identity; points at wrapper that holds `owning_handle` | + +Do **not** key sidecars by current SoA row integer alone without a stable id. + +### 7.3 Sidecar contents by entity + +**Point_process** + +| Field | Notes | +|-------|-------| +| `Object* ob` | HOC object | +| `Section* sec`, `Node* node` | Location; invalidate on topology change | +| `Prop* prop` | Mechanism instance bridge during dual-write | +| `void* presyn_`, `nvi_`, `_vnt` | Until indices replace them | + +During Phase 1 dual-write, the existing `struct Point_process` **is** the sidecar + legacy shell; SoA holds Instance/MechType/ThreadId. Migration moves fields out of the struct into maps/`owning_handle` wrappers over time. + +**NetCon** + +| Field | Notes | +|-------|-------| +| `Object* obj_` | HOC | +| `PreSyn* src_` | Dual-write reverse edge; long-term `SrcPreSyn` index and/or `PreSyn::handle` (§5.4.1) — not hot-path delivery | +| `Point_process* target_` | Dual-write; long-term `Target` index (+ optional `PointProcess::handle` on wrapper) | +| `double* weight_` | Dual-write alias into Weight SoA (then removed) | + +**PreSyn** + +| Field | Notes | +|-------|-------| +| `data_handle thvar_` | **Keep** as canonical threshold source | +| `Object* osrc_`, `Section* ssrc_` | Source identity | +| `IvocVect* tvec_`, `idvec_`, `HocCommand* stmt_` | Recording | +| `NrnThread* nt_`, `hoc_Item* hi_th_` | Thread / threshold list | +| `NetConPList dil_` | Dual-write until `NcIndex`/`NcCount` | +| MPI / MUSIC / multisend unions | Sidecar or later phase | + +### 7.4 What must never be duplicated as “second graph” + +- Do not keep a parallel `std::vector` fanout **and** `NcIndex`/`NcCount` as equally authoritative after Phase 3. One source of truth: SoA ranges. +- Do not allocate per-NetCon `new double[cnt]` after weight migration completes. + +--- + +## 8. Invalidation and sorting + +### 8.1 Unsorted triggers + +| Event | Action | +|-------|--------| +| Create/destroy PointProcess, NetCon, PreSyn | `mark_as_unsorted()` on affected network container(s) | +| NetCon retarget / weight count change | weights + netcons unsorted | +| Topology / `define_shape` / `tree_changed` | Invalidate sidecar caches (`Section*`, node location); mark network unsorted if thread membership may change | +| `nrn_threads_create` / repartition | Full network reorder | +| Mechanism permute that changes Instance rows | Update PointProcess `Instance` column (or mark unsorted and fix at sort) | + +Each network `storage` registers `set_unsorted_callback` → `neuron::cache::model.reset()` (same as nodes). + +### 8.2 Sort / freeze policy + +Extend `nrn_ensure_model_data_are_sorted()`: + +1. Issue frozen tokens for node, mechanisms, **and** network containers. +2. If any network container is unsorted, run network permute/repack (§6.3). +3. Hold frozen tokens through integration like mechanisms. + +**Freeze:** same `frozen_token_type` pattern — no row insert/delete while sorted token is held for that container. + +### 8.3 Handle death + +| Situation | Behavior | +|-----------|----------| +| `owning_handle` destroyed | Row freed; non-owning handles / `data_handle`s to that row become invalid (`operator bool` false) | +| PointProcess destroyed while NetCons target it | Existing NetCon disconnect rules; `Target = -1` | +| Weight block freed | NetCon weight fields cleared; HOC weight access errors | +| Node voltage dies under `thvar_` | Existing `notify_when_handle_dies` path | + +### 8.4 Permutation vs mechanism sort (open Q2 resolved) + +**Order inside global sort:** + +1. Nodes (existing). +2. Mechanisms (existing) — Instance rows stable for this step’s PointProcess fixup. +3. **Network:** PointProcess (by thread), Weights (repack by NetCon), NetCon (by src PreSyn / thread), PreSyn (by thread) + fanout ranges. + +PointProcess `Instance` is rewritten if mechanism permute moved rows (stable mech handles preferred when available). + +--- + +## 9. `pnt_receive` signature migration + +### 9.1 Today + +```cpp +typedef void (*pnt_receive_t)(Point_process*, double* weight, double flag); +// POINT_RECEIVE(type, tar, w, f) (*pnt_receive[type])(tar, w, f) +``` + +### 9.2 Target (align CoreNEURON) + +```cpp +// Conceptual end state +void pnt_receive(int type, int pnt_instance /* or pnt row */, int weight_index, double flag); +// body loads weights via global/thread weight base + weight_index +``` + +### 9.3 Migration steps + +| Step | Action | +|------|--------| +| M1 | Dual-write weights into SoA; `NetCon::weight_` points into SoA (`data()` + index) so existing `pnt_receive` still gets `double*` | +| M2 | Delivery uses weight_index internally; still materializes `double*` for generated code | +| M3 | Codegen / MOD translation accepts weight_index (Phase 4); legacy pointer form kept until mods regenerated | +| M4 | Drop per-NetCon heap weights | + +Phase 1 only needs M1 scaffolding (container + optional pointer into SoA). **No MOD codegen changes in Phase 1.** + +--- + +## 10. SaveState / BBSaveState (note for Phase 2+) + +| Concern | Impact | +|---------|--------| +| `NetConSave` weight pointer tables | Live helper `weight2netcon(double*)` remains; SaveState SelfEvent **file** identity is NetCon obj index + `weight_index2netcon`. Call `NetConSave::invalid()` when heap bases reallocate. | +| SaveState weight values | Save prefers Weight SoA; restore writes heap + SoA (`restorenet`). | +| `PreSynSave` / `hi_index_` | Keep index tables; rebuild on unsorted. | +| BBSaveState | Weight SoA sync on IO; SelfEvent DEList `ncindex` + heap/`weight_index` match (Commit B done). | +| Binary layout | SelfEvent line already stores `ncindex`; keep compatible. | + +Minimal Phase 1 impact: none if PointProcess/Weight SoA is not yet referenced by SaveState. + +--- + +## 11. Handle API and HOC wrappers + +| HOC-facing type | Wrapper holds | Example | +|-----------------|---------------|---------| +| Point process object | `PointProcess::owning_handle` (+ legacy `Point_process*` during dual-write) | `instance()` → mech row | +| `NetCon` | `NetCon::owning_handle` | `weight(i)` → `weights[WeightIndex+i]` | +| `PreSyn` | `PreSyn::owning_handle` | `threshold` ↔ SoA column; `thvar_` stays `data_handle` | + +**CPU delivery target:** + +```cpp +// After Phase 4 +pnt_receive[type](/* target identity */, weight_index, flag); +``` + +--- + +## 12. Migration strategy (execution order) + +| Step | Action | +|------|--------| +| M1 dual-write | Allocate SoA row when creating legacy object | +| M2 dual-read | Hot path prefers SoA; HOC via wrapper/legacy | +| M3 remove hot-path pointers | Drop `weight_` heap, `dil_` iteration | +| M4 sidecars only | Interpreter extras only in sidecars | + +**Order:** weights + Point_process → NetCon → PreSyn → SelfEvent. + +--- + +## 13. Test plan + +### 13.1 Unit (Phase 1) + +| Test | Expectation | +|------|-------------| +| Create N PointProcess rows; set Instance/MechType/ThreadId | Round-trip via handles | +| `apply_reverse_permutation` on PointProcess storage | Handle logical values unchanged; storage order may change | +| Create weight block of size K; permute | Values via handles stable | +| Destroy `owning_handle` | Non-owning handle / data_handle invalid | +| `find_container_info` / `data_handle` promote | Weight `Value` and PointProcess fields discoverable | + +### 13.2 Integration (Phase 2+) + +| Gate | Command / check | +|------|-----------------| +| Phase 2 | CPU ringtest delivery (NetCon → ExpSyn) | +| Phase 3 | CPU spike parity @ 100 ms vs baseline | +| Phase 4 | ExpSyn receive @ 1.025 ms CPU | +| Regression | `ctest` network-related; SaveState smoke if touched | + +### 13.3 Non-tests on this branch + +- GPU `net_buf_receive`, ringtest GPU network buffers. + +--- + +## 14. Phase 0 deliverables checklist + +- [x] Final field tag list per container (§5.2–5.6) +- [x] `Model` ownership and file layout (`src/neuron/container/network/`) +- [x] Handle typedef names (`PointProcess::handle`, `Weight::owning_handle`, …) +- [x] Thread offset fields on `NrnThread` / cache (§6) +- [x] Sidecar map design + invalidation rules (§7–§8) +- [x] `pnt_receive` signature migration plan (§9) +- [x] SaveState / BBSaveState impact note (§10) +- [x] Test plan (§13) + +--- + +## 15. Phase 1 code scaffold + +``` +src/neuron/container/network/ + point_process.hpp # field tags, storage, handle API + weights.hpp # field tags, storage, handle API +``` + +Hook points (implementation after scaffold): + +1. `Model` members + accessors + `find_container_info` + unsorted callbacks. +2. Unit test: permute survival. +3. Later: `nrn_point_process` / NetCon ctor dual-write (not required for empty scaffold compile). + +--- + +## 16. Resolved design questions + +| # | Question | Decision | +|---|----------|----------| +| 1 | Global vs per-thread weight pool | **Per-thread slices** of one global SoA (CoreNEURON-compatible); absolute indices during dual-write OK | +| 2 | Network permute vs mechanism permute | **After** nodes + mechanisms inside `nrn_ensure_model_data_are_sorted` | +| 3 | NetCon as `DiscreteEvent` | **Keep subclass** through Phase 2–3 for queue compatibility | +| 4 | `InputPreSyn` | **Role yes, dual class optional**; thin gid→fanout after cpu-net-soa — §5.5.1 | +| 5 | Weight contiguity under permute | **Repack at sort** into contiguous per-NetCon blocks; rewrite `WeightIndex` | +| 6 | PreSyn threshold | **Keep `data_handle`**; optional `ThVarRow` cache for scans | +| 7 | NetCon `target_` / `src_` | **Indices on hot path**; **handles** for stable/HOC refs; **not** `data_handle`; pointers dual-write only — §5.4.1 | + +--- + +## 17. Out-of-scope reminder + +Do **not** implement GPU `net_buf_receive` on `local/cpu-network-soa`. Stage 1 `NetReceiveBuffer` on the gpu-native branch remains a valid pattern; indexing will align after SoA merges. + +--- + +*This document is the Phase 0 gate. Phase 1 implements PointProcess + Weight containers and proves handles survive permutation.* diff --git a/doc/network-soa/README.md b/doc/network-soa/README.md new file mode 100644 index 0000000000..b25a3f34f9 --- /dev/null +++ b/doc/network-soa/README.md @@ -0,0 +1,62 @@ +# Network SoA — developer docs (map) + +**Audience:** people working on `local/cpu-network-soa` +**Status tracker / session handoff:** `GROK-NETWORK-SOA.md` (repo root) +**Phase-0 field tags & checklist:** `doc/network-soa-phase0.md` (detailed, longer) + +Docs here are **segregated by height**. Prefer the shortest document that answers your question. + +| Level | Doc | Length / purpose | +|-------|-----|------------------| +| **L0 — North star** | this file §North star | One screen: goals, non-goals, current posture | +| **L1 — Topology** | [topology.md](topology.md) | PreSyn → NetCon → Point_process; CoreNEURON mapping; fanout authority; rank/thread placement | +| **L2 — Dual-write & heap** | [dual-write-and-heap.md](dual-write-and-heap.md) | What is primary where; why `weight_` still exists; FOR_NETCONS / INITIAL / SaveState | +| **L3 — Sort & packing** | [sort-and-packing.md](sort-and-packing.md) | What `nrn_ensure_model_data_are_sorted` does to network containers | +| **L4 — Spec detail** | `../network-soa-phase0.md` | Field tags, sidecars, SaveState notes, open questions resolved | +| **Ops** | `../../GROK-NETWORK-SOA.md` | Build/test gates, commit status, “what next” | + +--- + +## North star (L0) + +**Goal:** One CPU backing store for *integration-hot* network data (columns + indices), with HOC wrappers as permutation-stable handles — same *spirit* as node/mechanism SoA and CoreNEURON’s per-thread flat arrays. + +**Integration reference:** CoreNEURON (`NrnThread::{pntprocs,netcons,weights,presyns}`, `PreSyn::{nc_index_,nc_cnt_}`, `NetCon::u.weight_index_`). + +**Not the first PR’s job (unless we change strategy):** + +- Full free of `NetCon::weight_` under default **nocmodl** ABI +- GPU net buffers (sibling track after SoA is usable) +- Thin remote `InputPreSyn` / gid→fanout table + +**Current posture (2026-07):** Dual-write complete for CoreNEURON-shaped columns; network participates in global sort; SaveState/BBSaveState dual-write; **`weight_` remains long-lived MOD scratch** because generated `pnt_receive` / `net_send(..., _w)` still use `double*`. + +**PR vs master:** A PR can document progress and gather review without claiming “land now.” Landing criteria (performance/space *and* preferably a path to GPU) are product decisions separate from “is the dual-write design sound?” + +--- + +## Two graphs (memorize this) + +```text +HOT PATH (spike → delivery) — CoreNEURON shape, indices: + PreSyn ──(NcIndex,NcCount)──► ordered NetCon fanout + NetCon ──Target──► PointProcess row + NetCon ──WeightIndex──► contiguous weight block + +MANAGEMENT / HOC (interactive NEURON) — pointers/handles still exist in dual-write: + PreSyn.dil_ / NetCon.src_ / NetCon.target_ / Object* +``` + +**Fanout authority** = which of the two is allowed to decide “who receives this spike” on the hot path. See [topology.md](topology.md). + +--- + +## Related code + +| Concern | Location | +|---------|----------| +| Containers | `src/neuron/container/network/*.hpp` | +| Sort / repack | `src/nrncvode/network_soa_sort.cpp` | +| Fanout dual-write | `PreSyn::ensure_fanout_order` in `netcvode.cpp` | +| Receive by index | `nrn_pnt_receive_by_weight_index` | +| CoreNEURON layouts | `src/coreneuron/sim/multicore.hpp`, `network/netcon.hpp` | diff --git a/doc/network-soa/dual-write-and-heap.md b/doc/network-soa/dual-write-and-heap.md new file mode 100644 index 0000000000..379fbd71a3 --- /dev/null +++ b/doc/network-soa/dual-write-and-heap.md @@ -0,0 +1,59 @@ +# L2 — Dual-write and the weight heap + +Brevity: medium. Topology: [topology.md](topology.md). + +--- + +## 1. Dual-write rule of thumb + +| Plane | Primary for… | Examples | +|-------|----------------|----------| +| **Weight SoA** | HOC `weight[]`, SaveState values, future GPU upload | `weight_soa_`, `Model::weights()` | +| **`weight_` heap** | Generated MOD `double*` ABI, FOR_NETCONS walks, `net_send` capturing `_w` | `NetCon::weight_` | +| **Pointers (`target_`, `src_`, `dil_`)** | Queue / HOC / disconnect until dual-read done | Legacy shells | +| **Indices (`Target`, `WeightIndex`, `NcIndex`)** | CoreNEURON-shaped hot path once dual-read | SoA columns | + +**Invariant:** after any path that mutates weights for HOC visibility, SoA and heap agree *or* the next deliver materializes SoA→heap before MOD runs and heap→SoA after. + +--- + +## 2. Why “short-lived materialize only” failed + +Passing a **temporary** buffer into `pnt_receive` broke SelfEvent identity: nocmodl does `net_send(..., _w, ...)` and SaveState keyed SelfEvents by that pointer (`weight2netcon`). + +**Current policy:** long-lived `weight_` is **MOD scratch** (stable identity for `net_send`). SoA remains HOC-primary; deliver does SoA→heap→MOD→heap→SoA (or equivalent owner path in `nrn_pnt_receive_by_weight_index`). + +--- + +## 3. SaveState / BBSaveState (done on this branch) + +- **Values:** save prefer SoA; restore write heap **and** SoA. +- **SelfEvent identity:** NetCon object index / DEList `ncindex` + `weight_index`, not sole reliance on heap pointer (pointer still works as fallback while heap lives). + +--- + +## 4. Roadmap to full heap free (if chosen) + +Rough commit series (each gateable): + +1. **SelfEvent never stores identity-only-as-temp-pointer** — always set `weight_index_` (done in dual-write spirit); never pass non-owner buffers into `pnt_receive` if MOD can net_send. +2. **FOR_NETCONS** — either CoreNEURON-style weight index perm, or generate index-based FOR_NETCONS; stop requiring foreign NetCon `weight_` bases. +3. **INITIAL** — `pnt_receive_init` by index or materialize into owner heap only. +4. **Stop allocating `new double[cnt_]`** — optional thread-local scratch of `max(pnt_receive_size)` for MOD only if ABI still needs `double*`. +5. **Delete `weight_` field** — after (1–4) and tests (stdp/FOR_NETCONS, SaveState ring, netrec init). + +Until (2)–(3), “full drop” is premature. + +--- + +## 5. PR strategy (honest) + +| Option | Pros | Cons | +|--------|------|------| +| **PR series without heap free** | Reviewable design; CI; early feedback; doesn’t pretend GPU is done | Master doesn’t get space win yet | +| **Hold until heap free** | Stronger “finished dual-write” story | Large blocked-on-codegen risk; long private branch | +| **Hold until GPU** | End-to-end performance story | Couples orthogonal tracks; slowest | + +Recommendation: **document dual-write+sort as a reviewable PR (or draft PR), keep landing bar explicit** (“not for merge until X”). Heap-free and GPU as **follow-on PR chains**, not prerequisites for *having* a PR. + +If the only merge criterion is measured space/time, then next engineering is **benchmarks** (ringtest CPU memory + spike path) *and* heap-free or GPU — not more silent dual-write surface area. diff --git a/doc/network-soa/sort-and-packing.md b/doc/network-soa/sort-and-packing.md new file mode 100644 index 0000000000..38ff3feed4 --- /dev/null +++ b/doc/network-soa/sort-and-packing.md @@ -0,0 +1,33 @@ +# L3 — Sort and packing + +Brevity: short–medium. Spec detail: phase0 §6, §8. + +--- + +## When + +`nrn_ensure_model_data_are_sorted()` after nodes + mechanisms. Network containers issue frozen tokens; if any unsorted → `sort_network_data`. + +## Order of work + +1. Sync dual-write shells → SoA (`ob2pntproc_0` for HOC PP; `soa_sync` NetCon/PreSyn). +2. **PointProcess** — reverse-permute by `ThreadId`; set `cache.thread[i].point_process_offset`. +3. **Weight** — repack so each NetCon’s `weight_soa_` rows are contiguous; NetCons ordered by (target thread, src PreSyn); set `weight_offset`. +4. **NetCon** SoA — same order as weight packing; set `netcon_offset`; refresh indices. +5. **PreSyn** — by thread; set `presyn_offset`; rebuild fanout (`NcIndex`/`NcCount` + order table). + +## Why this order + +- Target-thread packing matches CoreNEURON **per-thread weight/netcon pools**. +- Fanout rebuild **after** PreSyn/NetCon permutes so ranges match current rows. +- Weight contiguity is required for `weight_index` base + `count` semantics (CoreNEURON). + +## What sort does *not* claim + +- Fanout physical layout ≡ NetCon SoA order (may use separate order array). +- FOR_NETCONS adjacency (separate perm, CoreNEURON-style, still TODO for index path). +- Freeing `weight_` heap. + +## Implementation + +`src/nrncvode/network_soa_sort.cpp`, header `neuron/container/network/sort.hpp`. diff --git a/doc/network-soa/topology.md b/doc/network-soa/topology.md new file mode 100644 index 0000000000..b751e6a373 --- /dev/null +++ b/doc/network-soa/topology.md @@ -0,0 +1,159 @@ +# L1 — Topology: PreSyn → NetCon → Point_process + +Brevity: medium. For field tags see `doc/network-soa-phase0.md` §5. + +--- + +## 1. What “fanout authority” means + +On a spike, NEURON must enumerate the NetCons that fire from a given source. + +| Representation | Role historically | Authority? | +|----------------|-------------------|------------| +| `PreSyn::dil_` (`vector`) | Interactive NEURON; append on connect; disconnect walks this | **Was** authority | +| `PreSyn` SoA `NcIndex`/`NcCount` + global order (`g_network_fanout_order` or, later, NetCon SoA range) | CoreNEURON-shaped fanout | **Should be** authority on hot path after dual-read | + +**Fanout authority** = the structure the spike path *must* trust. +Management code (connect, disconnect, HOC inspect) may still *write* `dil_` during dual-write, but must not leave delivery depending on a second, divergent truth. + +Today (dual-write): + +- Connect still fills `dil_`. +- `ensure_fanout_order()` rebuilds order + `NcIndex`/`NcCount` from `dil_` (and at sort / `init_events`). +- Spike path prefers SoA range when sorted, else falls back to `dil_`. + +**Target end state:** rebuild ranges from connectivity at sort (or from a single edge table); `dil_` becomes optional sidecar or dies. Delivery never walks `dil_`. + +That is the same *idea* as CoreNEURON: `nc_index_` / `nc_cnt_` into `netcon_in_presyn_order_[]`, not a per-PreSyn pointer vector. + +--- + +## 2. CoreNEURON layout (what to keep in mind) + +Per **thread** (after transfer), CoreNEURON holds flat arrays: + +```text +nt.pntprocs[n_pntproc] // Point_process { _type, _i_instance, _tid, ... } +nt.netcons[n_netcon] // NetCon { active, delay, target_*, weight_index } +nt.weights[n_weight] // flat double pool +nt.presyns[n_presyn] // local/output PreSyn { threshold, gid, nc_index, nc_cnt, ... } +// plus InputPreSyn for remote gid→fanout only +``` + +### Spike fanout (source-centric) + +```text +PreSyn.nc_index_ / nc_cnt_ + → netcon_in_presyn_order_[nc_index + i] // NetCon* (setup), or index into nt.netcons +``` + +Source is **implicit**: NetCons in that range belong to this PreSyn. There is **no** hot-path `src_` on NetCon. + +### Delivery (target-centric data, not fanout) + +```text +NetCon.target_ → Point_process* into nt.pntprocs +NetCon.u.weight_index_ → base into nt.weights[] +pnt_receive(target, weight_index, flag) // CoreNEURON: index, not double* +``` + +Weight **block length** is fixed by the target mechanism’s `NET_RECEIVE` arity (`pnt_receive_size[type]`). CoreNEURON does **not** store weight count on NetCon in the hot struct the same way; the size is known from `target_->_type`. + +### FOR_NETCONS (target-centric *permutation*) + +NetCons that share a target are **not** adjacent in construction order. CoreNEURON builds: + +- `_fornetcon_weight_perm` — indices into `weights` so one target’s NetCon weight groups become adjacent in *index space* +- `_fornetcon_perm_indices` — displacement into that perm for each target instance + +So: **fanout order is by source PreSyn; FOR_NETCONS order is a separate target-grouped view of weight indices.** Two orderings, one weight pool. + +### Rank / thread placement (your intuition) + +| Constraint | CoreNEURON / multicore NEURON | +|------------|-------------------------------| +| NetCon target PP must be local to the rank that owns the synapse | Yes — remote connectivity is **gid → InputPreSyn fanout → local NetCons**, not a NetCon with a remote target pointer | +| NetCon delivery thread = target PP’s thread | Yes — `PreSyn::send` enqueues on `PP2NT(target)`; cross-thread only as interthread event | +| Weights live with **target** thread’s pool | Yes — `nt.weights` is per-thread (CoreNEURON) | + +**NetCon is “about” the target PP** for: + +- weight arity (`NET_RECEIVE` args) +- delivery locality (same rank; same thread as target) +- `pnt_receive` / FOR_NETCONS + +**NetCon is “about” the source PreSyn** only for: + +- *who spikes it into existence* (fanout membership) +- delay / active on that edge + +So both of these are true: + +1. **Data plane packing:** prefer **target-thread** slices (weights, NetCon rows that deliver on that thread, PointProcess rows). +2. **Spike enumeration:** **source-centric** ranges (`NcIndex`/`NcCount`). + +NEURON’s global SoA + **thread partition by permutation** (like nodes/mechs) is the elegant way to get (1) without per-thread heap graphs. Fanout ranges (2) are a *logical* order that may be a separate index array (CoreNEURON’s `netcon_in_presyn_order_`) even if NetCon SoA rows are packed by target thread. + +--- + +## 3. How dual-write NEURON maps today + +| CoreNEURON | NEURON dual-write | +|------------|-------------------| +| `nt.pntprocs[]` | `Model::point_processes()` + legacy `Point_process*` shell | +| `nt.netcons[]` | `Model::netcons()` + `NetCon` DiscreteEvent shell | +| `nt.weights[]` | `Model::weights()` + long-lived `NetCon::weight_` MOD scratch | +| `nt.presyns[]` | `Model::presyns()` + `PreSyn` shell + `dil_` | +| `nc_index_/nc_cnt_` | SoA fields + `g_network_fanout_order` (NetCon* table) | +| `weight_index_` | SoA `WeightIndex` (+ `weight_soa_.front().current_row()`) | +| `pnt_receive(..., weight_index, flag)` | `nrn_pnt_receive_by_weight_index` → still calls nocmodl `double*` form | + +**Not yet CoreNEURON-true:** + +- NetCon SoA row order is not required to equal fanout order (we have a **separate** fanout pointer table). +- `pnt_receive` still takes `double*` (nocmodl). +- `weight_` heap still allocated per NetCon. +- Remote sources still fat PreSyn in `gid2in_` (InputPreSyn role deferred). + +--- + +## 4. Ordering arrangements (summary table) + +| Order | Key | Used for | +|-------|-----|----------| +| **Thread pack** | `thread_id` of target PP (weights/NetCons/PPs); PreSyn by its thread | Cache locality, future GPU upload slices, same as node/mech sort | +| **Spike fanout** | Source PreSyn (contiguous range) | `PreSyn::send` / deliver fanout | +| **FOR_NETCONS** | Target PP instance (perm of weight bases) | CoreNEURON `_fornetcon_*`; NEURON still walks heaps/pointers today | +| **HOC construction** | Object create order | SaveState lists, some BBSaveState assumptions — **not** hot-path locality | + +Sort wiring today: partition PP/PreSyn by thread; pack weight blocks by NetCon ordered (target thread, src PreSyn); rebuild fanout ranges. That is deliberately **compatible** with CoreNEURON’s split between target-thread pools and source-centric fanout indices. + +--- + +## 5. Implications for “full heap drop” + +Heap `weight_` is **not** required by topology. It is required by: + +1. **nocmodl ABI** — `pnt_receive(Point_process*, double*, flag)` and `net_send` capturing `_w` as SelfEvent identity. +2. **FOR_NETCONS** — generated code walks other NetCons’ `weight_` bases (or needs a CoreNEURON-like weight perm). +3. **INITIAL** — `pnt_receive_init(..., double*, ...)`. + +CoreNEURON already dropped (1) by generating `weight_index` receive. NEURON default builds use nocmodl, so (1) is a **codegen/ABI** project unless we keep a permanent materialize shim (SoA → scratch buffer that is *not* per-NetCon heap — e.g. thread-local scratch of max arity, *if* SelfEvent identity is always `weight_index` / NetCon id, never the scratch pointer). + +That last sentence is the real fork for “full drop without full GPU”: + +- **Path H (heap-free, nocmodl kept):** SelfEvent and FOR_NETCONS never store/compare `double* weight_`; only indices; scratch is ephemeral and never identity. +- **Path C (CoreNEURON ABI):** change generated receive to `weight_index` (nocmodl or NMODL project). + +Either can be a PR series; neither is required to *open* a design/review PR of dual-write+sort. + +--- + +## 6. Discussion anchors (for back-and-forth) + +1. **Should fanout order array stay `NetCon*` or become integer NetCon SoA rows?** + Integers match GPU/CoreNEURON export; pointers ease dual-write with DiscreteEvent queue. +2. **Should NetCon SoA physical order follow target-thread, source, or construction?** + Recommendation: **target-thread primary** (weights co-located); fanout stays a separate index range (CoreNEURON pattern). +3. **Is rank-local target enough, or must we also assert same-thread at connect?** + NEURON already delivers to `PP2NT(target)`; cross-thread NetCon is allowed as interthread send (same as CoreNEURON). diff --git a/src/neuron/cache/model_data.hpp b/src/neuron/cache/model_data.hpp index 675b9b4097..14d410aa6f 100644 --- a/src/neuron/cache/model_data.hpp +++ b/src/neuron/cache/model_data.hpp @@ -32,6 +32,14 @@ struct Thread { * @brief Offsets into global mechanism storage for this thread (one per mechanism) */ std::vector mechanism_offset{}; + /** + * @brief Offsets into global network SoA storages for this thread. + * @see doc/network-soa-phase0.md §6 + */ + std::size_t point_process_offset{}; + std::size_t weight_offset{}; + std::size_t netcon_offset{}; + std::size_t presyn_offset{}; }; struct Model { std::vector thread{}; diff --git a/src/neuron/container/memory_usage.hpp b/src/neuron/container/memory_usage.hpp index 33c3cfa5db..6e7c83afbb 100644 --- a/src/neuron/container/memory_usage.hpp +++ b/src/neuron/container/memory_usage.hpp @@ -76,9 +76,25 @@ struct ModelMemoryUsage { /// @brief The memory usage of all mechanisms. StorageMemoryUsage mechanisms{}; + /// @brief Network SoA: Point_process integration rows. + StorageMemoryUsage point_processes{}; + + /// @brief Network SoA: flat weight pool. + StorageMemoryUsage weights{}; + + /// @brief Network SoA: NetCon integration rows. + StorageMemoryUsage netcons{}; + + /// @brief Network SoA: PreSyn integration rows. + StorageMemoryUsage presyns{}; + const ModelMemoryUsage& operator+=(const ModelMemoryUsage& other) { nodes += other.nodes; mechanisms += other.mechanisms; + point_processes += other.point_processes; + weights += other.weights; + netcons += other.netcons; + presyns += other.presyns; return *this; } @@ -86,6 +102,10 @@ struct ModelMemoryUsage { VectorMemoryUsage compute_total() const { auto total = nodes.compute_total(); total += mechanisms.compute_total(); + total += point_processes.compute_total(); + total += weights.compute_total(); + total += netcons.compute_total(); + total += presyns.compute_total(); return total; } @@ -187,6 +207,10 @@ struct MemoryUsageSummary { void add(const ModelMemoryUsage& model) { add(model.nodes); add(model.mechanisms); + add(model.point_processes); + add(model.weights); + add(model.netcons); + add(model.presyns); } void add(const cache::ModelMemoryUsage& model) { diff --git a/src/neuron/container/network/netcon.hpp b/src/neuron/container/network/netcon.hpp new file mode 100644 index 0000000000..d626ff457e --- /dev/null +++ b/src/neuron/container/network/netcon.hpp @@ -0,0 +1,162 @@ +#pragma once +/** + * @file network/netcon.hpp + * @brief SoA storage for integration-hot NetCon fields. + * + * CoreNEURON reference: target_, u.weight_index_, delay_, active_. + * Design: doc/network-soa-phase0.md §5.4, §5.4.1. + * + * Phase 2 dual-write: C++ NetCon keeps DiscreteEvent + pointers; this SoA holds + * the CoreNEURON-shaped columns. HOC weight() steers into Weight SoA when + * weight_soa_ is populated (see netcvode.cpp). + */ +#include "neuron/container/data_handle.hpp" +#include "neuron/container/soa_container.hpp" +#include "neuron/container/view_utils.hpp" + +#include +#include + +namespace neuron::container::network::NetCon { +namespace field { + +/** @brief Row in PointProcess storage (CoreNEURON target). */ +struct Target { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Base row in Weight storage for this NetCon's weight block. */ +struct WeightIndex { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Number of weights (pnt_receive_size). */ +struct WeightCount { + using type = int; + constexpr type default_value() const { + return 0; + } +}; + +/** @brief Delivery delay (ms). */ +struct Delay { + using type = double; + constexpr type default_value() const { + return 1.0; + } +}; + +/** @brief Active flag as int (GPU-friendly). */ +struct Active { + using type = int; + constexpr type default_value() const { + return 1; + } +}; + +/** + * @brief Source PreSyn row, or -1. + * + * Phase 2: remains -1 until PreSyn SoA exists (Phase 3). Reverse edge only. + */ +struct SrcPreSyn { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +} // namespace field + +/** + * @brief Public API for NetCon handles (owning and non-owning). + * @tparam Identifier owning_identifier or non_owning_identifier for storage. + */ +template +struct handle_interface: handle_base { + using base_type = handle_base; + using base_type::base_type; + + [[nodiscard]] field::Target::type& target() { + return this->template get(); + } + [[nodiscard]] field::Target::type const& target() const { + return this->template get(); + } + + [[nodiscard]] field::WeightIndex::type& weight_index() { + return this->template get(); + } + [[nodiscard]] field::WeightIndex::type const& weight_index() const { + return this->template get(); + } + + [[nodiscard]] field::WeightCount::type& weight_count() { + return this->template get(); + } + [[nodiscard]] field::WeightCount::type const& weight_count() const { + return this->template get(); + } + + [[nodiscard]] field::Delay::type& delay() { + return this->template get(); + } + [[nodiscard]] field::Delay::type const& delay() const { + return this->template get(); + } + + [[nodiscard]] field::Active::type& active() { + return this->template get(); + } + [[nodiscard]] field::Active::type const& active() const { + return this->template get(); + } + + [[nodiscard]] field::SrcPreSyn::type& src_presyn() { + return this->template get(); + } + [[nodiscard]] field::SrcPreSyn::type const& src_presyn() const { + return this->template get(); + } + + friend std::ostream& operator<<(std::ostream& os, handle_interface const& handle) { + if (handle.id()) { + return os << "NetCon{" << handle.id() << '/' << handle.underlying_storage().size() + << " target=" << handle.target() << " widx=" << handle.weight_index() + << " wcnt=" << handle.weight_count() << " delay=" << handle.delay() + << " active=" << handle.active() << " src=" << handle.src_presyn() << '}'; + } + return os << "NetCon{null}"; + } +}; + +/** @brief Underlying storage for all NetCon integration rows. */ +struct storage: soa { + [[nodiscard]] std::string_view name() const { + return "network::NetCon"; + } +}; + +using handle = handle_interface>; + +struct owning_handle: handle_interface> { + using base_type = handle_interface>; + using base_type::base_type; + + [[nodiscard]] handle non_owning_handle() { + return non_owning_identifier{&underlying_storage(), id()}; + } +}; +} // namespace neuron::container::network::NetCon diff --git a/src/neuron/container/network/point_process.hpp b/src/neuron/container/network/point_process.hpp new file mode 100644 index 0000000000..805b85132c --- /dev/null +++ b/src/neuron/container/network/point_process.hpp @@ -0,0 +1,116 @@ +#pragma once +/** + * @file network/point_process.hpp + * @brief SoA storage for integration-hot Point_process fields. + * + * CoreNEURON reference: coreneuron::Point_process {_i_instance, _type, _tid}. + * Design: doc/network-soa-phase0.md §5.2. + * + * HOC location / Object* / Prop* remain on the legacy Point_process shell or + * sidecars during dual-write (Phase 1 does not replace interpreter objects). + */ +#include "neuron/container/data_handle.hpp" +#include "neuron/container/soa_container.hpp" +#include "neuron/container/view_utils.hpp" + +#include +#include + +namespace neuron::container::network::PointProcess { +namespace field { + +/** @brief Mechanism SoA instance row (_i_instance). */ +struct Instance { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Mechanism type (_type). */ +struct MechType { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Owning NrnThread id (_tid). */ +struct ThreadId { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +} // namespace field + +/** + * @brief Public API for PointProcess handles (owning and non-owning). + * @tparam Identifier owning_identifier or non_owning_identifier for storage. + */ +template +struct handle_interface: handle_base { + using base_type = handle_base; + using base_type::base_type; + + [[nodiscard]] field::Instance::type& instance() { + return this->template get(); + } + [[nodiscard]] field::Instance::type const& instance() const { + return this->template get(); + } + [[nodiscard]] data_handle instance_handle() { + return this->template get_handle(); + } + + [[nodiscard]] field::MechType::type& mech_type() { + return this->template get(); + } + [[nodiscard]] field::MechType::type const& mech_type() const { + return this->template get(); + } + [[nodiscard]] data_handle mech_type_handle() { + return this->template get_handle(); + } + + [[nodiscard]] field::ThreadId::type& thread_id() { + return this->template get(); + } + [[nodiscard]] field::ThreadId::type const& thread_id() const { + return this->template get(); + } + [[nodiscard]] data_handle thread_id_handle() { + return this->template get_handle(); + } + + friend std::ostream& operator<<(std::ostream& os, handle_interface const& handle) { + if (handle.id()) { + return os << "PointProcess{" << handle.id() << '/' << handle.underlying_storage().size() + << " instance=" << handle.instance() << " mech_type=" << handle.mech_type() + << " thread_id=" << handle.thread_id() << '}'; + } + return os << "PointProcess{null}"; + } +}; + +/** @brief Underlying storage for all PointProcess integration rows. */ +struct storage: soa { + [[nodiscard]] std::string_view name() const { + return "network::PointProcess"; + } +}; + +/** @brief Non-owning handle; stable across permute; invalid after owner dies. */ +using handle = handle_interface>; + +/** @brief Owning handle; destroys the storage row on destruction. */ +struct owning_handle: handle_interface> { + using base_type = handle_interface>; + using base_type::base_type; + + [[nodiscard]] handle non_owning_handle() { + return non_owning_identifier{&underlying_storage(), id()}; + } +}; +} // namespace neuron::container::network::PointProcess diff --git a/src/neuron/container/network/point_process_access.hpp b/src/neuron/container/network/point_process_access.hpp new file mode 100644 index 0000000000..1d6bd7f261 --- /dev/null +++ b/src/neuron/container/network/point_process_access.hpp @@ -0,0 +1,30 @@ +#pragma once +/** + * @file point_process_access.hpp + * @brief Internal helper to get a PointProcess SoA handle from Point_process*. + * + * Not for MOD files / section_fwd.hpp — only NEURON library TUs that already + * link model_data. + */ +#include "neuron/container/network/point_process.hpp" +#include "neuron/model_data.hpp" +#include "section_fwd.hpp" + +#include + +namespace neuron::container::network { + +/** @brief Non-owning handle to the dual-write SoA row for @p pnt. */ +inline PointProcess::handle point_process_soa(Point_process* pnt) { + assert(pnt); + assert(pnt->_soa_id); + return PointProcess::handle{ + non_owning_identifier{&neuron::model().point_processes(), + pnt->_soa_id}}; +} + +inline PointProcess::handle point_process_soa(Point_process const* pnt) { + return point_process_soa(const_cast(pnt)); +} + +} // namespace neuron::container::network diff --git a/src/neuron/container/network/presyn.hpp b/src/neuron/container/network/presyn.hpp new file mode 100644 index 0000000000..20493e16c7 --- /dev/null +++ b/src/neuron/container/network/presyn.hpp @@ -0,0 +1,173 @@ +#pragma once +/** + * @file network/presyn.hpp + * @brief SoA storage for integration-hot PreSyn fields. + * + * CoreNEURON reference: nc_index_, nc_cnt_, thvar_index_, threshold_, gid_. + * Design: doc/network-soa-phase0.md §5.5. + * + * Phase 3 dual-write: legacy PreSyn keeps dil_ and thvar_; this SoA holds + * CoreNEURON-shaped columns. Fanout order is rebuilt into a global NetCon* + * table (see netcvode.cpp); NcIndex/NcCount describe ranges in that table. + */ +#include "neuron/container/data_handle.hpp" +#include "neuron/container/soa_container.hpp" +#include "neuron/container/view_utils.hpp" + +#include +#include + +namespace neuron::container::network::PreSyn { +namespace field { + +/** @brief Spike threshold. */ +struct Threshold { + using type = double; + constexpr type default_value() const { + return 10.; + } +}; + +/** @brief Output gid, or -1. */ +struct Gid { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Start index into global fanout order (NetCon*). */ +struct NcIndex { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Fanout count (replaces dil_.size() on hot path when sorted). */ +struct NcCount { + using type = int; + constexpr type default_value() const { + return 0; + } +}; + +/** @brief CoreNEURON output_index_ (MPI / spike compression). */ +struct OutputIndex { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** + * @brief Optional denormalized node-voltage row for threshold scans. + * + * Canonical threshold source remains PreSyn::thvar_ (data_handle). -1 if unknown. + */ +struct ThVarRow { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +/** @brief Owning NrnThread id. */ +struct ThreadId { + using type = int; + constexpr type default_value() const { + return -1; + } +}; + +} // namespace field + +template +struct handle_interface: handle_base { + using base_type = handle_base; + using base_type::base_type; + + [[nodiscard]] field::Threshold::type& threshold() { + return this->template get(); + } + [[nodiscard]] field::Threshold::type const& threshold() const { + return this->template get(); + } + + [[nodiscard]] field::Gid::type& gid() { + return this->template get(); + } + [[nodiscard]] field::Gid::type const& gid() const { + return this->template get(); + } + + [[nodiscard]] field::NcIndex::type& nc_index() { + return this->template get(); + } + [[nodiscard]] field::NcIndex::type const& nc_index() const { + return this->template get(); + } + + [[nodiscard]] field::NcCount::type& nc_count() { + return this->template get(); + } + [[nodiscard]] field::NcCount::type const& nc_count() const { + return this->template get(); + } + + [[nodiscard]] field::OutputIndex::type& output_index() { + return this->template get(); + } + [[nodiscard]] field::OutputIndex::type const& output_index() const { + return this->template get(); + } + + [[nodiscard]] field::ThVarRow::type& thvar_row() { + return this->template get(); + } + [[nodiscard]] field::ThVarRow::type const& thvar_row() const { + return this->template get(); + } + + [[nodiscard]] field::ThreadId::type& thread_id() { + return this->template get(); + } + [[nodiscard]] field::ThreadId::type const& thread_id() const { + return this->template get(); + } + + friend std::ostream& operator<<(std::ostream& os, handle_interface const& handle) { + if (handle.id()) { + return os << "PreSyn{" << handle.id() << '/' << handle.underlying_storage().size() + << " thr=" << handle.threshold() << " gid=" << handle.gid() << " nc=[" + << handle.nc_index() << "," << handle.nc_count() << ")" + << " out=" << handle.output_index() << " tid=" << handle.thread_id() << '}'; + } + return os << "PreSyn{null}"; + } +}; + +struct storage: soa { + [[nodiscard]] std::string_view name() const { + return "network::PreSyn"; + } +}; + +using handle = handle_interface>; + +struct owning_handle: handle_interface> { + using base_type = handle_interface>; + using base_type::base_type; + + [[nodiscard]] handle non_owning_handle() { + return non_owning_identifier{&underlying_storage(), id()}; + } +}; +} // namespace neuron::container::network::PreSyn diff --git a/src/neuron/container/network/self_event.hpp b/src/neuron/container/network/self_event.hpp new file mode 100644 index 0000000000..3bc9444d31 --- /dev/null +++ b/src/neuron/container/network/self_event.hpp @@ -0,0 +1,82 @@ +#pragma once +/** + * @file network/self_event.hpp + * @brief Phase 4 SelfEvent index fields (not a long-lived SoA container). + * + * Design: doc/network-soa-phase0.md §5.6, §9. + * + * SelfEvents are short-lived queue/pool objects. Integration-relevant payload: + * - target PointProcess row (optional dual-write) + * - weight_index into Weight SoA (base of the NetCon weight block) + * - flag (NET_RECEIVE flag) + * + * Generated MOD code still receives double* for Phase 4 (M2 dual-write): + * delivery materializes SoA → heap around pnt_receive. + */ +#include "neuron/container/network/weights.hpp" +#include "neuron/model_data.hpp" + +#include + +namespace neuron::container::network::SelfEventFields { + +/** Field tags for documentation / future structural pool (not used as soa<> tags yet). */ +namespace field { +struct TargetPnt { + using type = int; + static constexpr type default_value() { + return -1; + } +}; +struct WeightIndex { + using type = int; + static constexpr type default_value() { + return -1; + } +}; +struct Flag { + using type = double; + static constexpr type default_value() { + return 0.; + } +}; +} // namespace field + +/** + * @brief Copy @p count consecutive Weight SoA rows starting at @p weight_index into @p out. + * + * Contiguity is the dual-write invariant from weight block allocation (Phase 1–2). + * After a full weight repack/sort this remains valid for a NetCon's block. + */ +inline void materialize_weight_block(int weight_index, int count, double* out) { + if (!out || count <= 0 || weight_index < 0) { + return; + } + auto& store = neuron::model().weights(); + auto const n = static_cast(store.size()); + for (int i = 0; i < count; ++i) { + int const row = weight_index + i; + out[i] = (row >= 0 && row < n) + ? store.get(static_cast(row)) + : 0.; + } +} + +/** + * @brief Write @p count heap values back into Weight SoA at @p weight_index. + */ +inline void store_weight_block(int weight_index, int count, double const* in) { + if (!in || count <= 0 || weight_index < 0) { + return; + } + auto& store = neuron::model().weights(); + auto const n = static_cast(store.size()); + for (int i = 0; i < count; ++i) { + int const row = weight_index + i; + if (row >= 0 && row < n) { + store.get(static_cast(row)) = in[i]; + } + } +} + +} // namespace neuron::container::network::SelfEventFields diff --git a/src/neuron/container/network/sort.hpp b/src/neuron/container/network/sort.hpp new file mode 100644 index 0000000000..fdaab21898 --- /dev/null +++ b/src/neuron/container/network/sort.hpp @@ -0,0 +1,34 @@ +#pragma once +/** + * @file network/sort.hpp + * @brief Sort / repack network SoA containers inside nrn_ensure_model_data_are_sorted. + * + * Design: doc/network-soa-phase0.md §6, §8.2–§8.4. + */ +#include "neuron/cache/model_data.hpp" +#include "neuron/model_data.hpp" + +namespace neuron::container::network { + +/** + * @brief Partition and repack network SoA for integration. + * + * Order (after nodes + mechanisms are already sorted): + * 1. PointProcess by thread + * 2. Weight blocks contiguous per NetCon, packed by target thread + * 3. NetCon by (target thread, src PreSyn) + * 4. PreSyn by thread + rebuild NcIndex/NcCount fanout ranges + * + * @param cache Working model cache (thread offsets filled here). + * @param pp_token Sole frozen token for PointProcess storage. + * @param w_token Sole frozen token for Weight storage. + * @param nc_token Sole frozen token for NetCon storage. + * @param ps_token Sole frozen token for PreSyn storage. + */ +void sort_network_data(neuron::cache::Model& cache, + PointProcess::storage::frozen_token_type& pp_token, + Weight::storage::frozen_token_type& w_token, + NetCon::storage::frozen_token_type& nc_token, + PreSyn::storage::frozen_token_type& ps_token); + +} // namespace neuron::container::network diff --git a/src/neuron/container/network/weight_block.hpp b/src/neuron/container/network/weight_block.hpp new file mode 100644 index 0000000000..15c70a9315 --- /dev/null +++ b/src/neuron/container/network/weight_block.hpp @@ -0,0 +1,48 @@ +#pragma once +/** + * @file network/weight_block.hpp + * @brief Helpers to dual-write NetCon weight blocks into Weight SoA. + * + * Kept separate from weights.hpp so model_data.hpp can include Weight storage + * without a circular include. + * + * Phase 1: heap `double* weight_` remains the delivery primary; SoA rows are + * owned in parallel for layout readiness (Phase 2 WeightIndex). + */ +#include "neuron/container/network/weights.hpp" +#include "neuron/model_data.hpp" + +#include +#include + +namespace neuron::container::network::Weight { + +/** + * @brief Allocate @p n weight SoA rows, optionally mirroring heap values. + */ +inline std::vector allocate_weight_rows(int n, double const* mirror = nullptr) { + std::vector rows; + if (n <= 0) { + return rows; + } + auto& store = neuron::model().weights(); + rows.reserve(static_cast(n)); + for (int i = 0; i < n; ++i) { + rows.emplace_back(store); + rows.back().value() = mirror ? mirror[i] : 0.; + } + return rows; +} + +/** @brief Copy heap weight values into already-allocated SoA rows. */ +inline void mirror_weights_to_soa(std::vector& rows, double const* heap, int n) { + if (!heap) { + return; + } + auto const m = std::min(static_cast(rows.size()), n); + for (int i = 0; i < m; ++i) { + rows[i].value() = heap[i]; + } +} + +} // namespace neuron::container::network::Weight diff --git a/src/neuron/container/network/weights.hpp b/src/neuron/container/network/weights.hpp new file mode 100644 index 0000000000..eaf0433d3c --- /dev/null +++ b/src/neuron/container/network/weights.hpp @@ -0,0 +1,79 @@ +#pragma once +/** + * @file network/weights.hpp + * @brief SoA storage for the flat NetCon weight pool. + * + * CoreNEURON reference: NrnThread::weights[n_weight]; NetCon.u.weight_index_ base. + * Design: doc/network-soa-phase0.md §5.3. + * + * Contiguity: a NetCon with weight_count = K owns K consecutive rows. + * Pack/repack at sort time; WeightIndex on NetCon is rewritten after permute. + */ +#include "neuron/container/data_handle.hpp" +#include "neuron/container/soa_container.hpp" +#include "neuron/container/view_utils.hpp" + +#include +#include + +namespace neuron::container::network::Weight { +namespace field { + +/** @brief One scalar weight entry in the flat pool. */ +struct Value { + using type = double; + constexpr type default_value() const { + return 0.; + } +}; + +} // namespace field + +/** + * @brief Public API for Weight handles (owning and non-owning). + * @tparam Identifier owning_identifier or non_owning_identifier for storage. + */ +template +struct handle_interface: handle_base { + using base_type = handle_base; + using base_type::base_type; + + [[nodiscard]] field::Value::type& value() { + return this->template get(); + } + [[nodiscard]] field::Value::type const& value() const { + return this->template get(); + } + [[nodiscard]] data_handle value_handle() { + return this->template get_handle(); + } + + friend std::ostream& operator<<(std::ostream& os, handle_interface const& handle) { + if (handle.id()) { + return os << "Weight{" << handle.id() << '/' << handle.underlying_storage().size() + << " value=" << handle.value() << '}'; + } + return os << "Weight{null}"; + } +}; + +/** @brief Underlying storage for all weight scalars. */ +struct storage: soa { + [[nodiscard]] std::string_view name() const { + return "network::Weight"; + } +}; + +/** @brief Non-owning handle; stable across permute; invalid after owner dies. */ +using handle = handle_interface>; + +/** @brief Owning handle; destroys the storage row on destruction. */ +struct owning_handle: handle_interface> { + using base_type = handle_interface>; + using base_type::base_type; + + [[nodiscard]] handle non_owning_handle() { + return non_owning_identifier{&underlying_storage(), id()}; + } +}; +} // namespace neuron::container::network::Weight diff --git a/src/neuron/model_data.hpp b/src/neuron/model_data.hpp index 4f35a64d5f..e074b7f175 100644 --- a/src/neuron/model_data.hpp +++ b/src/neuron/model_data.hpp @@ -2,6 +2,10 @@ #include "neuron/cache/model_data.hpp" #include "neuron/container/mechanism_data.hpp" #include "neuron/container/memory_usage.hpp" +#include "neuron/container/network/netcon.hpp" +#include "neuron/container/network/point_process.hpp" +#include "neuron/container/network/presyn.hpp" +#include "neuron/container/network/weights.hpp" #include "neuron/container/node_data.hpp" #include "neuron/model_data_fwd.hpp" @@ -31,6 +35,46 @@ struct Model { return m_node_data; } + /** @brief Access SoA storage for Point_process integration fields. + * @see doc/network-soa-phase0.md + */ + container::network::PointProcess::storage& point_processes() { + return m_point_processes; + } + container::network::PointProcess::storage const& point_processes() const { + return m_point_processes; + } + + /** @brief Access SoA storage for the flat NetCon weight pool. + * @see doc/network-soa-phase0.md + */ + container::network::Weight::storage& weights() { + return m_weights; + } + container::network::Weight::storage const& weights() const { + return m_weights; + } + + /** @brief Access SoA storage for NetCon integration fields (Phase 2). + * @see doc/network-soa-phase0.md §5.4 + */ + container::network::NetCon::storage& netcons() { + return m_netcons; + } + container::network::NetCon::storage const& netcons() const { + return m_netcons; + } + + /** @brief Access SoA storage for PreSyn integration fields (Phase 3). + * @see doc/network-soa-phase0.md §5.5 + */ + container::network::PreSyn::storage& presyns() { + return m_presyns; + } + container::network::PreSyn::storage const& presyns() const { + return m_presyns; + } + /** @brief Apply a function to each non-null Mechanism. */ template @@ -120,6 +164,10 @@ struct Model { void shrink_to_fit() { m_node_data.shrink_to_fit(); apply_to_mechanisms([](auto& mech_data) { mech_data.shrink_to_fit(); }); + m_point_processes.shrink_to_fit(); + m_weights.shrink_to_fit(); + m_netcons.shrink_to_fit(); + m_presyns.shrink_to_fit(); } private: @@ -151,6 +199,18 @@ struct Model { */ std::vector> m_mech_data{}; + /** @brief Network SoA: Point_process integration rows (Phase 1). */ + container::network::PointProcess::storage m_point_processes{}; + + /** @brief Network SoA: flat weight pool (Phase 1). */ + container::network::Weight::storage m_weights{}; + + /** @brief Network SoA: NetCon integration rows (Phase 2). */ + container::network::NetCon::storage m_netcons{}; + + /** @brief Network SoA: PreSyn integration rows (Phase 3). */ + container::network::PreSyn::storage m_presyns{}; + /** * @brief Backing storage for defer_delete helper. */ @@ -185,6 +245,17 @@ struct model_sorted_token { } container::Node::storage::frozen_token_type node_data_token; std::vector mech_data_tokens{}; + /** + * @brief Frozen tokens for network SoA containers (sorted with the model). + * + * Held as vectors (0 or 1 element) because frozen_token_type is not default- + * constructible; same pattern as mech_data_tokens. + */ + std::vector + point_process_tokens{}; + std::vector weight_tokens{}; + std::vector netcon_tokens{}; + std::vector presyn_tokens{}; private: std::reference_wrapper m_cache; diff --git a/src/nrncvode/netcon.h b/src/nrncvode/netcon.h index 17c49eee2b..35766aa34d 100644 --- a/src/nrncvode/netcon.h +++ b/src/nrncvode/netcon.h @@ -3,6 +3,11 @@ #undef check #include "neuron/container/data_handle.hpp" +#include "neuron/container/network/netcon.hpp" +#include "neuron/container/network/presyn.hpp" +#include "neuron/container/network/self_event.hpp" +#include "neuron/container/network/weight_block.hpp" +#include "neuron/model_data.hpp" #include "nrnmpi.h" #include "nrnneosm.h" #include "pool.hpp" @@ -110,6 +115,13 @@ class NetCon: public DiscreteEvent { void replace_src(PreSyn*); virtual void disconnect(Observable*); + /** @brief Sync legacy NetCon fields into NetCon + Weight SoA (Phase 2). */ + void soa_sync(); + /** @brief Copy Weight SoA values → heap weight_ (before pnt_receive). */ + void weights_soa_to_heap(); + /** @brief Copy heap weight_ → Weight SoA (after pnt_receive / MOD writes). */ + void weights_heap_to_soa(); + double delay_; PreSyn* src_; Point_process* target_; @@ -117,6 +129,10 @@ class NetCon: public DiscreteEvent { Object* obj_; int cnt_; bool active_; + /** @brief Phase 1 dual-write owners for Weight SoA rows (see weight_block.hpp). */ + std::vector weight_soa_{}; + /** @brief Phase 2 dual-write: NetCon integration row. */ + neuron::container::network::NetCon::owning_handle _soa{neuron::model().netcons()}; static unsigned long netcon_send_active_; static unsigned long netcon_send_inactive_; @@ -135,8 +151,17 @@ class NetConSave: public DiscreteEvent { NetCon* netcon_; static void invalid(); + /** @brief Map long-lived NetCon weight_ heap base → NetCon*. */ static NetCon* weight2netcon(double*); + /** @brief Map HOC NetCon object index → NetCon*. */ static NetCon* index2netcon(long); + /** + * @brief Map Weight SoA base row (weight_index) → NetCon*. + * + * Used when SelfEvent identity is carried as weight_index_ without a + * reliable weight_ heap pointer (dual-write / heap-drop path). + */ + static NetCon* weight_index2netcon(int weight_index); private: static NetConSaveWeightTable* wtable_; @@ -169,6 +194,10 @@ class SelfEvent: public DiscreteEvent { Point_process* target_; double* weight_; Datum* movable_; // pointed-to Datum holds TQItem* + /** @brief Phase 4: base row in Weight SoA (-1 if unknown / null weights). */ + int weight_index_{-1}; + /** @brief Phase 4: PointProcess SoA row of target_ (-1 if unknown). */ + int target_row_{-1}; static unsigned long selfevent_send_; static unsigned long selfevent_move_; @@ -291,6 +320,13 @@ class PreSyn: public ConditionEvent { double mindelay(); void fanout(double, NetCvode*, NrnThread*); // used by bbsavestate + /** @brief Sync legacy PreSyn fields into PreSyn SoA (Phase 3). */ + void soa_sync(); + /** @brief Mark global NetCon fanout order dirty (dil_ changed). */ + static void mark_fanout_unsorted(); + /** @brief Rebuild global fanout order from all PreSyn dil_ lists. */ + static void ensure_fanout_order(); + NetConPList dil_; double threshold_; double delay_; @@ -307,6 +343,8 @@ class PreSyn: public ConditionEvent { int rec_id_; int output_index_; int gid_; + /** @brief Phase 3 dual-write: PreSyn integration row. */ + neuron::container::network::PreSyn::owning_handle _soa{neuron::model().presyns()}; #if NRNMPI unsigned char localgid_; // compressed gid for spike transfer #endif diff --git a/src/nrncvode/netcvode.cpp b/src/nrncvode/netcvode.cpp index 18b2b5fdd4..0f82f32fd5 100644 --- a/src/nrncvode/netcvode.cpp +++ b/src/nrncvode/netcvode.cpp @@ -118,8 +118,12 @@ double nrn_netcon_get_delay(NetCon* nc) { } void nrn_netcon_set_delay(NetCon* nc, double d) { nc->delay_ = d; + nc->soa_sync(); } int nrn_netcon_weight(NetCon* nc, double** pw) { + // SoA is HOC source of truth when weight_soa_ is populated; keep heap current + // for MOD code that writes through the returned pointer. + nc->weights_soa_to_heap(); *pw = nc->weight_; return nc->cnt_; } @@ -141,6 +145,7 @@ double nrn_netcon_get_thresh(NetCon* nc) { void nrn_netcon_set_thresh(NetCon* nc, double th) { if (nc->src_) { nc->src_->threshold_ = th; + nc->src_->soa_sync(); } } @@ -158,6 +163,7 @@ int nrn_netcon_info(NetCon* nc, double** pw, Point_process** target, double** th *target = (nc->target_) ? nc->target_ : (Point_process*) 0; *th = (nc->src_) ? &(nc->src_->threshold_) : (double*) 0; *del = &nc->delay_; + nc->weights_soa_to_heap(); *pw = nc->weight_; return nc->cnt_; } @@ -644,8 +650,15 @@ static double nc_setpost(void* v) { } if (d->cnt_ != cnt) { d->cnt_ = cnt; - delete[] std::exchange(d->weight_, new double[d->cnt_]); + delete[] d->weight_; + d->weight_ = d->cnt_ ? new double[d->cnt_] : nullptr; + for (int i = 0; i < d->cnt_; ++i) { + d->weight_[i] = 0.0; + } + d->weight_soa_ = neuron::container::network::Weight::allocate_weight_rows(d->cnt_, + d->weight_); } + d->soa_sync(); return 0.; } @@ -663,6 +676,7 @@ static double nc_active(void* v) { bool a = d->active_; if (d->target_ && ifarg(1)) { d->active_ = bool(chkarg(1, 0, 1)); + d->soa_sync(); } hoc_return_type_code = HocReturnType::boolean; return double(a); @@ -775,6 +789,7 @@ static void steer_val(void* v) { Symbol* s = hoc_spop(); if (strcmp(s->name, "delay") == 0) { d->chksrc(); + // HOC writes delay_; soa_sync mirrors into NetCon SoA. hoc_pushpx(&d->delay_); d->src_->use_min_delay_ = 0; } else if (strcmp(s->name, "weight") == 0) { @@ -783,7 +798,12 @@ static void steer_val(void* v) { s->arayinfo->sub[0] = d->cnt_; index = hoc_araypt(s, SYMBOL); } - hoc_pushpx(d->weight_ + index); + // Phase 2: HOC weight() steers into flat Weight SoA when dual-write rows exist. + if (index >= 0 && index < static_cast(d->weight_soa_.size())) { + hoc_push(d->weight_soa_[index].value_handle()); + } else { + hoc_pushpx(d->weight_ + index); + } } else if (strcmp(s->name, "x") == 0) { static double dummy = 0.; d->chksrc(); @@ -2252,6 +2272,124 @@ void NetCvode::remove_event(TQItem* q, int tid) { p[tid].tqe_->remove(q); } +namespace { +/** Phase 4: fill SelfEvent dual-write index fields from weight* / Point_process. */ +void selfevent_set_indices(SelfEvent* se, Point_process* pnt, double* weight) { + se->target_row_ = nrn_point_process_soa_row(pnt); + se->weight_index_ = -1; + if (!weight) { + return; + } + // weight* is almost always a NetCon heap base; resolve Weight SoA index. + if (NetCon* nc = NetConSave::weight2netcon(weight)) { + if (!nc->weight_soa_.empty()) { + se->weight_index_ = static_cast(nc->weight_soa_.front().current_row()); + } else { + se->weight_index_ = nc->_soa.weight_index(); + } + } +} +} // namespace + +namespace { +bool type_has_fornetcon(int type) { + for (int i = 0; i < nrn_fornetcon_cnt_; ++i) { + if (nrn_fornetcon_type_[i] == type) { + return true; + } + } + return false; +} + +/** Sync all NetCon weight heaps ↔ SoA that share a target (FOR_NETCONS mutates them). */ +void sync_netcon_weights_for_target(Point_process* pnt, bool soa_to_heap) { + if (!pnt) { + return; + } + Symbol* sym = hoc_lookup("NetCon"); + if (!sym || !sym->u.ctemplate || !sym->u.ctemplate->olist) { + return; + } + hoc_Item* q = nullptr; + ITERATE(q, sym->u.ctemplate->olist) { + auto* nc = static_cast(OBJ(q)->u.this_pointer); + if (nc && nc->target_ == pnt) { + if (soa_to_heap) { + nc->weights_soa_to_heap(); + } else { + nc->weights_heap_to_soa(); + } + } + } +} +} // namespace + +void nrn_pnt_receive_by_weight_index(Point_process* pnt, + int weight_index, + double flag, + double* weight_heap) { + if (!pnt || !pnt->prop) { + return; + } + int const type = pnt->prop->_type; + if (!pnt_receive[type]) { + return; + } + int const n = pnt_receive_size[type]; + bool const fornet = type_has_fornetcon(type); + + // Prefer a long-lived NetCon weight_ heap buffer for pnt_receive. Temporary + // materialize buffers are unsafe: nocmodl NET_RECEIVE often does + // net_send(..., _w, ...) and SelfEvent must keep a stable weight_ identity + // for SaveState/BBSaveState (weight2netcon). + NetCon* owner = nullptr; + if (weight_heap) { + owner = NetConSave::weight2netcon(weight_heap); + } + if (!owner && weight_index >= 0) { + owner = NetConSave::weight_index2netcon(weight_index); + if (owner && owner->weight_) { + weight_heap = owner->weight_; + } + } + if (owner) { + if (fornet) { + // FOR_NETCONS walks weight_ of all NetCons with this target. + sync_netcon_weights_for_target(pnt, /*soa_to_heap*/ true); + } else { + owner->weights_soa_to_heap(); + } + POINT_RECEIVE(type, pnt, weight_heap ? weight_heap : owner->weight_, flag); + if (fornet) { + sync_netcon_weights_for_target(pnt, /*soa_to_heap*/ false); + } else { + owner->weights_heap_to_soa(); + } + return; + } + // No NetCon owner: flag-only SelfEvent (weight_ often nullptr from mech INITIAL + // net_send) or unresolved index. Preserve historical nullptr for zero-weight + // cases — do **not** invent a temp buffer when weight_index < 0, or MOD may + // net_send that stack pointer and corrupt later queue handling (TQueue UAF). + if (fornet) { + sync_netcon_weights_for_target(pnt, /*soa_to_heap*/ true); + } + double* buf = weight_heap; + std::vector tmp; + if (!buf && weight_index >= 0 && n > 0) { + tmp.resize(static_cast(n), 0.); + buf = tmp.data(); + neuron::container::network::SelfEventFields::materialize_weight_block(weight_index, n, buf); + } + POINT_RECEIVE(type, pnt, buf, flag); + if (weight_index >= 0 && n > 0 && buf) { + neuron::container::network::SelfEventFields::store_weight_block(weight_index, n, buf); + } + if (fornet) { + sync_netcon_weights_for_target(pnt, /*soa_to_heap*/ false); + } +} + // for threads, revised net_send to use absolute time (in the // mod file we add the thread time when we call it). void nrn_net_send(Datum* v, double* weight, Point_process* pnt, double td, double flag) { @@ -2262,6 +2400,7 @@ void nrn_net_send(Datum* v, double* weight, Point_process* pnt, double td, doubl se->flag_ = flag; se->target_ = pnt; se->weight_ = weight; + selfevent_set_indices(se, pnt, weight); se->movable_ = v; // needed for SaveState assert(net_cvode_instance); ++p.unreffed_event_cnt_; @@ -2289,6 +2428,7 @@ void artcell_net_send(Datum* v, double* weight, Point_process* pnt, double td, d se->flag_ = flag; se->target_ = pnt; se->weight_ = weight; + selfevent_set_indices(se, pnt, weight); se->movable_ = v; // needed for SaveState assert(net_cvode_instance); ++p.unreffed_event_cnt_; @@ -2780,6 +2920,9 @@ void NetCvode::init_events() { } } } + // Rebuild CoreNEURON-shaped fanout ranges before the run (not only on first spike). + PreSyn::mark_fanout_unsorted(); + PreSyn::ensure_fanout_order(); } // iterate over all NetCon in creation order to call // NETRECEIVE INITIAL blocks. @@ -2792,8 +2935,12 @@ void NetCvode::init_events() { ITERATE(q, nclist) { Object* obj = OBJ(q); auto* d = static_cast(obj->u.this_pointer); - if (d->target_) { - int type = d->target_->prop->_type; // somehow prop is non-deterministically-null here + // target_ may outlive Prop (unlocated / free_one_point); skip dead targets. + if (d->target_ && d->target_->prop) { + int type = d->target_->prop->_type; + // Dual-write: INITIAL and HOC weight[] must share one value stream. + // SoA is HOC-primary; heap is the buffer for generated pnt_receive_init. + d->weights_soa_to_heap(); if (pnt_receive_init[type]) { (*pnt_receive_init[type])(d->target_, d->weight_, 0); } else { @@ -2802,6 +2949,7 @@ void NetCvode::init_events() { d->weight_[j] = 0.; } } + d->weights_heap_to_soa(); } } if (gcv_) { @@ -2979,7 +3127,20 @@ void NetCon::deliver(double tt, NetCvode* ns, NrnThread* nt) { // printf("NetCon::deliver t=%g tt=%g %s\n", t, tt, hoc_object_name(target_->ob)); STATISTICS(netcon_deliver_); - POINT_RECEIVE(type, target_, weight_, 0); + // Phase 4: deliver via weight_index; use long-lived weight_ heap as MOD + // buffer so net_send(..., _w, ...) keeps a stable SelfEvent identity. + // SoA remains HOC-primary via materialize around pnt_receive. + int widx = _soa.weight_index(); + if (widx < 0 && !weight_soa_.empty()) { + widx = static_cast(weight_soa_.front().current_row()); + } + if (widx >= 0) { + nrn_pnt_receive_by_weight_index(target_, widx, 0., weight_); + } else { + weights_soa_to_heap(); + POINT_RECEIVE(type, target_, weight_, 0); + weights_heap_to_soa(); + } if (errno) { if (nrn_errno_check(type)) { hoc_warning("errno set during NetCon deliver to NET_RECEIVE", (char*) 0); @@ -2995,7 +3156,17 @@ void NetCon::pgvts_deliver(double tt, NetCvode* ns) { assert(target_); int type = target_->prop->_type; STATISTICS(netcon_deliver_); - POINT_RECEIVE(type, target_, weight_, 0); + int widx = _soa.weight_index(); + if (widx < 0 && !weight_soa_.empty()) { + widx = static_cast(weight_soa_.front().current_row()); + } + if (widx >= 0) { + nrn_pnt_receive_by_weight_index(target_, widx, 0., weight_); + } else { + weights_soa_to_heap(); + POINT_RECEIVE(type, target_, weight_, 0); + weights_heap_to_soa(); + } if (errno) { if (nrn_errno_check(type)) { hoc_warning("errno set during NetCon deliver to NET_RECEIVE", (char*) 0); @@ -3013,6 +3184,88 @@ void NetCon::pr(const char* s, double tt, NetCvode* ns) { Printf(" target=%s %.15g\n", (target_ ? hoc_object_name(target_->ob) : "nullptr"), tt); } +namespace { +// Global CoreNEURON-shaped fanout: NetCon* ranges described by PreSyn NcIndex/NcCount. +std::vector g_network_fanout_order; +bool g_network_fanout_sorted = true; + +template +void for_each_fanout_netcon(PreSyn* ps, F&& fn) { + PreSyn::ensure_fanout_order(); + int const base = ps->_soa.nc_index(); + int const cnt = ps->_soa.nc_count(); + if (base >= 0 && cnt == static_cast(ps->dil_.size()) && + base + cnt <= static_cast(g_network_fanout_order.size())) { + for (int i = 0; i < cnt; ++i) { + fn(g_network_fanout_order[static_cast(base + i)]); + } + } else { + // Fallback while topology is mid-update or PreSyn not in psl_. + for (NetCon* d: ps->dil_) { + fn(d); + } + } +} +} // namespace + +void PreSyn::mark_fanout_unsorted() { + g_network_fanout_sorted = false; +} + +void PreSyn::ensure_fanout_order() { + if (g_network_fanout_sorted) { + return; + } + g_network_fanout_order.clear(); + if (net_cvode_instance && net_cvode_instance->psl_) { + for (PreSyn* ps: *net_cvode_instance->psl_) { + // Refresh dual-write scalars (threshold may have been HOC-steered via double*). + ps->_soa.threshold() = ps->threshold_; + ps->_soa.gid() = ps->gid_; + ps->_soa.output_index() = ps->output_index_; + ps->_soa.thread_id() = ps->nt_ ? ps->nt_->id : -1; + if (ps->thvar_ && ps->thvar_.refers_to_a_modern_data_structure()) { + try { + ps->_soa.thvar_row() = static_cast(ps->thvar_.current_row()); + } catch (...) { + ps->_soa.thvar_row() = -1; + } + } else { + ps->_soa.thvar_row() = -1; + } + ps->_soa.nc_index() = static_cast(g_network_fanout_order.size()); + for (NetCon* nc: ps->dil_) { + g_network_fanout_order.push_back(nc); + // Keep NetCon reverse edge / delay dual-write current before sim. + nc->soa_sync(); + } + ps->_soa.nc_count() = static_cast(ps->dil_.size()); + } + } + g_network_fanout_sorted = true; +} + +void PreSyn::soa_sync() { + _soa.threshold() = threshold_; + _soa.gid() = gid_; + _soa.output_index() = output_index_; + _soa.thread_id() = nt_ ? nt_->id : -1; + _soa.nc_count() = static_cast(dil_.size()); + // NcIndex is owned by ensure_fanout_order when sorted. + if (!g_network_fanout_sorted) { + _soa.nc_index() = -1; + } + if (thvar_ && thvar_.refers_to_a_modern_data_structure()) { + try { + _soa.thvar_row() = static_cast(thvar_.current_row()); + } catch (...) { + _soa.thvar_row() = -1; + } + } else { + _soa.thvar_row() = -1; + } +} + void PreSyn::send(double tt, NetCvode* ns, NrnThread* nt) { int i; record(tt); @@ -3028,7 +3281,8 @@ void PreSyn::send(double tt, NetCvode* ns, NrnThread* nt) { } } else { STATISTICS(presyn_send_direct_); - for (const auto& d: dil_) { + // Phase 3: fanout via NcIndex/NcCount into global order (dual-write with dil_). + for_each_fanout_netcon(this, [&](NetCon* d) { if (d->active_ && d->target_) { NrnThread* n = PP2NT(d->target_); if (nt == n) { @@ -3037,7 +3291,7 @@ void PreSyn::send(double tt, NetCvode* ns, NrnThread* nt) { ns->p[n->id].interthread_send(tt + d->delay_, d, n); } } - } + }); } #endif // ndef USENCS #if USENCS || NRNMPI @@ -3091,7 +3345,7 @@ void PreSyn::deliver(double tt, NetCvode* ns, NrnThread* nt) { } // the thread is the one that owns the targets STATISTICS(presyn_deliver_netcon_); - for (const auto& d: dil_) { + for_each_fanout_netcon(this, [&](NetCon* d) { if (d->active_ && d->target_ && PP2NT(d->target_) == nt) { double dtt = d->delay_ - delay_; if (dtt == 0.) { @@ -3105,19 +3359,19 @@ void PreSyn::deliver(double tt, NetCvode* ns, NrnThread* nt) { ns->event(tt + dtt, d, nt); } } - } + }); } // used by bbsavestate since during restore, some NetCon spikes may // have already been delivered while others need to be delivered in // the future. Not implemented fof qthresh_ case. No statistics. void PreSyn::fanout(double td, NetCvode* ns, NrnThread* nt) { - for (const auto& d: dil_) { + for_each_fanout_netcon(this, [&](NetCon* d) { if (d->active_ && d->target_ && PP2NT(d->target_) == nt) { double dtt = d->delay_ - delay_; ns->bin_event(td + dtt, d, nt); } - } + }); } NrnThread* PreSyn::thread() { @@ -3135,7 +3389,7 @@ void PreSyn::pgvts_deliver(double tt, NetCvode* ns) { return; } STATISTICS(presyn_deliver_netcon_); - for (const auto& d: dil_) { + for_each_fanout_netcon(this, [&](NetCon* d) { if (d->active_ && d->target_) { double dtt = d->delay_ - delay_; if (dtt < 0.) { @@ -3145,7 +3399,7 @@ void PreSyn::pgvts_deliver(double tt, NetCvode* ns) { ns->event(tt + dtt, d, nt); } } - } + }); } void PreSyn::pr(const char* s, double tt, NetCvode* ns) { @@ -3163,13 +3417,24 @@ DiscreteEvent* SelfEvent::savestate_save() { se->flag_ = flag_; se->target_ = target_; se->weight_ = weight_; + se->weight_index_ = weight_index_; + se->target_row_ = target_row_; se->movable_ = movable_; return se; } void SelfEvent::savestate_restore(double tt, NetCvode* nc) { // pr("savestate_restore", tt, nc); - nrn_net_send(movable_, weight_, target_, tt, flag_); + // Prefer long-lived NetCon heap base when known; if only weight_index_ survived + // (heap-drop / index identity), rebind via NetConSave::weight_index2netcon. + double* w = weight_; + if (!w && weight_index_ >= 0) { + if (NetCon* owner = NetConSave::weight_index2netcon(weight_index_)) { + w = owner->weight_; + weight_ = w; + } + } + nrn_net_send(movable_, w, target_, tt, flag_); } DiscreteEvent* SelfEvent::savestate_read(FILE* f) { @@ -3184,9 +3449,18 @@ DiscreteEvent* SelfEvent::savestate_read(FILE* f) { 6); se->target_ = SelfEvent::index2pp(pptype, ppindex); se->weight_ = nullptr; + se->weight_index_ = -1; + se->target_row_ = nrn_point_process_soa_row(se->target_); if (ncindex >= 0) { + // File identity is NetCon object index (not weight_ pointer). NetCon* nc = NetConSave::index2netcon(ncindex); + assert(nc); se->weight_ = nc->weight_; + if (!nc->weight_soa_.empty()) { + se->weight_index_ = static_cast(nc->weight_soa_.front().current_row()); + } else { + se->weight_index_ = nc->_soa.weight_index(); + } } se->flag_ = flag; se->movable_ = (moff >= 0) ? (se->target_->prop->dparam + moff) : nullptr; @@ -3225,11 +3499,22 @@ void SelfEvent::savestate_write(FILE* f) { fprintf(f, "%d\n", SelfEventType); int const moff = movable_ ? (movable_ - target_->prop->dparam) : -1; int ncindex = -1; - // find the NetCon index for weight_ + // SelfEvent identity for SaveState: NetCon object index (stable), not weight_*. + // Prefer heap map; fall back to weight_index. Do not assert if unresolved — + // flag-only / ignored events use ncindex = -1 (BBSaveState uses -2 for ignore). + NetCon* owner = nullptr; if (weight_) { - NetCon* nc = NetConSave::weight2netcon(weight_); - assert(nc); - ncindex = nc->obj_->index; + owner = NetConSave::weight2netcon(weight_); + } + if (!owner && weight_index_ >= 0) { + owner = NetConSave::weight_index2netcon(weight_index_); + } + if (owner && owner->obj_) { + ncindex = owner->obj_->index; + // Rebind to long-lived heap if we resolved via index only. + if (!weight_ && owner->weight_) { + weight_ = owner->weight_; + } } fprintf(f, @@ -3280,7 +3565,10 @@ void SelfEvent::pgvts_deliver(double tt, NetCvode* ns) { } void SelfEvent::call_net_receive(NetCvode* ns) { STATISTICS(selfevent_deliver_); - POINT_RECEIVE(target_->prop->_type, target_, weight_, flag_); + // Always use dual-write receive path: even when weight_ is nullptr (e.g. + // mech INITIAL net_send without a NetCon weight), FOR_NETCONS types still + // mutate peer NetCon heaps and must sync back to Weight SoA. + nrn_pnt_receive_by_weight_index(target_, weight_index_, flag_, weight_); if (errno) { if (nrn_errno_check(target_->prop->_type)) { hoc_warning("errno set during SelfEvent deliver to NET_RECEIVE", (char*) 0); @@ -4617,6 +4905,13 @@ NetCon* NetCvode::install_deliver(neuron::container::data_handle dsrc, NetCon* d = new NetCon(ps, target); d->delay_ = delay; d->weight_[0] = magnitude; + // Dual-write: constructor arg is written to heap; HOC weight[] reads SoA. + // INITIAL (init_events) does soa_to_heap then heap_to_soa — SoA must already + // hold magnitude or weight[0] is wiped to 0. + if (!d->weight_soa_.empty()) { + d->weight_soa_[0].value() = magnitude; + } + d->soa_sync(); structure_change_cnt_ = 0; return d; } @@ -4699,6 +4994,48 @@ void DiscreteEvent::savestate_write(FILE* f) { fprintf(f, "%d\n", DiscreteEventType); } +void NetCon::weights_soa_to_heap() { + if (!weight_ || weight_soa_.empty()) { + return; + } + auto const n = std::min(cnt_, static_cast(weight_soa_.size())); + for (int i = 0; i < n; ++i) { + weight_[i] = weight_soa_[i].value(); + } +} + +void NetCon::weights_heap_to_soa() { + if (!weight_ || weight_soa_.empty()) { + return; + } + neuron::container::network::Weight::mirror_weights_to_soa(weight_soa_, weight_, cnt_); +} + +void NetCon::soa_sync() { + _soa.delay() = delay_; + _soa.active() = active_ ? 1 : 0; + _soa.weight_count() = cnt_; + if (target_) { + // Point_process dual-write row (Phase 1). + _soa.target() = nrn_point_process_soa_row(target_); + } else { + _soa.target() = -1; + } + // Phase 3: reverse edge into PreSyn SoA. + if (src_) { + _soa.src_presyn() = static_cast(src_->_soa.current_row()); + } else { + _soa.src_presyn() = -1; + } + if (!weight_soa_.empty()) { + _soa.weight_index() = static_cast(weight_soa_.front().current_row()); + // Do not mirror heap → SoA here: HOC weight() writes SoA first; heap is only + // a pnt_receive buffer. heap→SoA after MOD runs (weights_heap_to_soa). + } else { + _soa.weight_index() = -1; + } +} + NetCon::NetCon(PreSyn* src, Object* target) { NetConSave::invalid(); obj_ = nullptr; @@ -4707,6 +5044,7 @@ NetCon::NetCon(PreSyn* src, Object* target) { if (src_) { src_->dil_.push_back(this); src_->use_min_delay_ = 0; + PreSyn::mark_fanout_unsorted(); } if (target == nullptr) { target_ = nullptr; @@ -4714,6 +5052,8 @@ NetCon::NetCon(PreSyn* src, Object* target) { cnt_ = 1; weight_ = new double[cnt_]; weight_[0] = 0.0; + weight_soa_ = neuron::container::network::Weight::allocate_weight_rows(cnt_, weight_); + soa_sync(); return; } target_ = ob2pntproc(target); @@ -4731,7 +5071,9 @@ NetCon::NetCon(PreSyn* src, Object* target) { for (int i = 0; i < cnt_; ++i) { weight_[i] = 0.0; } + weight_soa_ = neuron::container::network::Weight::allocate_weight_rows(cnt_, weight_); } + soa_sync(); } NetCon::~NetCon() { @@ -4741,6 +5083,8 @@ NetCon::~NetCon() { if (cnt_) { delete[] weight_; } + weight_soa_.clear(); + // _soa owning_handle frees the NetCon SoA row. #if DISCRETE_EVENT_OBSERVER if (target_) { ObjObservable::Detach(target_->ob, this); @@ -4753,6 +5097,7 @@ void NetCon::rmsrc() { for (size_t i = 0; i < src_->dil_.size(); ++i) { if (src_->dil_[i] == this) { src_->dil_.erase(src_->dil_.begin() + i); + PreSyn::mark_fanout_unsorted(); if (src_->dil_.size() == 0 && src_->tvec_ == NULL && src_->idvec_ == NULL) { if (src_->output_index_ == -1) { delete std::exchange(src_, nullptr); @@ -4763,6 +5108,7 @@ void NetCon::rmsrc() { } } src_ = nullptr; + soa_sync(); } void NetCon::replace_src(PreSyn* p) { @@ -4771,7 +5117,9 @@ void NetCon::replace_src(PreSyn* p) { if (src_) { src_->dil_.push_back(this); src_->use_min_delay_ = 0; + PreSyn::mark_fanout_unsorted(); } + soa_sync(); } DiscreteEvent* NetCon::savestate_save() { @@ -4853,7 +5201,9 @@ NetCon* NetConSave::index2netcon(long id) { ITERATE(q, sym->u.ctemplate->olist) { Object* obj = OBJ(q); nc = (NetCon*) obj->u.this_pointer; - if (nc->weight_) { + // Index all NetCons by HOC object index (identity for SaveState), + // not only those with a weight_ heap (heap-drop readiness). + if (nc) { (*idxtable_)[obj->index] = nc; } } @@ -4868,6 +5218,33 @@ NetCon* NetConSave::index2netcon(long id) { } } +NetCon* NetConSave::weight_index2netcon(int weight_index) { + if (weight_index < 0) { + return nullptr; + } + Symbol* sym = hoc_lookup("NetCon"); + if (!sym || !sym->u.ctemplate || !sym->u.ctemplate->olist) { + return nullptr; + } + hoc_Item* q = nullptr; + ITERATE(q, sym->u.ctemplate->olist) { + auto* nc = static_cast(OBJ(q)->u.this_pointer); + if (!nc) { + continue; + } + int base = -1; + if (!nc->weight_soa_.empty()) { + base = static_cast(nc->weight_soa_.front().current_row()); + } else { + base = nc->_soa.weight_index(); + } + if (base == weight_index) { + return nc; + } + } + return nullptr; +} + void nrn_update_ps2nt() { net_cvode_instance->update_ps2nt(); } @@ -4885,6 +5262,7 @@ void NetCvode::ps_thread_link(PreSyn* ps) { } } if (!ps->nt_) { // premature, reorder_secorder() not called yet + ps->soa_sync(); return; } if (ps->thvar_) { @@ -4894,6 +5272,7 @@ void NetCvode::ps_thread_link(PreSyn* ps) { } ps->hi_th_ = hoc_l_insertvoid(p[i].psl_thr_, ps); } + ps->soa_sync(); } void NetCvode::update_ps2nt() { @@ -4970,10 +5349,13 @@ PreSyn::PreSyn(neuron::container::data_handle src, Object* osrc, Section nrn_notify_when_void_freed(osrc_, this); } #endif + soa_sync(); + mark_fanout_unsorted(); } PreSyn::~PreSyn() { PreSynSave::invalid(); + mark_fanout_unsorted(); // printf("~PreSyn %p\n", this); nrn_cleanup_presyn(this); delete std::exchange(stmt_, nullptr); @@ -5342,7 +5724,15 @@ void WatchCondition::deliver(double tt, NetCvode* ns, NrnThread* nt) { PP2t(pnt_) = tt; } STATISTICS(watch_deliver_); + // WATCH-driven NET_RECEIVE (e.g. flag=2) may run FOR_NETCONS, which mutates + // every NetCon weight_ heap sharing this target — keep SoA dual-write coherent. + if (type_has_fornetcon(type)) { + sync_netcon_weights_for_target(pnt_, /*soa_to_heap*/ true); + } POINT_RECEIVE(type, pnt_, nullptr, nrflag_); + if (type_has_fornetcon(type)) { + sync_netcon_weights_for_target(pnt_, /*soa_to_heap*/ false); + } if (errno) { if (nrn_errno_check(type)) { hoc_warning("errno set during WatchCondition deliver to NET_RECEIVE", (char*) 0); @@ -5433,7 +5823,13 @@ void WatchCondition::pgvts_deliver(double tt, NetCvode* ns) { } int type = pnt_->prop->_type; STATISTICS(watch_deliver_); + if (type_has_fornetcon(type)) { + sync_netcon_weights_for_target(pnt_, /*soa_to_heap*/ true); + } POINT_RECEIVE(type, pnt_, nullptr, nrflag_); + if (type_has_fornetcon(type)) { + sync_netcon_weights_for_target(pnt_, /*soa_to_heap*/ false); + } if (errno) { if (nrn_errno_check(type)) { hoc_warning("errno set during WatchCondition deliver to NET_RECEIVE", (char*) 0); diff --git a/src/nrncvode/network_soa_sort.cpp b/src/nrncvode/network_soa_sort.cpp new file mode 100644 index 0000000000..5d3610fc4e --- /dev/null +++ b/src/nrncvode/network_soa_sort.cpp @@ -0,0 +1,361 @@ +/** + * @file network_soa_sort.cpp + * @brief Network SoA sort / weight repack for nrn_ensure_model_data_are_sorted. + */ +#include "neuron/container/network/point_process_access.hpp" +#include "neuron/container/network/sort.hpp" +#include "netcon.h" +#include "netcvode.h" +#include "nrniv_mf.h" +#include "section.h" + +#include +#include +#include +#include + +extern cTemplate** nrn_pnt_template_; +extern int n_memb_func; +extern NetCvode* net_cvode_instance; + +namespace neuron::container::network { +namespace { + +constexpr std::size_t npos = std::numeric_limits::max(); + +/** @brief Sync all Point_process dual-write rows from HOC point-process lists. */ +void sync_all_point_processes() { + if (!nrn_pnt_template_) { + return; + } + for (int type = 0; type < n_memb_func; ++type) { + cTemplate* tmp = nrn_pnt_template_[type]; + if (!tmp || !tmp->olist) { + continue; + } + hoc_Item* q = nullptr; + ITERATE(q, tmp->olist) { + // HOC point-process templates store Point_process* in dataspace + // (not this_pointer); use ob2pntproc_0 for both C and HOC shells. + Point_process* pnt = ob2pntproc_0(OBJ(q)); + if (pnt) { + nrn_point_process_soa_sync(pnt); + } + } + } +} + +/** @brief Enumerate live NetCon shells from the HOC template list. */ +std::vector<::NetCon*> all_netcons() { + std::vector<::NetCon*> out; + Symbol* sym = hoc_lookup("NetCon"); + if (!sym || !sym->u.ctemplate || !sym->u.ctemplate->olist) { + return out; + } + out.reserve(static_cast(sym->u.ctemplate->count)); + hoc_Item* q = nullptr; + ITERATE(q, sym->u.ctemplate->olist) { + auto* nc = static_cast<::NetCon*>(OBJ(q)->u.this_pointer); + if (nc) { + out.push_back(nc); + } + } + return out; +} + +/** @brief Enumerate live PreSyn shells (psl_ when available). */ +std::vector<::PreSyn*> all_presyns() { + std::vector<::PreSyn*> out; + if (net_cvode_instance && net_cvode_instance->psl_) { + out.reserve(net_cvode_instance->psl_->size()); + for (::PreSyn* ps: *net_cvode_instance->psl_) { + if (ps) { + out.push_back(ps); + } + } + } + return out; +} + +int target_thread_id(::NetCon* nc) { + if (!nc || !nc->target_) { + return -1; + } + if (nc->target_->_vnt) { + return static_cast(nc->target_->_vnt)->id; + } + // Fall back to dual-write column if present. + int const row = nrn_point_process_soa_row(nc->target_); + if (row >= 0) { + return neuron::model().point_processes().get( + static_cast(row)); + } + return -1; +} + +int src_presyn_row(::NetCon* nc) { + if (!nc || !nc->src_) { + return -1; + } + return static_cast(nc->src_->_soa.current_row()); +} + +/** + * @brief Build perm[old_row] = new_row grouping rows by integer key in [0, nkey). + * @return Number of rows placed in the keyed region (orphans follow). + */ +template +std::vector partition_by_key(std::size_t n, + int nkey, + GetKey&& get_key, + std::vector& key_offsets) { + key_offsets.assign(static_cast(std::max(nkey, 0)), 0); + std::vector perm(n, npos); + std::size_t global_i = 0; + for (int k = 0; k < nkey; ++k) { + key_offsets[static_cast(k)] = global_i; + for (std::size_t row = 0; row < n; ++row) { + if (get_key(row) == k) { + perm[row] = global_i++; + } + } + } + for (std::size_t row = 0; row < n; ++row) { + if (perm[row] == npos) { + perm[row] = global_i++; + } + } + assert(global_i == n); + return perm; +} + +template +void apply_if_needed(Storage& store, Token& token, std::vector perm) { + if (store.size() == 0) { + store.mark_as_sorted(token); + return; + } + // apply_reverse_permutation always leaves the container sorted. + store.apply_reverse_permutation(std::move(perm), token); +} + +} // namespace + +void sort_network_data(neuron::cache::Model& cache, + PointProcess::storage::frozen_token_type& pp_token, + Weight::storage::frozen_token_type& w_token, + NetCon::storage::frozen_token_type& nc_token, + PreSyn::storage::frozen_token_type& ps_token) { + auto& model = neuron::model(); + auto& pp_store = model.point_processes(); + auto& w_store = model.weights(); + auto& nc_store = model.netcons(); + auto& ps_store = model.presyns(); + + // Ensure cache.thread is sized (caller should have done this; be defensive). + if (cache.thread.size() < static_cast(nrn_nthread)) { + cache.thread.resize(static_cast(nrn_nthread)); + } + + // ------------------------------------------------------------------ + // 0. Dual-write sync from legacy shells (values / thread membership). + // ------------------------------------------------------------------ + sync_all_point_processes(); + auto netcons = all_netcons(); + for (::NetCon* nc: netcons) { + // Keep WeightIndex from handles; do not heap→SoA wipe HOC-primary values. + nc->soa_sync(); + } + auto presyns = all_presyns(); + for (::PreSyn* ps: presyns) { + ps->soa_sync(); + } + + // ------------------------------------------------------------------ + // 1. PointProcess: partition by thread_id. + // ------------------------------------------------------------------ + { + std::vector pp_offsets; + auto perm = partition_by_key( + pp_store.size(), + nrn_nthread, + [&](std::size_t row) { return pp_store.get(row); }, + pp_offsets); + for (int tid = 0; tid < nrn_nthread; ++tid) { + cache.thread[static_cast(tid)].point_process_offset = + tid < static_cast(pp_offsets.size()) + ? pp_offsets[static_cast(tid)] + : 0; + } + apply_if_needed(pp_store, pp_token, std::move(perm)); + } + + // Order NetCons by (target thread, src PreSyn SoA row, NetCon SoA row). + struct NcSortKey { + ::NetCon* nc{}; + int tid{-1}; + int src_row{-1}; + std::size_t soa_row{0}; + }; + std::vector ordered; + ordered.reserve(netcons.size()); + for (::NetCon* nc: netcons) { + NcSortKey k; + k.nc = nc; + k.tid = target_thread_id(nc); + k.src_row = src_presyn_row(nc); + k.soa_row = nc->_soa.current_row(); + ordered.push_back(k); + } + std::stable_sort(ordered.begin(), ordered.end(), [](NcSortKey const& a, NcSortKey const& b) { + if (a.tid != b.tid) { + // Unassigned (-1) after real threads. + int const at = a.tid < 0 ? nrn_nthread : a.tid; + int const bt = b.tid < 0 ? nrn_nthread : b.tid; + if (at != bt) { + return at < bt; + } + } + if (a.src_row != b.src_row) { + return a.src_row < b.src_row; + } + return a.soa_row < b.soa_row; + }); + + // ------------------------------------------------------------------ + // 2. Weight repack: contiguous blocks per NetCon, packed by target thread. + // ------------------------------------------------------------------ + { + std::size_t const wsize = w_store.size(); + std::vector desired; // old rows in new order + desired.reserve(wsize); + std::vector used(wsize, 0); + std::vector first_pos(static_cast(std::max(nrn_nthread, 0)), + npos); + + for (auto const& k: ordered) { + if (k.tid >= 0 && k.tid < nrn_nthread) { + auto& fp = first_pos[static_cast(k.tid)]; + if (fp == npos) { + fp = desired.size(); + } + } + for (auto& wh: k.nc->weight_soa_) { + if (!wh.id()) { + continue; + } + auto const row = wh.current_row(); + if (row < wsize && !used[row]) { + used[row] = 1; + desired.push_back(row); + } + } + } + // Orphan weight rows (not owned by any live NetCon) go at the end. + for (std::size_t row = 0; row < wsize; ++row) { + if (!used[row]) { + desired.push_back(row); + } + } + assert(desired.size() == wsize); + + std::size_t running = 0; + for (int tid = 0; tid < nrn_nthread; ++tid) { + auto const fp = first_pos[static_cast(tid)]; + if (fp != npos) { + running = fp; + } + cache.thread[static_cast(tid)].weight_offset = running; + } + + std::vector perm(wsize, npos); + for (std::size_t new_i = 0; new_i < desired.size(); ++new_i) { + perm[desired[new_i]] = new_i; + } + apply_if_needed(w_store, w_token, std::move(perm)); + } + + // ------------------------------------------------------------------ + // 3. NetCon SoA: same order as weight packing (target thread, src, …). + // ------------------------------------------------------------------ + { + std::size_t const n = nc_store.size(); + std::vector perm(n, npos); + std::vector first_pos(static_cast(nrn_nthread), npos); + std::size_t global_i = 0; + std::vector seen(n, 0); + for (auto const& k: ordered) { + auto const old = k.soa_row; + if (old < n && !seen[old]) { + if (k.tid >= 0 && k.tid < nrn_nthread) { + auto& fp = first_pos[static_cast(k.tid)]; + if (fp == npos) { + fp = global_i; + } + } + perm[old] = global_i++; + seen[old] = 1; + } + } + for (std::size_t row = 0; row < n; ++row) { + if (perm[row] == npos) { + perm[row] = global_i++; + } + } + assert(global_i == n); + std::size_t running = 0; + for (int tid = 0; tid < nrn_nthread; ++tid) { + auto const fp = first_pos[static_cast(tid)]; + if (fp != npos) { + running = fp; + } + cache.thread[static_cast(tid)].netcon_offset = running; + } + apply_if_needed(nc_store, nc_token, std::move(perm)); + } + + // Refresh dual-write indices after PP/weight/NetCon permutes. + for (::NetCon* nc: netcons) { + nc->soa_sync(); + } + + // ------------------------------------------------------------------ + // 4. PreSyn by thread + fanout ranges (NcIndex/NcCount). + // ------------------------------------------------------------------ + { + // Refresh thread_id on PreSyn shells before partition. + for (::PreSyn* ps: presyns) { + ps->soa_sync(); + } + std::vector ps_offsets; + auto perm = partition_by_key( + ps_store.size(), + nrn_nthread, + [&](std::size_t row) { return ps_store.get(row); }, + ps_offsets); + for (int tid = 0; tid < nrn_nthread; ++tid) { + cache.thread[static_cast(tid)].presyn_offset = + tid < static_cast(ps_offsets.size()) + ? ps_offsets[static_cast(tid)] + : 0; + } + apply_if_needed(ps_store, ps_token, std::move(perm)); + } + + // Fanout order uses NetCon* ranges; rebuild after topology/sort. + ::PreSyn::mark_fanout_unsorted(); + ::PreSyn::ensure_fanout_order(); + + // Final NetCon reverse-edge refresh (src PreSyn rows may have moved). + for (::NetCon* nc: netcons) { + nc->soa_sync(); + } + + // Ensure all four are marked sorted even if empty / trivial perm paths. + pp_store.mark_as_sorted(pp_token); + w_store.mark_as_sorted(w_token); + nc_store.mark_as_sorted(nc_token); + ps_store.mark_as_sorted(ps_token); +} + +} // namespace neuron::container::network diff --git a/src/nrniv/bbsavestate.cpp b/src/nrniv/bbsavestate.cpp index 81ded04c7e..f2dd5f7d2d 100644 --- a/src/nrniv/bbsavestate.cpp +++ b/src/nrniv/bbsavestate.cpp @@ -1107,13 +1107,57 @@ class SEWrap: public DiscreteEvent { double tt; int ncindex; // in the DEList or -1 if no NetCon for self event. }; + +/** True if SelfEvent is bound to this NetCon (heap base and/or Weight SoA index). */ +static bool selfevent_matches_netcon(SelfEvent const* se, NetCon const* nc) { + if (!se || !nc) { + return false; + } + // Legacy: long-lived heap base pointer identity. + if (se->weight_ && nc->weight_ && se->weight_ == nc->weight_) { + return true; + } + // Dual-write / heap-drop: Weight SoA base row (weight_index). + int se_widx = se->weight_index_; + if (se_widx < 0) { + return false; + } + int nc_widx = -1; + if (!nc->weight_soa_.empty()) { + nc_widx = static_cast(nc->weight_soa_.front().current_row()); + } else { + nc_widx = nc->_soa.weight_index(); + } + return nc_widx == se_widx; +} + +/** Bind SelfEvent weight_ / weight_index_ from a NetCon (after BBSaveState restore). */ +static void selfevent_bind_netcon(SelfEvent* se, NetCon* nc) { + if (!se) { + return; + } + if (!nc) { + se->weight_ = nullptr; + se->weight_index_ = -1; + return; + } + se->weight_ = nc->weight_; + if (!nc->weight_soa_.empty()) { + se->weight_index_ = static_cast(nc->weight_soa_.front().current_row()); + } else { + se->weight_index_ = nc->_soa.weight_index(); + } +} + SEWrap::SEWrap(const TQItem* tq, DEList* dl) { tt = tq->t_; se = (SelfEvent*) tq->data_; - if (se->weight_) { + // SelfEvent identity for BBSaveState: index into target's NetCon DEList + // (same policy as SaveState NetCon object index), not weight_* alone. + if (se->weight_ || se->weight_index_ >= 0) { ncindex = 0; for (; dl && dl->de && dl->de->type() == NetConType; dl = dl->next, ++ncindex) { - if (se->weight_ == ((NetCon*) dl->de)->weight_) { + if (selfevent_matches_netcon(se, static_cast(dl->de))) { return; } } @@ -2201,7 +2245,18 @@ void BBSaveState::netrecv_pp(Point_process* pp) { f->s(buf, 1); for (; dl && dl->de->type() == NetConType; dl = dl->next) { NetCon* nc = (NetCon*) dl->de; + // Dual-write: HOC weight[] is SoA-primary. Materialize SoA → heap before + // OUT/CNT so f->d sees current values; after IN, mirror heap → SoA. + if (f->type() != BBSS_IO::IN) { + nc->weights_soa_to_heap(); + } f->d(nc->cnt_, nc->weight_); + if (f->type() == BBSS_IO::IN) { + nc->weights_heap_to_soa(); + if (!nc->weight_soa_.empty()) { + nc->soa_sync(); + } + } if (f->type() != BBSS_IO::IN) { // writing, counting DblList* db = 0; int j = 0; @@ -2277,7 +2332,7 @@ void BBSaveState::netrecv_pp(Point_process* pp) { // since the queue has been cleared. for (int i = 0; i < cnt; ++i) { int ncindex, moff; - double flag, tt, *w; + double flag, tt; f->s(buf); f->d(1, flag); f->d(1, tt); @@ -2302,16 +2357,16 @@ void BBSaveState::netrecv_pp(Point_process* pp) { se->movable_ = movable; } if (ncindex == -1) { - w = NULL; + selfevent_bind_netcon(se, nullptr); } else { int j; for (j = 0, dl1 = dliter->second; j < ncindex; ++j, dl1 = dl1->next) { ; } assert(dl1 && dl1->de->type() == NetConType); - w = ((NetCon*) dl1->de)->weight_; + // ncindex is position in target DEList (file identity), not weight_*. + selfevent_bind_netcon(se, static_cast(dl1->de)); } - se->weight_ = w; } } if (debug) { diff --git a/src/nrniv/memory_usage.cpp b/src/nrniv/memory_usage.cpp index 49361bbd67..cfef3abc88 100644 --- a/src/nrniv/memory_usage.cpp +++ b/src/nrniv/memory_usage.cpp @@ -13,7 +13,12 @@ ModelMemoryUsage memory_usage(const Model& model) { auto mechanisms = StorageMemoryUsage(); model.apply_to_mechanisms([&mechanisms](const auto& md) { mechanisms += memory_usage(md); }); - return {nodes, mechanisms}; + auto point_processes = memory_usage(model.point_processes()); + auto weights = memory_usage(model.weights()); + auto netcons = memory_usage(model.netcons()); + auto presyns = memory_usage(model.presyns()); + + return {nodes, mechanisms, point_processes, weights, netcons, presyns}; } cache::ModelMemoryUsage memory_usage(const std::optional& model) { @@ -117,6 +122,23 @@ std::string format_memory_usage(const MemoryUsage& usage) { os << " data " << format_memory_usage(model.mechanisms.heavy_data) << "\n"; os << " stable_identifiers " << format_memory_usage(model.mechanisms.stable_identifiers) << "\n"; + os << " network::PointProcess \n"; + os << " data " << format_memory_usage(model.point_processes.heavy_data) + << "\n"; + os << " stable_identifiers " + << format_memory_usage(model.point_processes.stable_identifiers) << "\n"; + os << " network::Weight \n"; + os << " data " << format_memory_usage(model.weights.heavy_data) << "\n"; + os << " stable_identifiers " << format_memory_usage(model.weights.stable_identifiers) + << "\n"; + os << " network::NetCon \n"; + os << " data " << format_memory_usage(model.netcons.heavy_data) << "\n"; + os << " stable_identifiers " << format_memory_usage(model.netcons.stable_identifiers) + << "\n"; + os << " network::PreSyn \n"; + os << " data " << format_memory_usage(model.presyns.heavy_data) << "\n"; + os << " stable_identifiers " << format_memory_usage(model.presyns.stable_identifiers) + << "\n"; os << "cache::Model \n"; os << " threads " << format_memory_usage(cache_model.threads) << "\n"; os << " mechanisms " << format_memory_usage(cache_model.mechanisms) << "\n"; diff --git a/src/nrniv/netpar.cpp b/src/nrniv/netpar.cpp index b274d641b6..54048abb51 100644 --- a/src/nrniv/netpar.cpp +++ b/src/nrniv/netpar.cpp @@ -1099,6 +1099,7 @@ void BBS::cell() { } else { ps->output_index_ = gid; } + ps->soa_sync(); } void BBS::outputcell(int gid) { @@ -1108,6 +1109,7 @@ void BBS::outputcell(int gid) { assert(ps); ps->output_index_ = gid; ps->gid_ = gid; + ps->soa_sync(); } void BBS::spike_record(int gid, IvocVect* spikevec, IvocVect* gidvec) { diff --git a/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp b/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp index 169ef8ba1f..64e10d02ef 100644 --- a/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp +++ b/src/nrniv/nrncore_write/callbacks/nrncore_callbacks.cpp @@ -143,6 +143,8 @@ void nrnthreads_all_weights_return(std::vector& weights) { for (int i = 0; i < nc->cnt_; ++i) { nc->weight_[i] = weights[ith][iw[ith]++]; } + // Dual-write: HOC weight[] is SoA-primary; keep Weight SoA in sync. + nc->weights_heap_to_soa(); } } @@ -508,6 +510,8 @@ int nrnthread_dat2_3(int tid, int iw = 0; for (int i = 0; i < n; ++i) { NetCon* nc = cg.netcons[i]; + // Dual-write: HOC may have written Weight SoA only; materialize heap. + nc->weights_soa_to_heap(); for (int j = 0; j < nc->cnt_; ++j) { weights[iw++] = nc->weight_[j]; } diff --git a/src/nrniv/savstate.cpp b/src/nrniv/savstate.cpp index bc6e9f2b6a..0967c77448 100644 --- a/src/nrniv/savstate.cpp +++ b/src/nrniv/savstate.cpp @@ -14,6 +14,8 @@ #include "vrecitem.h" #include "utils/enumerate.h" +#include + typedef void (*ReceiveFunc)(Point_process*, double*, double); #include "membfunc.h" @@ -929,8 +931,19 @@ void SaveState::savenet() { const NetCon* d = (NetCon*) ob->u.this_pointer; int n = ncs_[i].nstate; double* w = ncs_[i].state; - for (int j = 0; j < n; ++j) { - w[j] = d->weight_[j]; + // HOC weight[] is SoA-primary under dual-write; prefer SoA over heap. + if (!d->weight_soa_.empty()) { + int const m = std::min(n, static_cast(d->weight_soa_.size())); + for (int j = 0; j < m; ++j) { + w[j] = d->weight_soa_[j].value(); + } + for (int j = m; j < n; ++j) { + w[j] = d->weight_ ? d->weight_[j] : 0.; + } + } else if (d->weight_) { + for (int j = 0; j < n; ++j) { + w[j] = d->weight_[j]; + } } ++i; } @@ -974,8 +987,19 @@ void SaveState::restorenet() { NetCon* d = (NetCon*) ob->u.this_pointer; int n = ncs_[i].nstate; const double* w = ncs_[i].state; - for (int j = 0; j < n; ++j) { - d->weight_[j] = w[j]; + // Restore into heap buffer (nocmodl / FOR_NETCONS) and Weight SoA so HOC + // dual-write and materialize-on-deliver stay consistent. + if (d->weight_) { + for (int j = 0; j < n; ++j) { + d->weight_[j] = w[j]; + } + } + if (!d->weight_soa_.empty()) { + int const m = std::min(n, static_cast(d->weight_soa_.size())); + for (int j = 0; j < m; ++j) { + d->weight_soa_[j].value() = w[j]; + } + d->soa_sync(); // WeightIndex / reverse edges } ++i; } diff --git a/src/nrnoc/container.cpp b/src/nrnoc/container.cpp index a493a2f5b5..ab2d26e301 100644 --- a/src/nrnoc/container.cpp +++ b/src/nrnoc/container.cpp @@ -15,6 +15,10 @@ void invalidate_cache() { namespace neuron { Model::Model() { m_node_data.set_unsorted_callback(invalidate_cache); + m_point_processes.set_unsorted_callback(invalidate_cache); + m_weights.set_unsorted_callback(invalidate_cache); + m_netcons.set_unsorted_callback(invalidate_cache); + m_presyns.set_unsorted_callback(invalidate_cache); // needs some re-organisation if we ever want to support multiple Model instances assert(!container::detail::defer_delete_storage); container::detail::defer_delete_storage = &m_ptrs_for_deferred_deletion; @@ -38,6 +42,18 @@ std::unique_ptr Model::find_container_info(void return maybe_info; } } + if (auto maybe_info = m_point_processes.find_container_info(cont); maybe_info) { + return maybe_info; + } + if (auto maybe_info = m_weights.find_container_info(cont); maybe_info) { + return maybe_info; + } + if (auto maybe_info = m_netcons.find_container_info(cont); maybe_info) { + return maybe_info; + } + if (auto maybe_info = m_presyns.find_container_info(cont); maybe_info) { + return maybe_info; + } return {}; } @@ -143,6 +159,19 @@ generic_data_handle promote_or_clear(generic_data_handle gdh) { if (done) { return gdh; } + if (auto h = model.point_processes().find_data_handle(gdh); + h.refers_to_a_modern_data_structure()) { + return h; + } + if (auto h = model.weights().find_data_handle(gdh); h.refers_to_a_modern_data_structure()) { + return h; + } + if (auto h = model.netcons().find_data_handle(gdh); h.refers_to_a_modern_data_structure()) { + return h; + } + if (auto h = model.presyns().find_data_handle(gdh); h.refers_to_a_modern_data_structure()) { + return h; + } return {}; } } // namespace detail diff --git a/src/nrnoc/multicore.cpp b/src/nrnoc/multicore.cpp index 10286a6bc0..a71d144eff 100644 --- a/src/nrnoc/multicore.cpp +++ b/src/nrnoc/multicore.cpp @@ -616,11 +616,13 @@ printf("thread_memblist_setup %lx v_node_count=%d ncell=%d end=%d\n", (long)nth, for (int i = 0; i < tml->ml->nodecount; ++i) { auto* pnt = tml->ml->prop[i]->dparam[1].get(); pnt->_vnt = _nt; + nrn_point_process_soa_sync(pnt); } } else { for (int i = 0; i < tml->ml->nodecount; ++i) { auto* pnt = tml->ml->pdata[i][1].get(); pnt->_vnt = _nt; + nrn_point_process_soa_sync(pnt); } } } diff --git a/src/nrnoc/nrniv_mf.h b/src/nrnoc/nrniv_mf.h index 3d28a0cbec..b22b18b1b6 100644 --- a/src/nrnoc/nrniv_mf.h +++ b/src/nrnoc/nrniv_mf.h @@ -22,6 +22,18 @@ using ldifusfunc_t = void (*)(ldifusfunc2_t, neuron::model_sorted_token const&, typedef void (*pnt_receive_t)(Point_process*, double*, double); typedef void (*pnt_receive_init_t)(Point_process*, double*, double); +/** + * @brief Phase 4 dual-write: call pnt_receive after materializing Weight SoA → heap buffer. + * + * Generated MOD still uses (Point_process*, double*, double). weight_index is the + * base row in neuron::model().weights(); count is pnt_receive_size[type]. + * If weight_heap is non-null it is used as the buffer (and written back to SoA after). + */ +void nrn_pnt_receive_by_weight_index(Point_process* pnt, + int weight_index, + double flag, + double* weight_heap = nullptr); + extern Prop* need_memb_cl(Symbol*, int*, int*); extern Prop* prop_alloc(Prop**, int, Node*); void prop_update_ion_variables(Prop*, Node*); diff --git a/src/nrnoc/point.cpp b/src/nrnoc/point.cpp index b822181d5b..3eb29c4502 100644 --- a/src/nrnoc/point.cpp +++ b/src/nrnoc/point.cpp @@ -7,11 +7,13 @@ saves the pointtype as later argument to create and loc */ #include #include "membfunc.h" +#include "neuron/container/network/point_process_access.hpp" #include "nrniv_mf.h" #include "ocnotify.h" #include "parse_with_deps.hpp" #include "section.h" +#include extern char* pnt_map; extern Symbol** pointsym; /*list of variable symbols in s->u.ppsym[k] @@ -30,6 +32,60 @@ Prop* nrn_point_prop_; void (*nrnpy_o2loc_p_)(Object*, Section**, double*); void (*nrnpy_o2loc2_p_)(Object*, Section**, double*); +namespace { +// Owning SoA rows keyed by Point_process* so section_fwd.hpp stays MOD-light. +std::unordered_map + g_point_process_soa_owners; +} // namespace + +Point_process::Point_process() { + auto& owner = + g_point_process_soa_owners.emplace(this, neuron::model().point_processes()).first->second; + _soa_id = owner.id(); +} + +Point_process::~Point_process() { + g_point_process_soa_owners.erase(this); + _soa_id = {}; +} + +int nrn_point_process_soa_row(Point_process const* pnt) { + if (!pnt || !pnt->_soa_id) { + return -1; + } + return static_cast(pnt->_soa_id.current_row()); +} + +void nrn_point_process_soa_sync(Point_process* pnt) { + if (!pnt || !pnt->_soa_id) { + return; + } + // Owner must still be alive (owning_handle in g_point_process_soa_owners). + if (!g_point_process_soa_owners.count(pnt)) { + return; + } + auto h = neuron::container::network::point_process_soa(pnt); + // Only read Prop* when non-null and still linked to this Point_process. + // During free/relocate, prop may already be deleted or half-torn-down. + Prop* p = pnt->prop; + if (p && p->dparam && p->dparam[1].get() == pnt) { + h.mech_type() = p->_type; + // Prop for a point process owns a mechanism SoA row while live. + h.instance() = static_cast(p->current_row()); + } else { + h.mech_type() = -1; + h.instance() = -1; + } + // _vnt is NrnThread* when set; only use if it looks like a live thread. + auto* nt = static_cast(pnt->_vnt); + if (nt && nrn_threads && nrn_nthread > 0 && nt >= nrn_threads && + nt < nrn_threads + nrn_nthread) { + h.thread_id() = nt->id; + } else { + h.thread_id() = -1; + } +} + void* create_point_process(int pointtype, Object* ho) { auto* const pp = new Point_process{}; pp->ob = ho; @@ -40,6 +96,7 @@ void* create_point_process(int pointtype, Object* ho) { if (ho && ho->ctemplate->steer && ifarg(1)) { loc_point_process(pointtype, (void*) pp); } + nrn_point_process_soa_sync(pp); return pp; } @@ -132,6 +189,7 @@ void nrn_loc_point_process(int pointtype, Point_process* pnt, Section* sec, Node hoc_template_notify(pnt->ob, 2); } } + nrn_point_process_soa_sync(pnt); } static void create_artcell_prop(Point_process* pnt, short type) { @@ -148,6 +206,7 @@ static void create_artcell_prop(Point_process* pnt, short type) { hoc_template_notify(pnt->ob, 2); } } + nrn_point_process_soa_sync(pnt); } void nrn_relocate_old_points(Section* oldsec, Node* oldnode, Section* sec, Node* node) { @@ -310,9 +369,11 @@ void connect_point_process_pointer(void) { static void free_one_point(Point_process* pnt) { auto* p = pnt->prop; if (!p) { + // SoA row is released in ~Point_process; do not touch Prop/_vnt here. + pnt->_vnt = nullptr; return; } - if (!nrn_is_artificial_[p->_type]) { + if (!nrn_is_artificial_[p->_type] && pnt->node) { auto* p1 = pnt->node->prop; if (p1 == p) { pnt->node->prop = p1->next; @@ -337,10 +398,12 @@ static void free_one_point(Point_process* pnt) { delete p; pnt->prop = (Prop*) 0; pnt->node = (Node*) 0; + pnt->_vnt = nullptr; if (pnt->sec) { section_unref(pnt->sec); } pnt->sec = (Section*) 0; + // No soa_sync: reading prop/_vnt after teardown is UAF (ASan). Row freed in ~Point_process. } // called from prop_free diff --git a/src/nrnoc/section_fwd.hpp b/src/nrnoc/section_fwd.hpp index fe1a955f6c..7fbd01e36e 100644 --- a/src/nrnoc/section_fwd.hpp +++ b/src/nrnoc/section_fwd.hpp @@ -1,10 +1,15 @@ #pragma once #include "multicore.h" #include "neuron/container/generic_data_handle.hpp" +#include "neuron/container/non_owning_soa_identifier.hpp" #include "nrnredef.h" /** * @file section_fwd.hpp * @brief Forward declarations of Section, Node etc. to be included in translated MOD files. + * + * Keep this header light: MOD / nrnivmodl compile against build/include only. + * Network SoA ownership lives in point.cpp (see nrn_point_process_soa_*), not + * as a heavy owning_handle member here. */ struct Node; struct Prop; @@ -83,4 +88,24 @@ struct Point_process { void* presyn_{}; /* non-threshold presynapse for NetCon */ void* nvi_{}; /* NrnVarIntegrator (for local step method) */ void* _vnt{}; /* NrnThread* (for NET_RECEIVE and multicore) */ + /** + * @brief Non-owning id of the network::PointProcess SoA row (Phase 1 dual-write). + * + * Owning lifetime is managed in point.cpp so this header stays free of + * model_data / network SoA includes (required for nrnivmodl / demo MOD builds). + */ + neuron::container::non_owning_identifier_without_container _soa_id{}; + + Point_process(); + ~Point_process(); + Point_process(Point_process const&) = delete; + Point_process& operator=(Point_process const&) = delete; + Point_process(Point_process&&) = delete; + Point_process& operator=(Point_process&&) = delete; }; + +/** @brief Sync legacy Point_process fields into the network SoA row. */ +void nrn_point_process_soa_sync(Point_process* pnt); + +/** @brief Current SoA row of a Point_process dual-write entry (-1 if none). */ +int nrn_point_process_soa_row(Point_process const* pnt); diff --git a/src/nrnoc/treeset.cpp b/src/nrnoc/treeset.cpp index e24a45b196..80a2f67308 100644 --- a/src/nrnoc/treeset.cpp +++ b/src/nrnoc/treeset.cpp @@ -9,6 +9,7 @@ #include "neuron.h" #include "neuron/cache/mechanism_range.hpp" #include "neuron/cache/model_data.hpp" +#include "neuron/container/network/sort.hpp" #include "neuron/container/soa_container.hpp" #include "node_order_optim/node_order_optim.h" #include "nonvintblock.h" @@ -1694,6 +1695,7 @@ void v_setup_vectors(void) { pnt->_vnt = nrn_threads + nti; nti = (nti + 1) % nrn_nthread; } + nrn_point_process_soa_sync(pnt); auto const tid = static_cast(pnt->_vnt)->id; ++thread_counts[tid]; // pnt->_i_instance = j; @@ -2212,6 +2214,18 @@ neuron::model_sorted_token nrn_ensure_model_data_are_sorted() { mech_tokens.push_back(mech_data.issue_frozen_token()); already_sorted = already_sorted && mech_data.is_sorted(); }); + // Network SoA containers (Phase 1–3 dual-write; sort after nodes+mechs). + std::vector pp_tokens; + std::vector weight_tokens; + std::vector netcon_tokens; + std::vector presyn_tokens; + pp_tokens.push_back(model.point_processes().issue_frozen_token()); + weight_tokens.push_back(model.weights().issue_frozen_token()); + netcon_tokens.push_back(model.netcons().issue_frozen_token()); + presyn_tokens.push_back(model.presyns().issue_frozen_token()); + already_sorted = already_sorted && model.point_processes().is_sorted() && + model.weights().is_sorted() && model.netcons().is_sorted() && + model.presyns().is_sorted(); // Now the whole model is marked frozen/read-only, but it may or may not be // marked sorted (if it is, the cache should be valid, otherwise it should // not be). @@ -2256,11 +2270,23 @@ neuron::model_sorted_token nrn_ensure_model_data_are_sorted() { // Now that all the mechanism data is sorted we can fill in pdata caches model.apply_to_mechanisms( [&cache](auto& mech_data) { nrn_fill_mech_data_caches(cache, mech_data); }); + // Network: PointProcess / Weight repack / NetCon / PreSyn + fanout. + // doc/network-soa-phase0.md §6.3, §8.4 + neuron::container::network::sort_network_data( + cache, pp_tokens[0], weight_tokens[0], netcon_tokens[0], presyn_tokens[0]); + assert(model.point_processes().is_sorted()); + assert(model.weights().is_sorted()); + assert(model.netcons().is_sorted()); + assert(model.presyns().is_sorted()); // Move our working cache into the global storage. neuron::cache::model = std::move(cache); } // Move our tokens into the return value and be done with it. neuron::model_sorted_token ret{*neuron::cache::model, std::move(node_token)}; ret.mech_data_tokens = std::move(mech_tokens); + ret.point_process_tokens = std::move(pp_tokens); + ret.weight_tokens = std::move(weight_tokens); + ret.netcon_tokens = std::move(netcon_tokens); + ret.presyn_tokens = std::move(presyn_tokens); return ret; } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index ffa28da9b2..8826430f8f 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -21,6 +21,7 @@ add_executable( unit_tests/container/container.cpp unit_tests/container/generic_data_handle.cpp unit_tests/container/mechanism.cpp + unit_tests/container/network.cpp unit_tests/container/node.cpp unit_tests/node_order_optim/permutations.cpp unit_tests/utils/enumerate.cpp @@ -245,6 +246,14 @@ if(NRN_ENABLE_PYTHON) COMMAND "${exe}" ${pytest} "./test/${group}" SCRIPT_PATTERNS "test/${group}/*.json" "test/${group}/*.py") endforeach() + # Focused CPU gate for network SoA dual-write (NetStim→ExpSyn). + nrn_add_test( + GROUP pytest + NAME network_soa_delivery_py${pyver} + PRELOAD_SANITIZER + ENVIRONMENT "CC=${CMAKE_C_COMPILER}" + COMMAND "${exe}" ${pytest} "./test/pytest/test_network_soa_delivery.py" + SCRIPT_PATTERNS "test/pytest/test_network_soa_delivery.py") endforeach() # Add some tests that are specifically aimed at NEURON - Python integration, covering different diff --git a/test/pytest/test_network_soa_delivery.py b/test/pytest/test_network_soa_delivery.py new file mode 100644 index 0000000000..28c8473425 --- /dev/null +++ b/test/pytest/test_network_soa_delivery.py @@ -0,0 +1,160 @@ +""" +CPU delivery gate for network SoA dual-write (local/cpu-network-soa). + +NetStim -> NetCon -> ExpSyn exercises: + - Weight SoA dual-write and HOC weight[] + - nrn_pnt_receive_by_weight_index around pnt_receive + - PreSyn fanout order rebuild at init_events +""" + +from neuron import h + + +def test_netstim_expsyn_delivery_near_1_025_ms(): + """Spike at t=1, delay 0.025 → receive ~1.025; g should peak soon after.""" + s = h.Section(name="soma") + s.insert("pas") + s.L = s.diam = 10 + + syn = h.ExpSyn(s(0.5)) + syn.tau = 1 + syn.e = 0 + + ns = h.NetStim() + ns.start = 1.0 + ns.number = 1 + ns.noise = 0 + + nc = h.NetCon(ns, syn) + # Write weight through HOC (Weight SoA dual-write path when available). + nc.weight[0] = 0.05 + nc.delay = 0.025 + + h.dt = 0.025 + h.finitialize(-65) + + g_max = 0.0 + t_gmax = 0.0 + while h.t < 3.0 - h.dt / 2: + h.fadvance() + g = float(syn.g) + if g > g_max: + g_max = g + t_gmax = float(h.t) + + assert g_max > 0.04, f"expected ExpSyn conductance peak, got g_max={g_max}" + assert ( + 1.0 <= t_gmax <= 1.2 + ), f"expected peak near 1.025 ms window, got t_gmax={t_gmax}" + + +def test_netcon_weight_hoc_roundtrip(): + """HOC weight set is visible to delivery (SoA dual-write + materialize).""" + s = h.Section(name="soma2") + s.insert("pas") + s.L = s.diam = 10 + syn = h.ExpSyn(s(0.5)) + ns = h.NetStim() + ns.start = 0.5 + ns.number = 1 + ns.noise = 0 + nc = h.NetCon(ns, syn) + nc.weight[0] = 0.12 + nc.delay = 0.0 + assert abs(nc.weight[0] - 0.12) < 1e-12 + + h.dt = 0.025 + h.finitialize(-65) + g_max = 0.0 + while h.t < 2.0 - h.dt / 2: + h.fadvance() + g_max = max(g_max, float(syn.g)) + # Larger weight → larger g peak than default-zero would give + assert g_max > 0.1, f"weight 0.12 should produce larger g, g_max={g_max}" + + +def test_savestate_weight_dualwrite_roundtrip(): + """SaveState restore must update Weight SoA, not only the weight_ heap.""" + s = h.Section(name="soma_ss") + s.insert("pas") + s.L = s.diam = 10 + syn = h.ExpSyn(s(0.5)) + syn.tau = 1 + syn.e = 0 + ns = h.NetStim() + ns.start = 1.0 + ns.number = 1 + ns.noise = 0 + nc = h.NetCon(ns, syn) + nc.weight[0] = 0.05 + nc.delay = 0.025 + + h.dt = 0.025 + h.finitialize(-65) + ss = h.SaveState() + ss.save() + + # Mutate after save (SoA-primary HOC path). + nc.weight[0] = 0.99 + assert abs(nc.weight[0] - 0.99) < 1e-12 + + ss.restore() + assert ( + abs(nc.weight[0] - 0.05) < 1e-12 + ), f"SaveState restore should restore HOC/SoA weight, got {nc.weight[0]}" + + # Delivery after restore must use restored weight (SoA materialize path). + g_max = 0.0 + while h.t < 3.0 - h.dt / 2: + h.fadvance() + g_max = max(g_max, float(syn.g)) + assert ( + g_max > 0.04 + ), f"restored weight 0.05 should produce ExpSyn g peak, g_max={g_max}" + assert ( + g_max < 0.5 + ), f"mutated weight 0.99 must not leak into delivery, g_max={g_max}" + + +def test_bbsavestate_weight_dualwrite_roundtrip(tmp_path): + """BBSaveState restore must update Weight SoA (NetCon on a gid cell).""" + + # BBSaveState requires a real cell with a gid (section parented to a cell object). + class Cell: + def __init__(self): + self.s = h.Section("soma", self) + self.s.insert("pas") + self.s.L = self.s.diam = 10 + self.syn = h.ExpSyn(self.s(0.5)) + self.syn.tau = 1 + self.syn.e = 0 + + pc = h.ParallelContext() + cell = Cell() + ns = h.NetStim() + ns.start = 1.0 + ns.number = 1 + ns.noise = 0 + nc = h.NetCon(ns, cell.syn) + nc.weight[0] = 0.05 + nc.delay = 0.025 + + gid = 7 + pc.set_gid2node(gid, pc.id()) + pc.cell(gid, h.NetCon(cell.s(0.5)._ref_v, None, sec=cell.s)) + + h.dt = 0.025 + h.finitialize(-65) + path = str(tmp_path / "bbss_weights.bbss") + bbss = h.BBSaveState() + bbss.save(path) + + nc.weight[0] = 0.99 + assert abs(nc.weight[0] - 0.99) < 1e-12 + + bbss.restore(path) + assert ( + abs(nc.weight[0] - 0.05) < 1e-12 + ), f"BBSaveState restore should restore HOC/SoA weight, got {nc.weight[0]}" + + pc.gid_clear() diff --git a/test/unit_tests/container/network.cpp b/test/unit_tests/container/network.cpp new file mode 100644 index 0000000000..e68cbf6f80 --- /dev/null +++ b/test/unit_tests/container/network.cpp @@ -0,0 +1,416 @@ +#include "neuron/cache/model_data.hpp" +#include "neuron/container/network/netcon.hpp" +#include "neuron/container/network/point_process.hpp" +#include "neuron/container/network/point_process_access.hpp" +#include "neuron/container/network/presyn.hpp" +#include "neuron/container/network/self_event.hpp" +#include "neuron/container/network/sort.hpp" +#include "neuron/container/network/weight_block.hpp" +#include "neuron/container/network/weights.hpp" +#include "neuron/model_data.hpp" +#include "nrn_ansi.h" +#include "section.h" + +#include + +#include +#include +#include +#include +#include + +using namespace neuron::container::network; + +TEST_CASE("SOA-backed PointProcess structure", + "[Neuron][data_structures][network][point_process]") { + auto& storage = neuron::model().point_processes(); + REQUIRE(storage.size() == 0); + + GIVEN("Default-constructed owning handles") { + PointProcess::owning_handle a{storage}; + PointProcess::owning_handle b{storage}; + THEN("Defaults match field tags") { + REQUIRE(a.instance() == -1); + REQUIRE(a.mech_type() == -1); + REQUIRE(a.thread_id() == -1); + REQUIRE(storage.size() == 2); + } + THEN("Fields can be set and read back via non-owning handles") { + a.instance() = 7; + a.mech_type() = 42; + a.thread_id() = 1; + auto ha = a.non_owning_handle(); + REQUIRE(ha.instance() == 7); + REQUIRE(ha.mech_type() == 42); + REQUIRE(ha.thread_id() == 1); + } + THEN("Destroying an owner invalidates non-owning handles to that row") { + auto ha = a.non_owning_handle(); + REQUIRE(ha.id()); + { + PointProcess::owning_handle tmp{std::move(a)}; + REQUIRE(ha.id()); + // tmp destroyed here + } + REQUIRE_FALSE(ha.id()); + REQUIRE(storage.size() == 1); // only b remains + } + } + + GIVEN("Several rows and a reverse permutation") { + constexpr int n = 8; + std::vector rows; + rows.reserve(n); + std::vector ref_instance(n); + for (int i = 0; i < n; ++i) { + rows.emplace_back(storage); + rows.back().instance() = 100 + i; + rows.back().mech_type() = i; + rows.back().thread_id() = i % 3; + ref_instance[i] = 100 + i; + } + { + auto token = storage.issue_frozen_token(); + storage.mark_as_sorted(token); + } + REQUIRE(storage.is_sorted()); + + WHEN("A rotate reverse-permutation is applied") { + std::vector perm(n); + std::iota(perm.begin(), perm.end(), 0); + std::rotate(perm.begin(), std::next(perm.begin()), perm.end()); + storage.apply_reverse_permutation(std::move(perm)); + + THEN("Handles still report the logical field values") { + for (int i = 0; i < n; ++i) { + REQUIRE(rows[i].instance() == ref_instance[i]); + REQUIRE(rows[i].mech_type() == i); + REQUIRE(rows[i].thread_id() == i % 3); + } + } + THEN("Underlying storage order differs from original creation order") { + bool same = true; + for (int i = 0; i < n; ++i) { + if (storage.get(i) != ref_instance[i]) { + same = false; + break; + } + } + REQUIRE_FALSE(same); + } + } + + WHEN("A random reverse-permutation is applied") { + std::vector perm(n); + std::iota(perm.begin(), perm.end(), 0); + std::mt19937 g{42}; + std::shuffle(perm.begin(), perm.end(), g); + storage.apply_reverse_permutation(std::move(perm)); + THEN("Handles survive and keep logical values") { + for (int i = 0; i < n; ++i) { + REQUIRE(rows[i].instance() == ref_instance[i]); + } + } + } + } +} + +TEST_CASE("Point_process dual-write into network SoA", + "[Neuron][data_structures][network][point_process][dualwrite]") { + auto& storage = neuron::model().point_processes(); + auto const before = storage.size(); + GIVEN("A default-constructed Point_process shell") { + // Allocates an SoA row via Point_process ctor dual-write map. + auto* pp = new Point_process{}; + THEN("SoA size grows by one and fields are defaults until prop is set") { + REQUIRE(storage.size() == before + 1); + REQUIRE(pp->_soa_id); + REQUIRE(nrn_point_process_soa_row(pp) >= 0); + auto h = neuron::container::network::point_process_soa(pp); + REQUIRE(h.mech_type() == -1); + REQUIRE(h.instance() == -1); + REQUIRE(h.thread_id() == -1); + } + WHEN("The Point_process is destroyed") { + delete pp; + THEN("The SoA row is released") { + REQUIRE(storage.size() == before); + } + } + } +} + +TEST_CASE("SOA-backed Weight structure", "[Neuron][data_structures][network][weights]") { + auto& storage = neuron::model().weights(); + REQUIRE(storage.size() == 0); + + GIVEN("A contiguous weight block (NetCon-style)") { + constexpr int weight_count = 5; + std::vector block; + block.reserve(weight_count); + for (int k = 0; k < weight_count; ++k) { + block.emplace_back(storage); + block.back().value() = static_cast(k + 1); // exact binary values + } + REQUIRE(storage.size() == static_cast(weight_count)); + + THEN("data_handle to Value is modern and yields correct values") { + auto dh = block[2].value_handle(); + REQUIRE(dh); + REQUIRE(dh.refers_to_a_modern_data_structure()); + REQUIRE(dh.refers_to(storage)); + REQUIRE(*dh == 3.0); + } + + WHEN("The weight storage is reverse-permuted") { + std::vector perm(weight_count); + std::iota(perm.begin(), perm.end(), 0); + std::rotate(perm.begin(), std::next(perm.begin()), perm.end()); + storage.apply_reverse_permutation(std::move(perm)); + + THEN("Handles still report the logical values") { + for (int k = 0; k < weight_count; ++k) { + REQUIRE(block[k].value() == static_cast(k + 1)); + } + } + THEN("data_handle still works after permute") { + auto dh = block[0].value_handle(); + REQUIRE(*dh == 1.0); + } + } + + WHEN("The owning block is destroyed") { + auto dh = block[0].value_handle(); + REQUIRE(dh); + block.clear(); + THEN("data_handles become invalid and storage is empty") { + REQUIRE_FALSE(dh); + REQUIRE(storage.size() == 0); + } + } + } + + GIVEN("allocate_weight_rows dual-write helper") { + double heap[3] = {1.5, 2.5, 3.5}; + auto rows = Weight::allocate_weight_rows(3, heap); + THEN("SoA mirrors the heap values") { + REQUIRE(rows.size() == 3); + REQUIRE(storage.size() == 3); + REQUIRE(rows[0].value() == 1.5); + REQUIRE(rows[1].value() == 2.5); + REQUIRE(rows[2].value() == 3.5); + } + WHEN("heap is updated and remirrored") { + heap[1] = 9.0; + Weight::mirror_weights_to_soa(rows, heap, 3); + REQUIRE(rows[1].value() == 9.0); + } + } +} + +TEST_CASE("SOA-backed NetCon structure", "[Neuron][data_structures][network][netcon]") { + auto& storage = neuron::model().netcons(); + auto& wstore = neuron::model().weights(); + REQUIRE(storage.size() == 0); + + GIVEN("Default-constructed NetCon SoA rows") { + NetCon::owning_handle a{storage}; + NetCon::owning_handle b{storage}; + THEN("Defaults match field tags") { + REQUIRE(a.target() == -1); + REQUIRE(a.weight_index() == -1); + REQUIRE(a.weight_count() == 0); + REQUIRE(a.delay() == 1.0); + REQUIRE(a.active() == 1); + REQUIRE(a.src_presyn() == -1); + REQUIRE(storage.size() == 2); + } + THEN("Fields round-trip and survive reverse permutation") { + a.target() = 3; + a.weight_index() = 10; + a.weight_count() = 2; + a.delay() = 0.5; + a.active() = 0; + b.target() = 7; + b.delay() = 2.0; + { + auto token = storage.issue_frozen_token(); + storage.mark_as_sorted(token); + } + std::vector perm{0, 1}; + std::rotate(perm.begin(), std::next(perm.begin()), perm.end()); + storage.apply_reverse_permutation(std::move(perm)); + REQUIRE(a.target() == 3); + REQUIRE(a.weight_index() == 10); + REQUIRE(a.weight_count() == 2); + REQUIRE(a.delay() == 0.5); + REQUIRE(a.active() == 0); + REQUIRE(b.target() == 7); + REQUIRE(b.delay() == 2.0); + } + } + + GIVEN("Weight block linked like Phase 2 dual-write") { + auto wrows = Weight::allocate_weight_rows(2, nullptr); + wrows[0].value() = 0.1; + wrows[1].value() = 0.2; + NetCon::owning_handle nc{storage}; + nc.weight_index() = static_cast(wrows[0].current_row()); + nc.weight_count() = 2; + THEN("WeightIndex points at first SoA weight row") { + REQUIRE(wstore.get(nc.weight_index()) == 0.1); + REQUIRE(wstore.get(nc.weight_index() + 1) == 0.2); + } + } +} + +TEST_CASE("Weight index materialize/store (Phase 4 SelfEvent path)", + "[Neuron][data_structures][network][selfevent]") { + auto& store = neuron::model().weights(); + REQUIRE(store.size() == 0); + auto rows = Weight::allocate_weight_rows(3, nullptr); + rows[0].value() = 1.0; + rows[1].value() = 2.0; + rows[2].value() = 3.0; + int const base = static_cast(rows[0].current_row()); + double buf[3]{}; + neuron::container::network::SelfEventFields::materialize_weight_block(base, 3, buf); + REQUIRE(buf[0] == 1.0); + REQUIRE(buf[1] == 2.0); + REQUIRE(buf[2] == 3.0); + buf[1] = 9.0; + neuron::container::network::SelfEventFields::store_weight_block(base, 3, buf); + REQUIRE(rows[1].value() == 9.0); +} + +TEST_CASE("SOA-backed PreSyn structure and fanout ranges", + "[Neuron][data_structures][network][presyn]") { + auto& storage = neuron::model().presyns(); + REQUIRE(storage.size() == 0); + + GIVEN("Default PreSyn SoA rows") { + PreSyn::owning_handle a{storage}; + PreSyn::owning_handle b{storage}; + THEN("Defaults match field tags") { + REQUIRE(a.threshold() == 10.0); + REQUIRE(a.gid() == -1); + REQUIRE(a.nc_index() == -1); + REQUIRE(a.nc_count() == 0); + REQUIRE(a.output_index() == -1); + REQUIRE(a.thvar_row() == -1); + REQUIRE(a.thread_id() == -1); + REQUIRE(storage.size() == 2); + } + THEN("Fanout range fields survive reverse permutation") { + // Simulate CoreNEURON-style contiguous fanout ranges in a global order. + a.nc_index() = 0; + a.nc_count() = 3; + a.threshold() = -20.; + a.gid() = 7; + b.nc_index() = 3; + b.nc_count() = 2; + b.gid() = 11; + { + auto token = storage.issue_frozen_token(); + storage.mark_as_sorted(token); + } + std::vector perm{0, 1}; + std::rotate(perm.begin(), std::next(perm.begin()), perm.end()); + storage.apply_reverse_permutation(std::move(perm)); + REQUIRE(a.nc_index() == 0); + REQUIRE(a.nc_count() == 3); + REQUIRE(a.threshold() == -20.); + REQUIRE(a.gid() == 7); + REQUIRE(b.nc_index() == 3); + REQUIRE(b.nc_count() == 2); + REQUIRE(b.gid() == 11); + } + } +} + +TEST_CASE("Network SoA sort partitions PointProcess by thread", + "[Neuron][data_structures][network][sort]") { + auto& pp = neuron::model().point_processes(); + auto& w = neuron::model().weights(); + auto& nc = neuron::model().netcons(); + auto& ps = neuron::model().presyns(); + + // Intentionally create in non-thread order: tid 1, 0, 1, 0 + std::vector rows; + rows.emplace_back(pp); + rows.back().thread_id() = 1; + rows.back().instance() = 10; + rows.emplace_back(pp); + rows.back().thread_id() = 0; + rows.back().instance() = 20; + rows.emplace_back(pp); + rows.back().thread_id() = 1; + rows.back().instance() = 11; + rows.emplace_back(pp); + rows.back().thread_id() = 0; + rows.back().instance() = 21; + + auto w0 = Weight::allocate_weight_rows(1, nullptr); + w0[0].value() = 1.0; + Weight::owning_handle orphan{w}; + orphan.value() = 99.0; + auto w1 = Weight::allocate_weight_rows(1, nullptr); + w1[0].value() = 2.0; + + extern int nrn_nthread; + { + auto pp_tok = pp.issue_frozen_token(); + auto w_tok = w.issue_frozen_token(); + auto nc_tok = nc.issue_frozen_token(); + auto ps_tok = ps.issue_frozen_token(); + + neuron::cache::Model cache{}; + cache.thread.resize(static_cast(std::max(nrn_nthread, 1))); + + neuron::container::network::sort_network_data(cache, pp_tok, w_tok, nc_tok, ps_tok); + + REQUIRE(pp.is_sorted()); + REQUIRE(w.is_sorted()); + REQUIRE(nc.is_sorted()); + REQUIRE(ps.is_sorted()); + + // Handles keep logical values after partition. + REQUIRE(rows[0].thread_id() == 1); + REQUIRE(rows[0].instance() == 10); + REQUIRE(rows[1].thread_id() == 0); + REQUIRE(rows[1].instance() == 20); + + if (nrn_nthread >= 2) { + // Storage order: thread 0 rows then thread 1 rows. + REQUIRE(pp.get(0) == 0); + REQUIRE(pp.get(1) == 0); + REQUIRE(pp.get(2) == 1); + REQUIRE(pp.get(3) == 1); + REQUIRE(cache.thread[0].point_process_offset == 0); + REQUIRE(cache.thread[1].point_process_offset == 2); + } + } // freeze tokens released before destroying owning handles +} + +TEST_CASE("nrn_ensure_model_data_are_sorted freezes network containers", + "[Neuron][data_structures][network][sort]") { + // Needs a minimal model (same precondition as other ensure_sorted unit tests). + REQUIRE(hoc_oc("create s\nfinitialize(-65)\n") == 0); + neuron::model().point_processes().mark_as_unsorted(); + neuron::model().weights().mark_as_unsorted(); + neuron::model().netcons().mark_as_unsorted(); + neuron::model().presyns().mark_as_unsorted(); + { + auto token = nrn_ensure_model_data_are_sorted(); + REQUIRE(neuron::model().point_processes().is_sorted()); + REQUIRE(neuron::model().weights().is_sorted()); + REQUIRE(neuron::model().netcons().is_sorted()); + REQUIRE(neuron::model().presyns().is_sorted()); + REQUIRE(neuron::cache::model); + REQUIRE_FALSE(token.point_process_tokens.empty()); + REQUIRE_FALSE(token.weight_tokens.empty()); + REQUIRE_FALSE(token.netcon_tokens.empty()); + REQUIRE_FALSE(token.presyn_tokens.empty()); + } // release freeze before structural cleanup + REQUIRE(hoc_oc("delete_section()\n") == 0); +}