Document native GPU gap junction CPU fixed-step fallback

Phase B docs previously described a hybrid GPU solve with host partrans
gather/scatter. When nrnthread_v_transfer_ is registered, fadvance.cpp
dispatches the full CPU fixed-step body instead of fadvance_gpu.cpp.

Update the Sphinx scope contract, design journal, limitations, future work,
test parity notes, and Phase B checklist to match.

Author: nrnhines

docs/dev/native-gpu-adoption/00-overview.md

@@ -24,6 +24,7 @@ See [04-known-limitations.md](04-known-limitations.md) and
 - GPU spike priority queue
 - Standalone `-DNRN_ENABLE_GPU=ON` configure without `-DNRN_ENABLE_CORENEURON=ON`
 - Full multi-thread OpenACC maturity
+- Device partrans gather/scatter (gap models use CPU fixed-step fallback today)
 - MPI-native modtest expansion beyond the current parity set
 
 ## Stopping point

docs/dev/native-gpu-adoption/01-design-decisions.md

@@ -46,23 +46,26 @@ after topology changes must invalidate cleanly.
 
 **Artifacts:** `device_state.cpp`, `upload.cpp`.
 
-## Gap junctions: three-phase partrans, host staging in Phase B
+## Gap junctions: CPU fixed-step fallback in Phase B
 
-**Decision:** Gap transfer follows CPU partrans phases when
-`pc.setup_transfer()` is active (not only when MPI or `nthread>1`):
+**Decision:** When `pc.setup_transfer()` registers `nrnthread_v_transfer_`,
+native GPU fixed-step dispatches the **full CPU body** in `nrn_fixed_step_thread`
+(`fadvance.cpp`) instead of `fadvance_gpu.cpp`. Partrans then runs unchanged on
+the CPU path:
 
 1. Gather sources into buffers (`mpi_transfer` / `outsrc_buf_`)
 2. `MPI_Alltoallv` when `nrnmpi_numprocs > 1` (skipped on one rank)
 3. Scatter to targets (`thread_transfer` in `nonvint`)
 
-Native GPU pulls source voltages device → host (`sync_gap_after_voltage_update`)
-before gather so `mpi_transfer` reads host memory. Scatter remains CPU in lastpart.
+`gpu.enable=True` still uploads model state; integration per step is CPU until
+device-resident gather/scatter is implemented.
 
-**Why:** Partrans predates device-resident gap buffers. GPU gather/scatter without
-host staging — especially when MPI is absent — is a future optimization, not
-Phase B.
+**Why:** A hybrid GPU solve plus host partrans gather/scatter diverged on Traub
+(`use_gap=1`) and could overwrite host voltages with stale device state. Full CPU
+fallback restores raster parity (`test_par_gj_native_gpu.py`, ringtest `-gap`) at
+~1× NEURON CPU cost. Device partrans buffers are follow-up work, not Phase B.
 
-**Artifacts:** `partrans.cpp`, `sync.cpp`, `fadvance_gpu.cpp`.
+**Artifacts:** `fadvance.cpp`, `partrans.cpp`, `sync.cpp`, `fadvance_gpu.cpp`.
 
 ## Batch download for recording
 

docs/dev/native-gpu-adoption/02-implementation-map.md

@@ -54,7 +54,7 @@ repo root.
 | Artifact | Role |
 |----------|------|
 | `src/neuron/gpu/post_solve.cpp` | `post_solve_needs_host_fallback()` |
-| `src/nrnoc/fadvance.cpp` | CPU `nrn_update_voltage` when fallback required |
+| `src/nrnoc/fadvance.cpp` | CPU `nrn_update_voltage` when fallback required; **gap CPU fixed-step fallback** when `nrnthread_v_transfer_` is set |
 
 ## L6 — Communication
 
@@ -64,8 +64,8 @@ repo root.
 | `src/neuron/gpu/net_events.hpp` | Spike / `NET_RECEIVE` policy comments |
 | `src/neuron/gpu/net_send_buffer.cpp` | GPU `net_send` buffering |
 | `src/neuron/gpu/net_send_buffer.hpp` | Buffer semantics |
-| `src/neuron/gpu/sync.cpp` | Gap / voltage device↔host sync |
-| `src/nrniv/partrans.cpp` | MPI gap transfer (host) |
+| `src/neuron/gpu/sync.cpp` | Gap voltage device↔host sync (hybrid path; unused on Phase B CPU fallback) |
+| `src/nrniv/partrans.cpp` | Partrans gather / MPI / scatter (CPU) |
 
 ## L7 — Recording
 

docs/dev/native-gpu-adoption/03-test-parity.md

@@ -63,6 +63,19 @@ tests without running the full CoreNEURON GPU matrix.
 
 Filter: `ctest -R testneuron_gpu`
 
+## Gap junction supplementary test
+
+Not part of the 19-ctest parity set; run manually after building with GPU:
+
+```console
+cd test/gjtests && nrnivmodl
+python3 test_par_gj_native_gpu.py
+```
+
+Compares `test_par_gj` dendrite voltages CPU vs `gpu.backend=native`. Ringtest
+`-gap -gpu-native` (single rank) is an additional raster check; CTest
+`neuron_gpu_native_mpi` does not pass `-gap` today.
+
 ## Ringtest native benchmark
 
 External ringtest supports `-gpu-native` (orthogonal to CoreNEURON `-gpu`).

docs/dev/native-gpu-adoption/04-known-limitations.md

@@ -31,11 +31,25 @@ not ruled out by the architecture, but Phase B does not certify DAE models on
 `gpu.backend="native"`. `post_solve_needs_host_fallback()` often forces CPU
 post-solve for DAE/LFP cases as well.
 
-## LFP / partrans post-solve fallback
+## Gap junctions: CPU fixed-step fallback
 
-When `nrnthread_vi_compute_` is registered (gap/partrans v+vext interpolation),
-post-solve runs on the host via `nrn_update_voltage` instead of
-`post_solve_on_device`. The matrix solve may still be on GPU for type-1 ODE models.
+When `pc.setup_transfer()` is active (`nrnthread_v_transfer_` registered), native
+GPU does **not** run `fadvance_gpu.cpp` for fixed-step integration. The CPU
+`nrn_fixed_step_thread` body handles matrix setup, solve, post-solve, partrans
+gather/MPI/scatter, and lastpart. Correctness is validated (e.g.
+`test/gjtests/test_par_gj_native_gpu.py`); performance matches NEURON CPU (~1×),
+not the GPU-accelerated non-gap path.
+
+Device-resident partrans gather/scatter — restoring GPU integration subphases while
+keeping host/MPI only for cross-rank `MPI_Alltoallv` — is deferred (see
+[05-future-work.md](05-future-work.md)).
+
+## LFP / extracellular post-solve fallback
+
+When `nrnthread_vi_compute_` is registered (extracellular `v+vext` sources for
+partrans), `post_solve_needs_host_fallback()` forces host `nrn_update_voltage`
+instead of `post_solve_on_device` on the **GPU integration path**. This applies
+only when the native GPU step runs (no `nrnthread_v_transfer_` CPU fallback).
 
 ## MPI-native modtest gap
 

docs/dev/native-gpu-adoption/05-future-work.md

@@ -12,6 +12,7 @@ Phase B checklist.
 | GPU spike priority queue | Requires MPI + device ordering design |
 | Full multi-thread OpenACC | Harden per-thread CUDA contexts; remove warning |
 | `sparse13` / extracellular / LFP on device | Eliminate per-step host fallbacks |
+| Device partrans gather/scatter | Remove gap CPU fixed-step fallback; host/MPI only when `nrnmpi_numprocs > 1` |
 | MPI-native modtest expansion | Beyond 19-test single-process parity |
 | Per-step matrix host sync elimination | Performance optimization in `treeset.cpp` |
 

docs/dev/native-gpu-adoption/PHASE-B-COMPLETE.md

@@ -8,7 +8,7 @@ Checklist for closing native GPU **fixed-step** Phase B on branch
 - [x] Fixed-step `pc.psolve` on `gpu.backend=native` without `coreneuron.enable`
 - [x] OpenACC matrix solve + mechanism state + post-solve on device
 - [x] CPU spike queues + MPI; GPU `NET_RECEIVE` and `net_send` buffer
-- [x] Gap junctions with device voltage sync before MPI
+- [x] Gap junctions: CPU fixed-step fallback when `setup_transfer` active (correctness; `test_par_gj_native_gpu`)
 - [x] Batch download API (`download_flush_interval`)
 - [x] CVode rejected when native GPU enabled
 - [x] Threading warning for `pc.nthread(n>1)`

docs/dev/native-gpu-fixed-step.rst

@@ -44,7 +44,9 @@ Phase B contract
 - ``NET_RECEIVE`` mechanism computation on GPU during the step
 - CPU spike priority queues + MPI exchange at the minimum NetCon delay interval
 - GPU ``net_send`` buffering; cross-cell events to CPU/MPI; self-events may flush on device
-- Gap junctions: device voltage pull, sparse MPI transfer, host fallback where required
+- Gap junctions: **correctness** on native GPU when ``pc.setup_transfer()`` is
+  active; fixed-step integration runs the **CPU body** (full fallback) until
+  device partrans gather/scatter is implemented
 - Batch host download API for ``Vector.record`` and graphs (``download_flush_interval``)
 
 **Out of scope (deferred)**
@@ -76,7 +78,7 @@ Supported and unsupported matrix
 +-------------------------------+------------------+---------------------------+
 | ``net_send`` self-events      | GPU buffer       | May flush on device       |
 +-------------------------------+------------------+---------------------------+
-| Gap junctions                 | Supported        | Sparse MPI voltage/ion    |
+| Gap junctions                 | Supported        | CPU fixed-step fallback   |
 +-------------------------------+------------------+---------------------------+
 | ``Vector.record``             | Supported        | Batch download API        |
 +-------------------------------+------------------+---------------------------+
@@ -154,8 +156,8 @@ not a rewrite of the whole native path.
 +---------------------------+--------------------+-------------------------------+
 | ``net_send`` buffer       | GPU                | GPU-capable in principle      |
 +---------------------------+--------------------+-------------------------------+
-| Gap junction transfer     | See gap phases     | Same (if gaps present)        |
-| (``setup_transfer``)      | below              |                               |
+| Gap junction transfer     | **CPU fixed-step   | Same (if gaps present)        |
+| (``setup_transfer``)      | fallback** (below) |                               |
 +---------------------------+--------------------+-------------------------------+
 | Recording flush           | GPU → host batch   | Same API                      |
 +---------------------------+--------------------+-------------------------------+
@@ -176,21 +178,22 @@ limitations (see matrix above).
 .. mermaid::
 
    flowchart TD
-     A[CPU: deliver_net_events] --> B[GPU: setup_tree_matrix + NMODL currents]
+     S{nrnthread_v_transfer_ registered?}
+     S -->|yes| CPUALL[CPU fixed-step: deliver, matrix, solve, update, gap transfer, lastpart]
+     S -->|no| A[CPU: deliver_net_events]
+     A --> B[GPU: setup_tree_matrix + NMODL currents]
      B --> C[GPU: tree solve solve_interleaved]
      C --> D{post_solve host fallback?}
      D -->|no| E[GPU: post_solve V, fast_imem, capacity]
      D -->|yes| F[CPU: post_solve nrn_update_voltage]
-     E --> G{gap transfer configured?}
-     F --> G
-     G -->|yes| H[CPU: gap gather, MPI if nhost>1, scatter in lastpart]
-     G -->|no| I[GPU: lastpart NET_RECEIVE + nonvint + vecplay]
-     H --> I
+     E --> I[GPU: lastpart NET_RECEIVE + nonvint + vecplay]
+     F --> I
      I --> J{download flush interval?}
      J -->|yes| K[batch_download_to_host]
      J -->|no| L[defer to next flush / psolve end]
      K --> M[advance step counter]
      L --> M
+     CPUALL --> M
 
 On this path ``nt.compute_gpu`` is set for integration. NMODL ``BREAKPOINT``
 currents (``nrn_cur_*`` OpenACC) and OpenACC axial assembly in ``setup_tree_matrix``
@@ -202,12 +205,22 @@ run on the GPU. **GPU tree solve** is ``solve_interleaved`` / ``solve_interleave
 during **lastpart** (after the solve/post-solve block). Cross-rank spike
 **scheduling** remains on CPU (see spike policy in :doc:`gpu-testing`).
 
-**Gap junction transfer** (``ParallelContext.setup_transfer``) is **not** gated on
-MPI or ``pc.nthread(n)>1`` alone. The diamond is true whenever
-``nrnthread_v_transfer_`` is registered (gap junctions or other partrans targets) —
-including single-process, single-thread models with gaps.
+**Gap junction transfer** (``ParallelContext.setup_transfer``) registers
+``nrnthread_v_transfer_`` whenever partrans targets exist — including
+single-process, single-thread models with gaps (not only when MPI or
+``pc.nthread(n)>1``).
 
-On CPU fixed-step, partrans uses three conceptual phases (``partrans.cpp``):
+Phase B native GPU **does not** run the hybrid GPU integration path above when
+``nrnthread_v_transfer_`` is set. ``nrn_fixed_step_thread`` in ``fadvance.cpp``
+dispatches the **full CPU fixed-step body** instead of ``fadvance_gpu.cpp`` until
+device-resident partrans gather/scatter is complete. ``gpu.enable=True`` still
+uploads mechanisms and state to device, but matrix setup, solve, post-solve, gap
+transfer, and lastpart execute on the CPU for correctness (validated by
+``test/gjtests/test_par_gj_native_gpu.py`` and ringtest ``-gap``). Runtime is
+therefore ~1× NEURON CPU for gap models, not the slower hybrid path seen on
+non-gap workloads.
+
+On that CPU path, partrans uses three phases (``partrans.cpp``):
 
 1. **Gather** source values into transfer buffers (``mpi_transfer``: ``outsrc_buf_[i]
    = *poutsrc_[i]``).
@@ -216,12 +229,11 @@ On CPU fixed-step, partrans uses three conceptual phases (``partrans.cpp``):
 3. **Scatter** to targets (``thread_transfer`` in ``nonvint``, per thread:
    ``*(ttd.tv[i]) = *(ttd.sv[i])``).
 
-Phase B native GPU: after post-solve, ``sync_gap_after_voltage_update`` pulls source
-voltages **GPU → host** so phase 1 can read them; thread 0 runs ``nrnmpi_v_transfer_``
-(MPI phase 2 when multi-rank); phase 3 still runs on the **CPU** inside lastpart
-``nonvint``. Gather and scatter are not on device yet — a future optimization could
-keep phases 1 and 3 on GPU and avoid host staging when **no MPI** (and possibly for
-intra-node thread transfers), but that is outside Phase B.
+A future optimization would restore the GPU integration subphases and move phases
+1 and 3 to device buffers, touching host/MPI only when ``nrnmpi_numprocs > 1``.
+The hybrid sync helpers in ``sync.cpp`` (``sync_gap_after_voltage_update``,
+``sync_gap_after_host_voltage_update``) remain for that path; they are not used on
+the Phase B CPU fallback dispatch.
 
 For DAE models, only the **matrix solve** subphase must swap to CPU ``sparse13``;
 other subphases are not inherently excluded by the subphase design, but Phase B