Skip to content

Commit 08835d0

Browse files
m9hclaude
andcommitted
Update CLAUDE.md: Slurm, related work, DANDI AI contribution gaps
Add Slurm submission docs, related work section (Magland et al. 2025 DANDI AI Notebooks, TVB projects, Beggs lab), and 5 specific gaps where BL-1 can contribute to the DANDI AI pipeline: simulation comparison, MEA-specific analysis, spike format fixes, differentiable fitting, and local GPU cost reduction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8d27ef7 commit 08835d0

1 file changed

Lines changed: 89 additions & 0 deletions

File tree

CLAUDE.md

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,3 +162,92 @@ python scripts/nsg_submit.py --download JOB_ID # get results
162162
```
163163

164164
Tool: `GPU_PY_EXPANSE` (Python on Expanse GPUs, V100s)
165+
166+
## Slurm (Local DGX Spark)
167+
168+
```bash
169+
# Submit per-recording training array (33 jobs)
170+
sbatch --array=0-32 scripts/slurm_train_sharf.sh
171+
172+
# Submit pooled condition training
173+
sbatch --job-name=bl1-pool-baseline --partition=gpu --gres=gpu:1 --mem=32G --time=02:00:00 \
174+
--wrap=".venv/bin/python scripts/train_pooled.py --condition baseline"
175+
176+
# Monitor
177+
squeue -o "%.8i %.20j %.4t %.10M %R"
178+
```
179+
180+
## Related Work & Collaboration Opportunities
181+
182+
### DANDI AI Notebooks (Magland et al. 2025)
183+
184+
Paper: "Facilitating analysis of open neurophysiology data on the DANDI Archive
185+
using large language model tools" (bioRxiv 2025.07.17.663965v3)
186+
187+
They built LLM-powered tools for automated DANDI dataset exploration and notebook
188+
generation. GPT-4.1 for chat exploration, Claude Sonnet 4 for notebook generation.
189+
Cost: ~$1.15/notebook. Tested on 12 datasets with expert review.
190+
191+
**Code:**
192+
- Notebook generator: https://github.com/dandi-ai-notebooks/dandi-ai-notebooks-study
193+
- Dandiset Explorer: https://github.com/dandi-ai-notebooks/dandiset-explorer
194+
- Generated notebooks: https://zenodo.org/records/16033603
195+
196+
**Gaps where BL-1 can contribute (see below).**
197+
198+
### Virtual Brain Projects (TVB)
199+
200+
- **tvboptim** (https://github.com/virtual-twin/tvboptim): JAX brain network simulation
201+
with gradient-based optimization. Wong-Wang, Jansen-Rit, Epileptor models. Uses optax.
202+
Relevant: `Parameter()` marking system, BOLD monitor, diffrax integration.
203+
- **vbjax** (https://github.com/ins-amu/vbjax): Lean JAX toolkit for virtual brain
204+
modeling. Euler/Heun/RK4 integrators, custom_vjp sparse matmul, delay helpers,
205+
BOLD/EEG monitors. Relevant: differentiable sparse ops, Heun integration.
206+
207+
### Beggs Lab (Indiana University)
208+
209+
John Beggs — discoverer of neuronal avalanches (Beggs & Plenz 2003). Book: "The Cortex
210+
and the Critical Point" (MIT Press 2022, open access). BL-1 validates against his
211+
published criticality metrics (branching ratio, -3/2 exponent). The criticality sweep
212+
notebook (notebooks/03_criticality_sweep.ipynb) demonstrates his theory in BL-1.
213+
214+
## Contribution Gaps: DANDI AI Notebooks x BL-1
215+
216+
The Magland et al. pipeline has specific gaps that BL-1 addresses:
217+
218+
### 1. No simulation comparison (biggest gap)
219+
Their notebooks show real data but never compare to a model. BL-1 can generate
220+
a "simulated counterpart" for any DANDI cortical culture recording — run a matched
221+
simulation with extracted targets and produce side-by-side rasters/statistics.
222+
This transforms their descriptive notebooks into model-validation notebooks.
223+
224+
### 2. No MEA-specific analysis
225+
Their tool is generic across all NWB datasets. For cortical culture MEA data
226+
specifically, BL-1 has specialized analysis: burst detection (Wagenaar method),
227+
criticality metrics, STP dynamics, E/I balance estimation. These could be
228+
contributed as "domain plugins" for their notebook generator.
229+
230+
### 3. Spike time format issues unhandled
231+
We discovered that DANDI 001611 stores spike times as sample indices (not seconds)
232+
and Sharf 2022 uses compound HDF5 datasets. Their pipeline likely hits the same
233+
issues. Our activity-window-aware loading and format auto-detection could be
234+
contributed upstream to pynwb or to their inspection tools.
235+
236+
### 4. No differentiable fitting
237+
Their notebooks are read-only analysis. BL-1's training pipeline could be
238+
integrated: after exploring a dataset, the agent generates a training script
239+
that fits BL-1 weights to match the recording's statistics. This is the
240+
"analysis → model → prediction" loop that their system lacks.
241+
242+
### 5. Cost/scale mismatch
243+
Their pipeline costs $1.15/notebook but relies on cloud LLMs. BL-1's analysis
244+
runs locally on GPU at zero marginal cost. Combining their LLM exploration with
245+
BL-1's local GPU analysis could reduce per-dataset cost while adding simulation.
246+
247+
### Concrete contribution plan
248+
1. Submit BL-1's MEA analysis functions as a PR to their notebook generator
249+
2. Add a "simulation comparison" template that their agent can use for
250+
cortical culture datasets
251+
3. Upstream our NWB loading fixes (sample-index detection, compound datasets)
252+
to pynwb or their get_nwbfile_info tool
253+
4. Propose a joint notebook: "From DANDI recording to fitted BL-1 simulation"

0 commit comments

Comments
 (0)