team I9D6: adding full set of scripts for afni_proc.py, etc.

narps_open/pipelines/team_I9D6/README.md

@@ -0,0 +1,42 @@
+# apaper_highlight_narps
+Scripts related to the following paper:
+
+  **Highlight Results, Don't Hide Them: Enhance interpretation, reduce
+  biases and improve reproducibility** \
+  by Paul A Taylor, Richard C Reynolds, Vince Calhoun, Javier
+  Gonzalez-Castillo, Daniel A Handwerker, Peter A Bandettini, Amanda F
+  Mejia, Gang Chen (2023) \
+  Neuroimage 274:120138. doi: 10.1016/j.neuroimage.2023.120138 \
+  https://pubmed.ncbi.nlm.nih.gov/37116766/
+
+---------------------------------------------------------------------------
+The input data comes from the NARPS project (Botvinik-Nezer et al., 2020): \
+  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7771346/ \
+This paper uses both the raw, unprocessed data as well as the
+participating teams' results, which were uploaded to NeuroVault (see
+the same paper for those details).
+
+---------------------------------------------------------------------------
+Essentially all scripts here use AFNI; one also uses FreeSurfer.
+
+The `scripts_biowulf` directory contains the main processing scripts,
+including:
++ Checking the data
++ Estimating nonlinear alignment to template space and skullstripping
+  with `@SSwarper`
++ Full FMRI time series processing through regression modeling and QC
+  generation with `afni_proc.py`
++ Group level modeling: both voxelwise (with cluster calcs) and
+  ROI-based (using `RBA`, in particular) 
+
+... and more.
+
+The `scripts_suppl_proc_vox` directory contains supplementary scripts
+for making images of the above-processed data, mainly for figure
+generation.
+
+The `scripts_suppl_proc_teams` directory contains scripts for
+processing the group-level results of the original participating Teams
+in the NARPS project.  Those public datasets were downloaded from
+NeuroVault.  The scripts make a lot of images and perform some simple
+similarity analyses.

narps_open/pipelines/team_I9D6/scripts_biowulf/README.txt

@@ -0,0 +1,118 @@
+This directory contains the main processing scripts for analyzing the
+raw NARPS data (task FMRI on two groups of 54 subj each).  These
+scripts include preliminary checks, full single subject processing
+through regression modeling, and a couple different forms of group
+level modeling.
+
+These scripts were run on the NIH's Biowulf computing cluster, hence
+there are considerations for batch processing with the slurm system.
+Each processing step is divided into a pair of associated scripts:
+
++ **do_SOMETHING.tcsh**: a script that mostly contains the processing
+  options and commands for a single subject, with subject ID and any
+  other relevant information passed in as a command line argument when
+  using it.  Most of the lines at the top of the file set up the
+  processing, directory structure (most every step generates a new
+  filetree called `data_SOMETHING/`), and usage of a scratch disk for
+  intermediate outputs.  At some point, actual processing commands are
+  run, and then there is a bit of checking, copying from the scratch
+  disk and verifying permissions, and then exiting.
+
++ **run_SOMETHING.tcsh**: mostly manage the group-level aspects of
+  things, to set up processing over all subjects of interest and start
+  a swarm job running on the cluster.
+
+---------------------------------------------------------------------------
+The enumeration in script names is to help to organize the order of
+processing (kind of a Dewey Decimal-esque system).  Gaps between
+numbers are fine---they just leave space for other processing steps to
+have been inserted, as might be necessary.  Loosely, each "decade" of
+enumeration corresponds to a different stage of processing:
+
++ the 00s are preliminary checks
++ the 10s are preliminary processing steps (parcellation,
+  skullstripping and nonlinear alignment for the anatomical)
++ the 20s are afni_proc.py processing of the FMRI data
++ the 50s are running some automatic quality control (QC) with
+  gen_ss_review_table.py
++ the 60s run voxelwise processing with ETAC (used in previous work,
+  not here, but included for fun) and 3dttest++
++ the 70s contain region-based processing, setting up for and
+  eventually running RBA.
+
+---------------------------------------------------------------------------
+
+The script details (recall: just listing the do_* scripts, since the
+run_* ones just correspond to a swarming that step):
+
++ do_01_gtkyd.tcsh
+  "Getting To Know Your Data" step, getting preliminary info about
+  voxel size, number of time points, and other fun properties;
+  consistency checks
+
++ do_02_deob.tcsh
+  Deoblique the anatomical volumes (so FS output matches with later
+  outputs)
+
++ do_12_fs.tcsh
+  Run FreeSurfer for anatomical parcellations
+
++ do_13_ssw.tcsh
+  Run AFNI's @SSwarper (SSW) for skullstripping (ss) and nonlinear
+  alignment (warping) to MNI template space
+
++ do_15_events.tcsh
+  Stimulus timing file creation
+
++ do_22_ap_task.tcsh
+  Run AFNI's afni_proc.py (AP) for full processing of the FMRI data,
+  through single subject regression modeling (here, without blurring,
+  to be used for ROI-based analyses); uses results of earlier stages;
+  also produces QC HTML
+
++ do_23_ap_task_b.tcsh
+  Run AFNI's afni_proc.py (AP) for full processing of the FMRI data,
+  through single subject regression modeling (here, with blurring, to
+  be used for voxelwise analyses); uses results of earlier stages;
+  also produces QC HTML
+
++ do_52_ap_qc.tcsh 
+  Run some automatic QC criteria selections on the "22_ap_task" output
+  with AFNI's gen_ss_review_table.py
+
++ do_53_ap_qc.tcsh
+  Run some automatic QC criteria selections on the "23_ap_task_b"
+  output with AFNI's gen_ss_review_table.py
+
++ do_61_etac_1grp.tcsh
+  Run ETAC for group level analysis on the "22_ap_task" output (not
+  used here; from a previous study, but script included); this applies
+  to the single-group hypotheses
+
++ do_62_etac_2grp.tcsh
+  Run ETAC for group level analysis on the "22_ap_task" output (not
+  used here; from a previous study, but script included); this applies
+  to the group contrast hypothesis
+
++ do_63_ttest_1grp.tcsh
+  Run 3dttest++ for group level analysis on the "23_ap_task_b" output,
+  for simple voxelwise analysis; this applies to the single-group
+  hypotheses
+
++ do_64_csim_1grp.tcsh
+  Run "3dttest++ -Clustsim ..." for getting standard cluster threshold
+  size for the t-test results
+
++ do_71a_rba_prep.tcsh
+  Prepare to run AFNI's RBA on the "22_ap_task" output, dumping
+  average effect estimate information for each ROI (for both of the
+  atlases used)
+
++ do_71b_rba_comb.tcsh
+  Prepare to run AFNI's RBA on the "22_ap_task" output, combining ROI
+  information across the group into a datatable (for both of the
+  atlases used)
+
++ do_71c_rba.tcsh 
+  Run AFNI's RBA on the "22_ap_task" output, using the created
+  datatable (for both of the atlases used)

narps_open/pipelines/team_I9D6/scripts_biowulf/do_01_gtkyd.tcsh

@@ -0,0 +1,174 @@
+#!/bin/tcsh
+
+# GTKYD: Getting To Know Your Data
+# -> preliminary info and QC
+
+# NOTES
+#
+# + This is a Biowulf script (has slurm stuff)
+# + Run this script in the scripts/ dir, via the corresponding run_*tcsh
+# + NO session level here.
+
+# ----------------------------- biowulf-cmd ---------------------------------
+# load modules
+source /etc/profile.d/modules.csh
+module load afni 
+
+# set N_threads for OpenMP
+# + consider using up to 4 threads, because of "-parallel" in recon-all
+setenv OMP_NUM_THREADS $SLURM_CPUS_PER_TASK
+
+# compress BRIK files
+setenv AFNI_COMPRESSOR GZIP
+
+# important for 'clean' output here
+setenv AFNI_NO_OBLIQUE_WARNING YES
+
+# initial exit code; we don't exit at fail, to copy partial results back
+set ecode = 0
+# ---------------------------------------------------------------------------
+
+# ---------------------------------------------------------------------------
+# top level definitions (constant across demo)
+# ---------------------------------------------------------------------------
+
+# labels: different than most other do*.tcsh scripts
+set file_all_epi   = $1
+set file_all_anat  = $2
+
+set template       = MNI152_2009_template_SSW.nii.gz 
+
+# upper directories
+set dir_inroot     = ${PWD:h}                        # one dir above scripts/
+set dir_log        = ${dir_inroot}/logs
+set dir_store      = /data/SSCC_NARPS/globus_sync/ds001205 # data on biowulf 
+set dir_basic      = ${dir_store}                    # holds all sub-* dirs
+set dir_gtkyd      = ${dir_inroot}/data_01_gtkyd
+
+# --------------------------------------------------------------------------
+# data and control variables
+# --------------------------------------------------------------------------
+
+# dataset inputs
+set all_anat      = `cat ${file_all_anat}`
+set all_epi       = `cat ${file_all_epi}`
+
+# 3dinfo params
+set all_info = ( n4 orient ad3 obliquity tr slice_timing \
+                 datum )
+
+# 3dBrickStat params
+set all_bstat = ( min max )
+
+# nifti_tool fields
+set all_nfield = ( datatype sform_code qform_code )
+
+
+# check available N_threads and report what is being used
+set nthr_avail = `afni_system_check.py -disp_num_cpu`
+set nthr_using = `afni_check_omp`
+
+echo "++ INFO: Using ${nthr_avail} of available ${nthr_using} threads"
+
+# ----------------------------- biowulf-cmd --------------------------------
+# 
+# *** not used here ***
+#
+# ---------------------------------------------------------------------------
+
+# ---------------------------------------------------------------------------
+# run programs
+# ---------------------------------------------------------------------------
+
+# report per data dir
+\mkdir -p ${dir_gtkyd}/anat
+\mkdir -p ${dir_gtkyd}/func
+
+# report both individual columns, and sort+uniq ones
+foreach info ( ${all_info} )
+    echo "++ 3dinfo -${info} ..."
+
+    set otxt = ${dir_gtkyd}/anat/rep_info_${info}_su.dat
+    echo "# 3dinfo -${info}" > ${otxt}
+    3dinfo -${info} ${all_anat} | sort | uniq >> ${otxt}
+
+    set otxt = ${dir_gtkyd}/anat/rep_info_${info}_detail.dat
+    echo "# 3dinfo -${info}" > ${otxt}
+    3dinfo -${info} -prefix ${all_anat} >> ${otxt}
+
+    set otxt = ${dir_gtkyd}/func/rep_info_${info}_su.dat
+    echo "# 3dinfo -${info}" > ${otxt}
+    3dinfo -${info} ${all_epi} | sort | uniq >> ${otxt}
+
+    set otxt = ${dir_gtkyd}/func/rep_info_${info}_detail.dat
+    echo "# 3dinfo -${info}" > ${otxt}
+    3dinfo -${info} -prefix ${all_epi} >> ${otxt}
+end
+
+if ( ${status} ) then
+    set ecode = 1
+    goto COPY_AND_EXIT
+endif
+
+
+# only sort+uniq at the moment
+foreach nfield ( ${all_nfield} )
+    echo "++ nifti_tool -disp_hdr -field ${nfield} ..."
+
+    set otxt = ${dir_gtkyd}/anat/rep_ntool_${nfield}_su.dat
+    echo "# nifti_tool -disp_hdr -field ${nfield}" > ${otxt}
+    nifti_tool -disp_hdr -field ${nfield} -quiet -infiles ${all_anat} \
+        | sort | uniq >> ${otxt}
+
+    set otxt = ${dir_gtkyd}/func/rep_ntool_${nfield}_su.dat
+    echo "# nifti_tool -disp_hdr -field ${nfield}" > ${otxt}
+    nifti_tool -disp_hdr -field ${nfield} -quiet -infiles ${all_epi} \
+        | sort | uniq >> ${otxt}
+end
+
+# report both individual columns, and sort+uniq ones
+foreach bstat ( ${all_bstat} )
+    echo "++ 3dBrickStat -slow -${bstat} ..."
+
+    set otxt = ${dir_gtkyd}/anat/rep_brickstat_${bstat}_detail.dat
+    foreach dset ( ${all_anat} )
+        set val  = `3dBrickStat -slow -${bstat} ${dset}`
+        set name = `3dinfo -prefix ${dset}`
+        printf "%12s %12s\n" "${val}" "${name}" >> ${otxt}
+    end
+
+    set otxt_su = ${dir_gtkyd}/anat/rep_brickstat_${bstat}_su.dat
+    cat ${otxt} | awk '{print $1}' | sort -n | uniq > ${otxt_su}
+
+    set otxt = ${dir_gtkyd}/func/rep_brickstat_${bstat}_detail.dat
+    foreach dset ( ${all_epi} )
+        set val  = `3dBrickStat -slow -${bstat} ${dset}`
+        set name = `3dinfo -prefix ${dset}`
+        printf "%12s %12s\n" "${val}" "${name}" >> ${otxt}
+    end
+
+    set otxt_su = ${dir_gtkyd}/func/rep_brickstat_${bstat}_su.dat
+    cat ${otxt} | awk '{print $1}' | sort -n | uniq > ${otxt_su}
+
+end
+
+echo "++ done proc ok"
+
+# ---------------------------------------------------------------------------
+
+COPY_AND_EXIT:
+
+# ----------------------------- biowulf-cmd --------------------------------
+# 
+# *** not used here ***
+#
+# ---------------------------------------------------------------------------
+
+if ( ${ecode} ) then
+    echo "++ BAD FINISH: GTKYD (ecode = ${ecode})"
+else
+    echo "++ GOOD FINISH: GTKYD"
+endif
+
+exit ${ecode}
+