Update README

Author: aparnabg

README.md

@@ -40,7 +40,9 @@ The project is organized as a research pipeline:
 Code: `src/sailsprep/id_tracking_model/`
 
 This part of the repository generates detection and pose caches, tracks people
-across frames, and selects the target child track.
+across frames, and selects the target child track. See
+[`src/sailsprep/id_tracking_model/README.md`](src/sailsprep/id_tracking_model/README.md)
+for the full pipeline and command reference.
 
 Important modules:
 
@@ -57,7 +59,10 @@ Important modules:
 Code: `src/sailsprep/tracking_pose_model_testing/`
 
 This directory contains scripts for testing or wrapping individual tracking and
-pose-estimation methods:
+pose-estimation methods. See
+[`src/sailsprep/tracking_pose_model_testing/README.md`](src/sailsprep/tracking_pose_model_testing/README.md)
+for which scripts are single-video demos versus CSV/SLURM batch pipelines,
+and the corresponding install groups:
 
 - YOLO pose
 - HRNet
@@ -76,7 +81,10 @@ and visualized outputs.
 Code: `src/sailsprep/action_model_testing/`
 
 This directory contains the model-training and inference scripts used for action
-classification experiments.
+classification experiments. See
+[`src/sailsprep/action_model_testing/README.md`](src/sailsprep/action_model_testing/README.md)
+for the full list of model folders, and each model folder's own README for
+setup and usage specific to that model.
 
 Included model families:
 
@@ -87,18 +95,23 @@ Included model families:
 - `slow_fast/` - SlowFast finetuning.
 - `motionbert/` - MotionBERT pose/action pipeline.
 - `mstcn2/` - MS-TCN++ sequence model over extracted features.
-- `open_tad/` - temporal action detection experiments.
-- `pyskl/` - PySKL config/logit utilities.
-- `vlm_models/` - Qwen/Ovis vision-language model classifiers.
-- `feature_extraction/` - I3D/R(2+1)D and V-JEPA feature extraction.
-- `vjepa/` - vjepa model variants.
-- `dlc_action/` - DLC2Action data preparation workflow.
+- `open_tad/` - temporal action detection experiments (requires the OpenTAD
+  codebase).
+- `pyskl/` - PySKL config generation, training helpers, and logit fusion
+  (requires the PySKL codebase).
+- `vlm_models/` - Qwen2.5-VL/Ovis2 vision-language model classifiers.
+- `feature_extraction/` - I3D/R(2+1)D and V-JEPA2 feature extraction.
+- `vjepa/` - V-JEPA2 feature extraction, probe training, and fine-tuning
+  variants.
+- `dlc_action/` - DLC2Action data preparation and training pipeline.
 
 ### Fusion Models
 
 Code: `src/sailsprep/fusion_model/`
 
-Fusion scripts combine predictions from multiple model families.
+Fusion scripts combine predictions from multiple model families. See
+[`src/sailsprep/fusion_model/README.md`](src/sailsprep/fusion_model/README.md)
+for full usage.
 
 - `late_fusion/two_model.py` - late fusion between two prediction sources.
 - `late_fusion/three_model.py` - late fusion among three prediction sources.
@@ -111,7 +124,9 @@ Fusion scripts combine predictions from multiple model families.
 Code: `src/sailsprep/analysis/`
 
 Behavior-specific analysis scripts extract kinematic features from pose tracks
-and run statistical analyses. Behaviors include:
+and run statistical analyses. See
+[`src/sailsprep/analysis/README.md`](src/sailsprep/analysis/README.md) for
+run order, inputs, and outputs. Behaviors include:
 
 - walking
 - running
@@ -133,7 +148,9 @@ leave-one-subject-out classification.
 Code: `src/sailsprep/annotation/`
 
 The annotation tool is a FastAPI application with a browser UI for reviewing
-videos and assigning frame-level behavior labels.
+videos and assigning frame-level behavior labels. See
+[`src/sailsprep/annotation/README.md`](src/sailsprep/annotation/README.md)
+for the API endpoints and label set.
 
 ```bash
 uvicorn sailsprep.annotation.annotation:app --reload
@@ -159,35 +176,51 @@ poetry install --with dev
 poetry run pytest
 ```
 
-Many model pipelines require optional heavy dependencies. Install only the
-groups needed for the experiment you want to reproduce.
-
-Examples:
+Many model pipelines require optional heavy dependencies. `pyproject.toml`
+defines the following optional Poetry groups; install only the ones needed
+for the experiment you want to reproduce.
 
 ```bash
 # Pose/tracking experiments
 poetry install --with dev,pose-estimation,tracking
 
+# EntitySAM
+poetry install --with dev,entity-sam
+
+# ViTPose
+poetry install --with dev,vitpose
+
+# Clip-level tracker (MMDetection/MMPose-based)
+poetry install --with dev,clip_tracker
+
+# DLC2Action
+poetry install --with dev,dlc2action
+
 # Video Swin experiments
 poetry install --with dev,video_swin
 
-# VideoMAE / feature extraction style experiments
+# I3D / R(2+1)D / V-JEPA2 feature extraction
 poetry install --with dev,feature-extraction
 
-# Vision-language model experiments
+# Vision-language model experiments (Qwen2.5-VL / Ovis2)
 poetry install --with dev,vlm
 
 # OpenTAD experiments
 poetry install --with dev,opentad
 
+# Statistical analysis (statsmodels, optionally rpy2/pymc/arviz)
+poetry install --with dev,stats-analysis
+
 # Documentation
 poetry install --with docs
 ```
 
-Some third-party model stacks, especially PySKL, OpenTAD, MMDetection, MMPose,
-and CUDA-specific video libraries, may require separate environment setup on an
-HPC system. The job scripts in `jobs/` show the environments used for the
-original experiments.
+There is no Poetry group for VideoMAE2, InternVideo2, SlowFast, MotionBERT,
+MS-TCN++, or PySKL — those model folders list their own dependencies to
+`pip install` directly in their README. Some third-party model stacks,
+especially PySKL, OpenTAD, MMDetection, MMPose, and CUDA-specific video
+libraries, may require separate environment setup on an HPC system. The job
+scripts in `jobs/` show the environments used for the original experiments.
 
 ## Data Requirements
 
@@ -314,40 +347,35 @@ need to be generated.
 
 ### 5. Train Action Models
 
-Representative examples:
+The scripts in `action_model_testing/` mostly import their shared helpers with
+bare imports (e.g. `from common.utils import ...`, `from utils.bbox import
+...`), so each script is meant to be run with its own folder as the current
+directory, not as a `python -m` module. See the model's own README (linked
+from
+[`src/sailsprep/action_model_testing/README.md`](src/sailsprep/action_model_testing/README.md))
+for exact commands. Representative examples:
 
 ```bash
 # Video Swin full-video sliding-window classifier
-python -m sailsprep.action_model_testing.video_swin.sliding_window.video_swin_fullvideo_sliding \
-  --task loco \
-  --seed 42
+cd src/sailsprep/action_model_testing/video_swin/sliding_window
+python video_swin_fullvideo_sliding.py --task loco --seed 42
 
 # Video Swin binary N/A vs non-N/A classifier
-python -m sailsprep.action_model_testing.video_swin.sliding_window.video_swin_binary_sliding \
-  --task rmm \
-  --seed 42
+python video_swin_binary_sliding.py --task rmm --seed 42
 
 # Video Swin two-stage classifier
-python -m sailsprep.action_model_testing.video_swin.sliding_window.video_swin_twostage_joint \
-  --task loco \
-  --seed 42
+python video_swin_twostage_joint.py --task loco --seed 42
 
 # VideoMAE2 full-video sliding-window classifier
-python -m sailsprep.action_model_testing.videomae2.videomae2_fullvideo_sliding \
-  --task loco \
-  --seed 42
+cd ../../videomae2
+python videomae2_fullvideo_sliding.py --task loco --seed 42
 
 # VideoMAE2 two-stage classifier
-python -m sailsprep.action_model_testing.videomae2.videomae2_twostage_sliding \
-  --task rmm \
-  --seed 42
+python videomae2_twostage_sliding.py --task rmm --seed 42
 
 # MS-TCN++ over extracted features
-python -m sailsprep.action_model_testing.mstcn2.mstcn2 \
-  --label loco \
-  --feature_type i3d \
-  --action train \
-  --seed 42
+cd ../mstcn2
+python mstcn2.py --label loco --feature_type i3d --action train --seed 42
 ```
 
 Most reported model experiments are run over three seeds:
@@ -451,14 +479,22 @@ Common edits:
 
 Representative job scripts:
 
-- `jobs/action_model_testing/video_swin/sliding_window/video_swin_full_video_multi_job.sh`
-- `jobs/action_model_testing/video_swin/sliding_window/video_swin_full_video_multi_job_twostage.sh`
-- `jobs/action_model_testing/video_swin/sliding_window/full_video_window_job_video_swin.sh`
+- `jobs/action_model_testing/Video_Swin/sliding_window/video_swin_full_video_multi_job.sh`
+- `jobs/action_model_testing/Video_Swin/sliding_window/video_swin_full_video_multi_job_twostage.sh`
+- `jobs/action_model_testing/Video_Swin/sliding_window/full_video_window_job_video_swin.sh`
+- `jobs/action_model_testing/videomae2/videomae2_job.sh`
 - `jobs/action_model_testing/videomae2/videomae2_submit_all.sh`
+- `jobs/action_model_testing/internvideo2/internvideo2_finetune.sh`
+- `jobs/action_model_testing/motionbert/motionbert.sh`
 - `jobs/action_model_testing/mstcn2/mstcn2.sh`
 - `jobs/action_model_testing/slow_fast/slow_fast.sh`
+- `jobs/action_model_testing/slow_fast/ablation/slowfast_ablation_job.sh`
+- `jobs/action_model_testing/feature_extraction/i3d_r2p1d_feature_extracion.sh`
+- `jobs/action_model_testing/feature_extraction/vjepa_feature_extracion.sh`
 - `jobs/action_model_testing/vlm_models/clips/submit_qwen_clip.sh`
 - `jobs/action_model_testing/vlm_models/clips/submit_ovis_clip.sh`
+- `jobs/action_model_testing/pyskl/pyskl_job.sh`
+- `jobs/action_model_testing/pyskl/test_job.sh`
 - `jobs/fusion_model/pyskl/train_pyskl_sw.sh`
 - `jobs/fusion_model/pyskl/test_pyskl_sw.sh`
 - `jobs/analysis/analysis_job.sh`

src/sailsprep/action_model_testing/README.md

@@ -1,82 +1,52 @@
-# pyskl custom pipeline
-
-This repo does **not** contain the [pyskl](https://github.com/kennymckormick/pyskl)
-codebase itself. It's a set of custom scripts (data conversion, config
-generation, SLURM job scripts, and late-fusion training) that must be dropped
-into an actual clone of the upstream `pyskl` repo to run.
-
-## Setup
-
-1. Clone the upstream repo and install it (see its README for full
-   environment setup, e.g. mmcv, torch, etc.):
-
-   ```bash
-   git clone https://github.com/kennymckormick/pyskl.git
-   cd pyskl
-   pip install -e .
-   ```
-
-2. Copy the contents of this repo into the root of that clone:
-
-   ```bash
-   cp -r scripts fusion jobs /path/to/pyskl/
-   ```
-
-   (`tools/dist_train.sh`, `tools/dist_test.sh`, and the `pyskl` Python
-   package used by `extract_logits.py` all come from the upstream clone —
-   they are not included here.)
-
-## Pipeline / run order
-
-All commands below are run **from the root of the upstream `pyskl` clone**
-(i.e. `cd /path/to/pyskl` first), after copying these files in — not from
-inside `scripts/` or `fusion/`. The scripts read/write paths like
-`configs/custom/...` and `work_dirs/...` relative to your current directory,
-not relative to the script's own location.
-
-1. **Convert raw pose JSONs to pyskl's `.pkl` annotation format:**
-
-   ```bash
-   python scripts/pyskl_dataset.py --task rmm  --out /path/to/rmm_pyskl.pkl
-   python scripts/pyskl_dataset.py --task loco --out /path/to/loco_pyskl.pkl
-   ```
-
-2. **Generate model configs** (writes into `configs/custom/<model>_<dataset>/`):
-
-   ```bash
-   python scripts/generate_stgcnpp_configs.py
-   python scripts/generate_ctrgcn_configs.py
-   python scripts/generate_posec3d_configs.py
-   ```
-
-3. **Train** all model/feature combinations across 3 seeds via SLURM
-   (edit `WORKSPACE` in the script, or set it as an env var, to match your
-   cluster paths):
-
-   ```bash
-   bash jobs/pyskl/pyskl_job.sh
-   ```
-
-4. **Test** the best checkpoint from each run:
-
-   ```bash
-   bash jobs/pyskl/test_job.sh
-   ```
-
-5. **Extract per-model logits** on val + test splits, for fusion:
-
-   ```bash
-   python fusion/extract_logits.py --dataset rmm  --split val
-   python fusion/extract_logits.py --dataset rmm  --split test
-   python fusion/extract_logits.py --dataset loco --split val
-   python fusion/extract_logits.py --dataset loco --split test
-   ```
-
-6. **Train the fusion MLP** over the concatenated logits and evaluate:
-
-   ```bash
-   python fusion/train_mlp_fusion.py --dataset rmm
-   python fusion/train_mlp_fusion.py --dataset loco
-   ```
-
-
+# Action Model Testing
+
+Training and inference scripts for the action-recognition experiments run on
+the SAILS locomotion (`loco`) and repetitive motor movement (`rmm`) tasks.
+Each subfolder below is a separate model family with its own inputs, and most
+have their own README with setup and run instructions.
+
+## Model folders
+
+| Folder | Model | README |
+|---|---|---|
+| `video_swin/` | Video Swin Transformer (clip-based, binary/full-video/two-stage sliding window) | [video_swin/README.md](video_swin/README.md) |
+| `videomae2/` | VideoMAE V2 finetuning (per-clip, full-video sliding window, two-stage) | [videomae2/README.md](videomae2/README.md) |
+| `internvideo2/` | InternVideo2-6B finetuning | [internvideo2/README.md](internvideo2/README.md) |
+| `slow_fast/` | SlowFast finetuning (single clip pipeline + ablation experiments) | [slow_fast/README.md](slow_fast/README.md) |
+| `motionbert/` | MotionBERT 2D->3D pose lifting + skeleton action recognition | [motionbert/README.md](motionbert/README.md) |
+| `mstcn2/` | MS-TCN++ frame-level action segmentation over pre-extracted features | [mstcn2/README.md](mstcn2/README.md) |
+| `feature_extraction/` | I3D, R(2+1)D, and V-JEPA2 per-frame feature extraction | [feature_extraction/README.md](feature_extraction/README.md) |
+| `vjepa/` | V-JEPA2 feature extraction, attentive-probe training, and fine-tuning variants | [vjepa/README.md](vjepa/README.md) |
+| `open_tad/` | Feature-based temporal action detection with OpenTAD | [open_tad/README.md](open_tad/README.md) |
+| `pyskl/` | PySKL skeleton-based action recognition configs, training helpers, and logit fusion | [pyskl/README.md](pyskl/README.md) |
+| `vlm_models/` | Qwen2.5-VL / Ovis2 vision-language model classifiers | [vlm_models/README.md](vlm_models/README.md) |
+| `dlc_action/` | DLC2Action pipeline for pose-based locomotion classification | [dlc_action/README.md](dlc_action/README.md) |
+
+## Tasks
+
+Nearly every script here trains for one of two label sets from the SAILS
+annotation CSVs:
+
+| Task flag | Annotation column | Classes |
+|---|---|---|
+| `loco` | `Locomotion` | Crawling, Cruising, Running, Vehicle, Walking |
+| `rmm` | `Repetitive_Motor_Movements` | Hands_flapping, Jumping, Rocking, Spinning |
+
+## Conventions across these scripts
+
+- Most scripts read a single split CSV (with `video_path`, `label_path`, and a
+  `split` column of `train`/`val`/`test`) and take `--task`/`--label` plus
+  `--seed` as CLI flags. Paths to that CSV, along with checkpoint/output
+  directories, are set as constants near the top of each script rather than
+  passed as flags — edit those constants for your environment.
+- Reported experiments are generally run over three seeds: `42`, `123`, `456`.
+- Several folders import shared helpers with bare imports (e.g.
+  `from common.utils import ...`, `from utils.bbox import ...`). Those scripts
+  must be run with their own folder as the working directory, not with
+  `python -m`. Each model's README states which convention it uses.
+- `pyskl/` and `open_tad/` are standalone scripts that operate inside a
+  separate upstream repo checkout (PySKL, OpenTAD) rather than importing
+  `sailsprep` directly — see their READMEs for the copy-in setup.
+
+Corresponding SLURM job scripts for these models live under
+`jobs/action_model_testing/`.