Skip to content

README.md

Latest commit

 

History

History
307 lines (226 loc) · 13.2 KB

README.md

File metadata and controls

307 lines (226 loc) · 13.2 KB

AdipoInsights

Is a command-line pipeline for adipocyte detection and analysis from .ndpi digital histological images (DHIs) of adipocyte depots. It performs automated segmentation, quantification, and analysis of adipocytes. It supports subcutaneous (sWAT), perigonadal (pWAT), and inguinal (iWAT) white adipose tissues, producing both numerical data and visual summaries for downstream analysis.

🚀 Features

🔹 Core Pipeline

  • The .ndpi should have good quality and high resolution (20× magnification or above) for good detection of adipocytes
  • Accepts a single .ndpi file, a directory of .ndpi slides, a CSV list of .ndpi paths, or a comma-separated list on the command line
  • Generates TIFFs, tissue masks, cropped regions, and adipocyte detection outputs
  • Configurable for subcutaneous white adipose tissue (sWAT) or perigonadal white adipose tissue (pWAT)
  • Automatic filename sanitization: replaces any non-alphanumeric character with _, collapses repeats, and trims

🔹 Adipo Viewer (Interactive Analysis Platform)

  • Streamlit-based interface to explore results interactively
  • Accepts either:
    • Folder path containing outputs, or
    • Uploaded .csv + image files
  • Visualizes histograms, box/violin plots, scatter plots
  • Built-in filters for outlier_score and mean_dist
  • Statistical tests: Welch t-test, Mann–Whitney U, KS, ANOVA/LM
  • Optional metadata integration (case_id, Sex, Genotype, or any columns)

🛠️ Requirements

  1. Python>=3.9
  2. ndpisplit -> move file to /usr/local/bin
  3. TensorFlow / Keras (required by segmentation models)

📁 Repository Structure

AdipoInsights/
├── Analyzer/
│   ├── configs/                     # JSON config files
│   ├── data_extraction/             # Scripts for data extraction
│   ├── optimize/                    # Optimization scripts
│   ├── segmentation/                # Segmentation model code
│   ├── training/                    # Training scripts and notebooks
│   └── utils/                       # Utility functions and helpers
├── adipo_viewer/                    # Streamlit-based interactive platform
│   ├── app.py                       # Main application entry point
│   ├── main_app.py                  # Page controllers and rendering logic
│   ├── config.py                    # Configuration and constants
│   ├── models.py                    # Data models and structures
│   ├── data_loaders.py              # Data loading and assembly
│   ├── visualizations.py            # Plotting and chart generation
│   ├── statistics.py                # Statistical tests and analysis
│   ├── ui_components.py             # Reusable UI components
│   ├── ui_styles.py                 # UI styling and theming
│   ├── image_processing.py          # Image handling for outlier viz
│   └── requirements.txt             # Viewer dependencies
├── ndpisplit/                       # Helpers to split large .ndpi files
├── parent_dir/
│   └── models/                      # Pretrained models (download via Zenodo)
├── requirements.txt                 # Full pipeline dependencies
├── script_notebooks/
│   ├── run_adipocytes_pipeline.sh   # Bash wrapper for CLI pipeline
│   └── testing_WAT_inf.py           # Main Python inference script
└── README.md

⚙️ Configuration

  1. Model directory: Place your pretrained models under models:

    • adipocytes_IsolationForest.pkl
    • skin_tissue_segmentation.ckpt
    • wat_tissue_segmentation.ckpt

    Model weights download: You can download the pretrained model weights from Zenodo: https://doi.org/10.5281/zenodo.15321697

  2. Config files: Update thresholds and parameters in Analyzer/configs/*.json if needed.

🔧 Config File Parameters

Each JSON config (e.g., Analyzer/configs/subcutaneous_wat.json) defines key parameters that you can tweak directly:

  • tissue_type: Type of WAT to process. Options: "sWAT" or "pWAT".
  • shape_ratio_thresh: Maximum ratio of minor/major axis for cell shape filtering (lower values enforce rounder cells).
  • max_20xsize: Maximum allowed pixel area at 20× magnification; cells larger than this are ignored.
  • device: Compute device for models. Use "cpu" or e.g. "gpu" if you have a GPU, or "mps" for macOS.
  • num_tissues: Number of separate tissue regions to segment per slide.
  • edge_detector: Method for edge detection ("canny", "otsu", etc.).
  • cell_segmentation_interval_binary: [min, max] pixel-area range for initial binary segmentation.
  • cell_segmentation_interval_binary_inv: [min, max] range for inverse segmentation mask.
  • min_adipocyte_intensity: Normalized intensity threshold (0–1) for removal of non-adipocyte structures.
  • outlier_threshold: Isolation forest outlier cutoff for cell outlier removal.

You can edit these values directly in the JSON to customize processing without changing any code.

⚙️ Installation

Using miniconda

  1. Clone github repository to local:

    git clone git@github.com:ExperimentalGenetics/AdipoInsights.git

  2. Using conda create a virtual environment

    a. Install Miniconda (if required)

    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
source miniconda3/etc/profile.d/conda.sh

    b. Create conda environment and install dependencies

    conda create --name wat -y python=3.9
conda activate wat
pip install -r requirements.txt

🏃‍♂️ Usage

First go into the cloned directory:

cd /path/to/AdipoInsights

Running the Pipeline

1. Directory of multiple DHIs

script_notebooks/run_adipocytes_pipeline.sh \
  /path/to/ndpi_folder \
  /path/to/output_dir \
  sWAT \
  /path/to/parent_dir

2. CSV listing slides

Create a CSV (slides.csv) with one column of .ndpi paths:

ndpi_path
/data/A.ndpi
/data/B.ndpi

Run:

script_notebooks/run_adipocytes_pipeline.sh /path/to/slides.csv /path/to/output sWAT /path/to/parent_dir

3. Comma-separated list

script_notebooks/run_adipocytes_pipeline.sh "A.ndpi,B.ndpi,C.ndpi" /path/to/output_dir sWAT /parent_dir

4. Single file

script_notebooks/run_adipocytes_pipeline.sh /data/slideX.ndpi /path/to/output_dir pWAT /parent_dir

📝 Pipeline Output

For each slide, the pipeline creates a folder named after the sanitized filename. Inside you will find:

  • *_fixed.tif, *_resized.tif – intermediate images
  • *_wat.png, *_wat_post.png – tissue masks
  • *_x5_cropped.tif, *_x20_cropped.tif – cropped ROIs
  • *_wat_cropped.png – final mask
  • *_shape_data.npz – raw adipocyte shape data
  • *_mean.csv, *_all.csv – aggregated and per-cell features
  • *_x20_wat.jpg, *_cells.jpg – visualizations

🧭 Adipo Viewer — Interactive Analysis Platform

Once your pipeline generates the *_all.csv and *_mean.csv files, you can explore results using the Adipo Viewer — a comprehensive Streamlit-based analysis platform.

▶️ Launch the Viewer

cd adipo_viewer
streamlit run app.py

This launches the app at http://localhost:8501.

📋 Platform Overview

The Adipo Viewer has three main tabs:

1. 📊 Analysis Tab

Six collapsible sections for comprehensive visualization:

  • Group Summary: Adipocyte counts and sample sizes by Sex × Genotype
  • Cell-level Distributions: Box plots of features across groups
  • Per-Mouse Analysis: Side-by-side comparisons for males vs. females
  • Stratified Histograms: 4-panel view (Mutants/Controls × Males/Females)
  • Mouse-Level Analysis: Scatter plots and box plots with independent feature selectors
  • Spatial Heatmaps: Visualize cell property distributions by location

2. 🔍 Outlier Analysis Tab

  • Histogram of cells per outlier score bin (configurable bins)
  • Bin inspection with data tables and image visualization
  • Bounding box overlays on example images

3. 📈 Statistics Tab

Cell-Level Tests: KS test, Cohen's d, t-Test, Two-way ANOVA
Sample-Level Tests: Mann-Whitney, Two-way ANOVA, Cohen's d, Linear Models with covariate adjustment (Age/Bodyweight)

Features: Automatic p-value highlighting, effect size interpretation, descriptive statistics

🎛️ Sidebar Controls

📁 Data Source: Choose folder path or upload files
📊 Metadata (optional): Upload CSV, specify path, or use inline editor. Supports case_id, Sex, Genotype, Age, Bodyweight with automatic column mapping.
🔍 Filters: Outlier score threshold (≥) and mean distance threshold (≤)
🎯 Sample Selection: Select all or choose specific samples via multi-select dropdown

💡 Usage Tips

  1. Start with Group Summary: Get an overview of your data distribution
  2. Use Filters: Refine your dataset by adjusting outlier score and mean distance thresholds
  3. Explore Collapsible Sections: Each section can be expanded/collapsed for focused analysis
  4. Metadata is Optional: The viewer works without metadata but provides richer analysis with it
  5. Covariate Analysis: If Age or Bodyweight data is available, ANOVA automatically shows adjusted models
  6. Independent Plot Controls: In mouse-level analysis, scatter and box plots have separate feature selectors
  7. Outlier Investigation: Use the Outlier tab to visually inspect cells with specific outlier scores
  8. Statistical Power: Use cell-level tests for large sample effects, sample-level for biological interpretation

🧹 Cleanup

If you use the upload option, clear temporary staged files using "🗑️ Clear uploaded files" in the sidebar.

🧱 Example Visuals

Below are examples of the visualizations available through the Adipo Viewer interface:

Visualization Description
📊 Box Plots Compare per-cell features such as area, perimeter, or solidity between Genotype × Sex groups with outlier detection.
🐭 Per-Mouse Analysis Side-by-side comparison of features across individual mice, split by sex with color-coded genotypes.
📈 Scatter Plots Explore relationships between sample-level variables with regression lines and confidence intervals.
📉 Histograms 4-panel stratified view showing distributions across all Sex × Genotype combinations.
🗺️ Spatial Heatmaps Visualize spatial distribution of cell properties using bounding box coordinates.
🔬 Outlier Inspection Interactive bin selection with visual examples showing cell images with bounding box overlays.
🧮 Statistical Tests Comprehensive test suite with automatic p-value highlighting and effect size interpretation.
📊 Covariate Analysis Adjust for Age and Bodyweight while testing Sex and Genotype effects.

🐛 Troubleshooting

Below are common issues you might encounter while running AdipoInsights or the Adipo Viewer, along with suggested fixes:

Pipeline Issues

Issue Possible Fix
"Skin region could not be identified" Adjust mask thresholds in your configuration JSON file under Analyzer/configs/.
Missing dependencies or import errors Ensure all packages are installed: pip install -r requirements.txt
openslide-python errors Install both openslide-python and the openslide-bin system library. On macOS: brew install openslide.

Viewer Issues

Issue Possible Fix
Streamlit fails to launch Verify you're in the adipo_viewer directory with the virtual environment active: streamlit run app.py
No data after loading Check that your folder contains valid *_all.csv and *_mean.csv files with correct column headers.
Plots not displaying or empty Ensure selected filters aren't excluding all data. Try resetting filter values to defaults.
"Permission denied" errors Ensure read/write permissions for input/output directories.
Images not showing in Outlier tab Verify image files (*_cells.jpg, *_shape_data.npz) exist in the same directory as CSVs. Check that sample IDs in data match file names.
Metadata not matching Ensure the ID column in metadata matches the sample ID column in your CSV files exactly. Use the column mapping UI to verify.
Viewer not updating after upload Use "Clear uploaded files" in the sidebar to reset cached data, then re-upload.
Slow performance Limit the number of samples or use sample selection. Close unused expander sections.
Statistical tests failing Ensure you have sufficient samples per group (minimum 2-3 per group for most tests). Check that Sex and Genotype columns have valid values.
Covariate models not showing Verify Age or Bodyweight columns contain numeric values and have sufficient variability.

📄 License

MIT License