Implementing NMF and Ataptive Thresholding

.github/dependabot.yml

@@ -0,0 +1,13 @@
+version: 2
+updates:
+  # Maintain dependencies for GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "monthly"
+
+  # Maintain dependencies for Composer
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "monthly"

.github/workflows/build_docs.yml

@@ -0,0 +1,69 @@
+name: Build Sphinx Documentation
+
+on:
+    push:
+        branches:
+        - main
+    pull_request:
+        branches:
+        - 2-add-continuous-integration
+        - main
+
+jobs:
+  build:
+    runs-on: ubuntu-latest  # You can also use `windows-latest` or `macos-latest` if needed
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v3
+
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'  # Set your preferred Python version
+
+    - name: Add contourusv to PYTHONPATH
+      run: echo "PYTHONPATH=$PYTHONPATH:$(pwd)/contourusv" >> $GITHUB_ENV
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install sphinx sphinx_rtd_theme opencv-python numpy pandas tqdm scipy matplotlib codecarbon scikit-learn
+
+
+    - name: Build the documentation
+      run: |
+
+        # Create a new branch for the documentation
+        git checkout --orphan gh-pages
+        # Generate reStructuredText files from the source code
+        sphinx-apidoc -o sphinx/ contourusv/ -f
+
+        rm -rf docs/
+
+        # Ensure the docs directory exists
+        mkdir -p docs
+
+        # Build the HTML documentation
+        sphinx-build -b html sphinx/ docs/
+        pwd
+        ls
+
+        # Add a .nojekyll file to bypass Jekyll processing on GitHub Pages
+        touch ./docs/.nojekyll
+
+    - name: Commit and push generated docs
+      run: |
+        ls
+        git config --global user.email "[email protected]"
+        git config --global user.name "Dallas Wade"
+        git config --global pull.rebase false  # Or true, or --ff-only based on your preference
+
+        # Ensure docs/ is staged properly
+        git add docs/
+        git status  # Debugging: See if anything is staged
+        git commit -m "Update Sphinx documentation" || echo "No changes to commit."
+
+        # Push the changes
+        git push origin gh-pages --force
+

.gitignore

@@ -1 +1,5 @@
-src/__pycache__
+contourusv/__pycache__
+venv
+contourusv/output/
+contourusv/Experiments/
+contourusv/test.ipynb

contourusv/contourusv.egg-info/PKG-INFO

@@ -0,0 +1,226 @@
+Metadata-Version: 2.2
+Name: contourusv
+Version: 0.1.0
+Summary: USV Detection Pipeline
+Author-email: Sabah Anis <[email protected]>
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: mne
+Requires-Dist: matplotlib
+Requires-Dist: scipy
+Requires-Dist: pandas
+Requires-Dist: opencv-python
+Requires-Dist: tqdm
+Requires-Dist: codecarbon
+Requires-Dist: seaborn
+Requires-Dist: pillow
+
+# ContourUSV: Ultrasonic Vocalization Detection Pipeline
+
+![Python Version](https://img.shields.io/badge/python-%3E%3D3.9-blue.svg)
+![License](https://img.shields.io/badge/license-MIT-green.svg)
+
+ContourUSV is an automated pipeline for detecting ultrasonic vocalizations (USVs) in audio recordings. The system uses spectrogram analysis combined with advanced image processing techniques to identify and classify 22kHz and 50kHz USVs.
+
+<!-- ## Features
+
+- Audio preprocessing with bandpass filtering and normalization
+- Spectrogram generation with customizable parameters
+- Advanced image processing for noise reduction:
+  - Median filtering
+  - Otsu's thresholding
+  - Contrast Limited Adaptive Histogram Equalization (CLAHE)
+  - Morphological operations
+- Contour-based USV detection
+- Annotation generation from multiple file formats (HTML, Excel, CSV)
+- Comprehensive evaluation metrics:
+  - Precision, Recall, F1 Score, Specificity
+  - Carbon emissions tracking via CodeCarbon
+- Parallel processing support -->
+
+## Installation
+
+1. **Clone the repository:**
+   ```bash
+   git clone https://github.com/yourusername/contourusv.git
+   cd contourusv
+   ```
+
+2. **Create and activate virtual environment:**
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # Linux/MacOS
+   venv\Scripts\activate     # Windows
+   ```
+
+3. **Install dependencies:**
+   ```bash
+   pip install -e .
+   ```
+
+## Data Directory Structure
+
+Organize your input data using the following structure:
+
+```
+root_path/                  # Passed via --root_path argument
+├── EXPERIMENT_NAME/        # Passed via --experiment argument
+│   └── TRIAL_NAME/         # Passed via --trial argument
+│       ├── *.wav           # Audio recordings
+│       ├── *.WAV           # (Alternative capitalization)
+│       ├── *.html          # HTML annotations
+│       ├── *.xlsx          # Excel annotations
+│       └── *.csv           # CSV annotations
+```
+
+**Example Concrete Structure:**
+```
+/Users/username/data/
+└── PTSD16/
+    └── ACQ/
+        ├── rat12_day1.wav
+        ├── rat12_day1.html
+        ├── rat13_day1.WAV
+        └── rat13_day1.html
+```
+
+**File Requirements:**
+- Audio files: Must have `.wav` or `.WAV` extension
+- Annotation files: Must match audio filenames and reside in same directory
+- Supported annotation formats:
+  - HTML
+  - Excel
+  - CSV
+
+## Output Structure
+
+```
+output/
+├── EXPERIMENT_NAME/
+│   ├── TRIAL_NAME/
+│   │   ├── contour_detections/
+│   │   │   └── *.csv (detection annotations)
+│   │   ├── evaluation_results/
+│   │   │   └── Evaluation_*.csv (performance metrics)
+│   │   └── spectrograms/
+│   │       └── *.png (annotated spectrograms)
+│   └── ground_truth_annotations/
+│       └── *.csv (processed ground truth)
+```
+
+## Usage
+
+In the `src` directory execute the following command to run the detection pipeline.
+
+### Basic Command
+```bash
+python main.py \
+  --root_path /path/to/your/data \
+  --experiment EXPERIMENT_NAME \
+  --trial TRIAL_NAME \
+  --file_ext ANNOTATION_FILE_EXT
+```
+
+### Example Command
+```bash
+python main.py \
+  --root_path /Users/username/data \
+  --experiment PTSD16 \
+  --trial ACQ \
+  --file_ext .html
+```
+
+### Required Arguments
+| Argument      | Description                               | Example          |
+|---------------|-------------------------------------------|------------------|
+| `--root_path` | Root directory containing experiment data | `/data/studies`  |
+| `--experiment`| Name of the experiment                    | `PTSD16`         |
+| `--trial`     | Name of the trial/condition               | `ACQ`            |
+| `--file_ext`  | Annotation file extension (`.html`, `.xlsx`, `.csv`) | `.html` |
+
+### Optional Parameters
+| Parameter      | Default | Description                              |
+|----------------|---------|------------------------------------------|
+| `--overlap`    | 3       | Overlap duration between windows (seconds) |
+| `--winlen`     | 10      | Window length for processing (seconds)   |
+| `--freq_min`   | 15      | Minimum frequency for detection (kHz)    |
+| `--freq_max`   | 115     | Maximum frequency for detection (kHz)    |
+| `--wsize`      | 2500    | Window size for processing               |
+| `--th_perc`    | 95      | Percentile threshold for noise reduction |
+
+<!-- ## Pipeline Architecture
+
+1. **Preprocessing**
+   - Audio normalization and filtering
+   - Spectrogram generation
+   - Noise reduction using:
+     - Median filtering
+     - Otsu's thresholding
+     - CLAHE contrast enhancement
+
+2. **Detection**
+   - Contour detection using OpenCV
+   - USV classification (22kHz vs 50kHz)
+   - Bounding box annotation
+   - Temporal and spectral feature extraction
+
+3. **Annotation Generation**
+   - Supports multiple input formats:
+     - HTML
+     - Excel
+     - CSV
+
+4. **Evaluation**
+   - Precision/Recall calculations
+   - F1 Score and Specificity metrics
+   - Carbon emissions tracking
+   - Energy consumption monitoring
+
+## Evaluation Metrics
+
+The pipeline calculates four key performance metrics:
+- **Precision**: Ratio of correct USV detections to total detections
+- **Recall**: Ratio of detected USVs to total actual USVs
+- **F1 Score**: Harmonic mean of precision and recall
+- **Specificity**: Ability to identify true negative segments
+
+Example output:
+```
+Mean Precision: 0.92 ± 0.05
+Mean Recall: 0.88 ± 0.07
+Mean F1 Score: 0.90 ± 0.04
+Mean Specificity: 0.95 ± 0.03
+```
+
+## Environmental Impact Tracking
+
+The pipeline integrates with CodeCarbon to monitor:
+- CO₂ emissions (kg)
+- Energy consumption (kWh)
+- Computational efficiency
+
+Sample output:
+```
+ContourUSV_Execution_Time_(s) = 452.783
+ContourUSV_Carbon_Emissions_(kgCO2) = 0.127
+ContourUSV_Total_Energy_Consumed_(kWh) = 0.342
+``` -->
+
+## Contributing
+
+Contributions are welcome! Please follow these steps:
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/your-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin feature/your-feature`)
+5. Open a Pull Request
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Contact
+
+Sabah Anis - [[email protected]](mailto:[email protected])

contourusv/contourusv.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+README.md
+pyproject.toml
+src/__init__.py
+src/detection.py
+src/evaluation.py
+src/generate_annotation.py
+src/main.py
+src/preprocessing.py
+src/contourusv.egg-info/PKG-INFO
+src/contourusv.egg-info/SOURCES.txt
+src/contourusv.egg-info/dependency_links.txt
+src/contourusv.egg-info/requires.txt
+src/contourusv.egg-info/top_level.txt

contourusv/contourusv.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

contourusv/contourusv.egg-info/requires.txt

@@ -0,0 +1,10 @@
+numpy
+mne
+matplotlib
+scipy
+pandas
+opencv-python
+tqdm
+codecarbon
+seaborn
+pillow

contourusv/contourusv.egg-info/top_level.txt

@@ -0,0 +1,6 @@
+__init__
+detection
+evaluation
+generate_annotation
+main
+preprocessing

src/detection.py → contourusv/detection.py

@@ -1,7 +1,7 @@
 import cv2
 
 def detect_contours(cleaned_image, start_time, end_time, freq_min, freq_max, 
-                                    file_name, annotations, call_type_defs=None):
+                                    file_name, annotations, call_type_defs=None, processing="adaptive"):
     """
     Detect and classify USVs in cleaned spectrogram images.
 
@@ -34,18 +34,22 @@ def detect_contours(cleaned_image, start_time, end_time, freq_min, freq_max,
          "22kHz": {"freq_min": 15,
                     "freq_max": 45,
                     "freq_span_max": 10, 
-                    "duration_min": 0.03,
+                    "duration_min": 0.03,  # .03 was original
                     "duration_max": 3.0},
          "50kHz": {"freq_min": 40,
                     "freq_max": 80,
                     "freq_span_max": 10, 
                     "duration_min": 0.01,
                     "duration_max": 0.3},         
+
          }
 
-    # Re-apply Otsu's Thresholding
-    ret, thresholded_image = cv2.threshold(
-        cleaned_image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
+    # Re-apply Otsu's Thresholding (Cant do if using adaptive)
+    if(processing == "Otsu"):
+        ret, thresholded_image = cv2.threshold(
+            cleaned_image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
+    else:
+        thresholded_image = cleaned_image
 
     contours, _ = cv2.findContours(
         thresholded_image, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)