crow-tools-reference.txt

# Reference snapshot of the crow-tools repository (https://github.com/CrowCall/crow-tools). Used for development context — not part of the Orpheus codebase.

==========================================
CROW-TOOLS REPOSITORY SNAPSHOT
Generated: 2025-12-05 10:35:38
==========================================

<FULL TREE>
.
|-- classifier
|   |-- models
|   |   +-- best_model.ckpt
|   |-- classify.py
|   |-- cluster.py
|   |-- dataset.py
|   |-- evaluate.py
|   |-- model.py
|   |-- optuna_tuning.py
|   |-- preview_auto_labels.py
|   |-- remove_dupes.py
|   |-- RESULTS.txt
|   +-- train.py
|-- denoiser
|   +-- denoise_crows.py
|-- detector
|   |-- detect.py
|   |-- detect_all.py
|   +-- detect_all_birdnet.py
|-- docs
|   |-- images
|   |   |-- crow-animation.gif
|   |   |-- crow-emote.png
|   |   |-- crow-tools-presentation-cover.png
|   |   |-- denoiser.png
|   |   |-- detector-timeline.png
|   |   |-- embeddings.png
|   |   |-- labeler.png
|   |   |-- separator.png
|   |   +-- transcriber.png
|   +-- videos
|       +-- embeddings.gif
|-- downloader
|   |-- download_backgrounds.py
|   |-- download_crows_ebird.py
|   +-- download_crows_xeno.py
|-- embedder
|   |-- ispa
|   |   |-- models
|   |   |   |-- aves-base-bio.torchaudio.model_config.json
|   |   |   |-- aves-base-bio.torchaudio.pt
|   |   |   |-- c2p.aves.json
|   |   |   |-- c2p.mfcc.json
|   |   |   |-- kmeans.aves.pkl
|   |   |   +-- kmeans.mfcc.pkl
|   |   |-- __init__.py
|   |   |-- acoustics.py
|   |   |-- features.py
|   |   +-- utils.py
|   |-- __init__.py
|   |-- analyze.py
|   |-- embed.py
|   +-- embed_all.py
|-- labeler
|   |-- js
|   |   |-- components
|   |   |   |-- filterComponent.js
|   |   |   |-- menu.js
|   |   |   |-- pagination.js
|   |   |   +-- segmentCard.js
|   |   +-- app.js
|   |-- embeddings.html
|   |-- index.html
|   |-- package-lock.json
|   |-- package.json
|   |-- server.js
|   +-- transcriptions.html
|-- separator
|   |-- models
|   |   +-- best_model_epoch=94.ckpt
|   |-- samples
|   |   |-- non-overlapping-multiple-crows.mp3
|   |   |-- overlapping-crows-1.wav
|   |   |-- overlapping-crows-2.wav
|   |   |-- overlapping-crows-3.wav
|   |   +-- softsong.mp3
|   |-- dataset.py
|   |-- migrate.py
|   |-- mix.py
|   |-- separate.py
|   +-- train.py
|-- .gitattributes
|-- CITATION.cff
|-- condense_for_llm.sh
|-- crow-tools-condensed-for-llm.out
|-- get-data.py
|-- LICENSE.md
|-- README.md
+-- requirements.txt


<FILE SUMMARY>
Total files included in this condensed view:
Files: 50

<FILES BY DIRECTORY>
(root): 6 files
classifier: 10 files
denoiser: 1 files
detector: 3 files
downloader: 3 files
embedder/ispa/models: 3 files
embedder/ispa: 4 files
embedder: 4 files
labeler/js/components: 4 files
labeler/js: 1 files
labeler: 6 files
separator: 5 files


<FILE CONTENTS>

==========================================
<CITATION.cff>
Size: 541 bytes | Lines:       19
==========================================
cff-version: 1.2.0
message: "If crow-tools was useful for your research, we'd love a citation!"
title: "crow-tools: open-source tools for analyzing crow vocalizations"
authors:
  - family-names: Thomas
    given-names: Jonathan
    email: crows@openshot.org
  - family-names: Thomas
    given-names: Madeline
    email: crows@owlmaddie.com
date-released: 2025-03-22
version: "1.0.0"
repository-code: https://github.com/CrowCall/crow-tools
keywords:
  - crows
  - bioacoustics
  - machine learning
  - audio analysis
  - animal communication

==========================================
</CITATION.cff>
==========================================

==========================================
<LICENSE.md>
Size: 1092 bytes | Lines:       21
==========================================
MIT License

Copyright (c) 2025 OpenShot Studios LLC, owlmaddie LLC

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

==========================================
</LICENSE.md>
==========================================

==========================================
<README.md>
Size: 8532 bytes | Lines:      160
==========================================
# Decoding Crow Communication with AI  

## Crows are among the most intelligent and vocal birds on the planet. But what exactly are they saying? 

<img src="docs/images/crow-animation.gif" align="left" width="50%" title="Original animation by @owlmaddie">

This project aims to **explore** and **decode** the rich and mysterious world of **crow communication** using cutting-edge 
machine learning and audio analysis.

This repository is your toolkit for working with thousands of **American crow** ([Corvus brachyrhynchos](https://www.allaboutbirds.org/guide/American_Crow/overview)) **calls**—from downloading and cleaning audio, to 
detecting individual calls, generating embeddings, classifying vocalizations, and even manually labeling them 
through a web interface.

Whether you're a bird enthusiast, a researcher, or just someone fascinated by animal intelligence, this project 
offers a glimpse into the complex vocal language of these remarkable creatures—and the possibility of better 
understanding them through technology.

## Install Dependencies

This project was built on **Ubuntu 24.04** and **Python 3.8+**, however it should be compatible with most linux
and mac systems. **Git LFS** is required to correctly clone, pull, and inflate all models. **FFmpeg** is required
to run the denoiser.

```
sudo apt install ffmpeg git-lfs portaudio19-dev python3-tk
pip install -r requirements.txt
```

## Download and Prepare Data

Run this script to download, denoise, embed, and auto-label all crow audio files. 
NOTE: This will download more than **30 GB** of data into a local `.cache` directory.

```
python get-data.py
```

## Introduction
[![crow-tools-presentation-cover.png](docs/images/crow-tools-presentation-cover.png)](https://docs.google.com/presentation/d/15q6MwZquIA3LaXqK29C7NRGKmcn5BTSwK5TFsem-QGk/present)

## Module Overview

### Downloader

The downloader module retrieves a large collection of crow vocalizations (13+ GB) from multiple public repositories. 
It handles the complexities of connecting to each source, downloading the audio files, and storing relevant metadata 
for proper attribution. Credits and licensing info for all files are saved in the `.cache/csv/` directory.

Crow-tools relies on openly available datasets for research and development. We gratefully acknowledge the following sources:

- [Macaulay Library – American Crow (*Corvus brachyrhynchos*)](https://search.macaulaylibrary.org/catalog?taxonCode=amecro&mediaType=audio)  
  © Cornell Lab of Ornithology. A comprehensive archive of wildlife recordings, used in accordance with licensing terms for non-commercial research.

- [Xeno-Canto – American Crow (*Corvus brachyrhynchos*)](https://xeno-canto.org/species/corvus-brachyrhynchos)  
  A global, community-powered collection of bird calls, shared under various Creative Commons licenses. Many thanks to the recordists who make this work possible.

### Denoiser
The denoiser module cleans the crow audio files by removing unwanted background noise. This process improves the 
quality of the audio for subsequent processing steps by focusing on the relevant crow sounds. It also enables
the creation of mixes (overlapping crow sounds) to train our separator model. This module utilizes the [biodenoising](https://github.com/earthspecies/biodenoising-inference)
module created by [Earth Species Project](https://earthspecies.org/).

![denoiser.png](docs/images/denoiser.png)

### Classifier
The classifier module analyzes crow call embeddings and categorizes them (i.e. auto labels) into various types such as 
alert, number of calls, age indicators, rattles, soft songs, and quality of audio. It processes the embedded data and 
applies machine learning techniques to identify and label crow vocalizations.

```python
    {
      "crowCount": int in [1,2,3,4],   # 1 = single, 2 = two crows, 3 = unused, 4 = crowd
      "crowAge": int in [1,2],         # 1 = adult, 2 = juvenile
      "alert": bool,                   # alert calls
      "begging": bool,                 # food related calls
      "softSong": bool,                # sub songs | soft sounds
      "rattle": bool,                  # rattle sounds
      "mob": bool,                     # anger calls | mob | attack
      "quality": int in [1,2,3],       # 1 = bad, 2 = good, 3 = unused
    }
```

### Detector
The detector module leverages our custom trained crow classifier, to quickly find all crow sounds across an audio file.
By isolating these segments, the module enables more focused analysis and processing of individual crow calls and vocalizations.
We also include an interactive crow timeline app to review and listen to the detections:

![detector-timeline.png](docs/images/detector-timeline.png)

```json
[
    {
        "start_time": 37.0,
        "end_time": 38.0,
        "crowCount": 2,
        "crowAge": 1,
        "alert": false,
        "begging": false,
        "softSong": false,
        "rattle": true,
        "mob": false,
        "quality": 2
    }
]
```

### Embedder
The embedder module transforms each crow call into a 768-dimensional vector using the [AVES](https://github.com/earthspecies/aves?tab=readme-ov-file#birdaves) embedding model. This 
transformation creates a numerical representation of the audio, which is essential for further analysis and machine 
learning applications.

![embeddings.gif](docs/videos/embeddings.gif)

### Labeler
The labeler module provides a web interface for manual labeling of crow calls. This interface is designed for 
human labeling and review, ensuring that the training data for the classifier is accurate and reliable. It also
provides a 3D interactive embedding feature. Built with Vue v3 and Node.js.

![labeler.png](docs/images/labeler.png)

### Transcriber
The transcriber module provides a web interface for segmenting crow audio and labeling calls with a custom notation 
system and limited vocabulary tokenizer. It supports waveform and spectrogram views, audio playback, and SRT export 
for training crow-to-text models. Built with Vue v3 and Node.js.

![transcriber.png](docs/images/transcriber.png)

### Separator
The separator module is responsible for separating overlapping crow calls into distinct audio files. This process 
enables clearer analysis by isolating individual calls that may be mixed together in the original recordings.

![separator.png](docs/images/separator.png)

## Directory Structure
```
crow-tools/
├── classifier/  # classify types of crow calls (alert, count, age, rattle, soft/sub song, quality)
├── denoiser/    # denoise crow audio files (remove background noises with biodenoising model)
├── detector/    # detect crow audio segments (1 second each, uses classifier model)
├── downloader/  # download library of crow audio files
├── embedder/    # embed crow calls into 768 dimensions (AVES embedding model)
├── labeler/     # human labeling web app (for training classifier, vieweing embeddings, and transcribing crow language)
├── separator/   # separate overlapping crow calls into seaparate audio files (train and inference)
└── .cache/      # all downloaded and generated files (30+ GB)
```

## Authors

This project is a collaborative effort driven by a shared curiosity and love for crows — with the ultimate goal of better understanding their complex and intelligent vocal language.

- **[Jonathan Thomas](mailto:crows@openshot.org)** is the creator of [OpenShot Video Editor](https://www.openshot.org/) and brings deep experience in software development and artificial intelligence. Through his company, **OpenShot Studios LLC**, he leads the technical development of tools for large-scale audio analysis, machine learning, and crow call classification.

- **[Madeline Thomas](mailto:crows@owlmaddie.com) (@owlmaddie)** is a professional [artist / animator](https://www.owlmaddie.com/) and crow enthusiast. Through **owlmaddie LLC**, she contributes her creative talents to the project — labeling thousands of crow calls, designing expressive crow animations, and helping to communicate the science through visual storytelling.

Together, we're building a **state-of-the-art toolkit for decoding and exploring crow communication**, blending AI, design, and a shared passion for one of nature’s most fascinating birds.

## Citations

Give our crows a shout-out — cite `crow-tools` in your work! GitHub provides a citation file — just click the **Cite this repository** button in the sidebar at the top of this page.


==========================================
</README.md>
==========================================

==========================================
<classifier/RESULTS.txt>
Size: 1779 bytes | Lines:       85
==========================================
/home/jonathan/apps/earthspecies/crow-tools/.venv/bin/python /home/jonathan/apps/earthspecies/crow-tools/classifier/evaluate.py
Seed set to 18202

=== Label Statistics ===
Total labels loaded: 3894

crowCount distribution:
  Class 0: 1245
  Class 1: 2048
  Class 2: 345
  Class 4: 256

crowAge distribution:
  Class 1: 3550
  Class 2: 344

quality distribution:
  Class 1: 1347
  Class 2: 2547

alert count: 593
begging count: 307
softSong count: 612
rattle count: 543
mob count: 753


Evaluating on 468 VALIDATE samples

=== Overall Accuracy ===
crowCount :  82.91% (388/468)
crowAge   :  96.15% (450/468)
quality   :  91.67% (429/468)
alert     :  90.81% (425/468)
begging   :  96.15% (450/468)
softSong  :  94.02% (440/468)
rattle    :  95.51% (447/468)
mob       :  88.68% (415/468)

=== crowCount Breakdown ===
  Class 0:  89.21% (124/139)
  Class 1:  87.69% (228/260)
  Class 2:  40.48% (17/42)
  Class 3:   0.00% (0/0)
  Class 4:  70.37% (19/27)

=== crowAge Breakdown ===
  Class 1:  98.56% (410/416)
  Class 2:  76.92% (40/52)

=== quality Breakdown ===
  Class 1:  85.91% (128/149)
  Class 2:  94.36% (301/319)

=== alert Breakdown ===
  Value 0:  93.37% (366/392)
  Value 1:  77.63% (59/76)

=== begging Breakdown ===
  Value 0:  98.11% (415/423)
  Value 1:  77.78% (35/45)

=== softSong Breakdown ===
  Value 0:  97.18% (379/390)
  Value 1:  78.21% (61/78)

=== rattle Breakdown ===
  Value 0:  96.94% (380/392)
  Value 1:  88.16% (67/76)

=== mob Breakdown ===
  Value 0:  95.00% (361/380)
  Value 1:  61.36% (54/88)

=== Composite Score ===
Overall composite score: 84.39%
Individual task scores:
  crowCount : 71.94%
  crowAge   : 87.74%
  quality   : 90.13%
  alert     : 85.50%
  begging   : 87.94%
  softSong  : 87.69%
  rattle    : 92.55%
  mob       : 78.18%

==========================================
</classifier/RESULTS.txt>
==========================================

==========================================
<classifier/classify.py>
Size: 2101 bytes | Lines:       64
==========================================
import os
import torch
import numpy as np
from classifier.model import CrowClassifier

# Set up paths.
PATH = os.path.dirname(__file__)
checkpoint_path = os.path.join(PATH, "models", "best_model.ckpt")

# Determine device.
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load the model only once.
if os.path.exists(checkpoint_path):
    _model = CrowClassifier.load_from_checkpoint(checkpoint_path)
else:
    print(f"Model checkpoint not found: {checkpoint_path}")
    _model = CrowClassifier()
_model.to(device)
_model.eval()


def predict_embedding(embedding, device=device):
    """
    Given an embedding (numpy array of shape [768]), perform inference using the
    pre-loaded model and return a predicted label dictionary in the new format:

    {
      "crowCount": int in [1,2,3,4],
      "crowAge": int in [1,2],
      "alert": bool,
      "begging": bool,
      "softSong": bool,
      "rattle": bool,
      "mob": bool,
      "quality": int in [1,2,3],
    }
    """
    # Ensure the embedding is a float32 numpy array and add a batch dimension.
    embedding_tensor = torch.from_numpy(embedding.astype(np.float32)).unsqueeze(0).to(device)

    # Perform inference.
    with torch.no_grad():
        outputs = _model(embedding_tensor)

    new_pred = {}
    new_pred["crowCount"] = torch.argmax(outputs["crowCount"], dim=1).item()
    new_pred["crowAge"] = torch.argmax(outputs["crowAge"], dim=1).item() + 1
    new_pred["quality"] = torch.argmax(outputs["quality"], dim=1).item() + 1

    new_pred["alert"] = (outputs["alert"].squeeze() > 0).item()
    new_pred["begging"] = (outputs["begging"].squeeze() > 0).item()
    new_pred["softSong"] = (outputs["softSong"].squeeze() > 0).item()
    new_pred["rattle"] = (outputs["rattle"].squeeze() > 0).item()
    new_pred["mob"] = (outputs["mob"].squeeze() > 0).item()

    return new_pred


if __name__ == "__main__":
    # Example: Replace with your actual numpy array of 768 numbers.
    dummy_embedding = np.random.randn(768)
    predicted_label = predict_embedding(dummy_embedding)
    print("Predicted Label:", predicted_label)

==========================================
</classifier/classify.py>
==========================================

==========================================
<classifier/cluster.py>
Size: 17340 bytes | Lines:      428
==========================================
import os
import glob
import json
import random
import numpy as np
import librosa
import sounddevice as sd
from sklearn.decomposition import PCA
import faiss
from collections import defaultdict

# Paths
BASE_PATH = os.path.dirname(__file__)
EMBEDDINGS_DIR = os.path.join(BASE_PATH, "..", ".cache", "embeddings-denoised")
VOLUMES_DIR = os.path.join(BASE_PATH, "..", ".cache", "embeddings-denoised-volumes")
OUTPUT_SEGMENTS = os.path.join(BASE_PATH, "..", ".cache", "cluster_segments.json")
OUTPUT_LABELS = os.path.join(BASE_PATH, "..", ".cache", "cluster_labels.json")
AUDIO_DIR = os.path.join(BASE_PATH, "..", ".cache", "library-denoised")
INDEX_PATH = os.path.join(BASE_PATH, "..", ".cache", "faiss_index.index")
LABEL_TEMPLATE_FILE = os.path.join(BASE_PATH, "..", ".cache", "cluster_segments_labels.json")

# Parameters
STARTING_CLUSTER_ID = 65
VOLUME_THRESHOLD = 0.0002
SUBSAMPLE_FACTOR = 1.0
PCA_COMPONENTS = 75
MAX_CLUSTER_SIZE = 500  # Maximum leaf size before splitting stops.
MERGE_THRESHOLD = 0.15  # Merge leaves if cosine distance < 0.15 (single pass)
PREVIEW_SEEDS = False
PREVIEW_CLUSTERS = False
ONLY_OUTPUT_SEEDS = True
PREVIEW_PER_CLUSTER = 10
NUM_REPRESENTATIVE = 10  # Number of segments per merged cluster for labeling

# Seed examples for new clusters.
SEED_EXAMPLES = [
    # Rattles
    #{"file_id": "365208991", "start": 13.0, "end": 14.0}, # Good, 9 similar
    #{"file_id": "227497211", "start": 48.0, "end": 49.0}, # Good, 7 similar
    #{"file_id": "124568031", "start": 7.0, "end": 8.0}, # Good 10+ similar, (had to lower volume to 0.0002)
    #{"file_id": "431165421", "start": 12.0, "end": 13.0},  # Okay, 3 similar
    #{"file_id": "58460", "start": 12.0, "end": 13.0},  # Good, 8 similar
    #{"file_id": "504976401", "start": 4.0, "end": 5.0},  # Great, 9 similar
    #{"file_id": "122364731", "start": 3.0, "end": 4.0},  # Okay, 3 similar
    #{"file_id": "163637", "start": 1.0, "end": 2.0}, # Okay, 3-4 similar
    #{"file_id": "496356", "start": 9.0, "end": 10.0}, # Good, 5 similar
    #{"file_id": "156527", "start": 30.0, "end": 31.0}, # Okay, 3 or 4 similar



{"file_id": "227497211", "start": 48.0, "end": 49.0},
{"file_id": "365208991", "start": 27.0, "end": 28.0},
{"file_id": "361178511", "start": 27.0, "end": 28.0},
{"file_id": "361178511", "start": 13.0, "end": 14.0},
{"file_id": "156527", "start": 6.0, "end": 7.0},
{"file_id": "92055", "start": 15.0, "end": 16.0},
{"file_id": "619206184", "start": 17.0, "end": 18.0},

    # Sub/Soft Song


{"file_id": "229159", "start": 74.0, "end": 75.0},
{"file_id": "542024451", "start": 7.0, "end": 8.0},
{"file_id": "539550101", "start": 12.0, "end": 13.0},
{"file_id": "535466271", "start": 1.0, "end": 2.0},
{"file_id": "408950861", "start": 21.0, "end": 22.0},
{"file_id": "319547721", "start": 100.0, "end": 101.0},
{"file_id": "167792", "start": 20.0, "end": 21.0},

    #{"file_id": "408950861", "start": 20.0, "end": 21.0},
    #{"file_id": "13123", "start": 34.0, "end": 35.0},
    #{"file_id": "984442", "start": 33.0, "end": 34.0},
    #{"file_id": "984442", "start": 6.0, "end": 7.0},

    # Juvenile begging
    #{"file_id": "32684421", "start": 39.0, "end": 40.0},
    #{"file_id": "32684421", "start": 22.0, "end": 23.0},
    #{"file_id": "32684421", "start": 10.0, "end": 11.0},
    #{"file_id": "32684421", "start": 3.0, "end": 4.0}
]


def play_audio_preview(audio_file, start_time, duration):
    try:
        y, sr = librosa.load(audio_file, sr=None, offset=start_time, duration=duration)
        sd.play(y, sr)
        sd.wait()
    except Exception as e:
        print(f"Error playing {audio_file}: {e}")


def load_non_silent_embeddings(emb_dir, vol_dir, thresh):
    emb_list, ids, total = [], [], 0
    for path in sorted(glob.glob(os.path.join(emb_dir, "*.npy"))):
        file_id = os.path.splitext(os.path.basename(path))[0]
        vol_path = os.path.join(vol_dir, f"{file_id}.npy")
        if not os.path.exists(vol_path):
            continue
        try:
            emb_data = np.load(path)
            vol_data = np.load(vol_path)
            if vol_data.ndim > 1:
                vol_data = vol_data.squeeze(-1)
        except Exception:
            continue
        total += emb_data.shape[0]
        for i in range(emb_data.shape[0]):
            if vol_data[i] > thresh:
                emb_list.append(emb_data[i])
                ids.append((file_id, i))
    return np.array(emb_list, dtype=np.float32), ids, total


def random_subsample(embeddings, ids, factor):
    seed_file_ids = {seed["file_id"] for seed in SEED_EXAMPLES}
    seed_indices = [i for i, (file_id, _) in enumerate(ids) if file_id in seed_file_ids]
    other_indices = [i for i, (file_id, _) in enumerate(ids) if file_id not in seed_file_ids]
    if factor >= 1.0:
        selected_other = other_indices
    else:
        n_keep = int(len(other_indices) * factor)
        selected_other = sorted(random.sample(other_indices, n_keep))
    selected_indices = sorted(seed_indices + selected_other)
    return embeddings[selected_indices], [ids[i] for i in selected_indices]


def reduce_dim_pca(embeddings, n_components=64):
    pca = PCA(n_components=n_components, random_state=42)
    reduced = pca.fit_transform(embeddings)
    print("Explained variance (first 5):", pca.explained_variance_ratio_[:5])
    return reduced, pca


def normalize_embeddings(embeddings):
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    norms = np.clip(norms, 1e-8, None)
    return embeddings / norms


def hierarchical_split(embeddings, indices, max_size=500, level=0):
    if len(indices) <= max_size:
        return {"indices": indices, "level": level, "size": len(indices)}
    k = 2
    d = embeddings.shape[1]
    kmeans = faiss.Kmeans(d, k, niter=20, verbose=False, seed=42)
    subset = embeddings[indices]
    kmeans.train(subset)
    D, I = kmeans.index.search(subset, 1)
    assignments = I.flatten()
    clusters = {i: [] for i in range(k)}
    for j, assign in enumerate(assignments):
        clusters[assign].append(indices[j])
    result = {"level": level, "size": len(indices), "children": {}}
    for i in range(k):
        result["children"][i] = hierarchical_split(embeddings, clusters[i], max_size, level + 1)
    return result


def collect_leaves(hierarchy):
    leaves = []
    if "children" not in hierarchy:
        leaves.append(hierarchy)
    else:
        for child in hierarchy["children"].values():
            leaves.extend(collect_leaves(child))
    return leaves


def compute_leaf_center(norm_emb, indices):
    center = np.mean(norm_emb[indices], axis=0)
    center /= np.linalg.norm(center)
    return center


def merge_leaves_once(leaves, merges):
    merged = set()
    new_leaves = []
    for (i, j, dist) in merges:
        if i not in merged and j not in merged:
            union_indices = leaves[i]["indices"] + leaves[j]["indices"]
            new_leaf = {"indices": union_indices,
                        "level": min(leaves[i]["level"], leaves[j]["level"]),
                        "size": len(union_indices)}
            new_leaves.append(new_leaf)
            merged.add(i)
            merged.add(j)
    for i in range(len(leaves)):
        if i not in merged:
            new_leaves.append(leaves[i])
    return new_leaves


def process_seed_examples(norm_emb, ids, faiss_index, max_per_file=4):
    # Process each seed example individually.
    seed_clusters = {}
    default_template = {
        "crowCount": 1, "crowAge": 1,
        "alert": False, "begging": False,
        "softSong": False, "rattle": False, "mob": False,
        "quality": 2, "reviewed": False
    }
    for i, seed in enumerate(SEED_EXAMPLES):
        target_index = None
        for idx, (file_id, sec) in enumerate(ids):
            if file_id == seed["file_id"] and int(sec) == int(seed["start"]):
                target_index = idx
                break
        if target_index is None:
            print(f"Seed example {seed} not found.")
            continue
        vec = norm_emb[target_index].reshape(1, norm_emb.shape[1])
        D, I = faiss_index.search(vec, NUM_REPRESENTATIVE)
        results = []
        file_counts = defaultdict(int)
        for file_id, sec in [ids[idx] for idx in I[0]]:
            if file_counts[file_id] < max_per_file:
                results.append({"file_id": file_id, "start_time": float(sec), "end_time": float(sec + 1)})
                file_counts[file_id] += 1
        if results:
            seed_clusters[str(i)] = results
    return seed_clusters


def build_and_save_clusters(merged_leaves, seed_clusters, ids, norm_emb):
    """
    Build segments and labels from both merged clusters and seed clusters.
    Existing segments and labels are loaded (if available) and new records are
    appended only if they are not already present.
    - For merged clusters, segments are sorted in similarity order for preview,
      then the top representative segments are chosen.
    - For seed clusters, segments are added if not already present.
    - Segments in each file are sorted by start_time before saving.
    - Duplicate segment keys (file, start, end) are avoided.
    """
    from collections import defaultdict

    # Load existing segments and labels if the files exist.
    if os.path.exists(OUTPUT_SEGMENTS):
        with open(OUTPUT_SEGMENTS, "r") as f:
            existing_segments = json.load(f)
    else:
        existing_segments = {}

    if os.path.exists(OUTPUT_LABELS):
        with open(OUTPUT_LABELS, "r") as f:
            existing_labels = json.load(f)
    else:
        existing_labels = {}

    # New segments and labels to add.
    new_segments = defaultdict(list)
    new_labels = {}

    if os.path.exists(LABEL_TEMPLATE_FILE):
        with open(LABEL_TEMPLATE_FILE, "r") as f:
            label_templates = json.load(f)
    else:
        label_templates = {}
    default_template = {
        "crowCount": 1, "crowAge": 1,
        "alert": False, "begging": False,
        "softSong": False, "rattle": False, "mob": False,
        "quality": 2, "reviewed": False
    }
    cluster_id = STARTING_CLUSTER_ID

    # Process merged clusters.
    for leaf in merged_leaves:
        center = compute_leaf_center(norm_emb, leaf["indices"])
        sorted_indices = sorted(leaf["indices"],
                                key=lambda idx: np.dot(norm_emb[idx], center),
                                reverse=True)
        if PREVIEW_CLUSTERS:
            while True:
                print(f"\nMerged Cluster {cluster_id} (Size {leaf['size']}):")
                for idx in sorted_indices[:PREVIEW_PER_CLUSTER]:
                    file_id, sec = ids[idx]
                    sim = np.dot(norm_emb[idx], center)
                    print(f"  File: {file_id}, Start: {sec}, Similarity: {sim:.4f}")
                    audio_file = os.path.join(AUDIO_DIR, f"{file_id}.wav")
                    play_audio_preview(audio_file, float(sec), 1.0)
                user_input = input("Press 'R' to repeat or Enter to continue: ")
                if user_input.strip().lower() != 'r':
                    break
        for idx in sorted_indices[:NUM_REPRESENTATIVE]:
            file_id, sec = ids[idx]
            start_time = float(sec)
            end_time = start_time + 1.0
            seg_key = f"{file_id}-{int(start_time)}-{int(end_time)}"
            # Avoid duplicate label keys in new labels.
            if seg_key in new_labels:
                continue
            seg = {"common_name": "American Crow",
                   "scientific_name": "Corvus brachyrhynchos",
                   "start_time": start_time,
                   "end_time": end_time,
                   "confidence": 0.0,
                   "cluster": cluster_id}
            new_segments[file_id].append(seg)
            if str(cluster_id) in label_templates:
                new_labels[seg_key] = label_templates[str(cluster_id)]
            else:
                new_labels[seg_key] = default_template.copy()
            new_labels[seg_key]["cluster"] = cluster_id
        cluster_id += 1

    # Process seed clusters.
    for seed_id, seg_list in seed_clusters.items():
        current_cluster_id = cluster_id
        for seg in seg_list:
            file_id = seg["file_id"]
            start_time = float(seg["start_time"])
            end_time = float(seg["end_time"])
            seg_key = f"{file_id}-{int(start_time)}-{int(end_time)}"
            if seg_key in new_labels:
                continue
            seg_entry = {"common_name": "American Crow",
                         "scientific_name": "Corvus brachyrhynchos",
                         "start_time": start_time,
                         "end_time": end_time,
                         "confidence": 0.0,
                         "cluster": current_cluster_id}
            new_segments[file_id].append(seg_entry)
            if str(cluster_id) in label_templates:
                new_labels[seg_key] = label_templates[str(cluster_id)]
            else:
                new_labels[seg_key] = default_template.copy()
            new_labels[seg_key]["cluster"] = current_cluster_id
        if PREVIEW_SEEDS:
            while True:
                print(f"\nSeed Cluster {current_cluster_id}:")
                for seg in seg_list[:PREVIEW_PER_CLUSTER]:
                    print(f"  File: {seg['file_id']}, Start: {seg['start_time']}, End: {seg['end_time']}")
                    audio_file = os.path.join(AUDIO_DIR, f"{seg['file_id']}.wav")
                    play_audio_preview(audio_file, seg["start_time"], 1.0)
                user_input = input("Press 'R' to repeat or Enter to continue: ")
                if user_input.strip().lower() != 'r':
                    break
        cluster_id += 1

    # Merge new segments with existing segments.
    for file_id, seg_list in new_segments.items():
        if file_id not in existing_segments:
            existing_segments[file_id] = seg_list
        else:
            # For each new segment, check if a segment with the same start and end exists.
            existing_keys = {(seg["start_time"], seg["end_time"]) for seg in existing_segments[file_id]}
            for seg in seg_list:
                key = (seg["start_time"], seg["end_time"])
                if key not in existing_keys:
                    existing_segments[file_id].append(seg)
                else:
                    print(f"Skipping duplicate segment for file {file_id} at {key}")

    # Merge new labels with existing labels.
    for key, label in new_labels.items():
        if key in existing_labels:
            print(f"Skipping duplicate label: {key}")
        else:
            existing_labels[key] = label

    # Ensure segments in each file are sorted by start_time.
    for file_id in existing_segments:
        existing_segments[file_id] = sorted(existing_segments[file_id], key=lambda x: x["start_time"])

    # Save updated JSON outputs.
    with open(OUTPUT_SEGMENTS, "w") as f:
        json.dump(existing_segments, f, indent=2)
    with open(OUTPUT_LABELS, "w") as f:
        json.dump(existing_labels, f, indent=2)

    print(f"\nSaved segments to {OUTPUT_SEGMENTS}")
    print(f"Saved cluster labels to {OUTPUT_LABELS}")
    return existing_segments, existing_labels


def main():
    random.seed(42)
    np.random.seed(42)

    # Load embeddings.
    embeddings, ids, total = load_non_silent_embeddings(EMBEDDINGS_DIR, VOLUMES_DIR, VOLUME_THRESHOLD)
    print(f"Processed {total} seconds. Non-silent segments: {embeddings.shape[0]}")
    embeddings, ids = random_subsample(embeddings, ids, SUBSAMPLE_FACTOR)
    print(f"After subsampling: {embeddings.shape[0]} segments")

    # Dimensionality reduction.
    reduced, _ = reduce_dim_pca(embeddings, PCA_COMPONENTS)

    # Normalize embeddings.
    norm_emb = normalize_embeddings(reduced)

    # Build a Faiss index for similarity search.
    d = norm_emb.shape[1]
    faiss_index = faiss.IndexFlatIP(d)
    faiss_index.add(norm_emb)

    # Hierarchical clustering via recursive k-means splitting.
    if not ONLY_OUTPUT_SEEDS:
        all_indices = list(range(len(norm_emb)))
        hierarchy = hierarchical_split(norm_emb, all_indices, max_size=MAX_CLUSTER_SIZE)
        leaves = collect_leaves(hierarchy)
        print(f"\nTotal leaf clusters: {len(leaves)}")

        # Compute leaf centers and determine merge candidates.
        centers = {i: compute_leaf_center(norm_emb, leaf["indices"]) for i, leaf in enumerate(leaves)}
        num_leaves = len(centers)
        merges = []
        for i in range(num_leaves):
            for j in range(i + 1, num_leaves):
                cosine_distance = 1 - np.dot(centers[i], centers[j])
                if cosine_distance < MERGE_THRESHOLD:
                    merges.append((i, j, cosine_distance))
        merged_leaves = merge_leaves_once(leaves, merges)
        print(f"After merging, total clusters: {len(merged_leaves)}")
    else:
        merged_leaves = []

    # Process seed examples.
    seed_clusters = process_seed_examples(norm_emb, ids, faiss_index, max_per_file=4)
    if seed_clusters:
        print(f"\nProcessed {len(seed_clusters)} seed clusters.")

    # Combine merged and seed clusters, preview as needed, sort and save JSON.
    build_and_save_clusters(merged_leaves, seed_clusters, ids, norm_emb)

if __name__ == "__main__":
    main()

==========================================
</classifier/cluster.py>
==========================================

==========================================
<classifier/dataset.py>
Size: 3969 bytes | Lines:      111
==========================================
import os
import json
import torch
from torch.utils.data import Dataset
import numpy as np

PATH = os.path.dirname(__file__)

embeddings_dir = os.path.join(PATH, "..", ".cache", "embeddings")
labels_file = os.path.join(PATH, "..", ".cache", "cluster_labels.json")

class CrowDataset(Dataset):
    def __init__(self):
        # Load the labels from the JSON file.
        with open(labels_file, 'r') as f:
            self.raw_labels = json.load(f)
            self.labels = { key: label for key, label in self.raw_labels.items() if "reviewed" in label and label["reviewed"] }
        self.keys = list(self.labels.keys())
        self.print_label_stats()

    def print_label_stats(self):
        total_labels = len(self.labels)
        counts = {
            "crowCount": {},
            "crowAge": {},
            "quality": {},
            "alert": 0,
            "begging": 0,
            "softSong": 0,
            "rattle": 0,
            "mob": 0
        }

        for key, label in self.labels.items():
            cc = label.get("crowCount", 1)
            counts["crowCount"][cc] = counts["crowCount"].get(cc, 0) + 1

            ca = label.get("crowAge", 1)
            counts["crowAge"][ca] = counts["crowAge"].get(ca, 0) + 1

            q = label.get("quality", 2)
            if q == 3:
                q = 2  # Force value 3 (HQ) to be a 2 (for training)
            counts["quality"][q] = counts["quality"].get(q, 0) + 1

            counts["alert"] += 1 if label.get("alert", False) else 0
            counts["begging"] += 1 if label.get("begging", False) else 0
            counts["softSong"] += 1 if label.get("softSong", False) else 0
            counts["rattle"] += 1 if label.get("rattle", False) else 0
            counts["mob"] += 1 if label.get("mob", False) else 0

        print("\n=== Label Statistics ===")
        print(f"Total labels loaded: {total_labels}\n")

        print("crowCount distribution:")
        for cls in sorted(counts["crowCount"]):
            print(f"  Class {cls}: {counts['crowCount'][cls]}")

        print("\ncrowAge distribution:")
        for cls in sorted(counts["crowAge"]):
            print(f"  Class {cls}: {counts['crowAge'][cls]}")

        print("\nquality distribution:")
        for cls in sorted(counts["quality"]):
            print(f"  Class {cls}: {counts['quality'][cls]}")

        print(f"\nalert count: {counts['alert']}")
        print(f"begging count: {counts['begging']}")
        print(f"softSong count: {counts['softSong']}")
        print(f"rattle count: {counts['rattle']}")
        print(f"mob count: {counts['mob']}\n")

    def __len__(self):
        return len(self.keys)

    def __getitem__(self, idx):
        key = self.keys[idx]
        file_id, start, end = key.split("-")

        # Load embedding.
        cached_path = os.path.join(embeddings_dir, f"{file_id}.npy")
        if os.path.exists(cached_path):
            embedding = np.load(cached_path)
            embedding = embedding[int(start):int(end)]

        # Convert embedding to a torch tensor.
        embedding_tensor = torch.from_numpy(embedding).float()

        # Get label info.
        label = self.labels[key]
        crowCount = label.get('crowCount', 1)
        crowAge = label.get('crowAge', 1)
        alert = 1 if label.get('alert', False) else 0
        begging = 1 if label.get('begging', False) else 0
        softSong = 1 if label.get('softSong', False) else 0
        rattle = 1 if label.get('rattle', False) else 0
        mob = 1 if label.get('mob', False) else 0
        quality = label.get('quality', 2)
        if quality == 3:
            quality = 2 # Force value 3 (HQ) to be a 2 (for training)

        return embedding_tensor, {
            "crowCount": crowCount,
            "crowAge": crowAge,
            "alert": alert,
            "begging": begging,
            "softSong": softSong,
            "rattle": rattle,
            "mob": mob,
            "quality": quality
        }

==========================================
</classifier/dataset.py>
==========================================

==========================================
<classifier/evaluate.py>
Size: 6087 bytes | Lines:      137
==========================================
import torch
from torch.utils.data import DataLoader, random_split
from dataset import CrowDataset
from model import CrowClassifier
import os

PATH = os.path.dirname(__file__)
checkpoint_path = os.path.join(PATH, "logs", "checkpoints", "best_model.ckpt")

def print_overall_metrics(metrics):
    print("\n=== Overall Accuracy ===")
    for key, counts in metrics.items():
        acc = counts["correct"] / counts["total"] if counts["total"] > 0 else 0.0
        print(f"{key:10s}: {acc * 100:6.2f}% ({counts['correct']}/{counts['total']})")


def print_breakdown(title, breakdown, label_prefix="Class"):
    print(f"\n=== {title} Breakdown ===")
    for cls, (correct, total) in breakdown.items():
        acc = correct / total if total > 0 else 0.0
        print(f"  {label_prefix} {cls}: {acc * 100:6.2f}% ({correct}/{total})")


def evaluate_model(model, dataloader, device):
    model.eval()
    # Overall metrics for each attribute.
    metrics = {
        "crowCount": {"correct": 0, "total": 0},
        "crowAge": {"correct": 0, "total": 0},
        "quality": {"correct": 0, "total": 0},
        "alert": {"correct": 0, "total": 0},
        "begging": {"correct": 0, "total": 0},
        "softSong": {"correct": 0, "total": 0},
        "rattle": {"correct": 0, "total": 0},
        "mob": {"correct": 0, "total": 0}
    }

    # For per-class breakdown.
    # For crowCount, we now have 5 classes (0-4), while crowAge and quality remain 2 and 3 classes (1-indexed)
    multi_class_keys = {"crowCount": 5, "crowAge": 2, "quality": 2}
    breakdown = {}
    for key, num in multi_class_keys.items():
        breakdown[key] = {cls: [0, 0] for cls in (range(num) if key == "crowCount" else [i + 1 for i in range(num)])}

    # For binary attributes breakdown.
    binary_keys = ["alert", "begging", "softSong", "rattle", "mob"]
    breakdown_binary = {key: {0: [0, 0], 1: [0, 0]} for key in binary_keys}

    with torch.no_grad():
        for batch in dataloader:
            embeddings, labels = batch
            embeddings = embeddings.to(device)
            for key in labels:
                labels[key] = labels[key].to(device)

            outputs = model(embeddings)
            # Multi-class predictions (0-indexed).
            pred_crowCount = torch.argmax(outputs["crowCount"], dim=1)
            pred_crowAge = torch.argmax(outputs["crowAge"], dim=1)
            pred_quality = torch.argmax(outputs["quality"], dim=1)
            # Binary predictions: threshold at 0 and force 1D.
            pred_alert = (outputs["alert"].squeeze() > 0).long().view(-1)
            pred_begging = (outputs["begging"].squeeze() > 0).long().view(-1)
            pred_softSong = (outputs["softSong"].squeeze() > 0).long().view(-1)
            pred_rattle = (outputs["rattle"].squeeze() > 0).long().view(-1)
            pred_mob = (outputs["mob"].squeeze() > 0).long().view(-1)

            # Update overall metrics (for multi-class, crowCount no longer subtracts 1).
            for key, pred, gt in [
                ("crowCount", pred_crowCount, labels["crowCount"]),
                ("crowAge", pred_crowAge, labels["crowAge"] - 1),
                ("quality", pred_quality, labels["quality"] - 1)
            ]:
                metrics[key]["correct"] += (pred == gt).sum().item()
                metrics[key]["total"] += gt.size(0)
                # Per-class breakdown.
                for cls in range(multi_class_keys[key]):
                    mask = (gt == cls) if key == "crowCount" else (gt == cls)
                    total = mask.sum().item()
                    correct = ((pred == cls) & mask).sum().item()
                    if key == "crowCount":
                        breakdown[key][cls][0] += correct
                        breakdown[key][cls][1] += total
                    else:
                        # For crowAge and quality, we display classes as 1-indexed.
                        breakdown[key][cls + 1][0] += correct
                        breakdown[key][cls + 1][1] += total

            # Update overall metrics for binary attributes.
            for key, pred in [
                ("alert", pred_alert),
                ("begging", pred_begging),
                ("softSong", pred_softSong),
                ("rattle", pred_rattle),
                ("mob", pred_mob)
            ]:
                metrics[key]["correct"] += (pred == labels[key]).sum().item()
                metrics[key]["total"] += labels[key].size(0)
                for val in [0, 1]:
                    mask = (labels[key] == val)
                    total = mask.sum().item()
                    correct = ((pred == val) & mask).sum().item()
                    breakdown_binary[key][val][0] += correct
                    breakdown_binary[key][val][1] += total

    print_overall_metrics(metrics)

    for key in multi_class_keys:
        print_breakdown(key, breakdown[key], label_prefix="Class")

    for key in binary_keys:
        print_breakdown(key, breakdown_binary[key], label_prefix="Value")
    return metrics, breakdown, breakdown_binary


if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = CrowClassifier.load_from_checkpoint(checkpoint_path)
    model.to(device)

    # Create the dataset.
    dataset = CrowDataset()

    train_size = int(0.88 * len(dataset))
    _, val_dataset = random_split(dataset, [train_size, len(dataset) - train_size])
    val_loader = DataLoader(val_dataset, batch_size=1, num_workers=3, shuffle=False)

    print(f"\nEvaluating on {len(val_loader)} VALIDATE samples")
    metrics, breakdown, breakdown_binary = evaluate_model(model, val_loader, device)

    # Instead of duplicating the composite score code, use the model's compute_composite_score
    composite, task_scores = model.compute_composite_score(breakdown, breakdown_binary)
    print("\n=== Composite Score ===")
    print(f"Overall composite score: {composite * 100:.2f}%")
    print("Individual task scores:")
    for task, score in task_scores.items():
        print(f"  {task:10s}: {score * 100:.2f}%")

==========================================
</classifier/evaluate.py>
==========================================

==========================================
<classifier/model.py>
Size: 10574 bytes | Lines:      234
==========================================
import torch
import torch.nn as nn
import pytorch_lightning as pl
from lightning_fabric import seed_everything

# Seed for reproducibility.
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False


class CrowClassifier(pl.LightningModule):
    def __init__(self, input_dim=768, hidden_dim=237, dropout_rate=0.3, seed=18202, lr=0.000145):
        super().__init__()

        seed_everything(seed, workers=True)
        self.lr = lr

        # A simple shared backbone.
        self.backbone = nn.Sequential(
            nn.Flatten(),
            nn.Linear(input_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(hidden_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout_rate)
        )

        # Output heads for each task.
        self.crowCount_head = nn.Linear(hidden_dim, 5)  # 5 classes (labels: 0,1,2,3,4)
        self.crowAge_head = nn.Linear(hidden_dim, 2)    # 2 classes: adult vs juvenile
        self.alert_head = nn.Linear(hidden_dim, 1)      # binary
        self.begging_head = nn.Linear(hidden_dim, 1)    # binary
        self.softSong_head = nn.Linear(hidden_dim, 1)   # binary
        self.rattle_head = nn.Linear(hidden_dim, 1)     # binary
        self.mob_head = nn.Linear(hidden_dim, 1)        # binary
        self.quality_head = nn.Linear(hidden_dim, 2)    # 3 classes: bad, average, HQ

        # Loss functions.
        self.loss_fn_class = nn.CrossEntropyLoss()
        self.loss_fn_bce = nn.BCEWithLogitsLoss()

    def forward(self, x):
        rep = self.backbone(x)
        out = {
            "crowCount": self.crowCount_head(rep),
            "crowAge": self.crowAge_head(rep),
            "alert": self.alert_head(rep),
            "begging": self.begging_head(rep),
            "softSong": self.softSong_head(rep),
            "rattle": self.rattle_head(rep),
            "mob": self.mob_head(rep),
            "quality": self.quality_head(rep)
        }
        return out

    def training_step(self, batch, batch_idx):
        embeddings, labels = batch
        outputs = self(embeddings)
        loss = (
            self.loss_fn_class(outputs["crowCount"], labels["crowCount"].long()) +
            self.loss_fn_class(outputs["crowAge"], (labels["crowAge"] - 1).long()) +
            self.loss_fn_bce(outputs["alert"].view(-1), labels["alert"].float().view(-1)) +
            self.loss_fn_bce(outputs["begging"].view(-1), labels["begging"].float().view(-1)) +
            self.loss_fn_bce(outputs["softSong"].view(-1), labels["softSong"].float().view(-1)) +
            self.loss_fn_bce(outputs["rattle"].view(-1), labels["rattle"].float().view(-1)) +
            self.loss_fn_bce(outputs["mob"].view(-1), labels["mob"].float().view(-1)) +
            self.loss_fn_class(outputs["quality"], (labels["quality"] - 1).long())
        )
        self.log('train_loss', loss)
        return loss

    def validation_step(self, batch, batch_idx):
        """
        Compute predictions and loss on the validation batch.
        Returns a dict of predictions, ground truths, and loss.
        """
        embeddings, labels = batch
        outputs = self(embeddings)
        loss = (
            self.loss_fn_class(outputs["crowCount"], labels["crowCount"].long()) +
            self.loss_fn_class(outputs["crowAge"], (labels["crowAge"] - 1).long()) +
            self.loss_fn_bce(outputs["alert"].view(-1), labels["alert"].float().view(-1)) +
            self.loss_fn_bce(outputs["begging"].view(-1), labels["begging"].float().view(-1)) +
            self.loss_fn_bce(outputs["softSong"].view(-1), labels["softSong"].float().view(-1)) +
            self.loss_fn_bce(outputs["rattle"].view(-1), labels["rattle"].float().view(-1)) +
            self.loss_fn_bce(outputs["mob"].view(-1), labels["mob"].float().view(-1)) +
            self.loss_fn_class(outputs["quality"], (labels["quality"] - 1).long())
        )

        # Multi-class predictions (0-indexed).
        pred_crowCount = torch.argmax(outputs["crowCount"], dim=1)
        pred_crowAge = torch.argmax(outputs["crowAge"], dim=1)
        pred_quality = torch.argmax(outputs["quality"], dim=1)
        # Binary predictions.
        pred_alert = (outputs["alert"].squeeze() > 0).long().view(-1)
        pred_begging = (outputs["begging"].squeeze() > 0).long().view(-1)
        pred_softSong = (outputs["softSong"].squeeze() > 0).long().view(-1)
        pred_rattle = (outputs["rattle"].squeeze() > 0).long().view(-1)
        pred_mob = (outputs["mob"].squeeze() > 0).long().view(-1)

        out = {
            "crowCount": {"pred": pred_crowCount, "gt": labels["crowCount"]},
            "crowAge": {"pred": pred_crowAge, "gt": labels["crowAge"] - 1},  # adjust as before
            "quality": {"pred": pred_quality, "gt": labels["quality"] - 1},
            "alert": {"pred": pred_alert, "gt": labels["alert"]},
            "begging": {"pred": pred_begging, "gt": labels["begging"]},
            "softSong": {"pred": pred_softSong, "gt": labels["softSong"]},
            "rattle": {"pred": pred_rattle, "gt": labels["rattle"]},
            "mob": {"pred": pred_mob, "gt": labels["mob"]},
            "loss": loss
        }
        # Append output to a temporary list (see on_validation_epoch_* hooks below).
        self._validation_outputs.append(out)
        return out

    def on_validation_epoch_start(self):
        # Clear the validation outputs at the start of the epoch.
        self._validation_outputs = []

    def on_validation_epoch_end(self):
        """
        Aggregate all validation_step outputs, compute per-task breakdowns,
        calculate the composite metric, and log the results.
        """
        # Expected number of classes for multi-class tasks.
        multi_class_keys = {"crowCount": 5, "crowAge": 2, "quality": 2}
        breakdown = {key: {cls: [0, 0] for cls in range(num)} for key, num in multi_class_keys.items()}
        # Binary tasks.
        binary_keys = ["alert", "begging", "softSong", "rattle", "mob"]
        breakdown_binary = {key: {0: [0, 0], 1: [0, 0]} for key in binary_keys}

        total_loss = 0.0
        batch_count = len(self._validation_outputs)
        for batch_out in self._validation_outputs:
            total_loss += batch_out["loss"].item()
            # Multi-class tasks.
            for task in multi_class_keys:
                preds = batch_out[task]["pred"].detach().cpu()
                gts = batch_out[task]["gt"].detach().cpu()
                for cls in range(multi_class_keys[task]):
                    mask = (gts == cls)
                    total = mask.sum().item()
                    correct = ((preds == cls) & mask).sum().item()
                    breakdown[task][cls][0] += correct
                    breakdown[task][cls][1] += total
            # Binary tasks.
            for task in binary_keys:
                preds = batch_out[task]["pred"].detach().cpu()
                gts = batch_out[task]["gt"].detach().cpu()
                for val in [0, 1]:
                    mask = (gts == val)
                    total = mask.sum().item()
                    correct = ((preds == val) & mask).sum().item()
                    breakdown_binary[task][val][0] += correct
                    breakdown_binary[task][val][1] += total

        avg_loss = total_loss / batch_count if batch_count > 0 else 0.0
        composite, task_scores = self.compute_composite_score(breakdown, breakdown_binary)

        self.log("val_loss", avg_loss, prog_bar=True)
        self.log("val_composite_score", composite, prog_bar=True)
        for task, score in task_scores.items():
            self.log(f"val_{task}_score", score)
        # (Optionally, you can also print or store these metrics.)

    def compute_composite_score(self, breakdown, breakdown_binary, weights=None):
        """
        Compute a composite score as a weighted average of:
          - Macro accuracy for multi-class tasks (crowCount, crowAge, quality)
          - Balanced accuracy for binary tasks (alert, begging, softSong, rattle, mob)
        Args:
            breakdown (dict): Breakdown dict for multi-class tasks.
            breakdown_binary (dict): Breakdown dict for binary tasks.
            weights (dict, optional): Task weights.
              Defaults to:
              {
                  "crowCount": 2,
                  "crowAge": 1,
                  "quality": 1,
                  "alert": 1.5,
                  "begging": 1.5,
                  "softSong": 1.5,
                  "rattle": 2,
                  "mob": 2
              }
        Returns:
            composite_score (float): Weighted composite score.
            task_scores (dict): Dictionary of individual task scores.
        """
        if weights is None:
            weights = {
                "crowCount": 2,
                "crowAge": 1,
                "quality": 1,
                "alert": 1.5,
                "begging": 1.5,
                "softSong": 1.5,
                "rattle": 2,
                "mob": 2
            }
        task_scores = {}

        # Multi-class tasks: macro accuracy = average accuracy over all classes.
        multi_class_tasks = ["crowCount", "crowAge", "quality"]
        for task in multi_class_tasks:
            class_accuracies = []
            for cls, (correct, total) in breakdown[task].items():
                if total > 0:
                    class_accuracies.append(correct / total)
            task_scores[task] = sum(class_accuracies) / len(class_accuracies) if class_accuracies else 0.0

        # Binary tasks: balanced accuracy = (acc_0 + acc_1) / 2.
        binary_tasks = ["alert", "begging", "softSong", "rattle", "mob"]
        for task in binary_tasks:
            acc_vals = []
            for val in [0, 1]:
                correct, total = breakdown_binary[task][val]
                acc_vals.append((correct / total) if total > 0 else 0.0)
            task_scores[task] = sum(acc_vals) / 2.0

        # Compute weighted average.
        total_weighted = 0.0
        total_weights = 0.0
        for task, score in task_scores.items():
            weight = weights.get(task, 1)
            total_weighted += weight * score
            total_weights += weight
        composite_score = total_weighted / total_weights if total_weights > 0 else 0.0
        return composite_score, task_scores

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.lr)
==========================================
</classifier/model.py>
==========================================

==========================================
<classifier/optuna_tuning.py>
Size: 3963 bytes | Lines:       93
==========================================
import optuna
import torch
from pytorch_lightning.callbacks import ModelCheckpoint
from torch.utils.data import DataLoader, random_split, Subset
import pytorch_lightning as pl
from pytorch_lightning.loggers import TensorBoardLogger

from dataset import CrowDataset
from model import CrowClassifier


def objective(trial: optuna.Trial):
    # Sample hyperparameters.
    hidden_dim = trial.suggest_int("hidden_dim", 237, 237)
    dropout_rate = trial.suggest_float("dropout_rate", 0.3, 0.3)
    random_seed = trial.suggest_int("random_seed", 0, 20000)
    learning_rate = trial.suggest_float("learning_rate", 0.000145, 0.000145)
    batch_size = trial.suggest_int("batch_size", 16, 30)
    rattle_oversample = trial.suggest_int("rattle_oversample", 1, 4)
    softsong_oversample = trial.suggest_int("softsong_oversample", 1, 4)
    begging_oversample = trial.suggest_int("begging_oversample", 1, 4)
    alert_oversample = trial.suggest_int("alert_oversample", 1, 4)
    mob_oversample = trial.suggest_int("mob_oversample", 1, 4)

    # Instantiate the model with sampled hyperparameters.
    model = CrowClassifier(hidden_dim=hidden_dim, dropout_rate=dropout_rate, seed=random_seed, lr=learning_rate)

    # Create the dataset and split into train/validation.
    dataset = CrowDataset()
    train_size = int(0.88 * len(dataset))
    val_size = len(dataset) - train_size
    train_dataset, val_dataset = random_split(dataset, [train_size, val_size])

    # Oversampling: Duplicate training indices for underrepresented labels.
    oversample_factors = {"rattle": rattle_oversample, "softSong": softsong_oversample, "begging": begging_oversample, "alert": alert_oversample, "mob": mob_oversample}
    oversampled_train_indices = []
    for idx in train_dataset.indices:
        _, label = dataset[idx]
        factor = 1
        for flag, dup_factor in oversample_factors.items():
            if label.get(flag, 0) == 1:
                factor *= dup_factor
        oversampled_train_indices.extend([idx] * factor)

    oversampled_train_dataset = Subset(dataset, oversampled_train_indices)

    train_loader = DataLoader(oversampled_train_dataset, batch_size=batch_size, num_workers=3, shuffle=True, drop_last=True)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, num_workers=3, shuffle=False, drop_last=True)

    # Set up checkpoint callback to monitor the composite score.
    checkpoint_callback = ModelCheckpoint(
        dirpath=f"logs/trial_{trial.number}/checkpoints",
        filename="best_model",
        monitor="val_composite_score",
        mode="max",  # We want to maximize composite score.
        save_top_k=1,
    )

    tb_logger = TensorBoardLogger("logs", name=f"crow-classify_trial_{trial.number}")

    trainer = pl.Trainer(
        max_epochs=16,
        logger=tb_logger,
        callbacks=[checkpoint_callback],
        enable_progress_bar=False,
    )

    trainer.fit(model, train_loader, val_loader)

    # Retrieve the best composite score.
    best_val_score = checkpoint_callback.best_model_score
    if best_val_score is None:
        # Fall back: try to get the last logged composite score.
        best_val_score = trainer.callback_metrics.get("val_composite_score")
        if best_val_score is None:
            raise ValueError("Validation composite score not found!")

    # Since we are maximizing the composite score, the higher the better.
    return best_val_score.item() if isinstance(best_val_score, torch.Tensor) else best_val_score


if __name__ == "__main__":
    # Change study direction to "maximize" because higher composite score is better.
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=60)

    print("Number of finished trials: ", len(study.trials))
    print("Best trial:")
    trial = study.best_trial
    print(f"  Value: {trial.value}")
    print("  Params: ")
    for key, value in trial.params.items():
        print(f"    {key}: {value}")

==========================================
</classifier/optuna_tuning.py>
==========================================

==========================================
<classifier/preview_auto_labels.py>
Size: 8449 bytes | Lines:      217
==========================================
#!/usr/bin/env python3
import os
import json
import librosa
import sounddevice as sd


def play_audio_preview(audio_file, start_time, duration):
    try:
        y, sr = librosa.load(audio_file, sr=None, offset=start_time, duration=duration)
        sd.play(y, sr)
        sd.wait()
    except Exception as e:
        print(f"Error playing {audio_file}: {e}")


def save_cluster_labels(cluster_labels, cluster_labels_file):
    with open(cluster_labels_file, "w") as f:
        json.dump(cluster_labels, f, indent=4)
    print(f"Progress saved to {cluster_labels_file}.")


def append_missing_segments(cluster_labels_file, cluster_segments_file):
    # Load cluster_labels.
    if os.path.exists(cluster_labels_file):
        with open(cluster_labels_file, "r") as f:
            cluster_labels = json.load(f)
    else:
        print(f"No cluster labels found at {cluster_labels_file}. Skipping appending segments.")
        return

    # Load or initialize cluster_segments.
    if os.path.exists(cluster_segments_file):
        with open(cluster_segments_file, "r") as f:
            cluster_segments = json.load(f)
    else:
        cluster_segments = {}

    # Process each detection in cluster_labels.
    for key, detection in cluster_labels.items():
        parts = key.split("-")
        if len(parts) != 3:
            print(f"Skipping key with unexpected format: {key}")
            continue
        file_id, start_str, end_str = parts
        try:
            start_time = float(start_str)
            end_time = float(end_str)
        except ValueError:
            print(f"Invalid start/end times in key: {key}")
            continue

        # Ensure the file_id exists in cluster_segments.
        if file_id not in cluster_segments:
            cluster_segments[file_id] = []

        # Check if the segment (by start and end times) already exists.
        segment_exists = any(
            seg.get("start_time") == start_time and seg.get("end_time") == end_time
            for seg in cluster_segments[file_id]
        )
        if not segment_exists:
            # Build a new segment entry following the correct format.
            segment_entry = {
                "common_name": "American Crow",
                "scientific_name": "Corvus brachyrhynchos",
                "start_time": start_time,
                "end_time": end_time,
                "confidence": detection.get("confidence", 0),
                "cluster": detection.get("cluster")
            }
            cluster_segments[file_id].append(segment_entry)
            print(f"Appended segment {key} to cluster_segments under file id {file_id}.")

    # Save the updated cluster_segments.
    with open(cluster_segments_file, "w") as f:
        json.dump(cluster_segments, f, indent=4)
    print(f"Updated cluster_segments saved to {cluster_segments_file}.")

def main(attribute, cluster, offset):
    base_dir = os.path.join(os.path.dirname(__file__), "..", ".cache")
    auto_labels_file = os.path.join(base_dir, "auto_labels.json")
    cluster_labels_file = os.path.join(base_dir, "cluster_labels.json")
    cluster_segments_file = os.path.join(base_dir, "cluster_segments.json")
    library_dir = os.path.join(base_dir, "library")

    # Load auto_labels.json.
    with open(auto_labels_file, "r") as f:
        auto_labels = json.load(f)

    # Load or initialize cluster_labels.json.
    if os.path.exists(cluster_labels_file):
        with open(cluster_labels_file, "r") as f:
            cluster_labels = json.load(f)
    else:
        cluster_labels = {}

    # Handle filter attribute with optional value.
    if ":" in attribute:
        attr_key, attr_val_str = attribute.split(":", 1)
        if attr_val_str == "":
            attr_val = True
        else:
            try:
                attr_val = int(attr_val_str)
            except ValueError:
                attr_val = attr_val_str
    else:
        attr_key = attribute
        attr_val = True

    # Filter for detections with the specified attribute.
    filtered_keys = [k for k, v in auto_labels.items() if v.get(attr_key) == attr_val]
    print(f"Found {len(filtered_keys)} detections with {attr_key} == {attr_val}.")

    # Set to store file_ids that should be skipped.
    skipped_file_ids = set()

    for key in filtered_keys[offset:]:
        parts = key.split("-")
        if len(parts) != 3:
            print(f"Skipping key with unexpected format: {key}")
            continue
        file_id, start, end = parts

        # Skip detections from file_ids already marked to be skipped.
        if file_id in skipped_file_ids:
            print(f"Skipping detection {key} because file id {file_id} is marked as skipped.")
            continue

        # Skip if already included.
        if key in cluster_labels:
            print(f"Skipped {key}: already in cluster_labels.")
            continue

        detection = auto_labels[key]
        audio_file = os.path.join(library_dir, f"{file_id}.mp3")
        if not os.path.exists(audio_file):
            print(f"Audio file not found: {audio_file}")
            continue

        try:
            start_time = float(start)
            end_time = float(end)
        except ValueError:
            print(f"Invalid start/end in key: {key}")
            continue

        duration = end_time - start_time
        print(f"\nPlaying detection {key} from {audio_file} (start: {start_time}, duration: {duration}) ...")
        print(detection)
        play_audio_preview(audio_file, start_time, duration)

        response = input("Include this detection? (Y = include, N = skip, Z = add as garbage, S = skip file, Q = quit): ").strip().lower()
        if response in ("q", "quit"):
            print("Quitting early. Saving progress...")
            save_cluster_labels(cluster_labels, cluster_labels_file)
            append_missing_segments(cluster_labels_file, cluster_segments_file)
            return
        elif response == "s":
            skipped_file_ids.add(file_id)
            print(f"Skipping all further detections for file id {file_id}.")
            continue
        elif response == "y":
            new_detection = detection.copy()
            new_detection["cluster"] = cluster
            cluster_labels[key] = new_detection
            print(f"Added {key} to cluster_labels with cluster {cluster}.")
            save_cluster_labels(cluster_labels, cluster_labels_file)
        elif response == "z":
            # Create a garbage label.
            garbage_detection = {
                "crowCount": 0,
                "crowAge": 1,
                "alert": False,
                "begging": False,
                "softSong": False,
                "rattle": False,
                "mob": False,
                "quality": 1,
                "cluster": cluster
            }
            cluster_labels[key] = garbage_detection
            print(f"Added {key} as garbage to cluster_labels with cluster {cluster}.")
            save_cluster_labels(cluster_labels, cluster_labels_file)
        else:
            print("Skipped.")

    print("\nAll detections processed.")
    save_cluster_labels(cluster_labels, cluster_labels_file)
    append_missing_segments(cluster_labels_file, cluster_segments_file)

if __name__ == "__main__":
    attr = input("Enter detection attribute to filter (e.g., rattle, softSong, begging, mob, alert, quality:1, crowCount:2, crowAge:1): ").strip()

    # Calculate default cluster value based on the largest existing cluster value + 1.
    base_dir = os.path.join(os.path.dirname(__file__), "..", ".cache")
    cluster_labels_file = os.path.join(base_dir, "cluster_labels.json")
    default_cluster = 1
    if os.path.exists(cluster_labels_file):
        with open(cluster_labels_file, "r") as f:
            existing_cluster_labels = json.load(f)
        cluster_values = []
        for detection in existing_cluster_labels.values():
            cluster_val = detection.get("cluster")
            if isinstance(cluster_val, int):
                cluster_values.append(cluster_val)
        if cluster_values:
            default_cluster = max(cluster_values) + 1

    clus_input = input(f"Enter cluster number to assign [default: {default_cluster}]: ").strip()
    clus = int(clus_input) if clus_input else default_cluster

    offset_input = input("Enter starting offset [default: 0]: ").strip()
    offset = int(offset_input) if offset_input else 0

    main(attr, clus, offset)

==========================================
</classifier/preview_auto_labels.py>
==========================================

==========================================
<classifier/remove_dupes.py>
Size: 5596 bytes | Lines:      153
==========================================
#!/usr/bin/env python3
import json
import os
import numpy as np

# Adjust CACHE_DIR to point to your .cache folder.
CACHE_DIR = os.path.join(os.path.dirname(__file__), "..", ".cache")
SEGMENTS_FILE = os.path.join(CACHE_DIR, "cluster_segments.json")
LABELS_FILE = os.path.join(CACHE_DIR, "cluster_labels.json")

def remove_duplicate_segments(segments_data):
    """
    For each file_id in segments_data, remove duplicate segments by:
      1. Comparing start and end times.
      2. Checking that the embedding slice lengths (using .shape[0]) are identical.
      3. Comparing the embedding arrays with near-equality.
    Only one segment (and its corresponding label) is kept per unique embedding.
    """
    deduped = {}
    seen = {}
    for file_id, segments in segments_data.items():
        unique_segments = []
        emb_path = os.path.join(CACHE_DIR, "embeddings", f"{file_id}.npy")
        try:
            embedding_array = np.load(emb_path)
        except Exception as e:
            print(f"Error loading embedding for {file_id}: {e}")
            embedding_array = None

        # Dictionary grouping segments by (start, end, emb_length)
        for seg in segments:
            start = seg.get("start_time")
            end = seg.get("end_time")
            if embedding_array is not None:
                try:
                    s = int(start)
                    e = int(end)
                    seg_embedding = embedding_array[s:e]
                    emb_length = seg_embedding.shape[0]
                except Exception as ex:
                    print(f"Error processing embedding slice for {file_id} segment {seg}: {ex}")
                    seg_embedding = None
                    emb_length = None
            else:
                seg_embedding = None
                emb_length = None

            composite_key = (start, end, emb_length)
            duplicate_found = False
            if composite_key in seen:
                for existing_embedding in seen[composite_key]:
                    # Use near-equality check with a small tolerance.
                    if seg_embedding is not None and np.allclose(seg_embedding, existing_embedding, atol=1e-5):
                        print(f"Removing duplicate segment in {file_id}: {seg}")
                        duplicate_found = True
                        break
            if duplicate_found:
                continue
            seen.setdefault(composite_key, []).append(seg_embedding)
            unique_segments.append(seg)
        deduped[file_id] = unique_segments
    return deduped

def remove_duplicate_labels(labels_data):
    """
    Remove duplicate label entries from a labels dictionary.
    The keys are expected to be strings of the form "file_id-start-end".
    """
    deduped = {}
    for key, label in labels_data.items():
        try:
            parts = key.split("-")
            if len(parts) != 3:
                print(f"Skipping label with unexpected key format: {key}")
                continue
            file_id, start, end = parts
            key_tuple = (file_id, float(start), float(end))
        except Exception as e:
            print(f"Error parsing key {key}: {e}")
            continue

        if key_tuple in deduped:
            print(f"Removing duplicate label: {key} -> {label}")
        else:
            deduped[key_tuple] = label

    new_labels = {}
    for key_tuple, label in deduped.items():
        new_key = f"{key_tuple[0]}-{int(key_tuple[1])}-{int(key_tuple[2])}"
        new_labels[new_key] = label
    return new_labels

def remove_missing_labels(labels_data, segments_data):
    """
    Remove any label entry for which there is no corresponding segment.
    A corresponding segment is defined as one in segments_data with the same file_id,
    start_time, and end_time.
    """
    new_labels = {}
    for key, label in labels_data.items():
        try:
            file_id, start, end = key.split("-")
            start = float(start)
            end = float(end)
        except Exception as e:
            print(f"Error parsing label key {key}: {e}")
            continue

        if file_id not in segments_data:
            print(f"Removing label {key}: file_id {file_id} not found in segments.")
            continue

        segments = segments_data[file_id]
        found = any(seg.get("start_time") == start and seg.get("end_time") == end for seg in segments)
        if not found:
            print(f"Removing label {key}: corresponding segment not found.")
        else:
            new_labels[key] = label
    return new_labels

def main():
    # Load segments file.
    try:
        with open(SEGMENTS_FILE, "r") as f:
            segments_data = json.load(f)
    except Exception as e:
        print(f"Error loading segments file: {e}")
        return

    # Remove duplicates from segments.
    segments_data = remove_duplicate_segments(segments_data)
    with open(SEGMENTS_FILE, "w") as f:
        json.dump(segments_data, f, indent=4)
    print("Updated segments file.")

    # Load labels file.
    try:
        with open(LABELS_FILE, "r") as f:
            labels_data = json.load(f)
    except Exception as e:
        print(f"Error loading labels file: {e}")
        return

    # Remove duplicate labels.
    labels_data = remove_duplicate_labels(labels_data)
    # Remove label entries for missing segments.
    labels_data = remove_missing_labels(labels_data, segments_data)
    with open(LABELS_FILE, "w") as f:
        json.dump(labels_data, f, indent=4)
    print("Updated labels file.")

if __name__ == "__main__":
    main()

==========================================
</classifier/remove_dupes.py>
==========================================

==========================================
<classifier/train.py>
Size: 2133 bytes | Lines:       58
==========================================
import torch
from pytorch_lightning.callbacks import ModelCheckpoint
from torch.utils.data import DataLoader, random_split, Subset
import pytorch_lightning as pl
from pytorch_lightning.loggers import TensorBoardLogger
from dataset import CrowDataset
from model import CrowClassifier

# Instantiate the model.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = CrowClassifier()
model.to(device)

# Create the dataset.
dataset = CrowDataset()

# Split dataset into training and validation (ensuring unique records in each).
train_size = int(0.88 * len(dataset))
val_size = len(dataset) - train_size
train_dataset, val_dataset = random_split(dataset, [train_size, val_size])

# Oversampling: Duplicate training indices for underrepresented labels.
oversample_factors = {"rattle": 4, "softSong": 4, "begging": 1, "alert": 3, "mob": 1}

oversampled_train_indices = []
# train_dataset.indices gives the list of indices from the original dataset in the training subset.
for idx in train_dataset.indices:
    _, label = dataset[idx]
    factor = 1
    # Multiply factors for each applicable flag.
    for flag, dup_factor in oversample_factors.items():
        if label.get(flag, 0) == 1:
            factor *= dup_factor
    # Add this index 'factor' times.
    oversampled_train_indices.extend([idx] * factor)

# Create a new training subset with the oversampled indices.
oversampled_train_dataset = Subset(dataset, oversampled_train_indices)

# Create DataLoaders.
train_loader = DataLoader(oversampled_train_dataset, batch_size=21, num_workers=3, shuffle=True, drop_last=True)
val_loader = DataLoader(val_dataset, batch_size=21, num_workers=3, shuffle=False, drop_last=True)

# Create TensorBoard logger.
tb_logger = TensorBoardLogger("logs", name="crow-classify")

# Checkpointing.
checkpoint_callback = ModelCheckpoint(
    dirpath="logs/checkpoints",
    filename="best_model",
    monitor="val_composite_score",
    mode="max",
    save_top_k=3
)

# Initialize the Trainer.
trainer = pl.Trainer(max_epochs=20, logger=tb_logger, callbacks=[checkpoint_callback])
trainer.fit(model, train_loader, val_loader)

==========================================
</classifier/train.py>
==========================================

==========================================
<condense_for_llm.sh>
Size: 5379 bytes | Lines:      194
==========================================
#!/bin/bash
# Generates a consolidated view of the crow-tools repository for LLM ingestion.

set -euo pipefail

OUTPUT_FILE="crow-tools-condensed-for-llm.out"

# Initialize/Clear the output file
> "$OUTPUT_FILE"

echo "Generating $OUTPUT_FILE..."

# Add header with metadata
{
    echo "=========================================="
    echo "CROW-TOOLS REPOSITORY SNAPSHOT"
    echo "Generated: $(date '+%Y-%m-%d %H:%M:%S')"
    echo "=========================================="
    echo ""
} >> "$OUTPUT_FILE"

# Generate full directory tree
echo "<FULL TREE>" >> "$OUTPUT_FILE"
if command -v python3 >/dev/null 2>&1; then
    python3 <<'PY' >> "$OUTPUT_FILE"
import os

ROOT = "."
SKIP_DIRS = {".git", "__pycache__", ".mypy_cache", ".pytest_cache", "node_modules", ".venv", "venv", ".cache"}
SKIP_FILES = {".DS_Store", "*.pyc", "*.pyo", "*.pyd", ".Python"}

def should_skip_file(name):
    return any(name.endswith(ext.lstrip('*')) for ext in SKIP_FILES if '*' in ext) or name in SKIP_FILES

def iter_entries(path):
    entries = []
    try:
        with os.scandir(path) as it:
            for entry in it:
                if entry.is_dir(follow_symlinks=False) and entry.name in SKIP_DIRS:
                    continue
                if entry.is_file(follow_symlinks=False) and should_skip_file(entry.name):
                    continue
                entries.append(entry)
    except PermissionError:
        pass
    entries.sort(key=lambda entry: (0 if entry.is_dir(follow_symlinks=False) else 1, entry.name.lower()))
    return entries

def walk(path, prefix=""):
    entries = iter_entries(path)
    total = len(entries)
    for index, entry in enumerate(entries):
        connector = "+-- " if index == total - 1 else "|-- "
        print(f"{prefix}{connector}{entry.name}")
        if entry.is_dir(follow_symlinks=False):
            extension = "    " if index == total - 1 else "|   "
            walk(os.path.join(path, entry.name), prefix + extension)

print(".")
walk(ROOT)
PY
else
    # Fallback to tree command or find
    if command -v tree >/dev/null 2>&1; then
        tree -I ".git|__pycache__|.mypy_cache|.pytest_cache|node_modules|.venv|venv|.cache" >> "$OUTPUT_FILE"
    else
        find . -not -path "*/.git/*" -not -path "*/__pycache__/*" -not -path "*/.cache/*" -not -path "*/node_modules/*" -print >> "$OUTPUT_FILE"
    fi
fi

echo "" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

# List of file patterns to include
INCLUDE_PATTERNS=(
    "*.py"
    "*.js"
    "*.html"
    "*.json"
    "*.md"
    "*.txt"
    "*.sh"
    "*.yml"
    "*.yaml"
    "*.toml"
    "*.cfg"
    "*.ini"
    "*.cff"
)

# Directories to exclude
EXCLUDE_DIRS=(
    ".git"
    "__pycache__"
    ".mypy_cache"
    ".pytest_cache"
    "node_modules"
    ".venv"
    "venv"
    ".cache"
    "docs/images"
    "docs/videos"
)

# Build find command with exclusions
FIND_CMD="find . -type f \("
for pattern in "${INCLUDE_PATTERNS[@]}"; do
    FIND_CMD="$FIND_CMD -iname \"$pattern\" -o"
done
FIND_CMD="${FIND_CMD% -o} \)"

for dir in "${EXCLUDE_DIRS[@]}"; do
    FIND_CMD="$FIND_CMD -not -path \"*/$dir/*\""
done

# Add file list summary
echo "<FILE SUMMARY>" >> "$OUTPUT_FILE"
echo "Total files included in this condensed view:" >> "$OUTPUT_FILE"
eval "$FIND_CMD" | wc -l | xargs echo "Files:" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

# Group files by directory
echo "<FILES BY DIRECTORY>" >> "$OUTPUT_FILE"
eval "$FIND_CMD" | sed 's|^\./||' | awk -F/ '{
    if (NF == 1) {
        dir = "(root)"
    } else {
        dir = $1
        for (i = 2; i < NF; i++) {
            dir = dir "/" $i
        }
    }
    dirs[dir]++
}
END {
    for (dir in dirs) {
        print dir ": " dirs[dir] " files"
    }
}' | sort >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

# Now output each file with clear delimiters
echo "<FILE CONTENTS>" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

eval "$FIND_CMD" | sort | while IFS= read -r file; do
    # Skip the output file itself
    if [[ "$file" == "./$OUTPUT_FILE" ]]; then
        continue
    fi
    
    # Get relative path
    rel_path="${file#./}"
    
    # Get file size
    if [[ "$OSTYPE" == "darwin"* ]]; then
        file_size=$(stat -f%z "$file" 2>/dev/null || echo "unknown")
    else
        file_size=$(stat -c%s "$file" 2>/dev/null || echo "unknown")
    fi
    
    # Get line count for text files
    line_count=$(wc -l < "$file" 2>/dev/null || echo "0")
    
    {
        echo "=========================================="
        echo "<$rel_path>"
        echo "Size: $file_size bytes | Lines: $line_count"
        echo "=========================================="
        
        # Check if file is likely binary
        if file "$file" | grep -q "text\|JSON\|ASCII\|UTF-8\|Python\|shell\|HTML\|JavaScript"; then
            cat "$file"
        else
            echo "[Binary or non-text file - content omitted]"
        fi
        
        echo ""
        echo "=========================================="
        echo "</$rel_path>"
        echo "=========================================="
        echo ""
    } >> "$OUTPUT_FILE"
done

echo "" >> "$OUTPUT_FILE"
echo "=========================================="
echo "END OF REPOSITORY SNAPSHOT"
echo "=========================================="

echo "Done! Content written to $OUTPUT_FILE"
echo "File size: $(du -h "$OUTPUT_FILE" | cut -f1)"

==========================================
</condense_for_llm.sh>
==========================================

==========================================
<denoiser/denoise_crows.py>
Size: 3203 bytes | Lines:       74
==========================================
import os
import torch
import torchaudio
import librosa
from biodenoising import pretrained
from biodenoising.denoiser.dsp import convert_audio
from tqdm import tqdm

# Set device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
PATH = os.path.dirname(__file__)

# Load the pre-trained model
model = pretrained.biodenoising16k_dns48().to(device)

def start_denoising():
    # Define directories
    library_dir = os.path.join(PATH, "..", ".cache", "library")
    output_dir = os.path.join(PATH, "..", ".cache", "library-denoised")
    os.makedirs(output_dir, exist_ok=True)

    # Get sorted list of files from the library directory
    files = sorted(os.listdir(library_dir))

    for filename in files:
        if filename.lower().endswith('.mp3'):
            base_name = os.path.splitext(filename)[0]
            output_file = os.path.join(output_dir, f"{base_name}.wav")

            # Skip if the output file already exists
            if os.path.exists(output_file):
                print(f"Skipping {filename} (denoised file exists)")
                continue

            input_file = os.path.join(library_dir, filename)
            print(f"Processing: {input_file}")

            # Load the MP3 file using librosa at 16kHz mono.
            # librosa.load returns a NumPy array of shape (samples,)
            audio_np, sr = librosa.load(input_file, sr=16000, mono=True)
            if sr != 16000:
                print(f"Warning: Sample rate mismatch for {filename}. Using SR={sr}.")
            # Convert the NumPy array to a torch tensor and add a channel dimension.
            wav = torch.from_numpy(audio_np).unsqueeze(0)

            # Convert audio to the model's expected sample rate and channel configuration.
            wav = convert_audio(wav, sr, model.sample_rate, model.chin).to(device)

            # Determine if we need to split the file (if longer than 1 minute).
            total_samples = wav.size(1)
            chunk_size = model.sample_rate * 60  # samples in 60 seconds
            if total_samples > chunk_size:
                print(f"Splitting {filename} into chunks...")
                denoised_chunks = []
                # Process each chunk individually.
                for start in tqdm(range(0, total_samples, chunk_size)):
                    end = min(start + chunk_size, total_samples)
                    chunk = wav[:, start:end]
                    with torch.no_grad():
                        # Add a batch dimension before denoising.
                        chunk_denoised = model(chunk.unsqueeze(0))[0]
                    denoised_chunks.append(chunk_denoised)
                # Reassemble the denoised chunks along the time dimension.
                denoised = torch.cat(denoised_chunks, dim=1)
            else:
                # Process the whole file if it is 1 minute or shorter.
                with torch.no_grad():
                    denoised = model(wav.unsqueeze(0))[0]

            # Save the denoised audio as a WAV file.
            torchaudio.save(output_file, denoised.cpu(), model.sample_rate)
            print(f"Saved denoised audio to: {output_file}")

if __name__ == "__main__":
    start_denoising()
==========================================
</denoiser/denoise_crows.py>
==========================================

==========================================
<detector/detect.py>
Size: 25198 bytes | Lines:      601
==========================================
#!/usr/bin/env python3
import json
import os
import sys
import time
import uuid
import shutil
from datetime import datetime
import tempfile

import librosa
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import sounddevice as sd
import torch
from tqdm import tqdm

# Import tkinter components for UI dialogs.
import tkinter as tk
from tkinter import messagebox, simpledialog

from classifier.classify import predict_embedding
from embedder.embed import generate_embeddings

def is_headless():
    return not os.environ.get("DISPLAY") and os.name != "nt"
if is_headless():
    matplotlib.use("Agg")
else:
    matplotlib.use("TkAgg")

###############################################################################
# Helper function to retrieve embeddings, volumes, audio and sample rate.
###############################################################################
def get_data(arg, public_path=None):
    """
    Given a file identifier or full file path, returns:
      - embeddings: a 2D numpy array (num_seconds x feature_dim)
      - volumes: a 1D numpy array of per-second volume metrics
      - audio: the audio waveform (mono)
      - sr: sample rate

    If arg is a valid file path, the function will:
      - Load the audio (forcing a sample rate of 8000 Hz for consistency)
      - Generate raw embeddings via generate_embeddings()
      - Average every 25 frames (i.e. ~1 second) into one embedding vector
      - Compute per‑second volume metrics from the waveform
      (No files are written to disk.)

    Otherwise, it assumes arg is a file ID and attempts to load cached files from:
      - {public_path}/embeddings-denoised/{file_id}.npy
      - {public_path}/embeddings-denoised-volumes/{file_id}.npy
      - {public_path}/library/{file_id}.mp3
    """
    if public_path is None:
        public_path = os.path.join(os.path.dirname(__file__), "..", ".cache")

    if os.path.isfile(arg):
        # Uncached mode: arg is a file path.
        print(f"Processing uncached file: {arg}")
        sample_rate = 8000  # force consistent sample rate (as in embed_all.py)
        cache_path = os.path.join(tempfile.gettempdir(), os.path.basename(arg) + ".avg_embeddings.npy")
        try:
            audio, sr = librosa.load(arg, sr=sample_rate, mono=True)
        except Exception as e:
            print(f"Error loading audio from {arg}: {e}")
            sys.exit(1)
        if os.path.exists(cache_path):
            embeddings = np.load(cache_path)
        else:
            print("Generating embeddings ...")
            full_embeddings = generate_embeddings(audio)
            full_embeddings = np.array(full_embeddings, dtype=np.float32)
            num_frames = full_embeddings.shape[0]
            # Average every 25 frames to get one embedding per second.
            chunk_size = 25
            num_chunks = int(np.ceil(num_frames / chunk_size))
            embedding_means = []
            for i in tqdm(range(num_chunks)):
                start_idx = i * chunk_size
                end_idx = min((i + 1) * chunk_size, num_frames)
                chunk = full_embeddings[start_idx:end_idx]
                mean_emb = np.mean(chunk, axis=0)
                embedding_means.append(mean_emb)
            embeddings = np.stack(embedding_means, axis=0)
            np.save(cache_path, embeddings)
        # Compute per-second volume metrics (mean absolute amplitude).
        total_samples = len(audio)
        total_seconds = int(np.ceil(total_samples / sr))
        volumes = []
        for sec in tqdm(range(total_seconds)):
            start_samp = sec * sr
            end_samp = min((sec + 1) * sr, total_samples)
            segment = audio[start_samp:end_samp]
            mean_amp = np.mean(np.abs(segment)) if len(segment) > 0 else 0.0
            volumes.append(mean_amp)
        volumes = np.array(volumes, dtype=np.float32)
        return embeddings, volumes, audio, sr

    else:
        # Cached mode: arg is a file ID.
        file_id = arg
        embeddings_path = os.path.join(public_path, "embeddings", f"{file_id}.npy")
        volumes_path = os.path.join(public_path, "embeddings-denoised-volumes", f"{file_id}.npy")
        library_dir = os.path.join(public_path, "library")
        audio_path = os.path.join(library_dir, f"{file_id}.mp3")

        if not os.path.exists(embeddings_path):
            print(f"Embedding file not found for file ID {file_id}")
            sys.exit(1)
        if not os.path.exists(volumes_path):
            print(f"Volume file not found for file ID {file_id}")
            sys.exit(1)
        if not os.path.exists(audio_path):
            print(f"Audio file not found for file ID {file_id}")
            sys.exit(1)

        embeddings = np.load(embeddings_path)
        volumes = np.load(volumes_path)
        if volumes.ndim > 1:
            volumes = volumes.squeeze(-1)
        try:
            audio, sr = librosa.load(audio_path, sr=None, mono=True)
        except Exception as e:
            print(f"Error loading audio: {e}")
            sys.exit(1)
        return embeddings, volumes, audio, sr

###############################################################################
# Detection function (common for both cached and uncached data)
###############################################################################
def detect_file_segments(arg, volume_threshold=0.0002, device=None, public_path=None):
    """
    Loads embeddings and volume data (and audio) using get_data() and then runs
    the classifier on each second where the volume exceeds the threshold.
    Returns:
      - detections: list of detection dictionaries
      - audio: the audio waveform
      - sr: sample rate
    """
    embeddings, volumes, audio, sr = get_data(arg, public_path)
    detections = []
    if device is None:
        device = "cuda" if torch.cuda.is_available() else "cpu"
    # Process seconds based on the lesser of embeddings length and volume length.
    num_seconds = min(embeddings.shape[0], len(volumes))
    for i in tqdm(range(num_seconds)):
        if volumes[i] <= volume_threshold:
            continue
        emb = embeddings[i]
        pred = predict_embedding(emb, device=device)
        # Only consider valid detections (quality > 1 and crowCount > 0).
        if pred["quality"] > 1 and pred["crowCount"] > 0:
            detection = {
                "start_time": float(i),
                "end_time": float(i + 1),
                "crowCount": pred["crowCount"],
                "crowAge": pred["crowAge"],
                "alert": pred["alert"],
                "begging": pred["begging"],
                "softSong": pred["softSong"],
                "rattle": pred["rattle"],
                "mob": pred["mob"],
                "quality": pred["quality"],
            }
            detections.append(detection)
    return detections, audio, sr

###############################################################################
# Generate detection summary (unchanged)
###############################################################################
def generate_detection_summary(detections, gap=5):
    """
    Returns a natural language paragraph summarizing high-quality detections.
    Only detections with quality==2 and at least one binary attribute True are considered.
    For each attribute (e.g., "softSong", "rattle", etc.) and for each natural description (crow count and age),
    only the first start time of each contiguous block (gaps > gap seconds) is listed.

    The start times are formatted as mm:ss.
    """
    # Define binary detection attributes.
    binary_keys = ["alert", "begging", "softSong", "rattle", "mob"]

    # Filter detections.
    filtered = [d for d in detections if d.get("quality") == 2 and any(d.get(attr, False) for attr in binary_keys)]
    if not filtered:
        return "No crow detections were found in the audio file."

    # Helper: Format seconds as mm:ss.
    def format_time(seconds):
        minutes = int(seconds // 60)
        secs = int(seconds % 60)
        return f"{minutes:02d}:{secs:02d}"

    # Helper: Collapse contiguous timestamps (sorted ascending) if difference <= gap seconds.
    def collapse_timestamps(times, gap=5):
        if not times:
            return []
        times = sorted(times)
        collapsed = [times[0]]
        last = times[0]
        for t in times[1:]:
            if t - last > gap:
                collapsed.append(t)
                last = t
        return collapsed

    # Helper: Natural language description for crow count.
    def crow_count_desc(count):
        if count == 1:
            return "a single crow"
        elif count == 2:
            return "a pair of crows"
        else:
            return f"a group of crows"

    # Helper: Natural language description for crow age.
    def crow_age_desc(age):
        return "adult" if age == 1 else "juvenile" if age == 2 else "of unknown age"

    # Group detections by binary attribute and natural description.
    groups = {}  # key: (attribute, description), value: list of start times (floats)
    for det in filtered:
        start = det.get("start_time")
        if start is None:
            continue
        count = det.get("crowCount", 1)
        age = det.get("crowAge", 1)
        description = f"{crow_count_desc(count)}, {crow_age_desc(age)}"
        for attr in binary_keys:
            if det.get(attr, False):
                key = (attr, description)
                groups.setdefault(key, []).append(float(start))

    # Build the paragraph summary.
    summary_parts = ["The following detections were observed in this recording:"]
    for (attr, description), times in groups.items():
        collapsed = collapse_timestamps(times, gap)
        if not collapsed:
            continue
        attr_nl = attr.replace("softSong", "subsong")
        times_str = ", ".join(format_time(t) for t in collapsed)
        num_groups = len(collapsed)
        summary_parts.append(f"{num_groups} {attr_nl} calls ({description}) at {times_str}.")
    return "\n".join(summary_parts)

###############################################################################
# INTERACTIVE TIMELINE PLAYER
###############################################################################
class TimelinePlayer:
    def __init__(self, label, detections, audio, sr, public_path=None, raw_file=None):
        """
        Args:
          label (str): file identifier or base filename.
          detections (list): list of detection dictionaries.
          audio (np.ndarray): audio waveform (mono).
          sr (int): sample rate.
          public_path (str): base path for .cache directory.
          raw_file (str): original file path if uncached raw file was provided;
                          if None, then a cached file ID was provided and the
                          "Add to Labeler" functionality is disabled.
        """
        self.label = label
        self.detections = detections
        self.audio = audio
        self.sr = sr
        self.total_duration = len(audio) / sr
        self.playing = False
        self.play_offset = 0.0  # seconds; playback start position
        self.play_start_time = None

        if public_path is None:
            self.public_path = os.path.join(os.path.dirname(__file__), "..", ".cache")
        else:
            self.public_path = public_path

        # Store the original raw file (if provided).
        self.source_file = raw_file

        # Define the boolean features to display.
        self.features = ["alert", "mob", "begging", "softSong", "rattle", "other"]
        self.feature_intervals = self.compute_feature_intervals()

        # Create a figure with two subplots: waveform and detection timeline.
        self.fig, (self.ax_wave, self.ax_det) = plt.subplots(2, 1, sharex=True, figsize=(19.2, 8), dpi=100)
        self.fig.subplots_adjust(left=0.1, right=0.85, top=0.9, bottom=0.2)
        self.fig.suptitle(f"File: {label} - Detection Timeline", fontsize=16)

        # Downsample waveform for efficiency.
        target_points = 1920
        if len(self.audio) > target_points:
            indices = np.linspace(0, len(self.audio) - 1, target_points).astype(int)
            downsampled_audio = self.audio[indices]
            t = np.linspace(0, self.total_duration, target_points)
        else:
            downsampled_audio = self.audio
            t = np.linspace(0, self.total_duration, len(self.audio))
        self.ax_wave.plot(t, downsampled_audio, color="blue", lw=0.8)
        self.ax_wave.set_ylabel("Amplitude")
        self.ax_wave.set_xlim(0, self.total_duration)
        self.ax_wave.set_title("Waveform")

        # Plot detection timeline.
        self.ax_det.set_title("Detections")
        self.ax_det.set_ylim(0, len(self.features))
        self.ax_det.set_yticks([i + 0.5 for i in range(len(self.features))])
        self.ax_det.set_yticklabels(self.features)
        self.ax_det.set_xlabel("Time (s)")
        self.ax_det.set_xticks(np.linspace(0, self.total_duration, num=5))
        self.ax_det.set_xlim(0, self.total_duration)
        self.ax_det.set_autoscale_on(False)

        # Draw detection bars.
        colors = {"alert": "red", "mob": "green", "begging": "orange",
                  "softSong": "purple", "rattle": "brown", "other": "gray"}
        for i, feat in enumerate(self.features):
            intervals = self.feature_intervals.get(feat, [])
            for (start, end) in intervals:
                self.ax_det.hlines(y=i + 0.5, xmin=start, xmax=end,
                                   colors=colors.get(feat, "gray"), linewidth=8)

        # Add a vertical playhead.
        self.playhead_line = self.ax_det.axvline(x=0, color="black", linestyle="--", lw=2)
        self.playhead_line.set_visible(False)
        self.fig.canvas.draw()
        self.background = self.fig.canvas.copy_from_bbox(self.ax_det.bbox)
        self.playhead_line.set_visible(True)

        # Add a Play/Pause button.
        self.button_ax = self.fig.add_axes([0.87, 0.05, 0.1, 0.05])
        self.play_button = plt.Button(self.button_ax, "Play")
        self.play_button.on_clicked(self.toggle_play)

        # Only add the "Add to Labeler" button if a raw file path was provided.
        if self.source_file is not None:
            self.add_button_ax = self.fig.add_axes([0.87, 0.12, 0.1, 0.05])
            self.add_button = plt.Button(self.add_button_ax, "Add to Labeler")
            self.add_button.on_clicked(self.add_to_labeler)

        # Connect mouse click and draw events.
        self.fig.canvas.mpl_connect("button_press_event", self.on_click)
        self.fig.canvas.mpl_connect("draw_event", self.on_draw)

        # Set up a timer to update the playhead.
        self.timer = self.fig.canvas.new_timer(interval=200)
        self.timer.add_callback(self.update_playhead)
        self.timer.start()

    def compute_feature_intervals(self):
        """
        Groups contiguous detection seconds for each feature.
        If no standard feature (alert, mob, begging, softSong, rattle) is true,
        the second is grouped under "other".
        """
        feat_seconds = {feat: [] for feat in self.features}
        standard_feats = {"alert", "mob", "begging", "softSong", "rattle"}

        for det in self.detections:
            t = det["start_time"]
            has_feature = any(det.get(feat, False) for feat in standard_feats)
            if has_feature:
                for feat in standard_feats:
                    if det.get(feat, False):
                        feat_seconds[feat].append(t)
            else:
                feat_seconds["other"].append(t)

        intervals = {}
        for feat, times in feat_seconds.items():
            times = sorted(times)
            if not times:
                continue
            intervals[feat] = []
            start = times[0]
            prev = times[0]
            for current in times[1:]:
                if current - prev > 1.5:
                    intervals[feat].append((start, prev + 1))
                    start = current
                prev = current
            intervals[feat].append((start, prev + 1))
        return intervals

    def on_draw(self, event):
        self.background = self.fig.canvas.copy_from_bbox(self.ax_det.bbox)

    def toggle_play(self, event):
        if self.playing:
            sd.stop()
            self.playing = False
            self.play_button.label.set_text("Play")
        else:
            self.start_playback()

    def start_playback(self):
        start_sample = int(self.play_offset * self.sr)
        remaining_audio = self.audio[start_sample:]
        if len(remaining_audio) == 0:
            return
        sd.play(remaining_audio, self.sr)
        self.play_start_time = time.time()
        self.playing = True
        self.play_button.label.set_text("Pause")

    def update_playhead(self):
        if self.playing:
            elapsed = time.time() - self.play_start_time
            current_time = self.play_offset + elapsed
            if current_time >= self.total_duration:
                current_time = self.total_duration
                self.playing = False
                self.play_button.label.set_text("Play")
            self.playhead_line.set_xdata(current_time)
            self.fig.canvas.restore_region(self.background)
            self.ax_det.draw_artist(self.playhead_line)
            self.fig.canvas.blit(self.ax_det.bbox)

    def on_click(self, event):
        if event.inaxes == self.ax_det:
            new_time = event.xdata
            if new_time is None:
                return
            self.play_offset = new_time
            self.playhead_line.set_xdata(new_time)
            self.fig.canvas.restore_region(self.background)
            self.ax_det.draw_artist(self.playhead_line)
            self.fig.canvas.blit(self.ax_det.bbox)
            if self.playing:
                sd.stop()
                self.start_playback()

    def add_to_labeler(self, event):
        """Callback for the 'Add to Labeler' button for uncached files."""
        base_path = self.public_path

        # --- Determine suggested cluster (largest existing cluster + 1) ---
        labels_file = os.path.join(base_path, "cluster_labels.json")
        suggestion = 1
        if os.path.exists(labels_file):
            try:
                with open(labels_file, "r") as f:
                    data = json.load(f)
                    clusters = [entry.get("cluster", 0) for entry in data.values()]
                    if clusters:
                        suggestion = max(clusters) + 1
            except Exception as e:
                print(f"Error reading {labels_file}: {e}")

        # Prompt the user via UI for a cluster number.
        root = tk.Tk()
        root.withdraw()
        cluster_int = simpledialog.askinteger("Cluster Input",
                                               f"Enter cluster number (suggested: {suggestion}):",
                                               initialvalue=suggestion,
                                               parent=root)
        root.destroy()
        if cluster_int is None:
            print("Cluster input cancelled. Operation aborted.")
            return

        # --- Generate GUID and add CSV record first ---
        short_guid = str(uuid.uuid4())[:8]
        csv_dir = os.path.join(base_path, "csv")
        os.makedirs(csv_dir, exist_ok=True)
        csv_file = os.path.join(csv_dir, "local.csv")
        header = "ML Catalog Number,Date,Latitude,Longitude,Recordist,Media notes,Age/Sex,Average Community Rating,Filename\n"
        today_date = datetime.now().strftime("%Y-%m-%d")
        # Use the GUID as the file ID for the CSV record.
        csv_row = f"{short_guid},{today_date},0,0,Unknown,N/A,N/A,0.0,{short_guid}\n"
        if not os.path.exists(csv_file):
            with open(csv_file, "w") as f:
                f.write(header)
                f.write(csv_row)
        else:
            with open(csv_file, "a") as f:
                f.write(csv_row)

        # --- Update cluster_labels.json using the new GUID for each detection ---
        if os.path.exists(labels_file):
            with open(labels_file, "r") as f:
                try:
                    cluster_labels = json.load(f)
                except json.JSONDecodeError:
                    cluster_labels = {}
        else:
            cluster_labels = {}

        for det in self.detections:
            if det['quality'] > 0:
                key = f"{short_guid}-{int(det['start_time'])}-{int(det['end_time'])}"
                det['cluster'] = cluster_int
                cluster_labels[key] = det
        with open(labels_file, "w") as f:
            json.dump(cluster_labels, f, indent=4)
        labels_count = len(self.detections)

        # --- Update cluster_segments.json using the GUID as key ---
        segments_file = os.path.join(base_path, "cluster_segments.json")
        if os.path.exists(segments_file):
            with open(segments_file, "r") as f:
                try:
                    cluster_segments = json.load(f)
                except json.JSONDecodeError:
                    cluster_segments = {}
        else:
            cluster_segments = {}

        segments_list = cluster_segments.get(short_guid, [])
        for det in self.detections:
            if det['quality'] > 0:
                segment = {
                    "common_name": "American Crow",
                    "scientific_name": "Corvus brachyrhynchos",
                    "start_time": det["start_time"],
                    "end_time": det["end_time"],
                    "confidence": 0,
                    "cluster": cluster_int
                }
                segments_list.append(segment)
        cluster_segments[short_guid] = segments_list
        with open(segments_file, "w") as f:
            json.dump(cluster_segments, f, indent=4)
        segments_count = len(segments_list)

        # --- File copy: copy the original raw file to .cache/library/GUID.mp3 ---
        library_dir = os.path.join(base_path, "library")
        os.makedirs(library_dir, exist_ok=True)
        dest_file = os.path.join(library_dir, f"{short_guid}.mp3")
        if os.path.exists(self.source_file):
            try:
                shutil.copy(self.source_file, dest_file)
                file_copy_msg = f"File copied from '{self.source_file}' to '{dest_file}'."
            except Exception as e:
                file_copy_msg = f"Failed to copy file: {e}"
        else:
            file_copy_msg = f"Source file '{self.source_file}' not found. File not copied."

        # --- Save embeddings to .cache/embeddings/GUID.npy ---
        try:
            # Reload embeddings (from the raw file) using get_data.
            embeddings, volumes, audio, sr = get_data(self.source_file, self.public_path)
            embeddings_dir = os.path.join(base_path, "embeddings")
            os.makedirs(embeddings_dir, exist_ok=True)
            emb_dest_file = os.path.join(embeddings_dir, f"{short_guid}.npy")
            np.save(emb_dest_file, embeddings)
            embeddings_msg = f"Embeddings saved to '{emb_dest_file}'."
        except Exception as e:
            embeddings_msg = f"Failed to save embeddings: {e}"

        # --- Summary Output ---
        print("\n=== Add to Labeler Summary ===")
        print(f"New File ID (GUID): {short_guid}")
        print(f"Cluster used: {cluster_int}")
        print(f"CSV record added for file ID: {short_guid}")
        print(f"Labels added: {labels_count}")
        print(f"Segments added: {segments_count}")
        print(file_copy_msg)
        print(embeddings_msg)
        print("==============================\n")

        # --- Confirmation Popup ---
        try:
            root = tk.Tk()
            root.withdraw()
            messagebox.showinfo("Confirmation", "All detections successfully added to Labeler")
            root.destroy()
        except Exception as e:
            print(f"Confirmation popup failed: {e}")

###############################################################################
# Main entry point.
###############################################################################
def main():
    if len(sys.argv) < 2:
        arg = input("Enter file ID or file path: ").strip()
    else:
        arg = sys.argv[1]

    # Use the .cache directory relative to this script.
    public_path = os.path.join(os.path.dirname(__file__), "..", ".cache")
    # Determine if arg is an uncached file path.
    if os.path.isfile(arg):
        raw_file = arg
    else:
        raw_file = None
    detections, audio, sr = detect_file_segments(arg, public_path=public_path)
    print(f"Found {len(detections)} detection segments.")
    print(json.dumps(detections, indent=4))

    description = generate_detection_summary(detections)
    print(f"Description:\n{description}\n")
    duration = len(audio) / sr
    print(f"Audio duration: {duration:.2f} seconds, Sample Rate: {sr} Hz")

    label = os.path.basename(arg)
    player = TimelinePlayer(label, detections, audio, sr, public_path=public_path, raw_file=raw_file)
    plt.show()

if __name__ == "__main__":
    main()

==========================================
</detector/detect.py>
==========================================

==========================================
<detector/detect_all.py>
Size: 9117 bytes | Lines:      209
==========================================
#!/usr/bin/env python3
import os
import json
import csv
import numpy as np
from detector.detect import detect_file_segments
from classifier.classify import predict_embedding

# Define paths.
PATH = os.path.dirname(__file__)
public_path = os.path.join(PATH, "..", ".cache")
csv_paths = [
    os.path.join(public_path, "csv", "crows.csv"),
    os.path.join(public_path, "csv", "crows-xeno-canto.csv"),
    os.path.join(public_path, "csv", "local.csv"),
]
library_dir = os.path.join(public_path, "library")
segments_path = os.path.join(public_path, "segments.json")
auto_labels_path = os.path.join(public_path, "auto_labels.json")

def compute_contiguous_stats(segments, auto_labels, target_crowCount, tolerance=0.01):
    """
    Compute contiguous group stats for segments whose auto_labels have a crowCount equal to target_crowCount.
    Contiguity is defined as int(curr["start_time"]) == int(prev["end_time"]).
    Returns a histogram (dict) where keys are the group sizes.
    """
    contiguous_group_hist = {}
    for file_id, seg_list in segments.items():
        # Filter segments by target crowCount.
        valid_segs = []
        for seg in seg_list:
            seg_key = f"{file_id}-{int(seg['start_time'])}-{int(seg['end_time'])}"
            label = auto_labels.get(seg_key)
            if label and label.get("crowCount") == target_crowCount:
                valid_segs.append(seg)
        if not valid_segs:
            continue

        # Sort valid segments by start_time.
        sorted_segs = sorted(valid_segs, key=lambda x: x["start_time"])
        groups = []
        current_group_size = 1

        for i in range(1, len(sorted_segs)):
            prev = sorted_segs[i - 1]
            curr = sorted_segs[i]
            # Using integer conversion as in the segment key.
            if int(curr["start_time"]) == int(prev["end_time"]):
                current_group_size += 1
            else:
                groups.append(current_group_size)
                current_group_size = 1
        groups.append(current_group_size)  # add the final group

        # Update histogram counts.
        for group_size in groups:
            contiguous_group_hist[group_size] = contiguous_group_hist.get(group_size, 0) + 1

    return contiguous_group_hist

def start_detections():
    # Load previously processed auto labels if available.
    if os.path.exists(auto_labels_path):
        with open(auto_labels_path, "r", encoding="utf-8") as f:
            auto_labels = json.load(f)
    else:
        auto_labels = {}

    # Load previously processed segments if available.
    if os.path.exists(segments_path):
        with open(segments_path, "r", encoding="utf-8") as f:
            segments = json.load(f)
    else:
        segments = {}

    # Use keys from auto_labels as already processed file IDs.
    processed_ids = set(auto_labels.keys())
    # (Since auto_labels keys are segment keys, we infer processed file IDs
    #  by collecting the file_id part from each key.)
    processed_file_ids = {key.split("-")[0] for key in processed_ids}

    # Iterate through CSVs for unique file IDs.
    for csv_path in csv_paths:
        with open(csv_path, "r", encoding="utf-8-sig") as f:
            reader = csv.DictReader(f)
            for row in reader:
                file_id = row["ML Catalog Number"]
                if file_id in processed_file_ids:
                    print(f"Skipping file {file_id} (already processed)")
                    continue

                # Run detection for this file.
                print(f"Detecting for file {file_id}")
                detections, audio, sr = detect_file_segments(file_id, public_path=public_path)
                if detections:
                    # For segments, we create a minimal dictionary per detection.
                    segments[file_id] = []
                    # Load the embeddings for the file.
                    embedding_file = os.path.join(public_path, "embeddings", f"{file_id}.npy")
                    if not os.path.exists(embedding_file):
                        print(f"Embedding file {embedding_file} not found for file {file_id}, skipping.")
                        continue
                    embeddings_array = np.load(embedding_file)

                    for det in detections:
                        st = det.get("start_time")
                        et = det.get("end_time")
                        # Construct a segment key (e.g., "fileid-44-45")
                        segment_key = f"{file_id}-{int(st)}-{int(et)}"
                        # Save minimal segment info.
                        seg_min = {
                            "start_time": st,
                            "end_time": et,
                            "confidence": 0.0,
                            "cluster": 0
                        }
                        segments[file_id].append(seg_min)

                        # Get the embedding corresponding to this segment.
                        idx = int(st)
                        if idx < 0 or idx >= embeddings_array.shape[0]:
                            print(f"Index {idx} out of range for file {file_id}, skipping segment {segment_key}.")
                            continue
                        emb = embeddings_array[idx]
                        # Predict the label.
                        predicted_label = predict_embedding(emb)
                        auto_labels[segment_key] = predicted_label

                    # Optional: print summary for this file.
                    features_to_report = ["alert", "mob", "begging", "softSong", "rattle"]
                    feature_counts = {feat: sum(1 for det in detections if det.get(feat, False))
                                      for feat in features_to_report}
                    feature_counts_str = ", ".join(f"{feat}: {count}"
                                                   for feat, count in feature_counts.items() if count > 0)
                    print(f">>>>> Found {len(detections)} detections in file {file_id} ({feature_counts_str})")
                else:
                    print(f"<<<<<< No detections in file {file_id}")

    # Save segments and auto labels.
    with open(segments_path, "w", encoding="utf-8") as f:
        json.dump(segments, f, indent=2)
    print(f"***** Saved segments for {len(segments)} files to {segments_path}")

    with open(auto_labels_path, "w", encoding="utf-8") as f:
        json.dump(auto_labels, f, indent=4)
    print(f"***** Saved auto labels for {len(auto_labels)} segments to {auto_labels_path}")

    # Recalculate summary statistics from auto_labels.
    total_detections = 0
    binary_totals = {
        "alert": 0,
        "begging": 0,
        "softSong": 0,
        "rattle": 0,
        "mob": 0,
    }

    # For non-binary attributes, group counts by unique value.
    grouped_counts = {
        "crowCount": {},
        "crowAge": {},
        "quality": {}
    }

    for segment_key, label in auto_labels.items():
        total_detections += 1
        # Update binary attributes.
        binary_totals["alert"] += int(label.get("alert", False))
        binary_totals["begging"] += int(label.get("begging", False))
        binary_totals["softSong"] += int(label.get("softSong", False))
        binary_totals["rattle"] += int(label.get("rattle", False))
        binary_totals["mob"] += int(label.get("mob", False))
        # Update non-binary grouped counts.
        for attr in ["crowCount", "crowAge", "quality"]:
            value = label.get(attr)
            if value is None:
                continue
            grouped_counts[attr][value] = grouped_counts[attr].get(value, 0) + 1

    # For total detection time, each detection is one second.
    hours = total_detections // 3600
    minutes = (total_detections % 3600) // 60

    print("\n===== Detection Summary =====")
    print(f"Total detections: {total_detections} ({int(hours):02d}:{int(minutes):02d} HH:MM)")
    print("\nTotals for non-binary attributes:")
    for attr, counts in grouped_counts.items():
        counts_str = ", ".join(f"{val} = {counts[val]}" for val in sorted(counts.keys()))
        print(f"  {attr}: {counts_str}")
    print("\nTotals for binary attributes:")
    for attr, total in binary_totals.items():
        hours = total // 3600
        minutes = (total % 3600) // 60
        print(f"  {attr}: {total} ({int(hours):02d}:{int(minutes):02d} HH:MM)")

    # --- New code: Compute contiguous segments group stats for different crowCount filters ---
    for target in [1, 2, 4]:
        stats = compute_contiguous_stats(segments, auto_labels, target_crowCount=target)
        print(f"\nContiguous Segments Groups (crowCount=={target}):")
        for group_size in sorted(stats.keys())[:10]:
            seg_label = "segment" if group_size == 1 else "segments"
            total_time = stats[group_size]
            hours = total_time // 3600
            minutes = (total_time % 3600) // 60
            print(f"  {group_size} {seg_label}: {total_time} ({int(hours):02d}:{int(minutes):02d} HH:MM)")


if __name__ == "__main__":
    start_detections()

==========================================
</detector/detect_all.py>
==========================================

==========================================
<detector/detect_all_birdnet.py>
Size: 3280 bytes | Lines:       82
==========================================
from birdnetlib import Recording
from birdnetlib.analyzer import Analyzer
from datetime import datetime
import csv
import json
import os

PATH = os.path.dirname(__file__)

public_path = os.path.join(PATH, "..", ".cache")
csv_paths = [os.path.join(public_path, "csv", "crows.csv"),
            os.path.join(public_path, "csv", "crows-xeno-canto.csv")]

library_dir = os.path.join(public_path, "library")
segments_path = os.path.join(public_path, "segments.json")
segments = {}

# Load and initialize the BirdNET-Analyzer models.
analyzer = Analyzer()


def start_detections():
    if os.path.exists(segments_path):
        with open(segments_path) as json_file:
            segments = json.load(json_file)

    for csv_path in csv_paths:
        with open(csv_path, "r", encoding="utf-8-sig") as f:
            reader = csv.DictReader(f)

            for row in reader:
                catalog_number = row["ML Catalog Number"]
                lat = float(row["Latitude"] or "0")
                long = float(row["Longitude"] or "0")
                if row["Date"]:
                    parsed_date = datetime.strptime(row["Date"], "%Y-%m-%d")
                else:
                    print("Invalid date")
                    continue

                filename = f"{catalog_number}.mp3"
                filepath = os.path.join(library_dir, filename)

                if os.path.exists(filepath) and catalog_number not in segments:
                    CONFIDENCE_THRESHOLD = 0.5
                    recording = Recording(
                     analyzer,
                     filepath,
                     lat=lat,
                     lon=long,
                     date=parsed_date,
                     min_conf=CONFIDENCE_THRESHOLD,
                    )
                    recording.analyze()

                    # Collect crow detections
                    detections = []
                    for d in recording.detection_list:
                        if d.confidence > CONFIDENCE_THRESHOLD and d.scientific_name == "Corvus brachyrhynchos":
                            detections.append({ "common_name": d.common_name, "scientific_name": d.scientific_name,
                                                 "start_time": d.start_time, "end_time": d.end_time, "confidence": d.confidence })

                    # {"common_name": "American Crow", "scientific_name": "Corvus brachyrhynchos", "start_time": 15.0, "end_time": 30.0, "confidence": 0.7989989399909974}
                    if detections:
                        print(f">>>>> Found {len(detections)} detections in {filename}")
                        segments[catalog_number] = detections
                    else:
                        print(f"<<<<<< No detections in {filename}")

                    if len(segments) % 100 == 0:
                        # Write output to JSON file
                        print(f"***** Saved Segments for Files: {len(segments)}")
                        with open(segments_path, 'w') as f:
                            json.dump(segments, f, indent=2)

    # Write output to JSON file
    print(f"***** Saved Segments for Files: {len(segments)}")
    with open(segments_path, 'w') as f:
        json.dump(segments, f, indent=2)


if __name__ == "__main__":
    start_detections()
==========================================
</detector/detect_all_birdnet.py>
==========================================

==========================================
<downloader/download_backgrounds.py>
Size: 3269 bytes | Lines:       89
==========================================
import csv
import os
import requests
import random

random.seed(42)
PATH = os.path.dirname(__file__)

def start_downloads():
    # Fetch all recordings
    # Background file names
    background_filenames = [
        "black-capped-chickadee.csv",
        "blue-jay.csv",
        "downy-woodpecker.csv",
        "european-starling.csv",
        "northern-cardinal.csv",
        "northern-mocking.csv",
        "robin.csv"
    ]

    # Limit downloads per file
    max_downloads = 50
    download_count = 0
    skipped_count = 0

    for background_filename in background_filenames:
        print(f"Downloading {background_filename} - {max_downloads} downloads")
        csv_path = os.path.join(PATH, "..", ".cache", "csv", background_filename)
        library_dir = os.path.join(PATH, "..", ".cache", "backgrounds")
        os.makedirs(library_dir, exist_ok=True)
        background_count = 0
        age_sexes = {}

        with open(csv_path, "r", encoding="utf-8-sig") as f:
            reader = csv.DictReader(f)
            rows = list(reader)  # Load all rows into a list
            random.shuffle(rows)  # Shuffle the list in-place

        for row in rows:
                # Now this should read the correct field name without BOM
                catalog_number = row["ML Catalog Number"]
                age_sex = row["Age/Sex"].lower()
                rating_str = row.get("Average Community Rating", "").strip()
                rating = float(rating_str) if rating_str else 0.0  # or use None
                download_url = f"https://cdn.download.ams.birds.cornell.edu/api/v2/asset/{catalog_number}/mp3"

                filename = f"{catalog_number}.mp3"
                filepath = os.path.join(library_dir, filename)

                if age_sex not in age_sexes:
                    age_sexes[age_sex] = 1
                else:
                    age_sexes[age_sex] += 1

                # Skip download if the file already exists
                if os.path.exists(filepath):
                    print(f"File already exists, skipping: {filepath}")
                    skipped_count += 1

                if "juv" in age_sex:
                    skipped_count += 1
                    continue

                if rating < 4.0:
                    skipped_count += 1
                    continue

                download_count += 1
                background_count += 1

                print(f"Downloading {download_url} to {filepath}")
                if not os.path.exists(filepath):
                    response = requests.get(download_url, stream=True)
                    if response.status_code == 200:
                        with open(filepath, "wb") as out_file:
                            for chunk in response.iter_content(chunk_size=8192):
                                out_file.write(chunk)
                    else:
                        print(f"Failed to download {download_url}: status code {response.status_code}")

                if background_count >= 100:
                    background_count = 0
                    print(f"Done with {background_filename} downloads")
                    break

    print(f"Downloaded {download_count} records, Skipped {skipped_count} records")

if __name__ == "__main__":
    start_downloads()
==========================================
</downloader/download_backgrounds.py>
==========================================

==========================================
<downloader/download_crows_ebird.py>
Size: 1654 bytes | Lines:       45
==========================================
import csv
import os
import requests

PATH = os.path.dirname(__file__)

def start_downloads():
    # Fetch all recordings
    csv_path = os.path.join(PATH, "..", ".cache", "csv", "crows.csv")
    library_dir = os.path.join(PATH, "..", ".cache", "library")
    os.makedirs(library_dir, exist_ok=True)
    download_count = 0
    skipped_count = 0

    with open(csv_path, "r", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)

        for row in reader:
            # Now this should read the correct field name without BOM
            catalog_number = row["ML Catalog Number"]
            download_url = f"https://cdn.download.ams.birds.cornell.edu/api/v2/asset/{catalog_number}/mp3"

            filename = f"{catalog_number}.mp3"
            filepath = os.path.join(library_dir, filename)

            # Skip download if the file already exists
            if os.path.exists(filepath):
                print(f"File already exists, skipping: {filepath}")
                skipped_count += 1
                continue

            download_count += 1
            print(f"Downloading {download_url} to {filepath}")
            response = requests.get(download_url, stream=True)
            if response.status_code == 200:
                with open(filepath, "wb") as out_file:
                    for chunk in response.iter_content(chunk_size=8192):
                        out_file.write(chunk)
            else:
                print(f"Failed to download {download_url}: status code {response.status_code}")

    print(f"Downloaded {download_count} records, Skipped {skipped_count} records")

if __name__ == "__main__":
    start_downloads()

==========================================
</downloader/download_crows_ebird.py>
==========================================

==========================================
<downloader/download_crows_xeno.py>
Size: 3963 bytes | Lines:      134
==========================================
import requests
import csv
import os
from datetime import datetime

PATH = os.path.dirname(__file__)

# -- CONFIG: Adjust as needed --
SPECIES_QUERY = "American+Crow"
OUTPUT_CSV = os.path.join(PATH, "..", ".cache", "csv", "crows-xeno-canto.csv")
OUTPUT_DIR = os.path.join(PATH, "..", ".cache", "library")
BASE_API_URL = "https://www.xeno-canto.org/api/2/recordings"

# Create output folder if not existing
os.makedirs(OUTPUT_DIR, exist_ok=True)

def fetch_all_recordings(query):
    """
    Fetch all pages from Xeno-Canto API for the given query (species).
    Returns a list of recording dictionaries.
    """
    all_recs = []
    page_number = 1

    while True:
        url = f"{BASE_API_URL}?query={query}&page={page_number}"
        resp = requests.get(url).json()
        recs = resp.get("recordings", [])
        if not recs:
            break

        all_recs.extend(recs)

        # Stop if we reached the last page
        if page_number >= int(resp["numPages"]):
            break

        page_number += 1

    return all_recs

def download_mp3(xc_id, mp3_url):
    """
    Download MP3 file to local OUTPUT_DIR/{xc_id}.mp3
    """
    if mp3_url.startswith("//"):
        mp3_url = "https:" + mp3_url  # Fix if it starts with //

    filename = f"{xc_id}.mp3"
    filepath = os.path.join(OUTPUT_DIR, filename)

    if not os.path.exists(filepath):
        try:
            print(f"Downloading {mp3_url}")
            response = requests.get(mp3_url, timeout=30)
            with open(filepath, 'wb') as f_out:
                f_out.write(response.content)
        except Exception as e:
            print(f"Error downloading {mp3_url}: {e}")

def start_downloads():
    # Fetch all recordings
    recordings = fetch_all_recordings(SPECIES_QUERY)
    print(f"Found {len(recordings)} recordings for '{SPECIES_QUERY}'.")

    fieldnames = [
        "ML Catalog Number",
        "Date",
        "Latitude",
        "Longitude",
        "Recordist",
        "Media notes",
        "Age/Sex",
        "Average Community Rating",
        "Filename"
    ]

    with open(OUTPUT_CSV, "w", newline="", encoding="utf-8") as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()

        for rec in recordings:
            xc_id = rec.get("id", "")
            if not xc_id:
                continue  # skip if no id

            # Parse date if it's valid
            date_str = rec.get("date", "")
            try:
                _ = datetime.strptime(date_str, "%Y-%m-%d")
            except ValueError:
                # If the date is missing or invalid, skip
                print(f"Skipping ID={xc_id}, invalid date '{date_str}'")
                continue

            # Convert lat/lng to floats (default to 0 if missing)
            lat_str = rec.get("lat", "0")
            lng_str = rec.get("lng", "0")
            try:
                lat = float(lat_str)
            except:
                lat = 0.0
            try:
                lng = float(lng_str)
            except:
                lng = 0.0

            # Age/Sex and rating not provided by Xeno-Canto
            age_sex = "unknown"
            rating = "0.0"

            # Download the MP3
            mp3_url = rec.get("file", "")
            if mp3_url:
                download_mp3(xc_id, mp3_url)

            # Write CSV row
            row = {
                "ML Catalog Number": xc_id,
                "Date": date_str,
                "Latitude": str(lat),
                "Longitude": str(lng),
                "Recordist": rec.get("rec", "") or "Unknown",
                "Media notes": rec.get("rmk", ""),
                "Age/Sex": age_sex,
                "Average Community Rating": rating,
                "Filename": f"{xc_id}.mp3"
            }
            writer.writerow(row)

    print(f"Done! CSV written to '{OUTPUT_CSV}'. MP3s downloaded to '{OUTPUT_DIR}/'.")

if __name__ == "__main__":
    start_downloads()

==========================================
</downloader/download_crows_xeno.py>
==========================================

==========================================
<embedder/__init__.py>
Size: 0 bytes | Lines:        0
==========================================
[Binary or non-text file - content omitted]

==========================================
</embedder/__init__.py>
==========================================

==========================================
<embedder/analyze.py>
Size: 11006 bytes | Lines:      309
==========================================
import json
import os
import random
import argparse

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sounddevice as sd
from sklearn.decomposition import PCA
from matplotlib.widgets import RadioButtons
from ispa import utils

PATH = os.path.dirname(__file__)

def is_headless():
    return not os.environ.get("DISPLAY") and os.name != "nt"
if is_headless():
    matplotlib.use("Agg")
else:
    matplotlib.use("TkAgg")


def print_label_stats(labels):
    # Convert labels dict into a DataFrame.
    df = pd.DataFrame.from_dict(labels, orient='index')

    # Define quality based on badQuality and human flags.
    def determine_quality(row):
        return 'bad' if row.get('quality') == 1 else 'good'

    df['quality_label'] = df.apply(determine_quality, axis=1)

    # Overall crosstab: frequency counts.
    overall_ct = pd.crosstab(df['quality_label'], df['crowCount'], margins=True)

    print("=== LABEL SUMMARY ===")
    print("\nOverall Counts (Quality vs. CrowCount):")
    print(overall_ct)
    print()

    # For each feature, compute frequency (number of True values) and percentages.
    for feat in ['rattle', 'begging', 'softSong', 'mob', 'alert']:
        feat_freq = df.groupby(['quality_label', 'crowCount'])[feat].sum().unstack()
        print(f"{feat.capitalize()} Counts (True):")
        print(feat_freq.fillna(0).astype(int))
        print()

    print("Total labels:", len(df))
    print()

def print_segment_stats(segments_dict):
    segments_list = []
    for segs in segments_dict.values():
        seg_list = segs if isinstance(segs, list) else [segs]
        for seg in seg_list:
            duration = seg["end_time"] - seg["start_time"]
            segments_list.append(duration)
    durations_series = pd.Series(segments_list)
    freq = durations_series.value_counts().sort_index()
    pct = durations_series.value_counts(normalize=True).sort_index() * 100
    freq_table = pd.DataFrame({'Count': freq, 'Percentage': pct.round(1)})

    print("=== SEGMENT SUMMARY ===")
    print(f"Total files   : {len(segments_dict)}")
    print(f"Total segments: {len(segments_list)}")
    print("\nSegment Duration Distribution (seconds):")
    print(freq_table)
    print()

def choose_color(label, mode='quality'):
    # For non-quality modes, hide bad quality points (make transparent)
    if mode != 'quality' and label.get('quality') == 1:
        return (0, 0, 0, 0)  # Invisible

    if mode == 'quality':
        if label.get('quality') == 1:
            return "red"
        else:
            return "limegreen"  # bright, but not overly neon

    elif mode == 'crowAge':
        age = label.get('crowAge')
        if age == 1:
            return "dodgerblue"  # a friendly, bright blue
        elif age == 2:
            return "gold"        # a warm, appealing yellow
        else:
            return "gray"

    elif mode == 'crowCount':
        count = label.get('crowCount', 0)
        if count == 0:
            return "lightgray"
        elif count == 1:
            return "#00bfff"
        elif count == 2:
            return "orange"
        elif count == 4:
            return "orchid"          # soft purple
        else:
            return "gray"

    elif mode == 'features':
        # Priority order for binary features: alert > mob > begging > softSong > rattle.
        features_priority = ['alert', 'mob', 'begging', 'softSong', 'rattle']
        features_colors = {
            'alert': "dodgerblue",       # bright blue
            'mob': "mediumorchid",       # appealing purple
            'begging': "goldenrod",      # warm golden tone
            'softSong': "mediumseagreen",# calm green
            'rattle': "tomato"           # pleasant red-orange
        }
        for feat in features_priority:
            if label.get(feat):
                return features_colors[feat]
        return "gray"

    return "gray"


def main(sample_size):
    segments_path = os.path.join("..", ".cache", "segments.json")
    with open(segments_path, encoding='utf-8', mode='r') as f:
        segments_dict = json.load(f)

    labels_path = os.path.join("..", ".cache", "auto_labels.json")
    with open(labels_path, encoding='utf-8', mode='r') as f:
        labels = json.load(f)

    print_label_stats(labels)
    print_segment_stats(segments_dict)

    denoised = True
    processed_seconds = 0

    valid_segments = []
    for file_id, segs in segments_dict.items():
        for segment in segs:
            segment_key = f"{file_id}-{segment.get('start_time'):.0f}-{segment.get('end_time'):.0f}"
            if segment_key not in labels:
                continue

            valid_segments.append({
                "file_id": file_id,
                "segment": segment,
                "segment_key": segment_key,
                "label": labels[segment_key]
            })

    if sample_size < len(valid_segments):
        valid_segments = random.sample(valid_segments, sample_size)
        print(f"Randomly sampled {sample_size} segments from available data.")
    else:
        print(f"Processing all {len(valid_segments)} available segments.")

    chunk_embeddings = []
    chunk_info = []  # Each item holds audio path, times, initial color, segment_key, and label.
    sample_rate = 8000

    for entry in valid_segments:
        file_id = entry["file_id"]
        segment = entry["segment"]
        segment_key = entry["segment_key"]
        label = entry["label"]

        segment_length = segment.get('end_time') - segment.get('start_time')
        processed_seconds += segment_length

        denoised_suffix = "-denoised" if denoised else ""
        audio_extension = "wav" if denoised else "mp3"
        audio_path = os.path.join(PATH, "..", ".cache", f"library{denoised_suffix}", f"{file_id}.{audio_extension}")

        if not os.path.exists(audio_path):
            print(f"Audio file {audio_path} not found, skipping")
            continue

        cached_path = os.path.join(PATH, "..", ".cache", f"embeddings{denoised_suffix}", f"{file_id}.npy")
        if not os.path.exists(cached_path):
            print(f"Skipping {segment_key}, no cached embedding file found")
            continue

        try:
            feature = np.load(cached_path, mmap_mode='r')
        except Exception as e:
            print(f"Error loading {cached_path}: {e}")
            continue

        # Set initial color using 'quality' mode.
        color = choose_color(label, mode='quality')

        start_idx = int(segment.get('start_time'))
        end_idx = int(segment.get('end_time'))
        chunk = feature[start_idx:end_idx]
        chunk_embeddings.append(chunk)
        chunk_info.append({
            "audio_path": audio_path,
            "chunk_start_time": segment.get('start_time'),
            "chunk_end_time": segment.get('end_time'),
            "color": color,
            "segment_key": segment_key,
            "label": label
        })

    if len(chunk_embeddings) == 0:
        print("No segments to process after filtering. Exiting.")
        return

    hours = int(processed_seconds // 3600)
    minutes = int((processed_seconds % 3600) // 60)
    print(f"Processed time: {hours:02d}:{minutes:02d} (HH:MM)")
    print(f"Average time per segment: {processed_seconds / len(chunk_embeddings):.2f} seconds")

    chunk_embeddings = np.array(chunk_embeddings)
    print("Total segments for PCA:", chunk_embeddings.shape[0])
    if len(chunk_embeddings.shape) != 2:
        chunk_embeddings = chunk_embeddings.squeeze(1)

    pca = PCA(n_components=3)
    embeddings_3d = pca.fit_transform(chunk_embeddings)

    output_json_path = os.path.join(PATH, "..", ".cache", "embeddings-3d.json")
    output_data = []
    for i, coords in enumerate(embeddings_3d.tolist()):
        output_data.append({
            "segment_key": chunk_info[i]["segment_key"],
            "coordinates": coords,
            "color": chunk_info[i]["color"]
        })

    with open(output_json_path, "w") as f:
        json.dump(output_data, f, indent=2)
    print(f"Saved 3D embeddings to {output_json_path}")

    fig = plt.figure(figsize=(10, 8))
    ax = fig.add_subplot(111, projection='3d')
    base_marker_size = 10
    colors = [info["color"] for info in chunk_info]
    sc = ax.scatter(embeddings_3d[:, 0], embeddings_3d[:, 1], embeddings_3d[:, 2],
                    s=base_marker_size, c=colors, picker=True)

    ax.set_title("3D PCA of AVES Embedding Chunks (Interactive)")
    ax.set_xlabel("PC 1")
    ax.set_ylabel("PC 2")
    ax.set_zlabel("PC 3")

    def update_output_json(new_colors):
        updated_data = []
        for i, coords in enumerate(embeddings_3d.tolist()):
            updated_data.append({
                "segment_key": chunk_info[i]["segment_key"],
                "coordinates": coords,
                "color": new_colors[i]
            })
        with open(output_json_path, "w") as f:
            json.dump(updated_data, f, indent=2)
        print(f"Re-saved updated output JSON to {output_json_path}")

    def update_mode(mode):
        new_colors = [choose_color(info["label"], mode=mode) for info in chunk_info]
        sc.set_color(new_colors)
        plt.draw()
        update_output_json(new_colors)

    ax_mode = plt.axes([0.05, 0.8, 0.12, 0.08])
    radio = RadioButtons(ax_mode, ('quality', 'crowAge', 'crowCount', 'features'), active=0)
    radio.on_clicked(update_mode)

    def on_scroll(event):
        if event.button == 'up':
            ax.dist = max(ax.dist / 1.1, 1)
        elif event.button == 'down':
            ax.dist = ax.dist * 1.1

        factor = 1.2 if event.button == 'up' else 1 / 1.2 if event.button == 'down' else 1
        current_sizes = sc.get_sizes()
        new_sizes = current_sizes * factor
        sc.set_sizes(new_sizes)
        plt.draw()

    fig.canvas.mpl_connect('scroll_event', on_scroll)

    def on_pick(event):
        ind = event.ind
        if len(ind) == 0:
            return
        idx = ind[0]
        info = chunk_info[idx]
        print(f"\nClicked segment #{idx}")
        print(f"File: {info['audio_path']}")
        print(f"Time: {info['chunk_start_time']:.2f}s to {info['chunk_end_time']:.2f}s")
        chunk_waveform, _ = utils.load_waveform(info["audio_path"],
                                                tgt_sr=sample_rate,
                                                start_sec=info["chunk_start_time"],
                                                end_sec=info["chunk_end_time"])
        chunk_np = chunk_waveform.squeeze(0).detach().numpy()
        sd.play(chunk_np, sample_rate)

    fig.canvas.mpl_connect('pick_event', on_pick)
    plt.show()


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Analyze Embedding Detections with Interactive Color Modes")
    parser.add_argument("--sample_size", type=int, default=10000,
                        help="Number of segments to sample (default: 10000)")
    args = parser.parse_args()
    main(args.sample_size)

==========================================
</embedder/analyze.py>
==========================================

==========================================
<embedder/embed.py>
Size: 2912 bytes | Lines:       72
==========================================
import numpy as np
import torch
from tqdm import tqdm
from embedder.ispa.features import FeatureBasedISPAPredictor
import os

PATH = os.path.dirname(__file__)

# Initialize the AVES feature predictor.
ispa_f_predictor = FeatureBasedISPAPredictor(
    feature_type='aves',
    kmeans_model=os.path.join(PATH, 'ispa', 'models', 'kmeans.aves.pkl'),
    phoneme_map=os.path.join(PATH, 'ispa', 'models', 'c2p.aves.json'),
    aves_config_path=os.path.join(PATH, 'ispa', 'models', 'aves-base-bio.torchaudio.model_config.json'),
    aves_model_path=os.path.join(PATH, 'ispa', 'models', 'aves-base-bio.torchaudio.pt')
)

def generate_embeddings(waveform):
    """
    Given an audio waveform and its sample rate, extract an embedding using the AVES feature extractor

    Args:
        waveform (torch.Tensor or np.ndarray): Audio waveform.
            - If a torch.Tensor, expected shape is (batch, time) or (time,).
            - If a numpy array, conversion to a tensor will be attempted.
        sample_rate (int): The sample rate of the waveform.

    Returns:
        np.ndarray: A 2D array where each row corresponds to the feature embeddings (25 embeddings per second)
    """
    # Ensure the waveform is a torch.Tensor (add batch dimension if needed).
    if isinstance(waveform, np.ndarray):
        waveform = torch.from_numpy(waveform)
    if waveform.dim() == 1:
        waveform = waveform.unsqueeze(0)

    # Remove the batch dimension
    waveform = waveform.squeeze(0)

    # Calculate the number of samples per chunk (120 seconds).
    chunk_size = 8000 * 120
    embeddings_chunks = []

    # Process the waveform in chunks (so large files don't crash)
    num_samples = waveform.shape[0]
    for start in tqdm(range(0, num_samples, chunk_size)):
        # Take a chunk (if the final chunk is shorter, that's fine)
        chunk = waveform[start:start + chunk_size]
        # Add batch dimension back for feature extraction.
        chunk = chunk.unsqueeze(0)  # shape (1, chunk_time)
        # Extract features; expected output shape: (1, time, feature_dim)
        features = ispa_f_predictor.feature_extractor(chunk)
        # Remove batch dimension.
        features = features.squeeze(0)  # shape: (time, feature_dim)
        # Convert to numpy.
        features_np = features.detach().numpy()
        embeddings_chunks.append(features_np)

    # Reassemble embeddings by concatenating along the time axis.
    full_embedding = np.concatenate(embeddings_chunks, axis=0)
    return full_embedding

# Example usage:
if __name__ == "__main__":
    # For demonstration, create a dummy waveform
    sample_rate = 8000
    duration_seconds = 10
    # Generate a random waveform as an example (normally you'd load real audio).
    dummy_waveform = torch.randn(sample_rate * duration_seconds)

    embedding = generate_embeddings(dummy_waveform)
    print("Generated embedding shape:", embedding.shape)

==========================================
</embedder/embed.py>
==========================================

==========================================
<embedder/embed_all.py>
Size: 4646 bytes | Lines:      110
==========================================
import os
import librosa
import numpy as np
from embedder.embed import generate_embeddings
from embedder.ispa import utils

PATH = os.path.dirname(__file__)

def start_embeddings(denoised=False):
    DENOISED_MODE = ""
    EXT = ".mp3"
    if denoised:
        DENOISED_MODE = "-denoised"
        EXT = ".wav"

    # Directory with the denoised .wav files
    library_path = os.path.join(PATH, "..", ".cache", f"library{DENOISED_MODE}")
    file_paths = sorted(os.listdir(library_path))

    # Where to save the averaged embeddings
    embeddings_path = os.path.join(PATH, "..", ".cache", f"embeddings{DENOISED_MODE}")
    if not os.path.exists(embeddings_path):
        os.makedirs(embeddings_path)

    # Where to save per-second volume data
    volumes_path = os.path.join(PATH, "..", ".cache", f"embeddings{DENOISED_MODE}-volumes")
    if not os.path.exists(volumes_path):
        os.makedirs(volumes_path)

    sample_rate = 8000
    chunk_size = 25  # Number of frames to average into one vector

    for file_name in file_paths:
        if not file_name.endswith(EXT):
            continue

        file_id = os.path.splitext(file_name)[0]
        audio_path = os.path.join(library_path, file_name)

        if not os.path.exists(audio_path):
            print(f"Audio file {audio_path} not found, skipping {file_id}.")
            continue

        # Embedding output
        embedding_out_path = os.path.join(embeddings_path, f"{file_id}.npy")
        # Volume output
        volume_out_path = os.path.join(volumes_path, f"{file_id}.npy")

        # Skip if both already exist
        if os.path.exists(embedding_out_path) and os.path.exists(volume_out_path):
            print(f"Skipping {file_id}; embedding & volume both exist.")
            continue

        print(f"Loading {audio_path} at {sample_rate} Hz.")
        waveform, sr = librosa.load(audio_path, sr=sample_rate, mono=True)
        if sr != sample_rate:
            print(f"Warning: Sample rate mismatch for {file_id}. Using SR={sr}.")

        ############################################################
        # 1) Generate and save embeddings (if not already existing)
        ############################################################
        if not os.path.exists(embedding_out_path):
            full_embedding = generate_embeddings(waveform)  # shape: (N_frames, 768)
            full_embedding = np.array(full_embedding, dtype=np.float32)
            num_frames = full_embedding.shape[0]
            print(f"{file_id}: got {num_frames} frames of embeddings.")

            # Chunk the frames in increments of 'chunk_size' and average
            num_chunks = int(np.ceil(num_frames / chunk_size))
            embedding_means = []
            for i in range(num_chunks):
                start_idx = i * chunk_size
                end_idx = min((i + 1) * chunk_size, num_frames)
                chunk = full_embedding[start_idx:end_idx]
                mean_emb = np.mean(chunk, axis=0)
                embedding_means.append(mean_emb)
            final_embedding = np.stack(embedding_means, axis=0)
            np.save(embedding_out_path, final_embedding)
            print(f"Saved embedding for {file_id} -> {embedding_out_path} shape={final_embedding.shape}")

        ############################################################
        # 2) Compute and save volume data (if not existing)
        ############################################################
        if not os.path.exists(volume_out_path):
            # Ensure waveform is 1D. librosa.load returns a 1D array if mono=True,
            # but in case it ends up 2D, squeeze the first axis.
            if waveform.ndim > 1:
                waveform = np.squeeze(waveform, axis=0)
            total_samples = waveform.shape[0]

            # Compute 1 volume metric per second (mean absolute amplitude)
            total_seconds = int(np.ceil(total_samples / sr))
            volumes = []
            for sec in range(total_seconds):
                start_samp = sec * sr
                end_samp = min((sec + 1) * sr, total_samples)
                segment = waveform[start_samp:end_samp]
                if len(segment) == 0:
                    volumes.append(0.0)
                else:
                    mean_amp = np.mean(np.abs(segment))
                    volumes.append(mean_amp)

            volumes = np.array(volumes, dtype=np.float32)
            np.save(volume_out_path, volumes)
            print(f"Saved volume data for {file_id} -> {volume_out_path} shape={volumes.shape}")

if __name__ == "__main__":
    start_embeddings(denoised=False)
    start_embeddings(denoised=True)

==========================================
</embedder/embed_all.py>
==========================================

==========================================
<embedder/ispa/__init__.py>
Size: 0 bytes | Lines:        0
==========================================
[Binary or non-text file - content omitted]

==========================================
</embedder/ispa/__init__.py>
==========================================

==========================================
<embedder/ispa/acoustics.py>
Size: 8789 bytes | Lines:      313
==========================================
from collections import defaultdict
from dataclasses import dataclass
import re

import librosa
import numpy as np
import pesto
import pandas as pd


def length_to_name(length):
    # convert length (multiples of 1/32 seconds) to note name
    # assuming bpm = 60
    return {
        1: '/32',
        2: '/16',
        4: '/8',
        8: '/4',
        16: '/2',
        32: '',      # quarter note
        64: 'x2',
        128: 'x4',
    }[length]


def bw_to_name(bw):
    # convert relative bandwidth to name
    if bw < 0.4:
        return 'U'
    elif bw < 0.8:
        return 'N'
    elif bw < 1.2:
        return 'M'
    elif bw < 1.6:
        return 'W'
    else:
        return 'X'


def estimate_pitch_pesto(waveform, sr):
    time, frequency, confidence, _ = pesto.predict(
        waveform, sr,
        step_size=31.25,
        reduction='alwa',
        convert_to_freq=True)

    # convert to df
    df = pd.DataFrame({
        'time': time,
        'frequency': frequency,
        'confidence': confidence})

    return df


def get_amplitude(df, waveform, sr):
    # get amplitude of waveform per frame
    frame_length = int(0.03125 * sr)      # 1/32 sec per frame
    amps = []
    for i in range(0, len(df)):
        st = int(df['time'][i] * sr)
        ed = st + frame_length
        # shift st and ed to the center of the frame
        st = max(0, st - frame_length // 2)
        ed = min(len(waveform[0]), ed + frame_length // 2)
        amp = np.mean(np.abs(waveform[0][st:ed].numpy()))
        # convert to dB
        amp = 20 * np.log10(amp + 1e-3)
        amps.append(amp)
    df['amplitude'] = amps

    return df


def get_relative_bandwidth(df, waveform, sr):
    frame_length = int(0.03125 * sr)      # 1/32 sec per frame
    bandwidth = librosa.feature.spectral_bandwidth(
        y=waveform[0].numpy(),
        n_fft=frame_length,
        hop_length=frame_length)
    centroid = librosa.feature.spectral_centroid(
        y=waveform[0].numpy(),
        n_fft=frame_length,
        hop_length=frame_length)

    centroid += 1e-3
    df['bandwidth'] = (bandwidth / centroid)[0]
    return df


def get_residual(x, y, freq, slope):
    if slope.value < 0:
        start_freq = freq * -slope.value
        end_freq = freq - (start_freq - freq)
    else:
        end_freq = freq * slope.value
        start_freq = freq - (end_freq - freq)
    # compute residual as sum of absolute error
    if x[-1] == x[0]:
        residual = 0.
    else:
        residual = np.sum(np.abs(y - (start_freq + (end_freq - start_freq) * x / (x[-1] - x[0]))))

    return residual


# dataclass for slope
@dataclass
class Slope:
    name: str
    value: float

    def __str__(self):
        return self.name

    def __repr__(self):
        return self.name

    def __eq__(self, other):
        return self.name == other.name

    def __hash__(self):
        return hash(self.name)


SLOPE_FLAT = Slope("=", 1)
SLOPES = [
    Slope("-3", -2),
    Slope("-2", -1.5),
    Slope("-1", -1.2),
    SLOPE_FLAT,
    Slope("+1", 1.2),
    Slope("+2", 1.5),
    Slope("+3", 2)
]


def enumerate_tokens(df, start):
    # enumerate list of tokens that start at start index
    # start: start index in df
    results = []
    for length in [1, 2, 4, 8, 16, 32, 64, 128]:    # multiple of 1/32 sec
        end = start + length
        if end > len(df):
            break
        if df['amplitude'][start:end].max() < -50:
            # rest
            results.append({
                'type': 'rest',
                'length': length,
                'freq': None,
                'res': None,
                'conf': None,
                'amp': None,
                'bw': None})
        else:
            if length == 1:
                slopes = [SLOPE_FLAT]
            else:
                slopes = SLOPES

            df_slice = df.iloc[start:end]
            x_slice = df_slice['time'].values
            x_slice = x_slice - x_slice[0]
            y_slice = df_slice['frequency'].values
            freq_slice = np.mean(y_slice)

            conf = np.mean(df_slice['confidence'].values)
            amp = np.mean(df_slice['amplitude'].values)
            bw = np.mean(df_slice['bandwidth'].values)

            for slope in slopes:
                # res = fit_sine_wave_with_fixed_slope(df, start, end, slope)
                # freq, res, conf, amp, bw = res

                res = get_residual(x_slice, y_slice, freq_slice, slope)
                freq = freq_slice

                results.append({
                    'type': 'note',
                    'length': length,
                    'freq': freq,
                    'slope': slope,
                    'res': res,
                    'conf': conf,
                    'amp': amp,
                    'bw': bw})

    return results


_BOS_TOKEN = {'type': 'BOS', 'length': 1, 'freq': None, 'res': None, 'conf': None, 'amp': None}
_EOS_TOKEN = {'type': 'EOS', 'length': 1, 'freq': None, 'res': None, 'conf': None, 'amp': None}


def cost_func(cnode, bnode=None):
    if cnode['token']['type'] == 'note':
        residual_cost = cnode['token']['res']
        conf_cost = 100 * (1 - cnode['token']['conf'])

    else:
        residual_cost = 0
        conf_cost = 0

    length_cost = 1_000 / (1 + np.log2(cnode['token']['length']))
    total_cost = residual_cost + length_cost

    return total_cost


def run_viterbi(df, cost_func=lambda node: 0):
    bos_node = {'start': -1, 'next': [], 'token': _BOS_TOKEN, 'cost': 0}
    end_node_list = defaultdict(list)
    end_node_list[0].append(bos_node)

    for i in range(0, len(df)+1):
        if i < len(df):
            tokens = enumerate_tokens(df, i)
        else:
            tokens = [_EOS_TOKEN]
        for token in tokens:
            cnode = {'start': i, 'next': [], 'token': token}
            min_cost = -1
            min_bnodes = []

            cnode_cost = cost_func(cnode)
            for bnode in end_node_list[i]:
                cost = bnode['cost'] + cnode_cost

                # allow ties
                # if min_cost == -1 or cost < min_cost:
                #     min_cost = cost
                #     min_bnodes = [bnode]
                # elif cost == min_cost:
                #     min_bnodes.append(bnode)

                # doesn't allow ties
                if min_cost == -1 or cost < min_cost:
                    min_cost = cost
                    min_bnodes = [bnode]

            if len(min_bnodes) > 0:
                for bnode in min_bnodes:
                    cnode['cost'] = min_cost
                    bnode['next'].append(cnode)

                end_nodes = end_node_list[i+token['length']]
                if not cnode in end_nodes:
                    end_nodes.append(cnode)

    solutions_cache = {}

    def _enum_solutions(node):
        # convert node to hashable
        cache_key = (node['start'], tuple(node['token'].items()))
        if cache_key in solutions_cache:
            return solutions_cache[cache_key]

        if node['token'] == _EOS_TOKEN:
            return [[_EOS_TOKEN]]

        solutions = []
        for next_node in node['next']:
            for solution in _enum_solutions(next_node):
                node['token']['start'] = node['start']
                solutions.append([node['token']] + solution)

        solutions_cache[cache_key] = solutions
        return solutions

    return _enum_solutions(bos_node)[0]


def convert_to_text(tokens):
    result = []
    prev_end = 0
    for token in tokens:
        if token['type'] == 'note':
            if token['start'] > prev_end:
                length_name = length_to_name(token['start']-prev_end)
                result.append(f"R{length_name}")
            bw_name = bw_to_name(token['bw'])
            note_name = librosa.hz_to_note(token['freq'], unicode=False)
            # extract just the octave number with regex
            note_name = re.sub(r'[A-G]#?', '', note_name)

            length_name = length_to_name(token['length'])
            slope_name = token['slope'].name
            result.append(f"{bw_name}{note_name}{length_name}{slope_name}")
            prev_end = token['start'] + token['length']
        elif token['type'] == 'BOS':
            pass
        elif token['type'] == 'rest':
            result.append(f"R{length_to_name(token['length'])}")
            prev_end = token['start'] + token['length']

    return ' '.join(result)


def run_inference(waveform, sr):
    df = estimate_pitch_pesto(waveform, sr)
    df = get_amplitude(df, waveform, sr)
    df = get_relative_bandwidth(df, waveform, sr)

    solution = run_viterbi(df, cost_func=cost_func)

    text = convert_to_text(solution)
    ispa_results = {
        'df': df,
        'solution': solution,
        'text': text
    }
    return ispa_results

==========================================
</embedder/ispa/acoustics.py>
==========================================

==========================================
<embedder/ispa/features.py>
Size: 7829 bytes | Lines:      240
==========================================
import os.path
from collections import defaultdict
import json
import pickle

import numpy as np
import torch
import torch.nn as nn
import torchaudio
from torchaudio.models import wav2vec2_model

PATH = os.path.dirname(__file__)

# default config and model paths
config_path = os.path.join(PATH, 'models', 'aves-base-bio.torchaudio.model_config.json')
model_path = os.path.join(PATH, 'models', 'aves-base-bio.torchaudio.pt')


class AvesFeatureExtractor(nn.Module):
    def __init__(self, aves_config_path=config_path, aves_model_path=model_path):
        super().__init__()
        self.config = self.load_config(config_path)
        self.model = wav2vec2_model(**self.config, aux_num_out=None)
        self.model.load_state_dict(torch.load(model_path, weights_only=False))
        self.model.feature_extractor.requires_grad_(False)
        self.model.eval()

    def load_config(self, config_path):
        with open(config_path, 'r') as ff:
            obj = json.load(ff)

        return obj

    def forward(self, sig):
        with torch.no_grad():
            # extract_feature in the torchaudio version will output all 12 layers' output, -1 to select the final one
            out = self.model.extract_features(sig)[0][-1]

        return out


class MFCCFeatureExtractor(nn.Module):

    def __init__(self, sr=16000, n_mfcc=40, n_fft=400, hop_length=320, power=2.0, **kwargs):
        super().__init__()

        self.sr = sr
        self.n_mfcc = n_mfcc
        self.n_fft = n_fft
        self.hop_length = hop_length
        self.power = power

        self.mfcc = torchaudio.transforms.MFCC(
            sample_rate=self.sr,
            n_mfcc=self.n_mfcc,
            melkwargs={'n_fft': self.n_fft, 'hop_length': self.hop_length, 'n_mels': self.n_mfcc},
        )

    def forward(self, sig):
        out = self.mfcc(sig)
        out = out.transpose(1, 2)
        return out


def enumerate_tokens(features, start, distance):
    # enumerate list of tokens that start at start index
    # start: start index in features
    results = []
    # 1 frame of MFCC/AVES = 20ms
    for length in [1, 2, 5, 10, 20, 50]:    # multiple of 20ms
        end = start + length
        if end > len(features):
            break
        # choose the cluster with the smallest distance
        c_min = distance[start:end, :].mean(axis=0).argmin()
        dist = distance[start:end, c_min].sum()

        results.append({
            'type': 'note',
            'length': length,
            'cluster': c_min,
            'dist': dist})
        # print(f"note: {start} {length} {c_min} {dist}")

    return results


_BOS_TOKEN = {'type': 'BOS', 'length': 1, 'cluster': None, 'dist': 0.}
_EOS_TOKEN = {'type': 'EOS', 'length': 1, 'cluster': None, 'dist': 0.}


def cost_func(cnode, bnode=None):
    distance_cost = cnode['token']['dist']
    length_cost = 10. / (1 + np.log10(cnode['token']['length']))
    total_cost = distance_cost + length_cost

    # print(f"cost: {cnode['start']} {cnode['token']['length']} {cnode['token']['dist']} {distance_cost} {length_cost} {total_cost} {total_cost / cnode['token']['length']}")
    return total_cost


def run_viterbi(features, kmeans, cost_func=lambda node: 0):
    # features: (time, feature)
    distance = kmeans.transform(features)   # (time, n_clusters)
    bos_node = {'start': -1, 'next': [], 'token': _BOS_TOKEN, 'cost': 0}
    end_node_list = defaultdict(list)
    end_node_list[0].append(bos_node)

    for i in range(0, len(features)+1):
        # print(f"i: {i}", file=sys.stderr)
        if i < len(features):
            tokens = enumerate_tokens(features, i, distance)
        else:
            tokens = [_EOS_TOKEN]

        for token in tokens:
            cnode = {'start': i, 'next': [], 'token': token}
            min_cost = -1
            min_bnodes = []

            for bnode in end_node_list[i]:
                cost = bnode['cost'] + cost_func(cnode, bnode)

                # allow ties
                # if min_cost == -1 or cost < min_cost:
                #     min_cost = cost
                #     min_bnodes = [bnode]
                # elif cost == min_cost:
                #     min_bnodes.append(bnode)

                # doesn't allow ties
                if min_cost == -1 or cost < min_cost:
                    min_cost = cost
                    min_bnodes = [bnode]

            if len(min_bnodes) > 0:
                for bnode in min_bnodes:
                    cnode['cost'] = min_cost
                    bnode['next'].append(cnode)

                end_nodes = end_node_list[i+token['length']]
                if not cnode in end_nodes:
                    end_nodes.append(cnode)

    solutions_cache = {}

    def _enum_solutions(node):
        # convert node to hashable
        cache_key = (node['start'], tuple(node['token'].items()))
        if cache_key in solutions_cache:
            return solutions_cache[cache_key]

        if node['token'] == _EOS_TOKEN:
            return [[_EOS_TOKEN]]

        solutions = []
        for next_node in node['next']:
            for solution in _enum_solutions(next_node):
                node['token']['start'] = node['start']
                solutions.append([node['token']] + solution)

        solutions_cache[cache_key] = solutions
        return solutions

    return _enum_solutions(bos_node)[0]

LEN2MARK = {
    1: '',
    2: ',',
    5: ':',
    10: ';',
    20: '.',
    50: '..',
}

def convert_to_text(tokens, phoneme_map=None):
    result = []
    for token in tokens:
        if token['type'] == 'note':
            if phoneme_map is not None:
                token_str = phoneme_map[str(token['cluster'])].lower()
            else:
                token_str = str(token['cluster'])
            result.append(f"{token_str}{LEN2MARK[token['length']]}")
        elif token['type'] == 'BOS':
            pass
        elif token['type'] == 'EOS':
            pass
        else:
            raise ValueError(f'Unknown token type: {token["type"]}')

    return ' '.join(result)


class FeatureBasedISPAPredictor:
    def __init__(self, feature_type=None, kmeans_model=None, phoneme_map=None, use_cuda=False, **kwargs):
        if feature_type == 'aves':
            self.feature_extractor = AvesFeatureExtractor(**kwargs)
        elif feature_type == 'mfcc':
            self.feature_extractor = MFCCFeatureExtractor()
        else:
            raise ValueError('feature_type must be "aves" or "mfcc"')
        pass

        with open(kmeans_model, 'rb') as f:
            self.kmeans = pickle.load(f)

        if phoneme_map is not None:
            with open(phoneme_map, 'r') as f:
                self.phoneme_map = json.load(f)

        self.use_cuda = use_cuda
        if self.use_cuda:
            self.feature_extractor = self.feature_extractor.cuda()

    def predict(self, waveform, variation=None):
        if variation not in {'raw', 'seg', 'phn'}:
            raise ValueError('variation must be "raw", "seg", or "phn"')

        # extract feature
        if self.use_cuda:
            waveform = waveform.cuda()
        feature = self.feature_extractor(waveform)   # (batch, time, feature)
        feature = feature.squeeze(0)            # (time, feature)

        # apply k-means
        if self.use_cuda:
            feature = feature.cpu()
        feature = feature.detach().numpy()

        if variation == 'raw':
            clusters = self.kmeans.predict(feature)
            text = ' '.join([str(x) for x in clusters])
        elif variation == 'seg':
            tokens = run_viterbi(feature, self.kmeans, cost_func=cost_func)
            text = convert_to_text(tokens)
        elif variation == 'phn':
            tokens = run_viterbi(feature, self.kmeans, cost_func=cost_func)
            text = convert_to_text(tokens, self.phoneme_map)

        return text

==========================================
</embedder/ispa/features.py>
==========================================

==========================================
<embedder/ispa/models/aves-base-bio.torchaudio.model_config.json>
Size: 999 bytes | Lines:       52
==========================================
{
    "extractor_mode": "group_norm",
    "extractor_conv_layer_config": [
        [
            512,
            10,
            5
        ],
        [
            512,
            3,
            2
        ],
        [
            512,
            3,
            2
        ],
        [
            512,
            3,
            2
        ],
        [
            512,
            3,
            2
        ],
        [
            512,
            2,
            2
        ],
        [
            512,
            2,
            2
        ]
    ],
    "extractor_conv_bias": false,
    "encoder_embed_dim": 768,
    "encoder_projection_dropout": 0.1,
    "encoder_pos_conv_kernel": 128,
    "encoder_pos_conv_groups": 16,
    "encoder_num_layers": 12,
    "encoder_num_heads": 12,
    "encoder_attention_dropout": 0.1,
    "encoder_ff_interm_features": 3072,
    "encoder_ff_interm_dropout": 0.0,
    "encoder_dropout": 0.1,
    "encoder_layer_norm_first": false,
    "encoder_layer_drop": 0.05
}
==========================================
</embedder/ispa/models/aves-base-bio.torchaudio.model_config.json>
==========================================

==========================================
<embedder/ispa/models/c2p.aves.json>
Size: 576 bytes | Lines:        1
==========================================
{"18": "t_d", "41": "a", "37": "i:", "15": "d", "46": "r-=", "10": "?", "5": "p", "34": "i", "47": "j", "49": "o", "19": "t_s-", "11": "t", "25": "I", "28": "N-", "48": "e", "22": "t_h", "31": "Q", "12": "s-", "3": "ts`", "32": "n", "21": "E", "24": "U", "2": "s", "43": "4", "8": "u:", "33": "f", "13": "N", "39": "O", "27": "ts", "4": "x", "1": "R", "35": "u@", "30": "@", "0": "t_j", "29": "k", "9": "l", "45": "D", "38": "z_j", "16": "m", "17": "1", "14": "b_o", "40": "b", "20": "v", "44": "n`", "26": "kp_}", "7": "V", "42": "t_d_h", "6": "s`", "23": "l_j", "36": "O~"}

==========================================
</embedder/ispa/models/c2p.aves.json>
==========================================

==========================================
<embedder/ispa/models/c2p.mfcc.json>
Size: 576 bytes | Lines:        1
==========================================
{"6": "t_d", "1": "a", "42": "i:", "29": "d", "4": "r-=", "9": "?", "31": "p", "3": "i", "24": "j", "14": "o", "37": "t_s-", "13": "t", "39": "I", "12": "N-", "47": "e", "32": "t_h", "11": "Q", "27": "s-", "7": "ts`", "41": "n", "21": "E", "35": "U", "5": "s", "36": "4", "18": "u:", "30": "f", "26": "N", "16": "O", "38": "ts", "49": "x", "23": "R", "28": "u@", "17": "@", "10": "t_j", "46": "k", "48": "l", "19": "D", "2": "z_j", "25": "m", "34": "1", "33": "b_o", "22": "b", "0": "v", "45": "n`", "20": "kp_}", "43": "V", "15": "t_d_h", "40": "s`", "8": "l_j", "44": "O~"}

==========================================
</embedder/ispa/models/c2p.mfcc.json>
==========================================

==========================================
<embedder/ispa/utils.py>
Size: 1566 bytes | Lines:       34
==========================================
import sys
import torch
import torchaudio
torchaudio.set_audio_backend("soundfile")

def load_waveform(path, tgt_sr=16_000, min_duration=None, max_duration=None, start_sec=None, end_sec=None):
    waveform, src_sr = torchaudio.load(path)
    # downsample to 16kHz
    if src_sr != tgt_sr:
        print(f"Resample from {src_sr} to {tgt_sr}", file=sys.stderr)
        waveform = torchaudio.transforms.Resample(src_sr, tgt_sr)(waveform)

    # downmix if stereo
    if len(waveform) > 1:
        print(f"Downmix from {len(waveform)} channels to 1", file=sys.stderr)
        waveform = torch.mean(waveform, dim=0, keepdim=True)

    # extract segment if start_sec or end_sec are provided
    if start_sec is not None or end_sec is not None:
        start_sample = int(start_sec * tgt_sr) if start_sec is not None else 0
        end_sample = int(end_sec * tgt_sr) if end_sec is not None else waveform.shape[1]
        waveform = waveform[:, start_sample:end_sample]

    # pad to min_duration
    if min_duration is not None and len(waveform[0]) < min_duration * tgt_sr:
        print(f"Pad from {len(waveform[0]) / tgt_sr} to {min_duration} secs", file=sys.stderr)
        waveform = torch.nn.functional.pad(waveform, (0, int(min_duration * tgt_sr - len(waveform[0]))))

    # truncate to max_duration
    if max_duration is not None and len(waveform[0]) > max_duration * tgt_sr:
        print(f"Truncate from {len(waveform[0]) / tgt_sr} to {max_duration} secs", file=sys.stderr)
        waveform = waveform[:, :int(max_duration * tgt_sr)]

    return waveform, tgt_sr

==========================================
</embedder/ispa/utils.py>
==========================================

==========================================
<get-data.py>
Size: 1352 bytes | Lines:       36
==========================================
import sys

print("""
This script will download and process a large amount of crow audio data.
It requires over 30 GB of disk space and will take several hours to complete.
Each audio file will be downloaded, denoised, segmented, embedded, and classified.

Files that are already processed will be skipped.
Are you ready to begin? (y/n)
""")
response = input("> ").strip().lower()
if response not in ("y", "yes"):
    print("Exiting. Please run the script again when you're ready.")
    sys.exit(0)

from downloader.download_backgrounds import start_downloads as start_downloads_background
from downloader.download_crows_ebird import start_downloads as start_downloads_ebird
from downloader.download_crows_xeno import start_downloads as start_downloads_xeno
from denoiser.denoise_crows import start_denoising
from detector.detect_all import start_detections
from embedder.embed_all import start_embeddings

# Download all crow audio and mix-related audio
for download in [start_downloads_background, start_downloads_ebird, start_downloads_xeno]:
    download()

# Denoise all crow audio
start_denoising()

# Embed all crow audio (both original and denoised)
start_embeddings(denoised=False)
start_embeddings(denoised=True)

# Detect 1 second crow segments (uses classifier + embeddings)
# This also creates the auto_labels.json file
start_detections()

==========================================
</get-data.py>
==========================================

==========================================
<labeler/embeddings.html>
Size: 10707 bytes | Lines:      231
==========================================
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Crow Embeddings | 3D Viewer</title>
    <link rel="icon" type="image/png" href="/images/crow-emote.png">
    <link rel="stylesheet" href="node_modules/bootstrap/dist/css/bootstrap.min.css">

    <style>
        body {
            margin: 0;
            background: linear-gradient(135deg, #000033, #000000, #330033);
        }
        #app {
            width: 100vw;
            height: 100vh;
        }
        canvas {
            display: block;
        }
    </style>
</head>
<body>
<div id="app">
    <menu-component title="Crow Embeddings"></menu-component>
</div>

<!-- Vue 3 (global build) -->
<script src="https://unpkg.com/vue@3/dist/vue.global.prod.js"></script>
<!-- Three.js and OrbitControls -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r134/three.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/three@0.134/examples/js/controls/OrbitControls.js"></script>
<!-- Bootstrap JS -->
<script src="node_modules/bootstrap/dist/js/bootstrap.bundle.min.js"></script>
<script src="js/components/menu.js"></script>
<script>
    const App = {
        data() {
            return {
                embeddings: [] // Each embedding should include coordinates, color, and segment (e.g. "FILE-START-END")
            };
        },
        methods: {
            async loadEmbeddings() {
                try {
                    const res = await fetch('/api/embeddings');
                    const data = await res.json();
                    if (data.success) {
                        this.embeddings = data.embeddings;
                        this.initThreeJS();
                    } else {
                        console.error("Error loading embeddings:", data.error);
                    }
                } catch (e) {
                    console.error("Fetch error:", e);
                }
            },
            playSegment(segmentKey) {
                // segmentKey must be in the format: FILE-START-END, e.g., "104365511-15-30"
                if (!segmentKey) return;
                const parts = segmentKey.split("-");
                if (parts.length !== 3) {
                    console.error("Invalid segment key:", segmentKey);
                    return;
                }
                const file = parts[0];
                const start = parseFloat(parts[1]);
                const end = parseFloat(parts[2]);
                const audio = new Audio("/cache/library/" + file + ".mp3");
                audio.addEventListener('loadedmetadata', () => {
                    audio.currentTime = start;
                    audio.play();
                });
                audio.addEventListener('timeupdate', () => {
                    if (audio.currentTime >= end) {
                        audio.pause();
                    }
                });
            },
            initThreeJS() {
                const container = document.getElementById('app');
                const scene = new THREE.Scene();

                const camera = new THREE.PerspectiveCamera(75, container.clientWidth / container.clientHeight, 0.1, 1000);
                camera.position.z = 50;

                const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
                renderer.setClearColor(0x000000, 0); // transparent background
                renderer.setSize(container.clientWidth, container.clientHeight);
                container.appendChild(renderer.domElement);

                const controls = new THREE.OrbitControls(camera, renderer.domElement);
                controls.enableDamping = true;
                controls.dampingFactor = 0.05;

                // Create a circular texture from an inline SVG.
                const circleSVG = `
            <svg xmlns="http://www.w3.org/2000/svg" width="64" height="64">
              <circle cx="32" cy="32" r="30" fill="white"/>
            </svg>
          `;
                const circleURL = 'data:image/svg+xml;charset=utf-8,' + encodeURIComponent(circleSVG);
                const textureLoader = new THREE.TextureLoader();
                const circleTexture = textureLoader.load(circleURL);

                // Build the points geometry.
                const positions = [];
                const colors = [];
                this.embeddings.forEach(item => {
                    positions.push(item.coordinates[0], item.coordinates[1], item.coordinates[2]);
                    const color = new THREE.Color(item.color);
                    colors.push(color.r, color.g, color.b);
                });
                const geometry = new THREE.BufferGeometry();
                geometry.setAttribute('position', new THREE.BufferAttribute(new Float32Array(positions), 3));
                geometry.setAttribute('color', new THREE.BufferAttribute(new Float32Array(colors), 3));

                // Points material using the circular texture.
                const pointsMaterial = new THREE.PointsMaterial({
                    size: 0.2,
                    vertexColors: true,
                    map: circleTexture,
                    alphaTest: 0.5,
                    transparent: true
                });
                const points = new THREE.Points(geometry, pointsMaterial);
                scene.add(points);

                // Line segments connecting the hovered point to its 5 nearest neighbors.
                const linePositions = new Float32Array(5 * 2 * 3); // 5 segments: 2 vertices each, 3 components per vertex.
                const lineGeometry = new THREE.BufferGeometry();
                lineGeometry.setAttribute('position', new THREE.BufferAttribute(linePositions, 3));
                const lineMaterial = new THREE.LineBasicMaterial({ color: 0xffffff });
                const lineSegments = new THREE.LineSegments(lineGeometry, lineMaterial);
                lineSegments.visible = false;
                scene.add(lineSegments);

                const raycaster = new THREE.Raycaster();
                const mouse = new THREE.Vector2();
                let selectedIndex = null;

                // Simplified mousemove: use default raycaster intersection.
                renderer.domElement.addEventListener('mousemove', event => {
                    const rect = renderer.domElement.getBoundingClientRect();
                    mouse.x = ((event.clientX - rect.left) / rect.width) * 2 - 1;
                    mouse.y = -((event.clientY - rect.top) / rect.height) * 2 + 1;
                    raycaster.setFromCamera(mouse, camera);
                    const intersects = raycaster.intersectObject(points);
                    if (intersects.length > 0) {
                        const intersect = intersects[0];
                        const hoveredPoint = intersect.point;
                        const hoveredIndex = intersect.index;
                        // Find the 5 nearest neighbors.
                        let neighbors = [];
                        for (let i = 0; i < geometry.attributes.position.count; i++) {
                            if (i === hoveredIndex) continue;
                            let pt = new THREE.Vector3().fromBufferAttribute(geometry.attributes.position, i);
                            // Convert to world coordinates.
                            points.localToWorld(pt);
                            neighbors.push({ pt, dist: hoveredPoint.distanceToSquared(pt) });
                        }
                        neighbors.sort((a, b) => a.dist - b.dist);
                        const five = neighbors.slice(0, 5);
                        // Update line geometry.
                        for (let i = 0; i < five.length; i++) {
                            linePositions[i * 6 + 0] = hoveredPoint.x;
                            linePositions[i * 6 + 1] = hoveredPoint.y;
                            linePositions[i * 6 + 2] = hoveredPoint.z;
                            linePositions[i * 6 + 3] = five[i].pt.x;
                            linePositions[i * 6 + 4] = five[i].pt.y;
                            linePositions[i * 6 + 5] = five[i].pt.z;
                        }
                        lineGeometry.attributes.position.needsUpdate = true;
                        lineSegments.visible = true;
                    } else {
                        lineSegments.visible = false;
                    }
                });

                // Simplified click: select/deselect point and play its audio segment.
                renderer.domElement.addEventListener('click', event => {
                    const rect = renderer.domElement.getBoundingClientRect();
                    mouse.x = ((event.clientX - rect.left) / rect.width) * 2 - 1;
                    mouse.y = -((event.clientY - rect.top) / rect.height) * 2 + 1;
                    raycaster.setFromCamera(mouse, camera);
                    const intersects = raycaster.intersectObject(points);
                    if (intersects.length > 0) {
                        const intersect = intersects[0];
                        const clickedIndex = intersect.index;
                        let pos = new THREE.Vector3().fromBufferAttribute(geometry.attributes.position, clickedIndex);
                        points.localToWorld(pos);
                        if (selectedIndex === clickedIndex) {
                            // Deselect.
                            selectedIndex = null;
                        } else {
                            selectedIndex = clickedIndex;
                            const embedding = this.embeddings[clickedIndex];
                            if (embedding && embedding.segment_key) {
                                this.playSegment(embedding.segment_key);
                            }
                        }
                    }
                });

                function animate() {
                    requestAnimationFrame(animate);
                    controls.update();
                    renderer.render(scene, camera);
                }
                animate();

                window.addEventListener('resize', () => {
                    camera.aspect = container.clientWidth / container.clientHeight;
                    camera.updateProjectionMatrix();
                    renderer.setSize(container.clientWidth, container.clientHeight);
                });
            }
        },
        mounted() {
            this.loadEmbeddings();
        }
    };

    const app = Vue.createApp(App);
    app.component('menu-component', window.MenuComponent);
    app.mount('#app');

</script>
</body>
</html>

==========================================
</labeler/embeddings.html>
==========================================

==========================================
<labeler/index.html>
Size: 9654 bytes | Lines:      260
==========================================
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Crow Audio Labeler</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" type="image/png" href="/images/crow-emote.png">
    <!-- Bootstrap CSS -->
    <link rel="stylesheet" href="node_modules/bootstrap/dist/css/bootstrap.min.css">
    <style>
        .segment-card {
            margin-bottom: 1rem;
        }
        /* We'll make the card borders thicker (4px) when colored. */
        .border-primary {
            border-width: 4px !important;
        }
        .border-danger {
            border-width: 4px !important;
        }
        .border-secondary {
            border-width: 1px !important;
        }
        body {
            margin: 0;
            background: linear-gradient(135deg, #000033, #000000, #330033);
            background-attachment: fixed;
            background-size: cover;
        }

        /* Modern, semi-transparent card style */
        .card {
            background-color: rgba(255, 255, 255, 0.75);
            border-radius: 10px;
            box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2);
            transition: transform 0.2s ease, box-shadow 0.2s ease;
        }
        
        /* Style for reviewed cards */
        .reviewed-card {
            background-color: rgba(200, 200, 200, 0.7) !important;
            box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
        }

        /* Update page title for better contrast */
        h1 {
            color: #fff;
            font-family: 'Montserrat', sans-serif; /* Ensure you include this font */
            text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.5);
        }

        .btn-check:not(:checked) + .btn-outline-primary,
        .btn-check:not(:checked) + .btn-outline-secondary,
        .btn-check:not(:checked) + .btn-outline-danger,
        .btn-check:not(:checked) + .btn-outline-success {
            background-color: #fff !important;
        }

        .btn-check:checked + .btn-feature {
            color: transparent;
            text-shadow: 0 0 0 white;
        }

        .stats-card {
            background-color: rgba(40, 40, 40, 0.9); /* dark, semi-transparent background */
            color: #f8f9fa; /* light text */
            border: none;
            border-radius: 10px;
            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.5);
        }

        .stats-card .card-body {
            padding: 0.75rem;
        }

        .stats-card .card-title {
            color: #fff;
            font-weight: bold;
        }

        .stats-card p {
            color: #ddd;
        }

        .stats-card .progress {
            background-color: rgba(255,255,255,0.2); /* subtle light progress track */
            height: 6px;
            border-radius: 3px;
            overflow: hidden;
        }

        .stats-card .progress-bar {
            background-color: #00aaff; /* a bright accent color */
        }

        .delete-btn {
          background-color: #555; /* Dark gray */
          border-color: #555;
          color: #fff;
        }

        .delete-btn:hover {
          background-color: #dc3545; /* Bootstrap danger red */
          border-color: #dc3545;
          color: #fff;
        }


    </style>
</head>
<body>
<div id="app" class="container my-4">
    <menu-component title="Crow Audio Labeler"></menu-component>

    <div v-if="errorMessage" class="alert alert-danger" role="alert">
        {{ errorMessage }}
    </div>

    <!-- Stats Row (3 columns) -->
    <div class="row mb-3">
        <!-- Files Card -->
        <div class="col-md-4">
            <div class="card stats-card mb-2">
                <div class="card-body p-1">
                    <h6 class="card-title mb-1">Files</h6>
                    <p class="m-0">
                        {{ stats.labeledFiles }}/{{ stats.totalFiles }} ({{ stats.filesPct }}%)
                    </p>
                    <div class="progress">
                        <div class="progress-bar" role="progressbar" :style="{ width: stats.filesPct + '%' }"></div>
                    </div>
                </div>
            </div>
        </div>
        <!-- Segments Card -->
        <div class="col-md-4">
            <div class="card stats-card mb-2">
                <div class="card-body p-1">
                    <h6 class="card-title mb-1">Segments</h6>
                    <p class="m-0">
                        {{ stats.labeledSegments }}/{{ stats.totalSegments }} ({{ stats.segmentsPct }}%)
                    </p>
                    <div class="progress">
                        <div class="progress-bar" role="progressbar" :style="{ width: stats.segmentsPct + '%' }"></div>
                    </div>
                </div>
            </div>
        </div>
        <!-- Hours Card -->
        <div class="col-md-4">
            <div class="card stats-card mb-2">
                <div class="card-body p-1">
                    <h6 class="card-title mb-1">Hours</h6>
                    <p class="m-0">
                        {{ stats.labeledHours.toFixed(2) }}/{{ stats.totalHours.toFixed(2) }} ({{ stats.hoursPct }}%)
                    </p>
                    <div class="progress">
                        <div class="progress-bar" role="progressbar" :style="{ width: stats.hoursPct + '%' }"></div>
                    </div>
                </div>
            </div>
        </div>
    </div>


    <div class="d-flex justify-content-between align-items-center mb-3">
        <!-- Left side: Paginator and Filter -->
        <div class="d-flex align-items-center gap-2">
            <pagination
                    :current-page="currentPage"
                    :total-pages="totalPages"
                    @prev="prevPage"
                    @next="nextPage"
                    @goto="handleGotoPage">
            </pagination>
        </div>

        <!-- Playback Speed Button Group on the right -->
        <div class="d-flex align-items-center" style="gap: 1rem;">
            <span style="color: #fff;">Speed:</span>
            <div class="btn-group btn-group-sm" role="group">
                <button type="button" class="btn"
                        :class="{'btn-primary': playbackSpeed === 1, 'btn-outline-primary': playbackSpeed !== 1}"
                        @click="setPlaybackSpeed(1)">1X</button>
                <button type="button" class="btn"
                        :class="{'btn-primary': playbackSpeed === 2, 'btn-outline-primary': playbackSpeed !== 2}"
                        @click="setPlaybackSpeed(2)">2X</button>
                <button type="button" class="btn"
                        :class="{'btn-primary': playbackSpeed === 3, 'btn-outline-primary': playbackSpeed !== 3}"
                        @click="setPlaybackSpeed(3)">3X</button>
                <button type="button" class="btn"
                        :class="{'btn-primary': playbackSpeed === 4, 'btn-outline-primary': playbackSpeed !== 4}"
                        @click="setPlaybackSpeed(4)">4X</button>
            </div>

            <!-- Filter Component -->
            <filter-component
                    :filters="filters"
                    :active-filters="activeFilters"
                    @filters-changed="updateFilters">
            </filter-component>
        </div>
    </div>


    <!-- 2-Column Layout for Segment Cards with equal height rows -->
    <div class="row row-cols-1 row-cols-lg-2 g-4">
        <div v-if="filteredSegments.length === 0" class="col-12 text-center py-5">
            <div class="card bg-light p-5">
                <h4 class="text-muted">No results found</h4>
                <p class="mb-0">Try adjusting your filters to see more segments</p>
            </div>
        </div>
        <div v-else-if="paginatedSegments.length === 0 && filteredSegments.length > 0" class="col-12 text-center py-5">
            <div class="card bg-light p-5">
                <h4 class="text-muted">No segments on this page</h4>
                <p class="mb-0">Try navigating to a different page</p>
            </div>
        </div>
        <div v-for="(segment, index) in paginatedSegments" :key="segment.id + '-' + segment.start_time + '-' + segment.end_time" class="col d-flex">
            <segment-card
              :segment="segment"
              :labels="labels"
              :playback-speed="playbackSpeed"
              :prev-labels="index > 0 ? labels[paginatedSegments[index - 1].id + '-' + paginatedSegments[index - 1].start_time + '-' + paginatedSegments[index - 1].end_time] : null"
              @labels-changed="saveLabelsToServer"
              @segment-deleted="handleSegmentDeleted"
              class="w-100">
            </segment-card>
        </div>
    </div>

    <!-- Pagination Controls -->
    <div class="mt-4">
      <pagination
              :current-page="currentPage"
              :total-pages="totalPages"
              @prev="prevPage"
              @next="nextPage"
              @goto="handleGotoPage"
      ></pagination>
    </div>

</div>

<!-- Vue 3 global build -->
<script src="node_modules/vue/dist/vue.global.js"></script>
<!-- Bootstrap JS -->
<script src="node_modules/bootstrap/dist/js/bootstrap.bundle.min.js"></script>
<!-- Papa Parse for CSV parsing (if needed) -->
<script src="https://unpkg.com/papaparse@5.4.1/papaparse.min.js"></script>
<!-- Components -->
<script src="js/components/pagination.js"></script>
<script src="js/components/segmentCard.js"></script>
<script src="js/components/filterComponent.js"></script>
<script src="js/components/menu.js"></script>
<!-- Main app script -->
<script src="js/app.js"></script>
</body>
</html>

==========================================
</labeler/index.html>
==========================================

==========================================
<labeler/js/app.js>
Size: 16595 bytes | Lines:      397
==========================================
const app = Vue.createApp({
    data() {
        return {
            segments: [],
            // Only a single cached filtered set:
            filteredSegmentsCache: [],
            labels: {},
            csvData: {},
            currentPage: 1,
            segmentsPerPage: 30,
            playbackSpeed: 1,
            totalPagesCached: 0,
            errorMessage: "",
            filters: {
                labelStatus: 'all',
                reviewStatus: 'all',
                callTypes: {
                    rattle: false,
                    softSong: false,
                    mob: false,
                    alert: false,
                    begging: false
                },
                crowCounts: {
                    0: false,
                    1: false,
                    2: false,
                    3: false,
                    4: false
                },
                crowAge: 'all',
                globalFilter: '',
                quality: 'all'
            },
            activeFilters: null
        };
    },
    computed: {
        totalPages() {
            return Math.ceil(this.filteredSegmentsCache.length / this.segmentsPerPage);
        },
        paginatedSegments() {
            const start = (this.currentPage - 1) * this.segmentsPerPage;
            return this.filteredSegmentsCache.slice(start, start + this.segmentsPerPage);
        },
        // Bridge for legacy template references.
        filteredSegments() {
            return this.filteredSegmentsCache;
        },
        stats() {
            const uniqueFileIDs = new Set();
            // Use filtered segments if active filters are applied, otherwise use all segments.
            const segmentsToCount = this.activeFilters ? this.filteredSegmentsCache : this.segments;
            let totalSegments = segmentsToCount.length;
            let labeledSegments = 0;
            let totalDurationSec = 0;
            let labeledDurationSec = 0;
            const fileIDtoIsLabeled = {};
            const isSegmentLabeled = (segKey) => {
                const lbl = this.labels[segKey];
                if (!lbl) return false;
                return lbl.reviewed === true;
            };
            for (const seg of segmentsToCount) {
                uniqueFileIDs.add(seg.id);
                const segKey = `${seg.id}-${seg.start_time}-${seg.end_time}`;
                const durationSec = seg.end_time - seg.start_time;
                totalDurationSec += durationSec;
                if (isSegmentLabeled(segKey)) {
                    labeledSegments++;
                    labeledDurationSec += durationSec;
                    fileIDtoIsLabeled[seg.id] = true;
                }
            }
            const totalFiles = uniqueFileIDs.size;
            let labeledFiles = 0;
            for (const fileID of uniqueFileIDs) {
                if (fileIDtoIsLabeled[fileID]) labeledFiles++;
            }
            const totalHours = totalDurationSec / 3600;
            const labeledHours = labeledDurationSec / 3600;
            const filesPct = totalFiles === 0 ? 0 : Math.round((labeledFiles / totalFiles) * 100);
            const segmentsPct = totalSegments === 0 ? 0 : Math.round((labeledSegments / totalSegments) * 100);
            const hoursPct = totalHours === 0 ? 0 : Math.round((labeledHours / totalHours) * 100);
            return {
                labeledFiles,
                totalFiles,
                filesPct,
                labeledSegments,
                totalSegments,
                segmentsPct,
                labeledHours,
                totalHours,
                hoursPct
            };
        }
    },
    methods: {
        segmentKey(segment) {
            return `${segment.id}-${segment.start_time}-${segment.end_time}`;
        },
        // Rewritten filtering function using only the activeFilters object.
        applyFiltersToSegments() {
            return this.segments.filter(segment => {
                const segKey = this.segmentKey(segment);
                const labelData = this.labels[segKey];

                // Label Status Filter
                if (this.activeFilters.labelStatus !== 'all') {
                    const isLabeled = labelData && (labelData.crowCount !== undefined);
                    if (this.activeFilters.labelStatus === 'labeled' && !isLabeled) return false;
                    if (this.activeFilters.labelStatus === 'unlabeled' && isLabeled) return false;
                }

                // Review Status Filter
                if (this.activeFilters.reviewStatus !== 'all') {
                    const isReviewed = labelData && labelData.reviewed === true;
                    if (this.activeFilters.reviewStatus === 'reviewed' && !isReviewed) return false;
                    if (this.activeFilters.reviewStatus === 'unreviewed' && isReviewed) return false;
                }

                // Call Types Filter (including begging)
                if (labelData) {
                    const callTypeFilters = this.activeFilters.callTypes;
                    const hasActiveCallTypeFilters = Object.values(callTypeFilters).some(val => val);
                    if (hasActiveCallTypeFilters) {
                        const matchesAType =
                            (callTypeFilters.rattle && labelData.rattle) ||
                            (callTypeFilters.softSong && labelData.softSong) ||
                            (callTypeFilters.mob && labelData.mob) ||
                            (callTypeFilters.alert && labelData.alert) ||
                            (callTypeFilters.begging && labelData.begging);
                        if (!matchesAType) return false;
                    }
                }

                // Crow Count Filter
                if (labelData && labelData.crowCount !== undefined) {
                    const crowCountFilters = this.activeFilters.crowCounts;
                    const hasActiveCrowCountFilters = Object.values(crowCountFilters).some(val => val);
                    if (hasActiveCrowCountFilters && !crowCountFilters[labelData.crowCount]) {
                        return false;
                    }
                }

                // Crow Age Filter
                if (this.activeFilters.crowAge !== 'all' && labelData && labelData.crowAge !== undefined) {
                    if (this.activeFilters.crowAge === 'adult' && labelData.crowAge !== 1) return false;
                    if (this.activeFilters.crowAge === 'juvenile' && labelData.crowAge !== 2) return false;
                }

                // Quality Filter
                if (this.activeFilters.quality !== 'all') {
                    // require a defined quality that matches the filter
                    const q = labelData?.quality;
                    if (q == null || Number(q) !== Number(this.activeFilters.quality)) {
                        return false;
                    }
                }

                // Global Filter: searches across media_notes, recordist (author), file ID and cluster.
                if (this.activeFilters.globalFilter && this.activeFilters.globalFilter.trim() !== '') {
                    const searchTerm = this.activeFilters.globalFilter.trim().toLowerCase();
                    const mediaNotes = (segment.media_notes || '').toLowerCase();
                    const recordist = (segment.recordist || '').toLowerCase();
                    const fileId = segment.id.toString().toLowerCase();
                    const cluster = (segment.cluster !== undefined ? segment.cluster.toString() : '').toLowerCase();
                    if (
                        !mediaNotes.includes(searchTerm) &&
                        !recordist.includes(searchTerm) &&
                        !fileId.includes(searchTerm) &&
                        !cluster.includes(searchTerm)
                    ) {
                        return false;
                    }
                }

                return true;
            });
        },
        updateFilters(newFilters) {
            // Deep copy new filters.
            this.activeFilters = JSON.parse(JSON.stringify(newFilters));
            this.currentPage = 1;
            localStorage.setItem('activeFilters', JSON.stringify(this.activeFilters));

            // Recalculate the filtered segments.
            this.filteredSegmentsCache = this.applyFiltersToSegments();
        },
        // Initialize filteredSegmentsCache once segments are loaded.
        initializeFilterCache() {
            if (this.activeFilters) {
                this.filteredSegmentsCache = this.applyFiltersToSegments();
            } else {
                this.filteredSegmentsCache = this.segments;
            }
        },
        loadCrowsCSV() {
            const files = ['/cache/csv/crows.csv', '/cache/csv/crows-xeno-canto.csv', '/cache/csv/local.csv' ];
            let remaining = files.length;
            files.forEach(file => {
                Papa.parse(file, {
                    download: true,
                    header: true,
                    complete: (results) => {
                        results.data.forEach(row => {
                            const id = row['ML Catalog Number'];
                            if (!id) return;
                            if (!this.csvData[id]) this.csvData[id] = {};
                            this.csvData[id].recorder = row['Recordist'] || this.csvData[id].recorder || 'Unknown';
                            this.csvData[id].mediaNotes = row['Media notes'] || this.csvData[id].mediaNotes || '';
                        });
                        if (!--remaining) this.attachCSVtoSegments();
                    },
                    error: err => {
                        console.error('Error parsing', file, err);
                        if (!--remaining) this.attachCSVtoSegments();
                    }
                });
            });
        },
        loadSegments() {
            fetch('/cache/cluster_segments.json')
                .then(r => r.json())
                .then(data => {
                    const segArray = [];
                    for (const [id, segs] of Object.entries(data)) {
                        segs.forEach(seg => {
                            seg.id = id;
                            segArray.push(seg);
                        });
                    }
                    
                    // Sort by cluster
                    segArray.sort((a, b) => {
                        const clusterA = a.cluster || 1;
                        const clusterB = b.cluster || 1;
                        return clusterA - clusterB;
                    });
                    
                    this.segments = segArray;
                    this.loadCrowsCSV();
                    this.totalPagesCached = Math.ceil(this.segments.length / this.segmentsPerPage);
                    this.initializeFilterCache();
                })
                .catch(err => console.error('Error loading segments:', err));
        },
        attachCSVtoSegments() {
            this.segments.forEach(seg => {
                if (this.csvData[seg.id]) {
                    seg.recordist = this.csvData[seg.id].recorder;
                    seg.media_notes = this.csvData[seg.id].mediaNotes;
                } else {
                    seg.recordist = 'Unknown';
                    seg.media_notes = '';
                }
            });
        },
        loadLabels() {
            fetch('/cache/cluster_labels.json')
                .then(r => r.json())
                .then(data => {
                    this.labels = data;
                    // Re-apply active filters now that labels have arrived
                    if (this.activeFilters) {
                        this.filteredSegmentsCache = this.applyFiltersToSegments();
                    }
                })
                .catch(err => {
                    console.warn('No labels json found, starting fresh.');
                    this.labels = {};
                });
        },
        saveLabelsToServer(payload) {
            fetch('/updateLabels', {
                method: 'POST',
                headers: {'Content-Type': 'application/json'},
                body: JSON.stringify(payload)
            })
                .then(res => {
                    if (!res.ok) {
                        return res.json().then(data => {
                            throw new Error(data.error || "Server error");
                        });
                    }
                    return res.json();
                })
                .then(data => {
                    if (!data.success) {
                        throw new Error("Failed to update labels on server.");
                    }
                    const segmentToUpdate = this.segments.find(seg => this.segmentKey(seg) === payload.segmentKey);
                    if (segmentToUpdate) {
                        segmentToUpdate.labelData = payload.labels;
                    }
                    this.errorMessage = "";
                })
                .catch(err => {
                    console.error('Error updating labels on server:', err);
                    this.errorMessage = "Error updating labels: " + err.message;
                });
        },
        prevPage() {
            if (this.currentPage > 1) {
                this.currentPage--;
                window.scrollTo({top: 0, behavior: 'smooth'});
            }
        },
        nextPage() {
            if (this.currentPage < this.totalPages) {
                this.currentPage++;
                window.scrollTo({top: 0, behavior: 'smooth'});
            }
        },
        handleGotoPage(pageNum) {
            if (pageNum >= 1 && pageNum <= this.totalPages) {
                this.currentPage = pageNum;
                window.scrollTo({top: 0, behavior: 'smooth'});
            }
        },
        setPlaybackSpeed(speed) {
            this.playbackSpeed = speed;
        },
        // Delete segment handling remains the same.
        handleSegmentDeleted(segmentKey) {
            this.segments = this.segments.filter(seg => this.segmentKey(seg) !== segmentKey);
            this.filteredSegmentsCache = this.filteredSegmentsCache.filter(seg => this.segmentKey(seg) !== segmentKey);
            delete this.labels[segmentKey];
            this.deleteSegmentFromServer(segmentKey);
        },
        deleteSegmentFromServer(segmentKey) {
            fetch('/deleteSegment', {
                method: 'DELETE',
                headers: {'Content-Type': 'application/json'},
                body: JSON.stringify({segmentKey})
            })
                .then(res => res.json())
                .then(data => {
                    if (!data.success) {
                        throw new Error("Failed to delete segment on server.");
                    }
                })
                .catch(err => {
                    console.error('Error deleting segment on server:', err);
                    this.errorMessage = "Error deleting segment: " + err.message;
                });
        }
    },
    watch: {
        currentPage(newVal) {
            localStorage.setItem('currentPage', newVal);
        },
        playbackSpeed(newVal) {
            localStorage.setItem('playbackSpeed', newVal);
        }
    },
    mounted() {
        // Retrieve saved current page and playback speed from localStorage
        const savedPage = localStorage.getItem('currentPage');
        if (savedPage) {
            this.currentPage = parseInt(savedPage, 10);
        }
        const savedSpeed = localStorage.getItem('playbackSpeed');
        if (savedSpeed) {
            this.playbackSpeed = parseInt(savedSpeed, 10);
        }
        
        // Load saved filters if they exist
        const savedFilters = localStorage.getItem('activeFilters');
        if (savedFilters) {
            try {
                this.activeFilters = JSON.parse(savedFilters);
            } catch (e) {
                console.error('Error parsing saved filters:', e);
                this.activeFilters = null;
            }
        }
        
        // Now loadSegments and labels
        this.loadSegments();
        this.loadLabels();
    }
});

app.component('pagination', window.Pagination);
app.component('segment-card', window.SegmentCard);
app.component('filter-component', window.FilterComponent);
app.component('menu-component', window.MenuComponent);

// Custom directive for auto focus
app.directive("focus", {
  mounted(el) {
    el.focus();
  }
});

app.mount('#app');

==========================================
</labeler/js/app.js>
==========================================

==========================================
<labeler/js/components/filterComponent.js>
Size: 13250 bytes | Lines:      220
==========================================
// js/components/filterComponent.js
window.FilterComponent = {
    props: ['filters', 'activeFilters'],
    template: `
    <div class="d-flex align-items-center">
        <button class="btn btn-sm" 
                :class="{'btn-outline-light': !hasActiveFilters, 'btn-primary': hasActiveFilters}"
                @click="openFilterModal" 
                data-bs-toggle="modal" 
                data-bs-target="#filterModal">
            <span style="font-size: 1.2rem;">🔍</span>
        </button>
        
        <!-- Filter Modal -->
        <div class="modal fade" id="filterModal" tabindex="-1" aria-labelledby="filterModalLabel" aria-hidden="true">
            <div class="modal-dialog modal-lg">
                <div class="modal-content">
                    <div class="modal-header">
                        <h5 class="modal-title" id="filterModalLabel">Filter Segments</h5>
                        <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close"></button>
                    </div>
                    <div class="modal-body">
                        <!-- Label Status -->
                        <div class="mb-3">
                            <label class="form-label">Label Status</label>
                            <div class="btn-group w-100" role="group">
                                <input type="radio" class="btn-check" name="labelStatus" id="all" value="all" v-model="localFilters.labelStatus">
                                <label class="btn btn-outline-primary" for="all">All</label>
                                <input type="radio" class="btn-check" name="labelStatus" id="labeled" value="labeled" v-model="localFilters.labelStatus">
                                <label class="btn btn-outline-primary" for="labeled">Labeled</label>
                                <input type="radio" class="btn-check" name="labelStatus" id="unlabeled" value="unlabeled" v-model="localFilters.labelStatus">
                                <label class="btn btn-outline-primary" for="unlabeled">Unlabeled</label>
                            </div>
                        </div>
                        
                        <!-- Review Status -->
                        <div class="mb-3">
                            <label class="form-label">Review Status</label>
                            <div class="btn-group w-100" role="group">
                                <input type="radio" class="btn-check" name="reviewStatus" id="allReview" value="all" v-model="localFilters.reviewStatus">
                                <label class="btn btn-outline-primary" for="allReview">All</label>
                                <input type="radio" class="btn-check" name="reviewStatus" id="reviewed" value="reviewed" v-model="localFilters.reviewStatus">
                                <label class="btn btn-outline-primary" for="reviewed">Reviewed</label>
                                <input type="radio" class="btn-check" name="reviewStatus" id="unreviewed" value="unreviewed" v-model="localFilters.reviewStatus">
                                <label class="btn btn-outline-primary" for="unreviewed">Unreviewed</label>
                            </div>
                        </div>
                        
                        <!-- Call Type (includes Begging) -->
                        <div class="mb-3">
                            <label class="form-label">Call Type</label>
                            <div class="d-flex flex-wrap" style="gap: 0.5rem;">
                                <div class="form-check form-switch">
                                    <input class="form-check-input" type="checkbox" id="rattleFilter" v-model="localFilters.callTypes.rattle">
                                    <label class="form-check-label" for="rattleFilter">Rattle</label>
                                </div>
                                <div class="form-check form-switch">
                                    <input class="form-check-input" type="checkbox" id="softsongFilter" v-model="localFilters.callTypes.softSong">
                                    <label class="form-check-label" for="softsongFilter">Softsong</label>
                                </div>
                                <div class="form-check form-switch">
                                    <input class="form-check-input" type="checkbox" id="mobFilter" v-model="localFilters.callTypes.mob">
                                    <label class="form-check-label" for="mobFilter">Mob</label>
                                </div>
                                <div class="form-check form-switch">
                                    <input class="form-check-input" type="checkbox" id="alertFilter" v-model="localFilters.callTypes.alert">
                                    <label class="form-check-label" for="alertFilter">Alert</label>
                                </div>
                                <div class="form-check form-switch">
                                    <input class="form-check-input" type="checkbox" id="beggingFilter" v-model="localFilters.callTypes.begging">
                                    <label class="form-check-label" for="beggingFilter">Begging</label>
                                </div>
                            </div>
                        </div>
                        
                        <!-- Crow Count -->
                        <div class="mb-3">
                            <label class="form-label">Crow Count</label>
                            <div class="btn-group w-100" role="group">
                                <input type="checkbox" class="btn-check" id="count0" v-model="localFilters.crowCounts[0]">
                                <label class="btn btn-outline-primary" for="count0">0</label>
                                <input type="checkbox" class="btn-check" id="count1" v-model="localFilters.crowCounts[1]">
                                <label class="btn btn-outline-primary" for="count1">1</label>
                                <input type="checkbox" class="btn-check" id="count2" v-model="localFilters.crowCounts[2]">
                                <label class="btn btn-outline-primary" for="count2">2</label>
                                <input type="checkbox" class="btn-check" id="count3" v-model="localFilters.crowCounts[3]">
                                <label class="btn btn-outline-primary" for="count3">3</label>
                                <input type="checkbox" class="btn-check" id="count4" v-model="localFilters.crowCounts[4]">
                                <label class="btn btn-outline-primary" for="count4">Crowd</label>
                            </div>
                        </div>
                        
                        <!-- Crow Age -->
                        <div class="mb-3">
                            <label class="form-label">Crow Age</label>
                            <div class="btn-group w-100" role="group">
                                <input type="radio" class="btn-check" name="ageFilter" id="allAge" value="all" v-model="localFilters.crowAge">
                                <label class="btn btn-outline-primary" for="allAge">All</label>
                                <input type="radio" class="btn-check" name="ageFilter" id="adult" value="adult" v-model="localFilters.crowAge">
                                <label class="btn btn-outline-primary" for="adult">Adult</label>
                                <input type="radio" class="btn-check" name="ageFilter" id="juvenile" value="juvenile" v-model="localFilters.crowAge">
                                <label class="btn btn-outline-primary" for="juvenile">Juvenile</label>
                            </div>
                        </div>
                        
                        <!-- Quality -->
                        <div class="mb-3">
                            <label class="form-label">Quality</label>
                            <div class="btn-group w-100" role="group">
                                <input type="radio" class="btn-check" name="quality" id="quality_all" value="all" v-model="localFilters.quality">
                                <label class="btn btn-outline-primary" for="quality_all">All</label>
                                <input type="radio" class="btn-check" name="quality" id="quality1" value="1" v-model="localFilters.quality">
                                <label class="btn btn-outline-primary" for="quality1">Bad</label>
                                <input type="radio" class="btn-check" name="quality" id="quality2" value="2" v-model="localFilters.quality">
                                <label class="btn btn-outline-primary" for="quality2">Average</label>
                                <input type="radio" class="btn-check" name="quality" id="quality3" value="3" v-model="localFilters.quality">
                                <label class="btn btn-outline-primary" for="quality3">Best</label>
                            </div>
                        </div>
                        
                        <!-- Global Text Filter: Media Notes, Author, File ID or Cluster -->
                        <div class="mb-3">
                            <label for="globalFilter" class="form-label">Media Notes, Author, File ID or Cluster</label>
                            <div class="input-group">
                                <input type="text" class="form-control" id="globalFilter" placeholder="Search..." v-model="localFilters.globalFilter">
                                <button class="btn btn-outline-secondary" type="button" @click="localFilters.globalFilter = ''">Clear</button>
                            </div>
                        </div>
                        
                    </div>
                    <div class="modal-footer">
                        <button type="button" class="btn btn-secondary" @click="resetFilters">Reset Filters</button>
                        <button type="button" class="btn btn-primary" @click="applyFilters" data-bs-dismiss="modal">Apply Filters</button>
                    </div>
                </div>
            </div>
        </div>
    </div>
    `,
    data() {
        return {
            localFilters: this.initLocalFilters(),
        };
    },
    computed: {
        hasActiveFilters() {
            return this.activeFilters && (
                this.activeFilters.labelStatus !== 'all' ||
                this.activeFilters.reviewStatus !== 'all' ||
                Object.values(this.activeFilters.callTypes).some(val => val) ||
                Object.values(this.activeFilters.crowCounts).some(val => val) ||
                this.activeFilters.crowAge !== 'all' ||
                this.activeFilters.quality !== 'all' ||
                (this.activeFilters.globalFilter && this.activeFilters.globalFilter.trim() !== '')
            );
        }
    },
    methods: {
        initLocalFilters() {
            return {
                labelStatus: this.filters?.labelStatus || 'all',
                reviewStatus: this.filters?.reviewStatus || 'all',
                callTypes: {
                    rattle: this.filters?.callTypes?.rattle || false,
                    softSong: this.filters?.callTypes?.softSong || false,
                    mob: this.filters?.callTypes?.mob || false,
                    alert: this.filters?.callTypes?.alert || false,
                    begging: this.filters?.callTypes?.begging || false
                },
                crowCounts: {
                    0: this.filters?.crowCounts?.[0] || false,
                    1: this.filters?.crowCounts?.[1] || false,
                    2: this.filters?.crowCounts?.[2] || false,
                    3: this.filters?.crowCounts?.[3] || false,
                    4: this.filters?.crowCounts?.[4] || false
                },
                crowAge: this.filters?.crowAge || 'all',
                quality: this.filters?.quality || 'all',
                globalFilter: this.filters?.globalFilter || ''
            };
        },
        openFilterModal() {
            // Modal is opened via Bootstrap data attributes
        },
        applyFilters() {
            this.$emit('filters-changed', this.localFilters);
        },
        resetFilters() {
            this.localFilters = {
                labelStatus: 'all',
                reviewStatus: 'all',
                callTypes: {
                    rattle: false,
                    softSong: false,
                    mob: false,
                    alert: false,
                    begging: false
                },
                crowCounts: {
                    0: false,
                    1: false,
                    2: false,
                    3: false,
                    4: false
                },
                crowAge: 'all',
                quality: 'all',
                globalFilter: ''
            };
            this.$emit('filters-changed', this.localFilters);
        }
    },
    watch: {
        filters: {
            handler(newFilters) {
                this.localFilters = this.initLocalFilters();
            },
            deep: true
        }
    }
};
==========================================
</labeler/js/components/filterComponent.js>
==========================================

==========================================
<labeler/js/components/menu.js>
Size: 1850 bytes | Lines:       51
==========================================
// js/components/menu.js
window.MenuComponent = {
  props: {
    title: { type: String, default: 'Crow Tools' }
  },
  template: `
    <nav class="navbar navbar-dark bg-dark mb-4">
      <div class="container-fluid">
        <!-- Brand acts as a refresh on the current page -->
        <a class="navbar-brand fs-2 d-flex align-items-center" href="#" @click.prevent="refreshPage">
          <img src="/images/crow-emote.png" width="40" height="40" class="d-inline-block align-text-top me-2" alt="">
          {{ title }}
        </a>
        <!-- Always-collapsed hamburger -->
        <button class="navbar-toggler" type="button" data-bs-toggle="collapse"
                data-bs-target="#mainNav" aria-controls="mainNav"
                aria-expanded="false" aria-label="Toggle navigation">
          <span class="navbar-toggler-icon"></span>
        </button>
        <div class="collapse navbar-collapse" id="mainNav">
          <ul class="navbar-nav ms-auto mb-2 mb-lg-0">
            <li class="nav-item">
              <a class="nav-link" :class="{ active: isActive('index.html') }" href="index.html">
                Labeler
              </a>
            </li>
            <li class="nav-item">
              <a class="nav-link" :class="{ active: isActive('transcriptions.html') }" href="transcriptions.html">
                Transcriber
              </a>
            </li>
            <li class="nav-item">
              <a class="nav-link" :class="{ active: isActive('embeddings.html') }" href="embeddings.html">
                Embeddings
              </a>
            </li>
          </ul>
        </div>
      </div>
    </nav>
  `,
  methods: {
    isActive(page) {
      const current = window.location.pathname.split('/').pop();
      return current === page;
    },
    refreshPage() {
      window.location.reload();
    }
  }
};

==========================================
</labeler/js/components/menu.js>
==========================================

==========================================
<labeler/js/components/pagination.js>
Size: 1067 bytes | Lines:       34
==========================================
// js/components/pagination.js
window.Pagination = {
    props: ['currentPage', 'totalPages'],
    data() {
        return {
            pageSelection: this.currentPage
        };
    },
    watch: {
        currentPage(newVal) {
            this.pageSelection = newVal;
        }
    },
    template: `
      <div class="mb-3 d-flex flex-wrap align-items-center text-nowrap" style="gap: 0.5rem; color: #fff;">
        <button class="btn btn-primary" @click="$emit('prev')" :disabled="currentPage === 1">
          Previous
        </button>
        <button class="btn btn-primary" @click="$emit('next')" :disabled="currentPage === totalPages">
          Next
        </button>
        <span>Page</span>
        <select class="form-select form-select-sm" style="width: auto;"
                v-model.number="pageSelection" @change="goToPage">
          <option v-for="p in totalPages" :value="p" :key="p">{{ p }}</option>
        </select>
      </div>
    `,
    methods: {
        goToPage() {
            this.$emit('goto', this.pageSelection);
        }
    }
};

==========================================
</labeler/js/components/pagination.js>
==========================================

==========================================
<labeler/js/components/segmentCard.js>
Size: 13330 bytes | Lines:      329
==========================================
window.SegmentCard = {
  props: ['segment', 'labels', 'playbackSpeed', 'prevLabels'],
  template: `
    <div :class="['card', 'h-100', 'mb-2', 'p-2', cardBorderClass, {'reviewed-card': currentLabels.reviewed}]" :style="[leftBorderStyle, cardBackgroundStyle, {position: 'relative'}]">
      <!-- NEW: Delete Button -->
      <button @click="deleteSegment" class="btn btn-sm delete-btn" style="position: absolute; top: 5px; right: 5px;">X</button>
      
      <!-- 1) Title Row -->
      <div class="d-flex flex-wrap align-items-center mb-1">
        <strong class="me-2">
          Recorder: {{ segment.recordist }} / ID: {{ segment.id }}
        </strong>
        <small class="text-muted">
          Time: {{ segment.start_time }} - {{ segment.end_time }},
          Conf: {{ segment.confidence.toFixed(2) }},
          Cluster: {{ segment.cluster }}
        </small>
      </div>

      <!-- 2) Media Notes -->
      <div class="mb-2" style="overflow: hidden; text-overflow: ellipsis; max-height: 3em; font-size: .85em;">
        <div class="text-muted" :title="segment.media_notes">{{ segment.media_notes || "N/A" }}</div>
      </div>

      <!-- 3) Play Button, Slider, and Copy Button -->
      <div class="d-flex align-items-center flex-grow-1 mb-2" style="gap: 0.5rem;">
        <button :class="['btn', 'btn-sm', playButtonClass]" style="width: 60px;" @click="handlePlay">
          <span v-if="!isPlaying">Play</span>
          <span v-else>Pause</span>
        </button>
        <input type="range" class="form-range"
               style="margin: 0;"
               :min="segment.start_time"
               :max="segment.end_time"
               :step="0.1"
               v-model.number="currentTime"
               @input="onSliderChange">
        <small class="text-nowrap">
          {{ formattedTime(currentTime - segment.start_time) }} /
          {{ formattedTime(segment.end_time - segment.start_time) }}
        </small>
        <button class="btn btn-sm btn-outline-secondary" style="width: 40px;" @click="copyPrevious" :disabled="!prevLabels">
          📋
        </button>
      </div>
      
      <!-- Top row: Crow Count, Crow Age, Behavior, and Quality -->
      <div class="d-flex flex-wrap align-items-center justify-content-evenly" style="gap: 0.5rem;">
        <!-- Crow Count Group -->
        <div class="btn-group btn-group-sm flex-fill mx-1" role="group">
           <input type="radio" class="btn-check"
                 :name="'crowCount' + segmentKey"
                 :id="'crowCount0_' + segmentKey"
                 :value="0"
                 v-model.number="currentLabels.crowCount"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowCount0_' + segmentKey" title="No Crows">0</label>
      
 
          <input type="radio" class="btn-check"
                 :name="'crowCount' + segmentKey"
                 :id="'crowCount1_' + segmentKey"
                 :value="1"
                 v-model.number="currentLabels.crowCount"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowCount1_' + segmentKey" title="1 Crow">1</label>
      
          <input type="radio" class="btn-check"
                 :name="'crowCount' + segmentKey"
                 :id="'crowCount2_' + segmentKey"
                 :value="2"
                 v-model.number="currentLabels.crowCount"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowCount2_' + segmentKey" title="2 Crows">2</label>
      
          <input type="radio" class="btn-check"
                 :name="'crowCount' + segmentKey"
                 :id="'crowCount3_' + segmentKey"
                 :value="3"
                 v-model.number="currentLabels.crowCount"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowCount3_' + segmentKey" title="3 Crows">3</label>
      
          <input type="radio" class="btn-check"
                 :name="'crowCount' + segmentKey"
                 :id="'crowCount4_' + segmentKey"
                 :value="4"
                 v-model.number="currentLabels.crowCount"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowCount4_' + segmentKey" title="Crowd">📣</label>
        </div>
      
        <!-- Crow Age Group -->
        <div class="btn-group btn-group-sm flex-fill mx-1" role="group">
          <input type="radio" class="btn-check"
                 :name="'crowAge' + segmentKey"
                 :id="'crowAgeAdult_' + segmentKey"
                 :value="1"
                 v-model.number="currentLabels.crowAge"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowAgeAdult_' + segmentKey" title="Adult"><img src="/images/crow-emote.png" style="width: 17px;"></label>
      
          <input type="radio" class="btn-check"
                 :name="'crowAge' + segmentKey"
                 :id="'crowAgeJuvenile_' + segmentKey"
                 :value="2"
                 v-model.number="currentLabels.crowAge"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'crowAgeJuvenile_' + segmentKey" title="Juvenile">🐣</label>
        </div>
      
        <!-- Behavior Group -->
        <div class="btn-group btn-group-sm flex-fill mx-1" role="group">
          <input type="checkbox" class="btn-check"
                 :id="'alert_' + segmentKey"
                 v-model="currentLabels.alert"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'alert_' + segmentKey" title="Alert">️🚨</label>
          
          <input type="checkbox" class="btn-check"
                 :id="'mob_' + segmentKey"
                 v-model="currentLabels.mob"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary btn-feature" :for="'mob_' + segmentKey" title="Mob / Attack">⚔️</label>
          
          <input type="checkbox" class="btn-check"
                 :id="'begging_' + segmentKey"
                 v-model="currentLabels.begging"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'begging_' + segmentKey" title="Food / Begging">🍇</label>

          <input type="checkbox" class="btn-check"
                 :id="'softSong_' + segmentKey"
                 v-model="currentLabels.softSong"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary btn-feature" :for="'softSong_' + segmentKey" title="Soft Song">🎵</label>

          <input type="checkbox" class="btn-check"
                 :id="'rattle_' + segmentKey"
                 v-model="currentLabels.rattle"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary btn-feature" :for="'rattle_' + segmentKey" title="Rattle">🪇</label>

        </div>
      
        <!-- Quality Group -->
        <div class="btn-group btn-group-sm flex-fill mx-1" role="group">
          <input type="radio" class="btn-check"
                 :name="'quality' + segmentKey"
                 :id="'quality1_' + segmentKey"
                 :value="1"
                 v-model.number="currentLabels.quality"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-danger btn-feature" :for="'quality1_' + segmentKey" title="Bad Quality">🚫</label>
      
          <input type="radio" class="btn-check"
                 :name="'quality' + segmentKey"
                 :id="'quality2_' + segmentKey"
                 :value="2"
                 v-model.number="currentLabels.quality"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-success" :for="'quality2_' + segmentKey" title="Average Quality">✅</label>
      
          <input type="radio" class="btn-check"
                 :name="'quality' + segmentKey"
                 :id="'quality3_' + segmentKey"
                 :value="3"
                 v-model.number="currentLabels.quality"
                 @change="onLabelsChanged">
          <label class="btn btn-outline-primary" :for="'quality3_' + segmentKey" title="Best Quality">⭐</label>
        </div>
      </div>
      
      <!-- Hidden audio element -->
      <audio ref="audio" style="display:none;"></audio>
    </div>
  `,
  data() {
    return {
      isPlaying: false,
      currentTime: this.segment.start_time,
      audio: null
    };
  },
  computed: {
    segmentKey() {
      return `${this.segment.id}-${this.segment.start_time}-${this.segment.end_time}`;
    },
    currentLabels() {
      // Otherwise use the labels object
      if (!this.labels[this.segmentKey]) {
        // Initialize with the new structure
        this.labels[this.segmentKey] = {
          crowCount: 1,
          crowAge: 1,
          scold: false,
          alert: false,
          begging: false,
          softSong: false,
          rattle: false,
          mob: false,
          quality: 2,
          reviewed: false
        };
      } else if (this.labels[this.segmentKey] && typeof this.labels[this.segmentKey].cluster === 'undefined') {
        // Ensure cluster property exists
        this.labels[this.segmentKey].cluster = 1;
      }
      return this.labels[this.segmentKey];
    },
    cardBorderClass() {
      // Show a primary (blue) border if the segment has been reviewed.
      return this.currentLabels.reviewed
          ? 'border-top border-right border-bottom border-primary'
          : 'border-top border-right border-bottom border-secondary';
    },
    playButtonClass() {
      return this.isPlaying ? 'btn-primary' : 'btn-secondary';
    },
    leftBorderStyle() {
      const colors = ["#007bff", "#28a745", "#ffc107", "#17a2b8", "#6610f2", "#fd7e14", "#6f42c1", "#20c997"];
      let fileId = String(this.segment.id);
      let hash = 0;
      for (let i = 0; i < fileId.length; i++) {
        hash = (hash * 31 + fileId.charCodeAt(i)) % colors.length;
      }
      return { borderLeft: `6px solid ${colors[hash]} !important` };
    },
    cardBackgroundStyle() {
      // Apply darker background for reviewed items
      if (this.currentLabels.reviewed) {
        return { backgroundColor: 'rgba(200, 200, 200, 0.9)' };
      }
      return {};
    }
  },
  methods: {
    formattedTime(seconds) {
      const minutes = Math.floor(seconds / 60);
      const secs = Math.floor(seconds % 60);
      return `${minutes}:${secs < 10 ? '0' : ''}${secs}`;
    },
    handlePlay() {
      // Mark segment as reviewed on play click.
      this.currentLabels.reviewed = true;
      this.onLabelsChanged();
      this.togglePlayback();
    },
    togglePlayback() {
      if (!this.audio) return;
      if (this.isPlaying) {
        this.audio.pause();
        this.isPlaying = false;
      } else {
        if (this.audio.currentTime >= this.segment.end_time) {
          this.audio.currentTime = this.segment.start_time;
          this.currentTime = this.segment.start_time;
        }
        this.audio.play();
        this.isPlaying = true;
      }
    },
    onSliderChange() {
      if (this.audio) {
        this.audio.currentTime = this.currentTime;
      }
    },
    onLabelsChanged() {
      // Whenever any label is changed, mark as reviewed and emit update.
      this.currentLabels.reviewed = true;
      this.$emit('labels-changed', { segmentKey: this.segmentKey, labels: this.currentLabels });

    },
    updateProgress() {
      if (this.audio) {
        this.currentTime = this.audio.currentTime;
        if (this.audio.currentTime >= this.segment.end_time) {
          this.audio.pause();
          this.audio.currentTime = this.segment.start_time;
          this.currentTime = this.segment.start_time;
          this.isPlaying = false;
        }
      }
    },
    copyPrevious() {
      if (this.prevLabels) {
        if (this.audio && this.isPlaying) {
          this.audio.pause();
          this.isPlaying = false;
        }
        // Copy all fields from prevLabels
        const fields = ['crowCount', 'crowAge', 'alert', 'begging', 'scold', 'softSong', 'rattle', 'mob', 'quality'];
        fields.forEach(field => {
          this.currentLabels[field] = this.prevLabels[field];
        });
        this.onLabelsChanged();
        this.audio.currentTime = this.segment.start_time;
        this.audio.play();
        this.isPlaying = true;
      }
    },
    deleteSegment() {
      if (confirm("Are you sure you want to delete this segment? This action cannot be undone.")) {
        this.$emit('segment-deleted', this.segmentKey);
      }
    }
  },
  mounted() {
    this.audio = this.$refs.audio;
    this.audio.src = `/cache/library/${this.segment.id}.mp3`;
    this.audio.currentTime = this.segment.start_time;
    this.audio.playbackRate = this.playbackSpeed;
    this.audio.addEventListener('timeupdate', this.updateProgress);
  },
  watch: {
    playbackSpeed(newSpeed) {
      if (this.audio) {
        this.audio.playbackRate = newSpeed;
      }
    }
  },
  beforeUnmount() {
    if (this.audio) {
      this.audio.pause();
      this.audio.removeEventListener('timeupdate', this.updateProgress);
      this.audio = null;
    }
  }
};

==========================================
</labeler/js/components/segmentCard.js>
==========================================

==========================================
<labeler/package-lock.json>
Size: 41916 bytes | Lines:     1120
==========================================
{
  "name": "crow-audio-labeler",
  "version": "1.0.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "crow-audio-labeler",
      "version": "1.0.0",
      "dependencies": {
        "bootstrap": "^5.3.0",
        "express": "^4.18.2",
        "vue": "^3.2.47"
      }
    },
    "node_modules/@babel/helper-string-parser": {
      "version": "7.25.9",
      "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.25.9.tgz",
      "integrity": "sha512-4A/SCr/2KLd5jrtOMFzaKjVtAei3+2r/NChoBNoZ3EyP/+GlhoaEGoWOZUmFmoITP7zOJyHIMm+DYRd8o3PvHA==",
      "engines": {
        "node": ">=6.9.0"
      }
    },
    "node_modules/@babel/helper-validator-identifier": {
      "version": "7.25.9",
      "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.25.9.tgz",
      "integrity": "sha512-Ed61U6XJc3CVRfkERJWDz4dJwKe7iLmmJsbOGu9wSloNSFttHV0I8g6UAgb7qnK5ly5bGLPd4oXZlxCdANBOWQ==",
      "engines": {
        "node": ">=6.9.0"
      }
    },
    "node_modules/@babel/parser": {
      "version": "7.26.9",
      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.26.9.tgz",
      "integrity": "sha512-81NWa1njQblgZbQHxWHpxxCzNsa3ZwvFqpUg7P+NNUU6f3UU2jBEg4OlF/J6rl8+PQGh1q6/zWScd001YwcA5A==",
      "dependencies": {
        "@babel/types": "^7.26.9"
      },
      "bin": {
        "parser": "bin/babel-parser.js"
      },
      "engines": {
        "node": ">=6.0.0"
      }
    },
    "node_modules/@babel/types": {
      "version": "7.26.9",
      "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.26.9.tgz",
      "integrity": "sha512-Y3IR1cRnOxOCDvMmNiym7XpXQ93iGDDPHx+Zj+NM+rg0fBaShfQLkg+hKPaZCEvg5N/LeCo4+Rj/i3FuJsIQaw==",
      "dependencies": {
        "@babel/helper-string-parser": "^7.25.9",
        "@babel/helper-validator-identifier": "^7.25.9"
      },
      "engines": {
        "node": ">=6.9.0"
      }
    },
    "node_modules/@jridgewell/sourcemap-codec": {
      "version": "1.5.0",
      "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.0.tgz",
      "integrity": "sha512-gv3ZRaISU3fjPAgNsriBRqGWQL6quFx04YMPW/zD8XMLsU32mhCCbfbO6KZFLjvYpCZ8zyDEgqsgf+PwPaM7GQ=="
    },
    "node_modules/@popperjs/core": {
      "version": "2.11.8",
      "resolved": "https://registry.npmjs.org/@popperjs/core/-/core-2.11.8.tgz",
      "integrity": "sha512-P1st0aksCrn9sGZhp8GMYwBnQsbvAWsZAX44oXNNvLHGqAOcoVxmjZiohstwQ7SqKnbR47akdNi+uleWD8+g6A==",
      "peer": true,
      "funding": {
        "type": "opencollective",
        "url": "https://opencollective.com/popperjs"
      }
    },
    "node_modules/@vue/compiler-core": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/compiler-core/-/compiler-core-3.5.13.tgz",
      "integrity": "sha512-oOdAkwqUfW1WqpwSYJce06wvt6HljgY3fGeM9NcVA1HaYOij3mZG9Rkysn0OHuyUAGMbEbARIpsG+LPVlBJ5/Q==",
      "dependencies": {
        "@babel/parser": "^7.25.3",
        "@vue/shared": "3.5.13",
        "entities": "^4.5.0",
        "estree-walker": "^2.0.2",
        "source-map-js": "^1.2.0"
      }
    },
    "node_modules/@vue/compiler-dom": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/compiler-dom/-/compiler-dom-3.5.13.tgz",
      "integrity": "sha512-ZOJ46sMOKUjO3e94wPdCzQ6P1Lx/vhp2RSvfaab88Ajexs0AHeV0uasYhi99WPaogmBlRHNRuly8xV75cNTMDA==",
      "dependencies": {
        "@vue/compiler-core": "3.5.13",
        "@vue/shared": "3.5.13"
      }
    },
    "node_modules/@vue/compiler-sfc": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/compiler-sfc/-/compiler-sfc-3.5.13.tgz",
      "integrity": "sha512-6VdaljMpD82w6c2749Zhf5T9u5uLBWKnVue6XWxprDobftnletJ8+oel7sexFfM3qIxNmVE7LSFGTpv6obNyaQ==",
      "dependencies": {
        "@babel/parser": "^7.25.3",
        "@vue/compiler-core": "3.5.13",
        "@vue/compiler-dom": "3.5.13",
        "@vue/compiler-ssr": "3.5.13",
        "@vue/shared": "3.5.13",
        "estree-walker": "^2.0.2",
        "magic-string": "^0.30.11",
        "postcss": "^8.4.48",
        "source-map-js": "^1.2.0"
      }
    },
    "node_modules/@vue/compiler-ssr": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/compiler-ssr/-/compiler-ssr-3.5.13.tgz",
      "integrity": "sha512-wMH6vrYHxQl/IybKJagqbquvxpWCuVYpoUJfCqFZwa/JY1GdATAQ+TgVtgrwwMZ0D07QhA99rs/EAAWfvG6KpA==",
      "dependencies": {
        "@vue/compiler-dom": "3.5.13",
        "@vue/shared": "3.5.13"
      }
    },
    "node_modules/@vue/reactivity": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/reactivity/-/reactivity-3.5.13.tgz",
      "integrity": "sha512-NaCwtw8o48B9I6L1zl2p41OHo/2Z4wqYGGIK1Khu5T7yxrn+ATOixn/Udn2m+6kZKB/J7cuT9DbWWhRxqixACg==",
      "dependencies": {
        "@vue/shared": "3.5.13"
      }
    },
    "node_modules/@vue/runtime-core": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/runtime-core/-/runtime-core-3.5.13.tgz",
      "integrity": "sha512-Fj4YRQ3Az0WTZw1sFe+QDb0aXCerigEpw418pw1HBUKFtnQHWzwojaukAs2X/c9DQz4MQ4bsXTGlcpGxU/RCIw==",
      "dependencies": {
        "@vue/reactivity": "3.5.13",
        "@vue/shared": "3.5.13"
      }
    },
    "node_modules/@vue/runtime-dom": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/runtime-dom/-/runtime-dom-3.5.13.tgz",
      "integrity": "sha512-dLaj94s93NYLqjLiyFzVs9X6dWhTdAlEAciC3Moq7gzAc13VJUdCnjjRurNM6uTLFATRHexHCTu/Xp3eW6yoog==",
      "dependencies": {
        "@vue/reactivity": "3.5.13",
        "@vue/runtime-core": "3.5.13",
        "@vue/shared": "3.5.13",
        "csstype": "^3.1.3"
      }
    },
    "node_modules/@vue/server-renderer": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/server-renderer/-/server-renderer-3.5.13.tgz",
      "integrity": "sha512-wAi4IRJV/2SAW3htkTlB+dHeRmpTiVIK1OGLWV1yeStVSebSQQOwGwIq0D3ZIoBj2C2qpgz5+vX9iEBkTdk5YA==",
      "dependencies": {
        "@vue/compiler-ssr": "3.5.13",
        "@vue/shared": "3.5.13"
      },
      "peerDependencies": {
        "vue": "3.5.13"
      }
    },
    "node_modules/@vue/shared": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/@vue/shared/-/shared-3.5.13.tgz",
      "integrity": "sha512-/hnE/qP5ZoGpol0a5mDi45bOd7t3tjYJBjsgCsivow7D48cJeV5l05RD82lPqi7gRiphZM37rnhW1l6ZoCNNnQ=="
    },
    "node_modules/accepts": {
      "version": "1.3.8",
      "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
      "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==",
      "dependencies": {
        "mime-types": "~2.1.34",
        "negotiator": "0.6.3"
      },
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/array-flatten": {
      "version": "1.1.1",
      "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",
      "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg=="
    },
    "node_modules/body-parser": {
      "version": "1.20.3",
      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.3.tgz",
      "integrity": "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g==",
      "dependencies": {
        "bytes": "3.1.2",
        "content-type": "~1.0.5",
        "debug": "2.6.9",
        "depd": "2.0.0",
        "destroy": "1.2.0",
        "http-errors": "2.0.0",
        "iconv-lite": "0.4.24",
        "on-finished": "2.4.1",
        "qs": "6.13.0",
        "raw-body": "2.5.2",
        "type-is": "~1.6.18",
        "unpipe": "1.0.0"
      },
      "engines": {
        "node": ">= 0.8",
        "npm": "1.2.8000 || >= 1.4.16"
      }
    },
    "node_modules/body-parser/node_modules/debug": {
      "version": "2.6.9",
      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
      "dependencies": {
        "ms": "2.0.0"
      }
    },
    "node_modules/body-parser/node_modules/iconv-lite": {
      "version": "0.4.24",
      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
      "dependencies": {
        "safer-buffer": ">= 2.1.2 < 3"
      },
      "engines": {
        "node": ">=0.10.0"
      }
    },
    "node_modules/body-parser/node_modules/ms": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
      "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
    },
    "node_modules/body-parser/node_modules/qs": {
      "version": "6.13.0",
      "resolved": "https://registry.npmjs.org/qs/-/qs-6.13.0.tgz",
      "integrity": "sha512-+38qI9SOr8tfZ4QmJNplMUxqjbe7LKvvZgWdExBOmd+egZTtjLB67Gu0HRX3u/XOq7UU2Nx6nsjvS16Z9uwfpg==",
      "dependencies": {
        "side-channel": "^1.0.6"
      },
      "engines": {
        "node": ">=0.6"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/bootstrap": {
      "version": "5.3.3",
      "resolved": "https://registry.npmjs.org/bootstrap/-/bootstrap-5.3.3.tgz",
      "integrity": "sha512-8HLCdWgyoMguSO9o+aH+iuZ+aht+mzW0u3HIMzVu7Srrpv7EBBxTnrFlSCskwdY1+EOFQSm7uMJhNQHkdPcmjg==",
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/twbs"
        },
        {
          "type": "opencollective",
          "url": "https://opencollective.com/bootstrap"
        }
      ],
      "peerDependencies": {
        "@popperjs/core": "^2.11.8"
      }
    },
    "node_modules/bytes": {
      "version": "3.1.2",
      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
      "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/call-bind-apply-helpers": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
      "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
      "dependencies": {
        "es-errors": "^1.3.0",
        "function-bind": "^1.1.2"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/call-bound": {
      "version": "1.0.4",
      "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz",
      "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==",
      "dependencies": {
        "call-bind-apply-helpers": "^1.0.2",
        "get-intrinsic": "^1.3.0"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/content-disposition": {
      "version": "0.5.4",
      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
      "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==",
      "dependencies": {
        "safe-buffer": "5.2.1"
      },
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/content-disposition/node_modules/safe-buffer": {
      "version": "5.2.1",
      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
      "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ]
    },
    "node_modules/content-type": {
      "version": "1.0.5",
      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
      "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/cookie": {
      "version": "0.7.1",
      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.1.tgz",
      "integrity": "sha512-6DnInpx7SJ2AK3+CTUE/ZM0vWTUboZCegxhC2xiIydHR9jNuTAASBrfEpHhiGOZw/nX51bHt6YQl8jsGo4y/0w==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/cookie-signature": {
      "version": "1.0.6",
      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz",
      "integrity": "sha512-QADzlaHc8icV8I7vbaJXJwod9HWYp8uCqf1xa4OfNu1T7JVxQIrUgOWtHdNDtPiywmFbiS12VjotIXLrKM3orQ=="
    },
    "node_modules/csstype": {
      "version": "3.1.3",
      "resolved": "https://registry.npmjs.org/csstype/-/csstype-3.1.3.tgz",
      "integrity": "sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw=="
    },
    "node_modules/depd": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
      "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/destroy": {
      "version": "1.2.0",
      "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz",
      "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==",
      "engines": {
        "node": ">= 0.8",
        "npm": "1.2.8000 || >= 1.4.16"
      }
    },
    "node_modules/dunder-proto": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
      "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
      "dependencies": {
        "call-bind-apply-helpers": "^1.0.1",
        "es-errors": "^1.3.0",
        "gopd": "^1.2.0"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/ee-first": {
      "version": "1.1.1",
      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
      "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
    },
    "node_modules/encodeurl": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
      "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/entities": {
      "version": "4.5.0",
      "resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz",
      "integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==",
      "engines": {
        "node": ">=0.12"
      },
      "funding": {
        "url": "https://github.com/fb55/entities?sponsor=1"
      }
    },
    "node_modules/es-define-property": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
      "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/es-errors": {
      "version": "1.3.0",
      "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
      "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/es-object-atoms": {
      "version": "1.1.1",
      "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz",
      "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==",
      "dependencies": {
        "es-errors": "^1.3.0"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/escape-html": {
      "version": "1.0.3",
      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
      "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="
    },
    "node_modules/estree-walker": {
      "version": "2.0.2",
      "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-2.0.2.tgz",
      "integrity": "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w=="
    },
    "node_modules/etag": {
      "version": "1.8.1",
      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
      "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/express": {
      "version": "4.21.2",
      "resolved": "https://registry.npmjs.org/express/-/express-4.21.2.tgz",
      "integrity": "sha512-28HqgMZAmih1Czt9ny7qr6ek2qddF4FclbMzwhCREB6OFfH+rXAnuNCwo1/wFvrtbgsQDb4kSbX9de9lFbrXnA==",
      "dependencies": {
        "accepts": "~1.3.8",
        "array-flatten": "1.1.1",
        "body-parser": "1.20.3",
        "content-disposition": "0.5.4",
        "content-type": "~1.0.4",
        "cookie": "0.7.1",
        "cookie-signature": "1.0.6",
        "debug": "2.6.9",
        "depd": "2.0.0",
        "encodeurl": "~2.0.0",
        "escape-html": "~1.0.3",
        "etag": "~1.8.1",
        "finalhandler": "1.3.1",
        "fresh": "0.5.2",
        "http-errors": "2.0.0",
        "merge-descriptors": "1.0.3",
        "methods": "~1.1.2",
        "on-finished": "2.4.1",
        "parseurl": "~1.3.3",
        "path-to-regexp": "0.1.12",
        "proxy-addr": "~2.0.7",
        "qs": "6.13.0",
        "range-parser": "~1.2.1",
        "safe-buffer": "5.2.1",
        "send": "0.19.0",
        "serve-static": "1.16.2",
        "setprototypeof": "1.2.0",
        "statuses": "2.0.1",
        "type-is": "~1.6.18",
        "utils-merge": "1.0.1",
        "vary": "~1.1.2"
      },
      "engines": {
        "node": ">= 0.10.0"
      },
      "funding": {
        "type": "opencollective",
        "url": "https://opencollective.com/express"
      }
    },
    "node_modules/express/node_modules/debug": {
      "version": "2.6.9",
      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
      "dependencies": {
        "ms": "2.0.0"
      }
    },
    "node_modules/express/node_modules/ms": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
      "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
    },
    "node_modules/express/node_modules/qs": {
      "version": "6.13.0",
      "resolved": "https://registry.npmjs.org/qs/-/qs-6.13.0.tgz",
      "integrity": "sha512-+38qI9SOr8tfZ4QmJNplMUxqjbe7LKvvZgWdExBOmd+egZTtjLB67Gu0HRX3u/XOq7UU2Nx6nsjvS16Z9uwfpg==",
      "dependencies": {
        "side-channel": "^1.0.6"
      },
      "engines": {
        "node": ">=0.6"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/express/node_modules/safe-buffer": {
      "version": "5.2.1",
      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
      "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/feross"
        },
        {
          "type": "patreon",
          "url": "https://www.patreon.com/feross"
        },
        {
          "type": "consulting",
          "url": "https://feross.org/support"
        }
      ]
    },
    "node_modules/finalhandler": {
      "version": "1.3.1",
      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.1.tgz",
      "integrity": "sha512-6BN9trH7bp3qvnrRyzsBz+g3lZxTNZTbVO2EV1CS0WIcDbawYVdYvGflME/9QP0h0pYlCDBCTjYa9nZzMDpyxQ==",
      "dependencies": {
        "debug": "2.6.9",
        "encodeurl": "~2.0.0",
        "escape-html": "~1.0.3",
        "on-finished": "2.4.1",
        "parseurl": "~1.3.3",
        "statuses": "2.0.1",
        "unpipe": "~1.0.0"
      },
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/finalhandler/node_modules/debug": {
      "version": "2.6.9",
      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
      "dependencies": {
        "ms": "2.0.0"
      }
    },
    "node_modules/finalhandler/node_modules/ms": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
      "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
    },
    "node_modules/forwarded": {
      "version": "0.2.0",
      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
      "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/fresh": {
      "version": "0.5.2",
      "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
      "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/function-bind": {
      "version": "1.1.2",
      "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
      "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/get-intrinsic": {
      "version": "1.3.0",
      "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
      "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
      "dependencies": {
        "call-bind-apply-helpers": "^1.0.2",
        "es-define-property": "^1.0.1",
        "es-errors": "^1.3.0",
        "es-object-atoms": "^1.1.1",
        "function-bind": "^1.1.2",
        "get-proto": "^1.0.1",
        "gopd": "^1.2.0",
        "has-symbols": "^1.1.0",
        "hasown": "^2.0.2",
        "math-intrinsics": "^1.1.0"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/get-proto": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
      "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
      "dependencies": {
        "dunder-proto": "^1.0.1",
        "es-object-atoms": "^1.0.0"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/gopd": {
      "version": "1.2.0",
      "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
      "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/has-symbols": {
      "version": "1.1.0",
      "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
      "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/hasown": {
      "version": "2.0.2",
      "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
      "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
      "dependencies": {
        "function-bind": "^1.1.2"
      },
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/http-errors": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.0.tgz",
      "integrity": "sha512-FtwrG/euBzaEjYeRqOgly7G0qviiXoJWnvEH2Z1plBdXgbyjv34pHTSb9zoeHMyDy33+DWy5Wt9Wo+TURtOYSQ==",
      "dependencies": {
        "depd": "2.0.0",
        "inherits": "2.0.4",
        "setprototypeof": "1.2.0",
        "statuses": "2.0.1",
        "toidentifier": "1.0.1"
      },
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/inherits": {
      "version": "2.0.4",
      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
      "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
    },
    "node_modules/ipaddr.js": {
      "version": "1.9.1",
      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
      "engines": {
        "node": ">= 0.10"
      }
    },
    "node_modules/magic-string": {
      "version": "0.30.17",
      "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.17.tgz",
      "integrity": "sha512-sNPKHvyjVf7gyjwS4xGTaW/mCnF8wnjtifKBEhxfZ7E/S8tQ0rssrwGNn6q8JH/ohItJfSQp9mBtQYuTlH5QnA==",
      "dependencies": {
        "@jridgewell/sourcemap-codec": "^1.5.0"
      }
    },
    "node_modules/math-intrinsics": {
      "version": "1.1.0",
      "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
      "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
      "engines": {
        "node": ">= 0.4"
      }
    },
    "node_modules/media-typer": {
      "version": "0.3.0",
      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
      "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/merge-descriptors": {
      "version": "1.0.3",
      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz",
      "integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==",
      "funding": {
        "url": "https://github.com/sponsors/sindresorhus"
      }
    },
    "node_modules/methods": {
      "version": "1.1.2",
      "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
      "integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/mime": {
      "version": "1.6.0",
      "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
      "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
      "bin": {
        "mime": "cli.js"
      },
      "engines": {
        "node": ">=4"
      }
    },
    "node_modules/mime-db": {
      "version": "1.52.0",
      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
      "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/mime-types": {
      "version": "2.1.35",
      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
      "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
      "dependencies": {
        "mime-db": "1.52.0"
      },
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/ms": {
      "version": "2.1.3",
      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
      "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
    },
    "node_modules/nanoid": {
      "version": "3.3.8",
      "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.8.tgz",
      "integrity": "sha512-WNLf5Sd8oZxOm+TzppcYk8gVOgP+l58xNy58D0nbUnOxOWRWvlcCV4kUF7ltmI6PsrLl/BgKEyS4mqsGChFN0w==",
      "funding": [
        {
          "type": "github",
          "url": "https://github.com/sponsors/ai"
        }
      ],
      "bin": {
        "nanoid": "bin/nanoid.cjs"
      },
      "engines": {
        "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
      }
    },
    "node_modules/negotiator": {
      "version": "0.6.3",
      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",
      "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/object-inspect": {
      "version": "1.13.4",
      "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
      "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/on-finished": {
      "version": "2.4.1",
      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
      "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
      "dependencies": {
        "ee-first": "1.1.1"
      },
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/parseurl": {
      "version": "1.3.3",
      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/path-to-regexp": {
      "version": "0.1.12",
      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.12.tgz",
      "integrity": "sha512-RA1GjUVMnvYFxuqovrEqZoxxW5NUZqbwKtYz/Tt7nXerk0LbLblQmrsgdeOxV5SFHf0UDggjS/bSeOZwt1pmEQ=="
    },
    "node_modules/picocolors": {
      "version": "1.1.1",
      "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
      "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="
    },
    "node_modules/postcss": {
      "version": "8.5.3",
      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.3.tgz",
      "integrity": "sha512-dle9A3yYxlBSrt8Fu+IpjGT8SY8hN0mlaA6GY8t0P5PjIOZemULz/E2Bnm/2dcUOena75OTNkHI76uZBNUUq3A==",
      "funding": [
        {
          "type": "opencollective",
          "url": "https://opencollective.com/postcss/"
        },
        {
          "type": "tidelift",
          "url": "https://tidelift.com/funding/github/npm/postcss"
        },
        {
          "type": "github",
          "url": "https://github.com/sponsors/ai"
        }
      ],
      "dependencies": {
        "nanoid": "^3.3.8",
        "picocolors": "^1.1.1",
        "source-map-js": "^1.2.1"
      },
      "engines": {
        "node": "^10 || ^12 || >=14"
      }
    },
    "node_modules/proxy-addr": {
      "version": "2.0.7",
      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
      "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
      "dependencies": {
        "forwarded": "0.2.0",
        "ipaddr.js": "1.9.1"
      },
      "engines": {
        "node": ">= 0.10"
      }
    },
    "node_modules/range-parser": {
      "version": "1.2.1",
      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
      "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/raw-body": {
      "version": "2.5.2",
      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.2.tgz",
      "integrity": "sha512-8zGqypfENjCIqGhgXToC8aB2r7YrBX+AQAfIPs/Mlk+BtPTztOvTS01NRW/3Eh60J+a48lt8qsCzirQ6loCVfA==",
      "dependencies": {
        "bytes": "3.1.2",
        "http-errors": "2.0.0",
        "iconv-lite": "0.4.24",
        "unpipe": "1.0.0"
      },
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/raw-body/node_modules/iconv-lite": {
      "version": "0.4.24",
      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
      "dependencies": {
        "safer-buffer": ">= 2.1.2 < 3"
      },
      "engines": {
        "node": ">=0.10.0"
      }
    },
    "node_modules/safer-buffer": {
      "version": "2.1.2",
      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
    },
    "node_modules/send": {
      "version": "0.19.0",
      "resolved": "https://registry.npmjs.org/send/-/send-0.19.0.tgz",
      "integrity": "sha512-dW41u5VfLXu8SJh5bwRmyYUbAoSB3c9uQh6L8h/KtsFREPWpbX1lrljJo186Jc4nmci/sGUZ9a0a0J2zgfq2hw==",
      "dependencies": {
        "debug": "2.6.9",
        "depd": "2.0.0",
        "destroy": "1.2.0",
        "encodeurl": "~1.0.2",
        "escape-html": "~1.0.3",
        "etag": "~1.8.1",
        "fresh": "0.5.2",
        "http-errors": "2.0.0",
        "mime": "1.6.0",
        "ms": "2.1.3",
        "on-finished": "2.4.1",
        "range-parser": "~1.2.1",
        "statuses": "2.0.1"
      },
      "engines": {
        "node": ">= 0.8.0"
      }
    },
    "node_modules/send/node_modules/debug": {
      "version": "2.6.9",
      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
      "dependencies": {
        "ms": "2.0.0"
      }
    },
    "node_modules/send/node_modules/debug/node_modules/ms": {
      "version": "2.0.0",
      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
      "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
    },
    "node_modules/send/node_modules/encodeurl": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
      "integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/serve-static": {
      "version": "1.16.2",
      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.2.tgz",
      "integrity": "sha512-VqpjJZKadQB/PEbEwvFdO43Ax5dFBZ2UECszz8bQ7pi7wt//PWe1P6MN7eCnjsatYtBT6EuiClbjSWP2WrIoTw==",
      "dependencies": {
        "encodeurl": "~2.0.0",
        "escape-html": "~1.0.3",
        "parseurl": "~1.3.3",
        "send": "0.19.0"
      },
      "engines": {
        "node": ">= 0.8.0"
      }
    },
    "node_modules/setprototypeof": {
      "version": "1.2.0",
      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
      "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="
    },
    "node_modules/side-channel": {
      "version": "1.1.0",
      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
      "integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==",
      "dependencies": {
        "es-errors": "^1.3.0",
        "object-inspect": "^1.13.3",
        "side-channel-list": "^1.0.0",
        "side-channel-map": "^1.0.1",
        "side-channel-weakmap": "^1.0.2"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/side-channel-list": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.0.tgz",
      "integrity": "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA==",
      "dependencies": {
        "es-errors": "^1.3.0",
        "object-inspect": "^1.13.3"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/side-channel-map": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
      "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
      "dependencies": {
        "call-bound": "^1.0.2",
        "es-errors": "^1.3.0",
        "get-intrinsic": "^1.2.5",
        "object-inspect": "^1.13.3"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/side-channel-weakmap": {
      "version": "1.0.2",
      "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
      "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
      "dependencies": {
        "call-bound": "^1.0.2",
        "es-errors": "^1.3.0",
        "get-intrinsic": "^1.2.5",
        "object-inspect": "^1.13.3",
        "side-channel-map": "^1.0.1"
      },
      "engines": {
        "node": ">= 0.4"
      },
      "funding": {
        "url": "https://github.com/sponsors/ljharb"
      }
    },
    "node_modules/source-map-js": {
      "version": "1.2.1",
      "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
      "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
      "engines": {
        "node": ">=0.10.0"
      }
    },
    "node_modules/statuses": {
      "version": "2.0.1",
      "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz",
      "integrity": "sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/toidentifier": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
      "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
      "engines": {
        "node": ">=0.6"
      }
    },
    "node_modules/type-is": {
      "version": "1.6.18",
      "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
      "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
      "dependencies": {
        "media-typer": "0.3.0",
        "mime-types": "~2.1.24"
      },
      "engines": {
        "node": ">= 0.6"
      }
    },
    "node_modules/unpipe": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
      "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/utils-merge": {
      "version": "1.0.1",
      "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
      "integrity": "sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==",
      "engines": {
        "node": ">= 0.4.0"
      }
    },
    "node_modules/vary": {
      "version": "1.1.2",
      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
      "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",
      "engines": {
        "node": ">= 0.8"
      }
    },
    "node_modules/vue": {
      "version": "3.5.13",
      "resolved": "https://registry.npmjs.org/vue/-/vue-3.5.13.tgz",
      "integrity": "sha512-wmeiSMxkZCSc+PM2w2VRsOYAZC8GdipNFRTsLSfodVqI9mbejKeXEGr8SckuLnrQPGe3oJN5c3K0vpoU9q/wCQ==",
      "dependencies": {
        "@vue/compiler-dom": "3.5.13",
        "@vue/compiler-sfc": "3.5.13",
        "@vue/runtime-dom": "3.5.13",
        "@vue/server-renderer": "3.5.13",
        "@vue/shared": "3.5.13"
      },
      "peerDependencies": {
        "typescript": "*"
      },
      "peerDependenciesMeta": {
        "typescript": {
          "optional": true
        }
      }
    }
  }
}

==========================================
</labeler/package-lock.json>
==========================================

==========================================
<labeler/package.json>
Size: 299 bytes | Lines:       14
==========================================
{
  "name": "crow-audio-labeler",
  "version": "1.0.0",
  "description": "Crow Audio Labeler with Vue 3, Node, and Express",
  "main": "index.html",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.18.2",
    "vue": "^3.2.47",
    "bootstrap": "^5.3.0"
  }
}

==========================================
</labeler/package.json>
==========================================

==========================================
<labeler/server.js>
Size: 6082 bytes | Lines:      179
==========================================
// server.js
const express = require('express');
const fs = require('fs');
const path = require('path');

const app = express();
app.use(express.json({
    limit: '200mb',
}));

// Serve static files from "public" and the project root
app.use('/cache', express.static(path.join(__dirname, '../.cache')));
app.use('/images', express.static(path.join(__dirname, '../docs/images')));
app.use(express.static('.'));

// Endpoint to update labels.json on disk
app.post('/updateLabels', (req, res) => {
  const update = req.body; // Expecting: { segmentKey, labels }
  const labelsFile = path.join(__dirname, '../', '.cache', 'cluster_labels.json');
  let allLabels = {};
  try {
    if (fs.existsSync(labelsFile)) {
      const data = fs.readFileSync(labelsFile, 'utf8');
      allLabels = JSON.parse(data);
    }
  } catch (e) {
    console.error("Error reading existing labels json:", e);
  }
  allLabels[update.segmentKey] = update.labels;
  try {
    fs.writeFileSync(labelsFile, JSON.stringify(allLabels, null, 4));
    res.json({ success: true });
  } catch (e) {
    console.error("Error writing labels json:", e);
    res.status(500).json({ success: false, error: 'Failed to update labels.' });
  }
});

// Add below your updateLabels endpoint in server.js
app.post('/updateNotationLabels', (req, res) => {
  // Expecting: { fileId: 'xxx', notations: { <timestamp>: "transcription text", ... } }
  const update = req.body;
  const notationFile = path.join(__dirname, '../', '.cache', 'notation_labels.json');
  let allNotations = {};
  try {
    if (fs.existsSync(notationFile)) {
      const data = fs.readFileSync(notationFile, 'utf8');
      allNotations = JSON.parse(data);
    }
  } catch (e) {
    console.error("Error reading existing notation labels:", e);
  }

  // Update the notation for the file
  allNotations[update.fileId] = update.notations;

  try {
    fs.writeFileSync(notationFile, JSON.stringify(allNotations, null, 4));
    res.json({ success: true });
  } catch (e) {
    console.error("Error writing notation labels:", e);
    res.status(500).json({ success: false, error: 'Failed to update notations.' });
  }
});

app.get('/getNotationLabels', (req, res) => {
  const fileId = req.query.fileId;
  const notationFile = path.join(__dirname, '../', '.cache', 'notation_labels.json');
  let allNotations = {};
  try {
    if (fs.existsSync(notationFile)) {
      const data = fs.readFileSync(notationFile, 'utf8');
      allNotations = JSON.parse(data);
    }
  } catch (e) {
    console.error("Error reading notation labels:", e);
    return res.status(500).json({ success: false, error: 'Error reading notation labels.' });
  }
  res.json({ success: true, notations: allNotations[fileId] || {} });
});

app.get('/api/embeddings', (req, res) => {
    const embeddingsFile = path.join(__dirname, '../', '.cache', 'embeddings-3d.json');
    if (fs.existsSync(embeddingsFile)) {
        fs.readFile(embeddingsFile, 'utf8', (err, data) => {
            if (err) {
                console.error("Error reading embeddings file:", err);
                return res.status(500).json({ success: false, error: err.message });
            }
            try {
                const embeddings = JSON.parse(data);
                res.json({ success: true, embeddings });
            } catch (parseErr) {
                console.error("Error parsing embeddings JSON:", parseErr);
                res.status(500).json({ success: false, error: parseErr.message });
            }
        });
    } else {
        res.status(404).json({ success: false, error: "Embeddings file not found" });
    }
});

// Endpoint to delete a segment and its label
app.delete('/deleteSegment', (req, res) => {
  const { segmentKey } = req.body;

  // ----- Remove from labels file -----
  const labelsFile = path.join(__dirname, '../', '.cache', 'cluster_labels.json');
  let allLabels = {};
  try {
    if (fs.existsSync(labelsFile)) {
      const data = fs.readFileSync(labelsFile, 'utf8');
      allLabels = JSON.parse(data);
    }
  } catch(e) {
    console.error("Error reading labels json for deletion:", e);
  }
  if (allLabels[segmentKey]) {
    delete allLabels[segmentKey];
  }
  try {
    fs.writeFileSync(labelsFile, JSON.stringify(allLabels, null, 4));
  } catch(e) {
    console.error("Error writing labels json for deletion:", e);
    return res.status(500).json({ success: false, error: 'Failed to delete segment label.' });
  }

  // ----- Remove from segments file -----
  const segmentsFile = path.join(__dirname, '../', '.cache', 'cluster_segments.json');
  let allSegments = {};
  try {
    if (fs.existsSync(segmentsFile)) {
      const data = fs.readFileSync(segmentsFile, 'utf8');
      allSegments = JSON.parse(data);
    }
  } catch(e) {
    console.error("Error reading segments json for deletion:", e);
  }

  // Expect segmentKey to be "id-start_time-end_time"
  const parts = segmentKey.split('-');
  const segId = parts[0];
  const segStart = Number(parts[1]);
  const segEnd = Number(parts[2]);

  if (allSegments[segId]) {
    // Filter out the segment from the array for that file id.
    allSegments[segId] = allSegments[segId].filter(seg => {
      return !(Number(seg.start_time) === segStart && Number(seg.end_time) === segEnd);
    });

    // If no segments remain for this file id, remove the key entirely.
    if (allSegments[segId].length === 0) {
      delete allSegments[segId];
    }
  }

  try {
    fs.writeFileSync(segmentsFile, JSON.stringify(allSegments, null, 4));
  } catch(e) {
    console.error("Error writing segments json for deletion:", e);
    return res.status(500).json({ success: false, error: 'Failed to delete segment from segments file.' });
  }

  res.json({ success: true });
});

const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
    console.log(`Server listening on port ${PORT}`);
});

// Generic error-handling middleware
app.use((err, req, res, next) => {
    console.error("Error occurred:", err);
    res.status(err.status || 500).json({
        error: err.message || "Internal Server Error"
    });
});

==========================================
</labeler/server.js>
==========================================

==========================================
<labeler/transcriptions.html>
Size: 31073 bytes | Lines:      773
==========================================
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Crow Transcriber</title>
  <!-- Bootstrap Dark Theme -->
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootswatch@5.3.0/dist/darkly/bootstrap.min.css">
  <!-- Optional Bootstrap Icons -->
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.5/font/bootstrap-icons.css">
  <link rel="icon" type="image/png" href="/images/crow-emote.png">
  <style>
    body {
      padding: 20px;
    }
    /* Waveform container (always in DOM) */
    #waveform {
      height: 80px;
      width: 100%;
      margin-bottom: 15px;
      position: relative;
      cursor: pointer;
    }
    /* Spectrogram container: its parent defines a fixed height */
    #spectrogram {
      width: 100%;
      height: 100%;
      border: 1px solid #444;
      background-color: #222;
    }
    /* Parent container for spectrogram */
    #spectrogram-container {
      width: 100%;
      height: 200px;
      margin-bottom: 15px;
      position: relative;
      cursor: pointer;
    }
    /* Transcription track container – in its own row */
    #transcription-track {
      width: 100%;
      height: 37px;
      border: 1px solid #444;
      background-color: #222;
      position: relative;
      cursor: pointer;
    }
    /* Playhead style – vertical line */
    .playhead {
      position: absolute;
      top: 0;
      bottom: 0;
      width: 3px;
      background-color: red;
      pointer-events: none;
      z-index: 100;
    }
    /* Notation entry styling */
    .notation-entry {
      position: absolute;
      top: 0;
      height: 25px;
      padding: 2px 4px 2px 12px;
      border: 2px solid #666;
      border-radius: 3px;
      background-color: #555;
      color: #fff;
      white-space: nowrap;
      user-select: none;
      font-size: 0.8em;
    }
    /* Extend left edge marker upward above the spectrogram */
    .notation-entry::before {
      content: "";
      position: absolute;
      top: -360px;
      left: 0;
      width: 1.5px;
      height: 383px;
      background-color: gold;
      z-index: 100;
    }
    /* Input styling for labels */
    .notation-entry input {
      border: none;
      background: transparent;
      color: #fff;
      width: 50px;
      font-size: 0.8em;
      padding: 2px;
    }
    /* Token styles */
    .token-valid {
      padding: 0 2px;
      border-radius: 2px;
      margin-right: 1px;
    }
    .token-invalid {
      background-color: darkred;
      text-decoration: line-through;
      padding: 0 2px;
      border-radius: 2px;
      margin-right: 1px;
    }
    /* Drag handle: only the left 10px is draggable */
    .drag-handle {
      position: absolute;
      top: 0;
      left: 0;
      width: 10px;
      height: 100%;
      cursor: move;
      background: rgba(255, 255, 255, 0.1);
      z-index: 10;
    }
    /* Context menu styling */
    .context-menu {
      position: fixed;
      background-color: #444;
      color: #fff;
      padding: 5px 10px;
      border-radius: 5px;
      z-index: 200;
    }
  </style>
</head>
<body class="bg-dark text-light">
  <div id="app" class="container">
    <menu-component title="Crow Transcriber"></menu-component>

    <!-- File ID Input Row -->
    <div class="row mb-3">
      <div class="col">
        <label for="fileIdInput" class="form-label">Enter File ID:</label>
        <div class="input-group">
          <input type="text" id="fileIdInput" class="form-control" v-model="fileId" placeholder="e.g. 12345">
          <button class="btn btn-primary" @click="initAudio">Load Audio</button>
        </div>
      </div>
    </div>
    <!-- Zoom & Navigation Controls Row (visible when audio is loaded) -->
    <div class="row mb-3" v-show="audioLoaded">
      <div class="col-12 col-md-6 d-flex align-items-center">
        <label for="zoomSlider" class="form-label me-2">Zoom:</label>
        <input type="range" id="zoomSlider" class="form-range" v-model.number="chunkDuration" min="3" max="30" step="1" @input="updateChunk">
        <span class="ms-2 text-nowrap">{{ chunkDuration }} seconds</span>
      </div>

      <div class="col-12 col-md-2 d-flex justify-content-end align-items-end">
        <!-- Vocabulary Modal Trigger Button -->
        <button class="btn btn-info ms-2" @click="openVocabModal">Vocabulary</button>
      </div>

      <div class="col-12 col-md-4 d-flex justify-content-end align-items-end">
        <!-- Previous Chunk button -->
        <button class="btn btn-secondary me-2" @click="prevChunk" :disabled="currentChunkStart === 0">Previous Chunk</button>
        <!-- Dropdown for chunk selection -->
        <select class="form-select me-2" style="width: auto;" v-model.number="currentChunkStart" @change="chunkSelectChanged">
          <option v-for="chunk in chunkOptions" :key="chunk" :value="chunk">
            {{ formatTimestamp(chunk) }}
          </option>
        </select>

        <!-- Next Chunk button -->
        <button class="btn btn-secondary" @click="nextChunk" :disabled="currentChunkStart + chunkDuration >= fileDuration">Next Chunk</button>
      </div>
    </div>
    <!-- Waveform Row -->
    <div class="row mb-3" v-show="audioLoading">
      <div class="col">
        <div id="waveform" @click="jumpToTime" @mousedown="startPlayheadDrag">
          <div id="playhead-waveform" class="playhead"></div>
        </div>
      </div>
    </div>
    <!-- Spectrogram Row -->
    <div class="row mb-3" v-show="audioLoading">
      <div class="col">
        <div id="spectrogram-container" @click="jumpToTime">
          <div id="spectrogram"></div>
          <div id="playhead-spectrogram" class="playhead"></div>
        </div>
      </div>
    </div>
    <!-- Transcription Track Row -->
    <div class="row mb-3" v-show="audioLoaded">
      <div class="col">
        <div id="transcription-track" @click="trackClick($event)">
          <!-- Display computed visible notations with computed positions and timestamp -->
          <div v-for="item in visibleNotations" :key="item.fullIndex"
               class="notation-entry"
               :style="{ left: item.computedPosition + 'px', top: item.entry.top + 'px' }"
               @contextmenu.prevent="showContextMenu(item.fullIndex, $event)">
            <div class="drag-handle" @mousedown.prevent="startDragging(item.fullIndex, $event)"></div>
            <div class="label-content" @dblclick="editEntry(item.fullIndex)">
              <template v-if="item.entry.editing">
                <input type="text" v-model="item.entry.token" v-focus
                       @keyup.enter="finishEditing(item.fullIndex)"
                       @blur="finishEditing(item.fullIndex)" @click.stop>
              </template>
              <template v-else>
                <span v-html="tokenizeAndColor(item.entry.token)"></span>
                <br>
                <small>{{ formatSeconds(item.entry.timestamp) }} sec</small>
              </template>
            </div>
          </div>
        </div>
      </div>
    </div>
    <!-- Control Buttons Row -->
    <div class="row mb-3" v-show="audioLoaded">
      <div class="col d-flex justify-content-between align-items-center">
        <div>
          <button class="btn btn-primary" @click="togglePlayPause">{{ isPlaying ? 'Pause' : 'Play' }}</button>
        </div>
        <button class="btn btn-success" @click="copySRT">Copy SRT</button>
      </div>
    </div>
    <!-- Context Menu -->
    <div v-if="contextMenu.visible" class="context-menu"
         :style="{ top: contextMenu.y + 'px', left: contextMenu.x + 'px' }">
      <button class="btn btn-sm btn-danger me-2" @click="deleteEntry(contextMenu.entryIndex)">Delete</button>
      <button class="btn btn-sm btn-info" @click="duplicateEntry(contextMenu.entryIndex)">Duplicate</button>
    </div>

    <!-- Vocabulary Modal -->
    <div v-if="vocabModalVisible" class="modal fade show d-block" tabindex="-1" style="background: rgba(0,0,0,0.5);">
      <div class="modal-dialog modal-lg">
        <div class="modal-content text-dark">
          <div class="modal-header">
            <h5 class="modal-title">Vocabulary</h5>
            <button type="button" class="btn-close" @click="closeVocabModal"></button>
          </div>
          <div class="modal-body">
            <!-- Individual Vocabulary Tokens -->
            <h6>Individual Tokens</h6>
            <div class="d-flex flex-wrap mb-3">
              <button v-for="(v, index) in vocabSegments" :key="index"
                      class="btn btn-sm m-1"
                      :style="{ 'background-color': tokenColors[v.token] || '#777', color: '#fff' }"
                      @click="playVocabSegment(v)">
                {{ v.token }} ({{ (v.end - v.start).toFixed(1) }} sec)
              </button>
            </div>
            <!-- Combination Examples using a Table and Button Group -->
            <h6>Combination Examples</h6>
            <table class="table table-dark table-bordered">
              <tbody>
                <tr v-for="(group, idx) in vocabExampleGroups" :key="idx">
                  <td style="width: 200px;">{{ group.label }}</td>
                  <td>
                    <div class="btn-group" role="group">
                      <button
                        v-for="(tokenObj, tIdx) in group.tokens" :key="tIdx"
                        type="button"
                        class="btn btn-sm"
                        :style="{ backgroundColor: tokenColors[tokenObj.token] || '#777', color: '#fff' }"
                        @click="playVocabSegment(tokenObj)">
                        {{ tokenObj.token }} ({{ (tokenObj.end - tokenObj.start).toFixed(1) }} sec)
                      </button>
                      <button
                        v-if="group.tokens.length > 0"
                        type="button"
                        class="btn btn-sm btn-outline-secondary"
                        @click="playVocabSegment(group.all)">
                        ALL ({{ (group.all.end - group.all.start).toFixed(1) }} sec)
                      </button>
                    </div>
                  </td>
                </tr>
              </tbody>
            </table>
          </div>
          <div class="modal-footer">
            <button type="button" class="btn btn-secondary" @click="closeVocabModal">Close</button>
          </div>
        </div>
      </div>
    </div>
  </div>

  <!-- Load Vue 3 -->
  <script src="https://unpkg.com/vue@3"></script>
  <!-- Bootstrap JS -->
  <script src="node_modules/bootstrap/dist/js/bootstrap.bundle.min.js"></script>
  <!-- Load Wavesurfer.js v6.6.4 and its spectrogram plugin -->
  <script src="https://unpkg.com/wavesurfer.js@6.6.4"></script>
  <script src="https://unpkg.com/wavesurfer.js@6.6.4/dist/plugin/wavesurfer.spectrogram.min.js"></script>
  <script src="js/components/menu.js"></script>
  <script>
    const app = Vue.createApp({
      data() {
        return {
          fileId: "13130",
          audioLoaded: false,
          audioLoading: false,
          wavesurfer: null,
          fullBuffer: null,
          chunkDuration: 3,      // visible seconds
          currentChunkStart: 0,  // current chunk start (seconds)
          fileDuration: 0,
          notations: [],
          isPlaying: false,
          isLooping: false,
          dragInfo: { dragging: false, entryIndex: null, offsetX: 0 },
          playheadDragging: false,
          vocabulary: [
              "ca", "ka", "cr", "kr", "w", "ugh", "a", "h",    // caw calls
              "wa", "we", "wo",                                // juvenile
              "ah", "oh", "uh", "ee", "kl",                    // vowels
              "rrr", "k",                                      // rattles
              "ta", "ak", "t",                                 // clacks
              "u", "g"                                         // gurgles
          ],
          tokenColors: {
              "ca": "#4caf50", "ka": "#2196f3", "cr": "#388e3c", "kr": "#1976d2",
              "w": "#ff9800", "ugh": "#9c27b0", "a": "#8bc34a", "h": "#66bb6a",
              "wa": "#e91e63", "we": "#e91e63", "wo": "#e91e63",
              "ah": "#ff5722", "oh": "#ff5722", "uh": "#ff5722", "ee": "#ff5722", "kl": "#ff5722",
              "rrr": "#9e9e9e", "k": "#9e9e9e",
              "ta": "#3f51b5", "ak": "#3f51b5", "t": "#3f51b5",
              "u": "#009688", "g": "#009688"
          },
          contextMenu: { visible: false, x: 0, y: 0, entryIndex: null },
          // Data for individual vocab segments (update start/end as needed)
          vocabSegments: [
            { token: "ca", start: 0.0, end: 0.0 },
            { token: "ka", start: 5.32, end: 5.5 },
            { token: "cr", start: 0.0, end: 0.0 },
            { token: "kr", start: 0.0, end: 0.0 },
            { token: "w", start: 0.0, end: 0.0 },
            { token: "ugh", start: 0.0, end: 0.0 },
            { token: "a", start: 0.0, end: 0.0 },
            { token: "h", start: 0.0, end: 0.0 }
          ],
          // New data for example call groups using a table and button group:
          vocabExampleGroups: [
            {
              label: "Short Call",
              tokens: [
                { token: "ca", start: 11.12, end: 11.4 },
              ],
              all: { start: 11.12, end: 11.4 }
            },
            { label: "Short-Medium", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Medium Call (Harsh)", tokens: [
                { token: "ka", start: 12.76, end: 13.0 },
                { token: "w", start: 13.0, end: 13.2 },
              ],
              all: { start: 12.76, end: 13.2 } },
            { label: "Long Call", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Harsh Call", tokens: [
                { token: "ka", start: 5.3, end: 5.6 },
                { token: "a", start: 5.6, end: 5.9 },
                { token: "a", start: 5.9, end: 6.2 },
                { token: "w", start: 6.2, end: 6.4 }
              ],
              all: { start: 5.3, end: 6.4 }
            },
            { label: "Ko Call", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Two-Syllable Call", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Double Short Call", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Rattle", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Juvenile Begging", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Adult Begging", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Subsong", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Klee-oh-ow", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Gurgle", tokens: [], all: { start: 0.0, end: 0.0 } },
            { label: "Snaps/Clacks", tokens: [], all: { start: 0.0, end: 0.0 } }
          ],
          vocabModalVisible: false
        };
      },
      computed: {
        zoomLevel() {
          const container = document.getElementById("spectrogram");
          return container ? container.offsetWidth / this.chunkDuration : 100;
        },
        // Compute visible notations with computed positions.
        visibleNotations() {
          const trackEl = document.getElementById("transcription-track");
          const trackWidth = trackEl ? trackEl.offsetWidth : 0;
          return this.notations
            .map((entry, fullIndex) => ({ entry, fullIndex }))
            .filter(({ entry }) =>
              entry.timestamp >= this.currentChunkStart &&
              entry.timestamp < this.currentChunkStart + this.chunkDuration
            )
            .map(({ entry, fullIndex }) => ({
              entry,
              fullIndex,
              computedPosition: ((entry.timestamp - this.currentChunkStart) / this.chunkDuration) * trackWidth
            }));
        },
        // Compute available chunk start times as dropdown options.
        chunkOptions() {
          const options = [];
          for (let t = 0; t < this.fileDuration; t += this.chunkDuration) {
            options.push(t);
          }
          return options;
        }
      },
      methods: {
        initAudio() {
          if (!this.fileId) {
            alert("Please enter a File ID");
            return;
          }
          if (this.wavesurfer) { this.wavesurfer.destroy(); }
          this.audioLoading = true;
          this.wavesurfer = WaveSurfer.create({
            container: "#waveform",
            waveColor: "#888",
            progressColor: "#555",
            height: 80,
            scrollParent: true,
            plugins: [
              WaveSurfer.spectrogram.create({
                container: "#spectrogram",
                labels: true,
                height: 200,
                splitChannels: false,
                scale: "mel",
                frequencyMax: 8000,
                frequencyMin: 0,
                fftSamples: 1024,
                labelsBackground: "rgba(0, 0, 0, 0.1)"
              })
            ]
          });
          this.wavesurfer.on("error", (err) => { console.error("Wavesurfer loading error:", err); });
          this.wavesurfer.on("play", () => {
            this.isPlaying = true;
            this.updatePlayhead();
          });
          this.wavesurfer.on("pause", () => {
            this.isPlaying = false;
            this.updatePlayhead();
            if (this.isLooping) { this.wavesurfer.play(0); }
          });
          this.wavesurfer.on("audioprocess", () => { this.updatePlayhead(); });
          const audioUrl = `/cache/library/${this.fileId}.mp3`;
          fetch(audioUrl)
            .then(response => response.arrayBuffer())
            .then(arrayBuffer => {
              const AudioContext = window.AudioContext || window.webkitAudioContext;
              const audioCtx = new AudioContext();
              return audioCtx.decodeAudioData(arrayBuffer);
            })
            .then(decodedBuffer => {
              this.fullBuffer = decodedBuffer;
              this.fileDuration = this.fullBuffer.duration;
              this.audioLoaded = true;
              this.currentChunkStart = 0;
              this.loadChunk();
              this.updateZoom();
              this.loadNotations();
            })
            .catch(e => console.error("Audio fetch/decode error:", e));
        },
        getChunkBuffer() {
          if (!this.fullBuffer) return null;
          const sampleRate = this.fullBuffer.sampleRate;
          const startSample = Math.floor(this.currentChunkStart * sampleRate);
          const chunkSamples = Math.floor(this.chunkDuration * sampleRate);
          const numChannels = this.fullBuffer.numberOfChannels;
          const AudioContext = window.AudioContext || window.webkitAudioContext;
          const tempCtx = new AudioContext();
          const newBuffer = tempCtx.createBuffer(numChannels, chunkSamples, sampleRate);
          for (let ch = 0; ch < numChannels; ch++) {
            const fullData = this.fullBuffer.getChannelData(ch);
            const newData = newBuffer.getChannelData(ch);
            for (let i = 0; i < chunkSamples; i++) {
              const sourceIndex = startSample + i;
              newData[i] = (sourceIndex < fullData.length) ? fullData[sourceIndex] : 0;
            }
          }
          tempCtx.close();
          return newBuffer;
        },
        loadChunk() {
          const chunkBuffer = this.getChunkBuffer();
          if (chunkBuffer) {
            this.wavesurfer.once("ready", () => { this.updateZoom(); });
            this.wavesurfer.loadDecodedBuffer(chunkBuffer);
            this.wavesurfer.seekTo(0);
          }
        },
        updateChunk() {
          this.loadChunk();
          this.updateZoom();
        },
        nextChunk() {
          if (this.currentChunkStart + this.chunkDuration < this.fileDuration) {
            this.currentChunkStart += this.chunkDuration;
            this.loadChunk();
            this.updateZoom();
            this.loadNotations();
            this.wavesurfer.play();
          }
        },
        prevChunk() {
          if (this.currentChunkStart - this.chunkDuration >= 0) {
            this.currentChunkStart -= this.chunkDuration;
            this.loadChunk();
            this.updateZoom();
            this.loadNotations();
          }
        },
        chunkSelectChanged() {
          this.loadChunk();
          this.updateZoom();
          this.loadNotations();
        },
        updateZoom() {
          const zoom = this.zoomLevel;
          if (this.wavesurfer) { this.wavesurfer.zoom(zoom); }
        },
        updatePlayhead() {
          const currentTime = this.wavesurfer.getCurrentTime();
          const ratio = currentTime / this.chunkDuration;
          const wf = document.getElementById("waveform");
          const phw = document.getElementById("playhead-waveform");
          if (wf && phw) { phw.style.left = (ratio * wf.offsetWidth) + "px"; }
          const spec = document.getElementById("spectrogram");
          const phs = document.getElementById("playhead-spectrogram");
          if (spec && phs) { phs.style.left = (ratio * spec.offsetWidth) + "px"; }
        },
        togglePlayPause() {
          this.wavesurfer?.playPause();
        },
        toggleLoop() {
          this.isLooping = !this.isLooping;
          if (this.isLooping && !this.isPlaying) { this.wavesurfer.play(); }
        },
        formatTimestamp(time) {
          const date = new Date(null);
          date.setSeconds(time);
          return date.toISOString().substr(11, 8) + ",000";
        },
        // New helper to format a timestamp (in seconds) with 2 decimals.
        formatSeconds(value) {
          return parseFloat(value).toFixed(2);
        },
        copySRT() {
          let srtText = "";
          let index = 1;
          for (let t = 0; t < this.fileDuration; t += this.chunkDuration) {
            const start = t;
            const end = Math.min(t + this.chunkDuration, this.fileDuration);
            const groupEntries = this.notations.filter(n => n.timestamp >= start && n.timestamp < end);
            const textLine = groupEntries.map(n => n.token).join(" ");
            srtText += `${index}\n${this.formatTimestamp(start)} --> ${this.formatTimestamp(end)}\n${textLine}\n\n`;
            index++;
          }
          navigator.clipboard.writeText(srtText).then(() => { alert("SRT copied to clipboard!"); });
        },
        trackClick(event) {
          if (event.target.closest(".notation-entry")) return;
          const trackRect = event.currentTarget.getBoundingClientRect();
          const x = event.clientX - trackRect.left;
          const newTimestamp = this.currentChunkStart + (x / trackRect.width) * this.chunkDuration;
          this.notations.push({ timestamp: newTimestamp, token: "", top: 5, editing: true });
          this.saveNotations();
        },
        editEntry(fullIndex) {
          this.notations[fullIndex].editing = true;
        },
        finishEditing(fullIndex) {
          const updated = { ...this.notations[fullIndex] };
          updated.token = updated.token.trim();
          updated.editing = false;
          this.notations.splice(fullIndex, 1, updated);
          this.saveNotations();
        },
        tokenizeAndColor(text) {
          let html = "";
          let remaining = text;
          const sortedVocab = [...this.vocabulary].sort((a, b) => b.length - a.length);
          while (remaining.length > 0) {
            let found = false;
            for (let token of sortedVocab) {
              if (remaining.startsWith(token)) {
                const color = this.tokenColors[token] || "#777";
                html += `<span class="token-valid" style="background-color: ${color};">${token}</span>`;
                remaining = remaining.slice(token.length);
                found = true;
                break;
              }
            }
            if (!found) {
              html += `<span class="token-invalid">${remaining[0]}</span>`;
              remaining = remaining.slice(1);
            }
          }
          return html;
        },
        startDragging(fullIndex, event) {
          if (this.dragInfo.dragging) return;
          this.dragInfo.dragging = true;
          this.dragInfo.entryIndex = fullIndex;
          const handleEl = event.currentTarget;
          const rect = handleEl.getBoundingClientRect();
          this.dragInfo.offsetX = event.clientX - rect.left;
          document.addEventListener("mousemove", this.onDrag);
          document.addEventListener("mouseup", this.stopDragging);
        },
        onDrag(event) {
          if (!this.dragInfo.dragging) return;
          const trackEl = document.getElementById("transcription-track");
          const trackRect = trackEl.getBoundingClientRect();
          let newX = event.clientX - trackRect.left - this.dragInfo.offsetX;
          newX = Math.max(0, Math.min(newX, trackRect.width));
          const newTimestamp = this.currentChunkStart + (newX / trackRect.width) * this.chunkDuration;
          this.notations[this.dragInfo.entryIndex].timestamp = newTimestamp;
        },
        stopDragging() {
          this.dragInfo.dragging = false;
          document.removeEventListener("mousemove", this.onDrag);
          document.removeEventListener("mouseup", this.stopDragging);
          this.saveNotations();
        },
        showContextMenu(fullIndex, event) {
          this.contextMenu.visible = true;
          this.contextMenu.x = event.clientX;
          this.contextMenu.y = event.clientY;
          this.contextMenu.entryIndex = fullIndex;
        },
        deleteEntry(fullIndex) {
          this.notations.splice(fullIndex, 1);
          this.contextMenu.visible = false;
          this.saveNotations();
        },
        duplicateEntry(fullIndex) {
          const entry = this.notations[fullIndex];
          const newEntry = { ...entry, timestamp: entry.timestamp + 0.1, editing: false };
          this.notations.splice(fullIndex + 1, 0, newEntry);
          this.contextMenu.visible = false;
          this.saveNotations();
        },
        saveNotations() {
          console.log("Saving notations:", this.notations);
          fetch('/updateNotationLabels', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ fileId: this.fileId, notations: this.notations })
          })
          .then(response => response.json())
          .then(data => {
            if (data.success) {
              console.log("Notations saved successfully.");
            } else {
              console.error("Error saving notations:", data.error);
            }
          })
          .catch(err => console.error("Error saving notations:", err));
        },
        loadNotations() {
          fetch(`/getNotationLabels?fileId=${this.fileId}`)
            .then(response => response.json())
            .then(data => {
              if (data.success) {
                this.notations = Array.isArray(data.notations) ? data.notations : [];
              } else {
                console.error("Error loading notations:", data.error);
              }
            })
            .catch(error => console.error("Error loading notations:", error));
        },
        jumpToTime(event) {
          if (this.playheadDragging) return;
          const rect = event.currentTarget.getBoundingClientRect();
          const x = event.clientX - rect.left;
          const ratio = x / rect.width;
          this.wavesurfer.seekTo(ratio);
          this.updatePlayhead();
          if (this.isPlaying) { this.wavesurfer.play(ratio); }
        },
        startPlayheadDrag(event) {
          event.preventDefault();
          this.playheadDragging = true;
          document.addEventListener("mousemove", this.onPlayheadDrag);
          document.addEventListener("mouseup", this.stopPlayheadDrag);
        },
        onPlayheadDrag(event) {
          const wfElem = document.getElementById("waveform");
          if (!wfElem) return;
          const rect = wfElem.getBoundingClientRect();
          const x = event.clientX - rect.left;
          const ratio = Math.min(Math.max(x / rect.width, 0), 1);
          this.wavesurfer.seekTo(ratio);
          this.updatePlayhead();
        },
        stopPlayheadDrag() {
          this.playheadDragging = false;
          document.removeEventListener("mousemove", this.onPlayheadDrag);
          document.removeEventListener("mouseup", this.stopPlayheadDrag);
        },
        onKeyDown(e) {
          if (e.code === "Space") {
            e.preventDefault();
            this.togglePlayPause();
          }
        },
        onWindowClick(e) {
          if (this.contextMenu.visible) { this.contextMenu.visible = false; }
        },
        // Methods for Vocabulary Modal
        openVocabModal() {
          this.vocabModalVisible = true;
          // Optionally, preload or re-read the audio buffer here.
        },
        closeVocabModal() {
          this.vocabModalVisible = false;
        },
        // Plays a given vocab segment (can be from individual tokens or from example groups)
        playVocabSegment(segment) {
          if (!this.fullBuffer) return;
          const AudioContext = window.AudioContext || window.webkitAudioContext;
          const ctx = new AudioContext();
          const source = ctx.createBufferSource();
          source.buffer = this.fullBuffer;
          source.connect(ctx.destination);
          const startTime = segment.start || 0;
          const endTime = segment.end || 0;
          const segmentDuration = endTime - startTime;
          if (segmentDuration <= 0) {
            alert("Vocab segment duration is not set. Please update the start and end times.");
            return;
          }
          source.start(0, startTime, segmentDuration);
          setTimeout(() => {
            source.stop();
            ctx.close();
          }, segmentDuration * 1000);
        }
      },
      mounted() {
        window.addEventListener("keydown", this.onKeyDown);
        window.addEventListener("click", this.onWindowClick);
      },
      beforeUnmount() {
        window.removeEventListener("keydown", this.onKeyDown);
        window.removeEventListener("click", this.onWindowClick);
      }
    });

    // Custom directive to auto-focus inputs.
    app.directive("focus", {
      mounted(el) { el.focus(); }
    });

    app.component('menu-component', window.MenuComponent);
    app.mount("#app");

    function generateBluePalette() {
      const palette = [];
      for (let i = 0; i < 256; i++) {
        const t = i / 255;
        const blue = Math.floor(100 + t * (255 - 100));
        const r = 30, g = 30, b = blue;
        const hex = '#' +
          r.toString(16).padStart(2, '0') +
          g.toString(16).padStart(2, '0') +
          b.toString(16).padStart(2, '0');
        palette.push(hex);
      }
      return palette;
    }
  </script>
</body>
</html>

==========================================
</labeler/transcriptions.html>
==========================================

==========================================
<requirements.txt>
Size: 125 bytes | Lines:       12
==========================================
birdnetlib
librosa
tensorflow
tqdm
resampy
biodenoising
norbert
asteroid
omegaconf
pesto-pitch==0.1.0
pandas
faiss-cpu
optuna
==========================================
</requirements.txt>
==========================================

==========================================
<separator/dataset.py>
Size: 3894 bytes | Lines:       98
==========================================
import os
import torch
import torchaudio
import numpy as np
import json

FIXED_N_SRC = 2


class CrowMixDataset:
    def __init__(self, json_path, merged_dir, separate_dir, sr=8000, transform=None):
        # Load the full mixes dataset.
        with open(json_path, "r", encoding="utf-8") as f:
            self.mixes = json.load(f)

        print(f"Found {len(self.mixes)} mixes.")

        self.merged_dir = merged_dir
        self.separate_dir = separate_dir
        self.sr = sr
        self.transform = transform

    def __len__(self):
        return len(self.mixes)

    def pad_sources(self, sources, n_src, expected_length):
        # sources: list of tensors of shape (channels, time)
        # expected_length: target time dimension length
        while len(sources) < n_src:
            # Create a silent source with the same number of channels (assume mono here) and expected length
            silence = torch.zeros(1, expected_length)
            sources.append(silence)
        # Stack along a new dimension; shape becomes (n_src, channels, time)
        stacked = torch.stack(sources, dim=0)
        # If sources are mono (channels == 1), squeeze that dimension to get shape (n_src, time)
        if stacked.shape[1] == 1:
            stacked = stacked.squeeze(1)
        return stacked

    def __getitem__(self, idx):
        mix_info = self.mixes[idx]
        mix_path = os.path.join(self.merged_dir, mix_info["mix"])
        mixture, sr_m = torchaudio.load(mix_path)
        if sr_m != self.sr:
            mixture = torchaudio.functional.resample(mixture, sr_m, self.sr)

        source_keys = sorted(mix_info["layers"].keys())[1:]
        sources = []
        for key in source_keys:
            src_filename = mix_info["layers"][key]
            src_path = os.path.join(self.separate_dir, src_filename)
            source, sr_s = torchaudio.load(src_path)
            if sr_s != self.sr:
                source = torchaudio.functional.resample(source, sr_s, self.sr)
            sources.append(source)

        # Determine expected length from the mixture
        expected_length = mixture.shape[1]

        # Pad if necessary
        sources = self.pad_sources(sources, FIXED_N_SRC, expected_length)

        return mixture, sources

def collate_fn(batch):
    mixtures, sources_list = zip(*batch)  # Unpack batch tuples

    # Convert to numpy arrays before torch conversion (avoids PyTorch warning)
    mixtures = [mixture.numpy() if isinstance(mixture, torch.Tensor) else mixture for mixture in mixtures]
    sources_list = [sources.numpy() if isinstance(sources, torch.Tensor) else sources for sources in sources_list]

    # Determine max time length among mixtures
    max_len = max(mixture.shape[1] for mixture in mixtures)

    # Preallocate tensors
    batch_size = len(batch)
    padded_mixtures = np.zeros((batch_size, mixtures[0].shape[0], max_len), dtype=np.float32)  # [batch, channels, time]
    padded_sources = np.zeros((batch_size, FIXED_N_SRC, max_len), dtype=np.float32)  # [batch, FIXED_N_SRC, time]

    for i, (mixture, sources) in enumerate(zip(mixtures, sources_list)):
        # Copy mixture data (instead of appending, preallocated)
        padded_mixtures[i, :, :mixture.shape[1]] = mixture

        # Squeeze channel dimension if sources are mono
        if sources.ndim == 3 and sources.shape[1] == 1:
            sources = sources.squeeze(1)  # Shape [n_src, time]

        # Pad sources in time
        sources_padded = np.zeros((FIXED_N_SRC, max_len), dtype=np.float32)
        sources_padded[:sources.shape[0], :sources.shape[1]] = sources[:FIXED_N_SRC]  # Truncate or pad

        padded_sources[i] = sources_padded  # Assign to preallocated array

    # Convert final arrays to tensors
    padded_mixtures = torch.tensor(padded_mixtures)
    padded_sources = torch.tensor(padded_sources)

    return padded_mixtures, padded_sources

==========================================
</separator/dataset.py>
==========================================

==========================================
<separator/migrate.py>
Size: 2865 bytes | Lines:       76
==========================================
import os
import json

# Set up paths.
PATH = os.path.dirname(__file__)
OLD_LABELS_PATHS = [os.path.join(PATH, "..", ".cache", "labels.json"),
                    os.path.join(PATH, "..", ".cache", "auto_labels.json")]
OUTPUT_PATH = os.path.join(PATH, "..", ".cache", "migrated_labels.json")

# Prepare a new dictionary.
new_labels = {}

for OLD_LABELS_PATH in OLD_LABELS_PATHS:
    # Load old labels.
    with open(OLD_LABELS_PATH, "r") as f:
        old_labels = json.load(f)

    for key, old in old_labels.items():
        # Check if any of the hard-to-find flags is True.
        file_id, start, end = key.split("-")
        segment_length = int(end) - int(start)
        reviewed = old.get("reviewed", False)
        if not reviewed and "auto_labels" in OLD_LABELS_PATH:
            continue
        if not (old.get("rattle", False) or old.get("softSong", False) or old.get("human", False) or old.get("mob", False) or old.get("begging", False) or old.get("quality", 2) in [1,3]):
            continue
        if not segment_length == 3.0:
            continue

        # Map crowCount: assume "multiple" becomes 4, else 1.
        if not reviewed:
            crow_count = 2 if old.get("crowCount", "").lower() == "multiple" else 1
            crow_age = 1 if old.get("crowAge", "").lower() == "adult" else 2
            quality = 1 if old.get("human", False) else 2
            if old.get("rattle", False) or old.get("softSong", False):
                quality = 2 # force rattle and softSong to average quality
        else:
            crow_count = old.get("crowCount", 1)
            crow_age = old.get("crowAge", 1)
            quality = old.get("quality", 2)

        # Determine cluster based on flag priority: human > rattle > softSong.
        if old.get("human", False):
            cluster = 100
        elif old.get("rattle", False):
            cluster = 200
        elif old.get("softSong", False):
            cluster = 300
        elif old.get("begging", False):
            cluster = 400
        elif old.get("mob", False):
            cluster = 500
        else:
            cluster = 0  # Fallback; should not occur given our filter.

        # Build new label dictionary.
        new_label = {
            "crowCount": crow_count,
            "crowAge": crow_age,
            "alert": False,
            "begging": old.get("begging", False),
            "softSong": old.get("softSong", False),
            "rattle": old.get("rattle", False),
            "mob": False,  # No corresponding old field; defaulting to False.
            "quality": quality,
            "reviewed": True,
            "cluster": cluster
        }

        new_labels[key] = new_label

# Write the new labels to the output file.
with open(OUTPUT_PATH, "w") as f:
    json.dump(new_labels, f, indent=4)

print(f"Migrated {len(new_labels)} labels to {OUTPUT_PATH}")

==========================================
</separator/migrate.py>
==========================================

==========================================
<separator/mix.py>
Size: 15469 bytes | Lines:      414
==========================================
#!/usr/bin/env python3
import os, random, json
import numpy as np
import librosa
import soundfile as sf
import sounddevice as sd

random.seed(42)

# ------------------------------
# Control switches and limits
# ------------------------------
PREVIEW = False
NUM_MIXES = 40000
SAMPLE_RATE = 8000
ENABLE_DENOISED = True
ENABLE_REVERB = False
ENABLE_RANDOM_VOLUME = False
ENABLE_BACKGROUND_SOUNDS = True
RANDOMIZE_BACKGROUND_VOLUME = True
LIMIT_SEGMENT_USE = True
LIMIT_FILE_ID_USE = True
OFFSET_SECONDS = (0.0, 0.5)
MAX_SEGMENT_USES = 1        # Each segment key can be used at most once.
MAX_FILE_ID_USES = 100      # Each file ID can be used at most once.
NUM_SEGMENTS_PER_MIX = 2
BACKGROUND_VOLUME_RANGE = (0.0, 0.8)

# ------------------------------
# Helper functions
# ------------------------------
def background_audio_generator(directory, min_length=3.0, sr=SAMPLE_RATE):
    files = os.listdir(directory)
    while True:
        random.shuffle(files)
        for file in files:
            if file.lower().endswith(".mp3"):
                path = os.path.join(directory, file)
                audio, _ = librosa.load(path, sr=sr)
                if len(audio) / sr >= min_length:
                    yield audio[:sr*4], file

def get_valid_segments(labels_path):
    with open(labels_path, "r", encoding="utf-8") as f:
        labels = json.load(f)

    # Group valid segments (assumed 1-second each) by file_id.
    segments_by_file = {}
    for key, attr in labels.items():
        # Only consider valid entries.
        # (Assume valid if crowCount == 1 and quality != 1)
        if attr.get("crowCount") == 1 and attr.get("quality") != 1:
            parts = key.split("-")
            if len(parts) == 3:
                file_id = parts[0]
                start_time = float(parts[1])
                end_time = float(parts[2])
                segments_by_file.setdefault(file_id, []).append((start_time, end_time))

    valid_segments = []
    tolerance = 0.01
    # For each file, sort and combine contiguous segments into blocks.
    # Then, for each block, apply the rule:
    #   - Discard blocks shorter than 2 seconds.
    #   - If duration is between 2 and 3 seconds, use the block as is.
    #   - If longer than 3 seconds, split into non-overlapping segments:
    #         full segments of 3 seconds each, and if the leftover is at least 2 seconds,
    #         include it as an extra segment.
    for file_id, segs in segments_by_file.items():
        segs.sort(key=lambda x: x[0])
        blocks = []
        block_start, block_end = segs[0]
        for s, e in segs[1:]:
            if abs(s - block_end) < tolerance:
                block_end = e
            else:
                blocks.append((block_start, block_end))
                block_start, block_end = s, e
        blocks.append((block_start, block_end))

        for b_start, b_end in blocks:
            block_duration = b_end - b_start
            if block_duration < 2.0:
                continue  # Discard blocks shorter than 2 seconds.
            elif block_duration <= 3.0:
                # Use the block as is.
                new_key = f"{file_id}-{b_start:.2f}-{b_end:.2f}"
                valid_segments.append({
                    "key": new_key,
                    "file_id": file_id,
                    "start_time": b_start,
                    "end_time": b_end
                })
            else:
                # Split block into full 3-second segments.
                num_full = int(block_duration // 3)
                for i in range(num_full):
                    seg_start = b_start + i * 3
                    seg_end = seg_start + 3
                    new_key = f"{file_id}-{seg_start:.2f}-{seg_end:.2f}"
                    valid_segments.append({
                        "key": new_key,
                        "file_id": file_id,
                        "start_time": seg_start,
                        "end_time": seg_end
                    })
                # Use the remainder if it's at least 2 seconds.
                remainder = block_duration - num_full * 3
                if remainder >= 2.0:
                    seg_start = b_start + num_full * 3
                    seg_end = b_end
                    new_key = f"{file_id}-{seg_start:.2f}-{seg_end:.2f}"
                    valid_segments.append({
                        "key": new_key,
                        "file_id": file_id,
                        "start_time": seg_start,
                        "end_time": seg_end
                    })
    return valid_segments

def choose_random_segments(valid_segments, segment_usage, file_id_usage, count=25):
    eligible_segments = []
    for seg in valid_segments:
        key = seg["key"]
        fid = seg["file_id"]
        seg_used = segment_usage.get(key, 0)
        file_used = file_id_usage.get(fid, 0)
        if LIMIT_SEGMENT_USE and seg_used >= MAX_SEGMENT_USES:
            continue
        if LIMIT_FILE_ID_USE and file_used >= MAX_FILE_ID_USES:
            continue
        eligible_segments.append(seg)
    if len(eligible_segments) < count:
        count = len(eligible_segments)
    return random.sample(eligible_segments, count) if eligible_segments else []

def adjust_segment(audio, sr, approx_start, approx_end, window_size=0.1, fraction=0.1):
    """
    Trim leading/trailing silence by computing a smoothed amplitude envelope.
    Returns (start_time_s, end_time_s) in seconds, or None if no active audio is found.
    """
    start_idx = int(approx_start * sr)
    end_idx = int(approx_end * sr)
    segment = audio[start_idx:end_idx]

    # Smooth envelope
    win_len = max(int(window_size * sr), 1)
    envelope = np.convolve(np.abs(segment), np.ones(win_len)/win_len, mode='same')

    # Dynamic threshold: min + fraction*(max-min)
    env_min, env_max = envelope.min(), envelope.max()
    threshold = env_min + fraction * (env_max - env_min)

    # Where does envelope exceed that threshold?
    active_indices = np.where(envelope > threshold)[0]
    if active_indices.size == 0:
        return None

    active_start = active_indices[0]
    active_end   = active_indices[-1] + 1  # +1 to include last active sample

    # Convert back to original audio indices
    new_start_idx = start_idx + active_start
    new_end_idx   = start_idx + active_end

    return new_start_idx / sr, new_end_idx / sr

def generate_random_ir(sr, ir_length_sec):
    ir_length = int(ir_length_sec * sr)
    t = np.linspace(0, ir_length_sec, ir_length)
    envelope = np.exp(-t / 0.3)
    random_variation = np.random.uniform(0.5, 1.0, size=ir_length)
    ir = envelope * random_variation
    ir[0] = 1.0
    return ir

def add_reverb(audio, sr, ir_length_sec=1.0, scale_rirs=10.0):
    reverb_amount = random.uniform(0.0, 0.6)
    ir = generate_random_ir(sr, ir_length_sec) * scale_rirs
    wet = np.convolve(audio, ir, mode="full")[:len(audio)]
    dry_rms = np.sqrt(np.mean(audio**2))
    wet_rms = np.sqrt(np.mean(wet**2))
    target_wet_rms = 0.3 * dry_rms
    if wet_rms > 0:
        wet = wet * (target_wet_rms / wet_rms)
    out = audio + wet
    out = (1 - reverb_amount) * audio + reverb_amount * out
    max_val = np.max(np.abs(out))
    if max_val > 1:
        out = out / max_val
    return out

def is_audio_silient(segment_audio, sr, min_duration=1.5, silence_threshold=1e-2, silence_fraction=0.95):
    # Check if the segment is too short.
    if len(segment_audio) < int(min_duration * sr):
        return True

    # Compute the fraction of silent samples.
    silent_ratio = np.mean(np.abs(segment_audio) < silence_threshold)
    return silent_ratio > silence_fraction

def adjust_volume(audio, factor=None):
    # Original volume adjustment function (unused if normalization is enabled)
    if factor is None:
        factor = random.uniform(0.6, 0.9)
    return audio * factor

def mix_audio(background, segments, sr=SAMPLE_RATE):
    """
    Mix each segment at OFFSET_SECONDS into the background,
    then crop the entire mix to the shortest length among all.
    """
    bg_len = len(background)
    seg_lens = [len(seg) for seg in segments]

    # The final mix length is the minimum among:
    #   - background length
    #   - each segment length
    final_length = min([bg_len] + seg_lens)

    # Truncate the background to final_length
    final_mix = background[:final_length].copy()

    # Create layers for each segment so we can also save them individually
    layers_out = []
    for seg in segments:
        layer = np.zeros(final_length, dtype=seg.dtype)
        seg_len = len(seg)

        # Random offset start
        offset_samples = int(random.uniform(*OFFSET_SECONDS) * sr)

        # Place the segment at OFFSET_SECONDS, but truncate if it goes beyond final_length
        start = offset_samples
        if start >= final_length:
            # No room to place this segment (offset is beyond final length)
            layers_out.append(layer)
            continue

        # If segment extends beyond final_length, crop it
        usable_len = min(seg_len, final_length - offset_samples)
        layer[start:start+usable_len] = seg[:usable_len]

        # Add layer into final mix
        final_mix += layer
        layers_out.append(layer)

    return final_mix, layers_out

# ------------------------------
# Main processing
# ------------------------------
def main():
    sr = SAMPLE_RATE
    total_seconds = 0.0
    mix_dataset_path = "../.cache/mixes/mix-dataset.json"
    backgrounds_dir = "../.cache/backgrounds"
    labels_json = "../.cache/auto_labels.json"
    library_dir = "../.cache/library"
    denoised_dir = "../.cache/library-denoised"
    merged_dir = "../.cache/mixes/merged"
    separate_dir = "../.cache/mixes/separate"
    os.makedirs(merged_dir, exist_ok=True)
    os.makedirs(separate_dir, exist_ok=True)

    # Dictionaries to track usage of segments and file IDs
    segment_usage = {}
    file_id_usage = {}

    if ENABLE_BACKGROUND_SOUNDS:
        bg_generator = background_audio_generator(backgrounds_dir, sr=sr)

    valid_segments = get_valid_segments(labels_json)

    mix_dataset = []
    mix_count = 0

    # Generate mixes (adjust the number as needed)
    for _ in range(NUM_MIXES):
        if ENABLE_BACKGROUND_SOUNDS:
            try:
                background, bg_file = next(bg_generator)
                if RANDOMIZE_BACKGROUND_VOLUME:
                    bg_volume_factor = random.uniform(*BACKGROUND_VOLUME_RANGE)
                    background = background * bg_volume_factor
            except StopIteration:
                break
        else:
            background = np.zeros(sr * 8)  # 8 seconds of silence
            bg_file = "silence"

        # Choose segments while respecting usage limits
        segments_info = choose_random_segments(valid_segments, segment_usage, file_id_usage, count=50)
        segments_list = []  # We'll store each trimmed segment audio here
        segments_details = []

        if not segments_info:
            print(f"No more valid segments found. Skipping...")
            break

        for seg in segments_info:
            if ENABLE_DENOISED:
                crow_path = os.path.join(denoised_dir, f"{seg['file_id']}.wav")
            else:
                crow_path = os.path.join(library_dir, f"{seg['file_id']}.mp3")
            if not os.path.exists(crow_path):
                print("Segment does not exist: {}".format(crow_path))
                continue

            crow_audio, _ = librosa.load(crow_path, sr=sr)
            start_time = seg['start_time']
            end_time = seg['end_time']
            if int(end_time * sr) > len(crow_audio):
                end_time = len(crow_audio) / sr

            # Normalize segment volume if enabled (else apply a random volume adjustment)
            if ENABLE_RANDOM_VOLUME:
                crow_audio = adjust_volume(crow_audio)

            start_sample = int(start_time * sr)
            end_sample = int(end_time * sr)
            segment_audio = crow_audio[start_sample:end_sample]

            # Skip segment if it's less than 1.0 seconds long or if it contains no nonzero values.
            if is_audio_silient(segment_audio, sr):
                print("Segment too short or silent: {}".format(crow_path))
                segment_usage[seg["key"]] = segment_usage.get(seg["key"], 0) + 1
                if PREVIEW:
                    print(f"Previewing too-short/silent segment {seg['key']}")
                    sd.play(segment_audio, sr)
                    sd.wait()
                continue

            # Apply reverb if enabled
            if ENABLE_REVERB:
                segment_audio = add_reverb(segment_audio, sr)

            segments_list.append(segment_audio)
            segments_details.append({
                "file_id": seg["file_id"],
                "original_key": seg["key"],
                "adjusted_start": start_time,
                "adjusted_end": end_time
            })

            # Update usage counts
            segment_usage[seg["key"]] = segment_usage.get(seg["key"], 0) + 1
            file_id_usage[seg["file_id"]] = file_id_usage.get(seg["file_id"], 0) + 1

            # For this mix, stop after adding X segments.
            if len(segments_list) == NUM_SEGMENTS_PER_MIX:
                break

        if not segments_list:
            print("No segments inserted")
            continue

        if len(segments_list) < NUM_SEGMENTS_PER_MIX:
            print("Not enough segments inserted: {}".format(len(segments_list)))
            continue

        # Now mix all audio at OFFSET_SECONDS, cropped to the shortest length
        mix, layer_arrays = mix_audio(background, segments_list, sr=sr)

        if PREVIEW:
            print("Previewing background")
            sd.play(background, sr)
            sd.wait()
            for i, seg in enumerate(segments_list):
                print(f"Previewing segment {i}")
                sd.play(seg, sr)
                sd.wait()
            print("Previewing final mix")
            sd.play(mix, sr)
            sd.wait()
            input("Press Enter to save this mix and continue...")

        mix_filename = f"mix_{mix_count}.wav"
        mix_path = os.path.join(merged_dir, mix_filename)
        sf.write(mix_path, mix, sr)

        mix_length = len(mix) / sr
        total_seconds += mix_length
        print(f"Saved {mix_filename}: {mix_count} - {len(layer_arrays)} layers: {mix_length} seconds")

        # Save individual layers.
        layer_files = {}
        bg_filename = f"mix_{mix_count}_background.wav"
        bg_path = os.path.join(separate_dir, bg_filename)
        if ENABLE_BACKGROUND_SOUNDS:
            sf.write(bg_path, background[:len(mix)], sr)
        layer_files["background"] = bg_filename

        for i, layer in enumerate(layer_arrays):
            seg_filename = f"mix_{mix_count}_segment_{i}.wav"
            seg_path = os.path.join(separate_dir, seg_filename)
            sf.write(seg_path, layer, sr)
            layer_files[f"segment_{i}"] = seg_filename

        mix_dataset.append({
            "mix": mix_filename,
            "layers": layer_files,
            "segments": segments_details,
            "background_file": bg_file
        })
        mix_count += 1

    with open(mix_dataset_path, "w") as f:
        json.dump(mix_dataset, f, indent=2)

    print(f"Saved {len(mix_dataset)} mix dataset, Total: {total_seconds/60.0} minutes")

if __name__ == "__main__":
    main()

==========================================
</separator/mix.py>
==========================================

==========================================
<separator/separate.py>
Size: 2086 bytes | Lines:       57
==========================================
import torch
import soundfile as sf
import torchaudio
from biodenoising.denoiser.dsp import convert_audio
from asteroid.models import DPRNNTasNet

# Choose device
device = "cuda" if torch.cuda.is_available() else "cpu"

# 69 = best so far
# 75 is great
#model_checkpoint = "models/best/best_model_epoch=46.ckpt" # BEST
model_checkpoint = "models/best_model_epoch=94.ckpt"

#input_audio = "samples/non-overlapping-multiple-crows.mp3"
input_audio = "samples/overlapping-crows-1.wav"
#input_audio = "samples/overlapping-crows-2.wav"
#input_audio = "samples/overlapping-crows-3.wav"


# Manually define the model architecture (match it exactly to your training setup)
FIXED_N_SRC = 2  # Adjust this to match how many sources you separated in training
model = DPRNNTasNet(n_src=FIXED_N_SRC).to(device)  # Move model to device

# Load the PyTorch Lightning checkpoint
checkpoint = torch.load(model_checkpoint, map_location="cpu")

# Extract the actual model state_dict (Lightning saves it under "model")
if "state_dict" in checkpoint:
    state_dict = {k.replace("model.", ""): v for k, v in checkpoint["state_dict"].items()}
else:
    raise ValueError(f"Invalid checkpoint format! Missing 'state_dict'. Available keys: {checkpoint.keys()}")

# Load weights into the model
model.load_state_dict(state_dict)
model.eval()

# Load the MP3 file
wav, sr = torchaudio.load(input_audio)

# Convert audio to the model's expected sample rate and channel configuration
wav = convert_audio(wav, sr, model.sample_rate, model.in_channels).to(device)

# Convert shape to (batch, channels, time)
mixture = wav.unsqueeze(0)  # (1, channels, time)

# Convert to torch tensor and run inference
with torch.no_grad():
    separated_sources = model.separate(mixture)

# Convert back to NumPy and save
separated_sources = separated_sources.squeeze(0).cpu().numpy()

for i, source in enumerate(separated_sources):
    output_filename = f"separated_source_{i + 1}.wav"
    sf.write(output_filename, source, model.sample_rate)  # Ensure correct sample rate
    print(f"✅ Saved separated source: {output_filename}")

==========================================
</separator/separate.py>
==========================================

==========================================
<separator/train.py>
Size: 2472 bytes | Lines:       75
==========================================
from torch import optim
from asteroid.models import DPRNNTasNet
from asteroid.losses import pairwise_neg_sisdr, PITLossWrapper
from asteroid.engine import System
from pytorch_lightning import Trainer, seed_everything
from pytorch_lightning.loggers import TensorBoardLogger
from pytorch_lightning.callbacks import ModelCheckpoint
from torch.utils.data import random_split, DataLoader
from dataset import *

# Seed for reproducibility
seed_everything(42, workers=True)

import os
os.environ["TF_ENABLE_ONEDNN_OPTS"] = "0"
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True,max_split_size_mb:128"

# Enable Tensor Core optimization
torch.set_float32_matmul_precision('high')

# Choose device
device = "cuda" if torch.cuda.is_available() else "cpu"

json_path = os.path.join("../.cache", "mixes", "mix-dataset.json")
merged_dir = os.path.join("../.cache", "mixes", "merged")
separate_dir = os.path.join("../.cache", "mixes", "separate")
sr = 8000
batch_size = 1

# Load dataset
dataset = CrowMixDataset(json_path, merged_dir, separate_dir, sr=sr)

# Determine sizes for training and validation sets.
train_size = int(0.9 * len(dataset))
val_size = len(dataset) - train_size

# Split the dataset.
train_dataset, val_dataset = random_split(dataset, [train_size, val_size])

# Get data loaders
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=3, persistent_workers=True, collate_fn=collate_fn)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False, num_workers=3, persistent_workers=True, collate_fn=collate_fn)

# Specify number of sources (e.g., 2 if you mix background and one crow call)
model = DPRNNTasNet(n_src=FIXED_N_SRC).to(device)

loss = PITLossWrapper(pairwise_neg_sisdr, pit_from="pw_mtx")
optimizer = optim.Adam(model.parameters(), lr=5e-4)

# Create the training system with your custom DataLoader(s)
tb_logger = TensorBoardLogger("logs", name="crow")
system = System(model, optimizer, loss, train_loader, val_loader)

# Checkpointing
checkpoint_callback = ModelCheckpoint(
    dirpath="logs/crow/checkpoints",
    filename="best_model",
    monitor="val_loss",
    mode="min",
    save_top_k=3
)

# Move Trainer to use CPU instead of GPU
trainer = Trainer(
    max_epochs=50,
    logger=tb_logger,
    log_every_n_steps=1,
    accumulate_grad_batches=2,
    precision="16-mixed",
    #accelerator="gpu",
    deterministic=True,
    callbacks=[checkpoint_callback]
)

trainer.fit(system)

==========================================
</separator/train.py>
==========================================