Add YAMNet subprocess backend and auto-resampling for classification#505
Conversation
- classification/doc.md: AST and YAMNet added to Models section (organized by category: scene, demographics, content) - model_registry.yaml: new audio_scene_classification section - model_registry.md: regenerated - Fixed generate_model_registry.py path (docs/ → src/senselab/) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the model registry generation script and introduces a new 'Audio Scene Classification' category, adding the AST and YAMNet models. Feedback identifies that YAMNet is incompatible with the current transformers-based API and suggests correcting a model link in the documentation for consistency.
| - name: YAMNet | ||
| task: audio_scene_classification | ||
| source: huggingface | ||
| model_id: google/yamnet | ||
| parameters: 3.2M | ||
| training_data: AudioSet (521 classes) | ||
| recommended_for: Lightweight audio scene classification (TensorFlow-based) |
There was a problem hiding this comment.
The google/yamnet model is a TensorFlow-based model and is not compatible with the transformers library. Since senselab currently uses HuggingFaceAudioClassifier.classify_audios_with_transformers for models with the huggingface source (as seen in src/senselab/audio/tasks/classification/api.py), this model will fail to load or run. Unless a TensorFlow backend is implemented, it should not be listed as a supported model in the registry.
|
|
||
| **Auditory Scene / Sound Event Classification:** | ||
| - [Audio Spectrogram Transformer (AST)](https://huggingface.co/MIT/ast-finetuned-audioset-10-10-0.4593) — 521 AudioSet classes (speech, music, environmental sounds, animals, etc.). Recommended for general-purpose scene analysis. | ||
| - [YAMNet](https://huggingface.co/google/yamnet) — Google's AudioSet classifier (521 classes). TensorFlow-based. |
There was a problem hiding this comment.
Listing YAMNet here as a supported model might be misleading if it cannot be used with the classify_audios API due to its TensorFlow dependency and lack of transformers compatibility. Consider removing it or adding a note that it is not yet supported via the senselab API.
| - [Adult vs. Child Speech](https://huggingface.co/bookbot/wav2vec2-adult-child-cls) | ||
|
|
||
| **Content:** | ||
| - [Music Genre](https://huggingface.co/agercas/distilhubert-finetuned-gtzan/blob/main/config.json) |
There was a problem hiding this comment.
The link for the Music Genre model points to the config.json file. It is better to point to the main model card page for consistency.
| - [Music Genre](https://huggingface.co/agercas/distilhubert-finetuned-gtzan/blob/main/config.json) | |
| - [Music Genre](https://huggingface.co/agercas/distilhubert-finetuned-gtzan) |
Without __all__, pdoc shows internal submodules (api, huggingface, etc.) as navigation items. With __all__, pdoc only shows the public API functions — what users actually need. Applied to 21 task modules + 1 workflow module across audio, video, text, and utils. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- YAMNet: noted as TensorFlow-based, not directly supported via classify_audios - Music genre model link: config.json → main model card - Model registry: YAMNet source changed from huggingface to tensorflow Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tion - YAMNet runs in isolated TF subprocess venv (tensorflow, tensorflow-hub) - Accessible via classify_audios(audios, model="yamnet") - Auto-resamples to 16kHz inside the loop (memory-efficient) - HF classifier now auto-resamples instead of erroring on wrong sample rate - Updated doc.md and model_registry to reflect YAMNet as fully supported Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
satra
changed the title
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 participant
Summary
yamnet.py)classify_audios(audios, model="yamnet")to the subprocess backend__all__to all task/workflow modules for clean pdoc navigationhf_hub_downloadstr cast)Test plan
scene_results_to_segmentsworks with YAMNet output🤖 Generated with Claude Code
Version
Published prerelease version:
1.3.1-alpha.26Changelog
🐛 Bug Fix
Authors: 1