Skip to content

Commit 6f676cc

Browse files
MASSIVEMAGNETICSMASSIVEMAGNETICS
authored
Merge pull request #1 from MASSIVEMAGNETICS/copilot/create-ai-audio-listening-concept
The AI Ear: enterprise-grade multi-modal AI audio listening system
2 parents f471f75 + cc178e9 commit 6f676cc

39 files changed

Lines changed: 6366 additions & 2 deletions

‎.gitignore‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
*.egg
7+
*.egg-info/
8+
dist/
9+
build/
10+
eggs/
11+
.eggs/
12+
lib/
13+
lib64/
14+
parts/
15+
sdist/
16+
var/
17+
wheels/
18+
*.egg-link
19+
.installed.cfg
20+
MANIFEST
21+
22+
# Virtual environments
23+
.venv/
24+
venv/
25+
ENV/
26+
env/
27+
28+
# Testing
29+
.pytest_cache/
30+
.coverage
31+
htmlcov/
32+
.tox/
33+
34+
# IDEs
35+
.idea/
36+
.vscode/
37+
*.swp
38+
*.swo
39+
40+
# OS
41+
.DS_Store
42+
Thumbs.db
43+
44+
# Whisper model cache
45+
~/.cache/whisper/
46+
47+
# Env files (may contain secrets)
48+
.env
49+
.env.*
50+
!.env.example

‎README.md‎

Lines changed: 314 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,314 @@
1-
# the-ai-ear
2-
a ear for ai
1+
# 🎧 The AI Ear
2+
3+
> *"Hear beyond words."*
4+
5+
**The AI Ear** is an outside-the-box, frontier, enterprise-grade AI system that gives machines the ability to **truly hear** — not just transcribe, but holistically understand the acoustic world in real time.
6+
7+
---
8+
9+
## What Makes It Different
10+
11+
Most "AI audio" systems stop at speech-to-text. The AI Ear goes further:
12+
13+
| Capability | What it means |
14+
|---|---|
15+
| **Multi-modal analysis** | Every audio window is simultaneously analysed for speech, emotion, acoustic environment, and music — in parallel |
16+
| **Temporal memory** | The system *remembers* what it has heard, building a rolling semantic context rather than processing isolated moments |
17+
| **Aural events** | State-machine transitions (speech started, music detected, environment changed, alarm sounded) surface as typed events for downstream alerting |
18+
| **LLM-ready context** | One call to `memory.context_summary()` produces a structured dict ready to inject into any LLM system prompt |
19+
| **Enterprise API** | FastAPI REST + WebSocket server with CORS, structured logging, and full OpenAPI docs |
20+
| **Pluggable analysers** | Bring your own model (BYOM) — swap in any `BaseAnalyzer` subclass at construction time |
21+
| **Zero-copy fan-out** | Results and events broadcast concurrently to all registered callbacks |
22+
23+
---
24+
25+
## Architecture
26+
27+
```
28+
┌─────────────────────────────────────────────────────────────────┐
29+
│ The AI Ear │
30+
│ │
31+
│ ┌─────────────┐ ┌──────────────────────────────────────┐ │
32+
│ │AudioListener│───▶│ AudioPipeline │ │
33+
│ │ (mic/file) │ │ ┌──────────┐ ┌──────────────────┐ │ │
34+
│ └─────────────┘ │ │ Speech │ │ Environment │ │ │
35+
│ │ │Analyzer │ │ Analyzer │ │ │
36+
│ ┌─────────────┐ │ │(Whisper) │ │ (heuristic + │ │ │
37+
│ │ REST API │ │ └──────────┘ │ DNN-ready) │ │ │
38+
│ │ /analyse │ │ ┌──────────┐ └──────────────────┘ │ │
39+
│ │ /memory/* │ │ │ Emotion │ ┌──────────────────┐ │ │
40+
│ │ /health │ │ │Analyzer │ │ MusicAnalyzer │ │ │
41+
│ └─────────────┘ │ │(wav2vec2)│ │ (librosa) │ │ │
42+
│ │ └──────────┘ └──────────────────┘ │ │
43+
│ ┌─────────────┐ │ │ concurrent asyncio.gather │ │
44+
│ │ WebSocket │ │ ▼ │ │
45+
│ │ /stream │ │ ┌────────────┐ ┌──────────────┐ │ │
46+
│ └─────────────┘ │ │ Fusion │──▶│ AuralMemory │ │ │
47+
│ │ │(AnalysisRe-│ │(rolling │ │ │
48+
│ │ │ sult + │ │ context + │ │ │
49+
│ │ │ semantic │ │ events) │ │ │
50+
│ │ │ tags) │ └──────────────┘ │ │
51+
│ │ └────────────┘ │ │
52+
│ └──────────────────────────────────────┘ │
53+
└─────────────────────────────────────────────────────────────────┘
54+
```
55+
56+
### Components
57+
58+
| Module | Role |
59+
|---|---|
60+
| `ai_ear/core/listener.py` | Non-blocking audio capture (microphone + file ingestion) |
61+
| `ai_ear/core/pipeline.py` | Concurrent multi-modal analysis engine |
62+
| `ai_ear/core/memory.py` | Temporal context memory with context summary + transcript |
63+
| `ai_ear/core/models.py` | Strongly-typed Pydantic data models for the entire system |
64+
| `ai_ear/core/config.py` | Pydantic-settings configuration (env-var overridable) |
65+
| `ai_ear/analyzers/speech.py` | OpenAI Whisper speech recognition + word timestamps |
66+
| `ai_ear/analyzers/emotion.py` | wav2vec2 speech emotion recognition |
67+
| `ai_ear/analyzers/environment.py` | Acoustic scene classification (silence/speech/music/alarm/crowd/traffic) |
68+
| `ai_ear/analyzers/music.py` | Tempo, key, energy, and genre-hint extraction via librosa |
69+
| `ai_ear/api/server.py` | FastAPI REST + WebSocket API server |
70+
| `ai_ear/utils/audio.py` | Pure-numpy DSP utilities (RMS, ZCR, spectral centroid, flatness) |
71+
72+
---
73+
74+
## Quick Start
75+
76+
### Install
77+
78+
```bash
79+
pip install -e ".[dev]"
80+
```
81+
82+
> **Heavy ML dependencies** (Whisper, PyTorch, transformers, librosa) are listed in
83+
> `requirements.txt` and `pyproject.toml`. For a lightweight evaluation, the
84+
> system degrades gracefully when they are absent — speech/emotion analysers
85+
> return empty results; environment/music analysers use fast numpy heuristics.
86+
87+
### Run the API server
88+
89+
```bash
90+
ai-ear serve --host 0.0.0.0 --port 8080
91+
```
92+
93+
Or programmatically:
94+
95+
```python
96+
import uvicorn
97+
from ai_ear.api.server import create_app
98+
from ai_ear.core.config import Settings
99+
100+
app = create_app(Settings())
101+
uvicorn.run(app, host="0.0.0.0", port=8080)
102+
```
103+
104+
Interactive API docs available at `http://localhost:8080/docs`.
105+
106+
### Analyse a file via REST
107+
108+
```bash
109+
curl -X POST http://localhost:8080/analyse \
110+
-F "file=@interview.wav" | python -m json.tool
111+
```
112+
113+
### Real-time microphone listening (Python)
114+
115+
```python
116+
import asyncio
117+
from ai_ear.core.listener import AudioListener
118+
from ai_ear.core.pipeline import AudioPipeline
119+
from ai_ear.core.memory import AuralMemory
120+
from ai_ear.analyzers.speech import SpeechAnalyzer
121+
from ai_ear.analyzers.emotion import EmotionAnalyzer
122+
from ai_ear.analyzers.environment import EnvironmentAnalyzer
123+
from ai_ear.analyzers.music import MusicAnalyzer
124+
125+
async def main():
126+
memory = AuralMemory()
127+
pipeline = AudioPipeline(
128+
analyzers=[
129+
SpeechAnalyzer(model_size="base"),
130+
EmotionAnalyzer(),
131+
EnvironmentAnalyzer(),
132+
MusicAnalyzer(),
133+
],
134+
memory=memory,
135+
)
136+
137+
async def on_result(result):
138+
print(f"Speech : {result.speech.text if result.speech else '—'}")
139+
print(f"Emotion: {result.emotion.dominant.value if result.emotion else '—'}")
140+
print(f"Tags : {result.semantic_tags}")
141+
142+
pipeline.on_result(on_result)
143+
await pipeline.start()
144+
145+
listener = AudioListener(sample_rate=16_000, chunk_duration_s=2.0)
146+
await listener.start()
147+
148+
try:
149+
await pipeline.process_stream(listener.chunks())
150+
finally:
151+
await listener.stop()
152+
await pipeline.stop()
153+
154+
asyncio.run(main())
155+
```
156+
157+
### WebSocket streaming (JavaScript client)
158+
159+
```javascript
160+
const ws = new WebSocket("ws://localhost:8080/stream");
161+
162+
ws.onopen = () => {
163+
// Send raw Float32 PCM chunks (16 kHz mono) as binary frames
164+
const pcmChunk = new Float32Array(32000); // 2 seconds @ 16 kHz
165+
ws.send(pcmChunk.buffer);
166+
};
167+
168+
ws.onmessage = (event) => {
169+
const result = JSON.parse(event.data);
170+
console.log("Heard:", result.speech?.text);
171+
console.log("Tags:", result.semantic_tags);
172+
};
173+
```
174+
175+
### LLM context injection
176+
177+
```python
178+
from ai_ear.core.memory import AuralMemory
179+
180+
memory: AuralMemory = ... # already receiving results from pipeline
181+
182+
# Inject into any LLM system prompt
183+
context = memory.context_summary(window_s=60)
184+
system_prompt = f"""
185+
You are an AI assistant with real-time acoustic awareness.
186+
You have been listening for the last {context['window_s']:.0f} seconds.
187+
188+
ACOUSTIC CONTEXT:
189+
Transcribed speech: "{context['transcript']}"
190+
Prevailing emotion: {context['dominant_emotions'][0][0] if context['dominant_emotions'] else 'neutral'}
191+
Environment: {context['dominant_environments'][0][0] if context['dominant_environments'] else 'unknown'}
192+
Music detected: {context['music_detected']}
193+
"""
194+
```
195+
196+
---
197+
198+
## Configuration
199+
200+
All settings are configurable via environment variables (prefix `AIEAR_`) or a `.env` file:
201+
202+
| Variable | Default | Description |
203+
|---|---|---|
204+
| `AIEAR_WHISPER_MODEL` | `base` | Whisper model size: `tiny`, `base`, `small`, `medium`, `large` |
205+
| `AIEAR_WHISPER_DEVICE` | `cpu` | PyTorch device: `cpu`, `cuda`, `mps` |
206+
| `AIEAR_WHISPER_LANGUAGE` | *(auto)* | Force language code (e.g. `en`) |
207+
| `AIEAR_EMOTION_ENABLED` | `true` | Enable emotion analysis |
208+
| `AIEAR_MUSIC_ENABLED` | `true` | Enable music analysis |
209+
| `AIEAR_ENVIRONMENT_ENABLED` | `true` | Enable environment classification |
210+
| `AIEAR_AUDIO_SAMPLE_RATE` | `16000` | Capture sample rate (Hz) |
211+
| `AIEAR_AUDIO_CHUNK_DURATION_S` | `2.0` | Analysis window size (seconds) |
212+
| `AIEAR_MEMORY_CONTEXT_WINDOW_S` | `60.0` | Rolling context window (seconds) |
213+
| `AIEAR_API_PORT` | `8080` | API server port |
214+
| `AIEAR_LOG_JSON` | `false` | Emit structured JSON logs |
215+
216+
---
217+
218+
## API Reference
219+
220+
| Method | Path | Description |
221+
|---|---|---|
222+
| `GET` | `/health` | Liveness / readiness probe |
223+
| `GET` | `/info` | Build info and configuration |
224+
| `POST` | `/analyse` | Analyse an uploaded audio file |
225+
| `GET` | `/memory/context` | Structured context summary |
226+
| `GET` | `/memory/transcript` | Plain-text recent speech |
227+
| `GET` | `/memory/events` | Recent aural events |
228+
| `GET` | `/pipeline/stats` | Pipeline throughput statistics |
229+
| `WS` | `/stream` | Real-time PCM audio streaming |
230+
231+
Full interactive docs: `http://localhost:8080/docs`
232+
233+
---
234+
235+
## Bring Your Own Model (BYOM)
236+
237+
```python
238+
from ai_ear.analyzers.base import BaseAnalyzer, SpeechResult
239+
from ai_ear.core.models import AudioChunk, SpeechSegment
240+
241+
class MyKeywordSpotter(BaseAnalyzer):
242+
name = "keyword_spotter"
243+
244+
async def load(self):
245+
# Load your custom model here
246+
self._model = load_my_model()
247+
248+
async def analyse(self, chunk: AudioChunk) -> SpeechResult:
249+
keyword = self._model.detect(chunk.samples)
250+
return SpeechResult(
251+
segment=SpeechSegment(text=keyword or "", confidence=0.95),
252+
confidence=0.95 if keyword else 0.0,
253+
)
254+
255+
# Inject into the pipeline
256+
from ai_ear.core.pipeline import AudioPipeline
257+
pipeline = AudioPipeline(analyzers=[MyKeywordSpotter(), ...])
258+
```
259+
260+
---
261+
262+
## Examples
263+
264+
```bash
265+
# Synthetic demo (no audio hardware required)
266+
python examples/basic_listening.py --demo
267+
268+
# Analyse a real audio file
269+
python examples/basic_listening.py path/to/audio.wav
270+
271+
# Enterprise integration patterns
272+
python examples/enterprise_integration.py custom-analyser
273+
python examples/enterprise_integration.py alerting
274+
python examples/enterprise_integration.py llm-prompt
275+
python examples/enterprise_integration.py serve
276+
```
277+
278+
---
279+
280+
## Testing
281+
282+
```bash
283+
# Run the full test suite
284+
pytest
285+
286+
# With coverage
287+
pytest --cov=ai_ear --cov-report=term-missing
288+
```
289+
290+
---
291+
292+
## Aural Events
293+
294+
The pipeline automatically surfaces discrete events for real-time alerting:
295+
296+
| Event | Description |
297+
|---|---|
298+
| `speech_started` | Voice activity detected |
299+
| `speech_ended` | Voice activity ceased |
300+
| `keyword_detected` | Registered keyword recognised |
301+
| `emotion_shift` | Dominant emotion changed |
302+
| `environment_change` | Acoustic scene changed |
303+
| `music_started` | Music onset detected |
304+
| `music_ended` | Music offset detected |
305+
| `alarm_detected` | Alarm sound detected (high severity) |
306+
| `silence_started` | Silence onset |
307+
| `silence_ended` | Silence offset |
308+
| `anomaly` | Unclassified acoustic anomaly |
309+
310+
---
311+
312+
## License
313+
314+
MIT

0 commit comments

Comments
 (0)