Gemini Embedding 2 - Demo

Audio-to-Anything: sound → track → Spotify → YouTube

A full-stack demo showing how Google Gemini Embedding 2 audio embeddings work in practice: upload or record a sound, and the system finds the most similar music tracks by comparing raw audio vectors (not text metadata)

audio query ──► Gemini Embedding 2 ──► Qdrant similarity search ──► top matches
                                                                          │
                                                               ┌──────────┴──────────┐
                                                          Spotify API          YouTube API
                                                         (enrichment)         (enrichment)

Demo

Demo.mp4

Quick Start (5 minutes)
API Key Setup — Google, Spotify, YouTube
Generate a Sample Audio File
Local Development (without Docker)
Docker Compose
How It Works (technical)
Project Structure
Environment Variables
API Endpoints
Tests
PoC Limitations

Quick Start (5 minutes)

Minimum requirements:

Python 3.11+, Node 18+, Docker (for Qdrant only), uv, ffmpeg
A Google API key with Gemini access (get one here) — free tier is sufficient

# 1. Install uv (fast Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Install ffmpeg
brew install ffmpeg          # macOS
# sudo apt install ffmpeg    # Ubuntu/Debian

# 3. Clone the repo
git clone <repo-url>
cd audio-to-anything-demo

# 4. Configure API keys (see section below for how to get them)
cp .env.example .env
# Open .env and set GOOGLE_API_KEY=...

# 5. Start Qdrant
docker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant

# 6. Start the backend
cd backend
uv venv && source .venv/bin/activate
uv pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000 &

# 7. Generate a sample audio file from the dataset
python scripts/save_sample_audio.py
# → saves samples/sample_query.wav

# 8. Start the frontend (separate terminal)
cd ../frontend
npm install && npm run dev

# 9. Open http://localhost:5173
#    → click "Load Dataset into Qdrant" (Sample mode, 30 tracks)
#    → wait for completion (~3-5 min)
#    → upload samples/sample_query.wav and click Search

API Key Setup

Google API Key (required — for embeddings)

Gemini Embedding 2 is the core engine of this demo. Without this key nothing works.

Go to Google AI Studio
Click Create API key
Select an existing Google Cloud project or create a new one
Copy the key and add it to .env:

GOOGLE_API_KEY=AIza...

Free quota: Gemini API free tier supports ~1,500 requests/minute — enough to ingest all 594 tracks in the dataset.

Model used: gemini-embedding-2-preview — natively multimodal (text, images, audio, video mapped into a single 3072-dim vector space). Currently in public preview; the model ID may change when it reaches GA.

Spotify (optional — enrichment)

Only needed to display Spotify cards (album cover, track link) on top matches. The app works fine without it and shows a non-blocking message in the UI.

Go to the Spotify Developer Dashboard
Click Create app
Choose any name; set http://localhost as the Redirect URI
Go to Settings → Client ID and Client Secret
Add to .env:

SPOTIFY_CLIENT_ID=a1b2c3...
SPOTIFY_CLIENT_SECRET=x9y8z7...

Uses the Client Credentials flow — no user login required, public track search only.

YouTube Data API (optional — enrichment)

Only needed to show video embeds and Shorts on top matches. The app works fine without it.

Go to Google Cloud Console
Select (or create) a project
Search for YouTube Data API v3 → Enable
Go to APIs & Services → Credentials → Create Credentials → API key
(Recommended) Add a restriction: YouTube Data API v3
Add to .env:

YOUTUBE_API_KEY=AIza...

Quota: each search.list call costs 100 units; the free tier gives 10,000 units/day (~100 searches). More than enough for demos.

Note: the same Google key used for Gemini embeddings does not work for YouTube — they are separate APIs requiring separate credentials, even within the same GCP project.

Complete `.env` file

# ── Required ──────────────────────────────────────────────────────────────────
GOOGLE_API_KEY=AIza...

# ── Qdrant (leave as-is for local) ───────────────────────────────────────────
QDRANT_HOST=localhost
QDRANT_PORT=6333
QDRANT_COLLECTION=fma_audio

# ── Embedding ─────────────────────────────────────────────────────────────────
EMBEDDING_MODEL=gemini-embedding-2-preview
EMBEDDING_DIM=3072

# ── Ingest ────────────────────────────────────────────────────────────────────
INGEST_SAMPLE_SIZE=30
CHUNK_DURATION_SEC=30

# ── Optional ──────────────────────────────────────────────────────────────────
SPOTIFY_CLIENT_ID=
SPOTIFY_CLIENT_SECRET=
YOUTUBE_API_KEY=

Generate a Sample Audio File

To test search without hunting for an audio file, the script save_sample_audio.py downloads the first track from the dataset and saves the first 10 seconds as a WAV:

cd backend
source .venv/bin/activate
python scripts/save_sample_audio.py

Output:

✓ Saved 10s clip → samples/sample_query.wav
✓ Metadata        → samples/sample_query.json

Track info:
  Title   : ...
  Artist  : ...
  Genre   : dark, cinematic
  Tags    : tense, ambient, suspenseful, ...
  Duration: 10.0s  (sr=22050)

After ingesting the dataset, drag samples/sample_query.wav into the UI uploader and click Search. The source track will appear in the top results with a very high similarity score — because it is literally a fragment of one of the indexed tracks.

The script uses HuggingFace streaming: it downloads only the first track (a few MB), not the full 2 GB dataset.

Local Development (without Docker)

System requirements

Tool	Min version	How to install
Python	3.11	python.org
Node.js	18	nodejs.org
Docker	any	docker.com — Qdrant only
uv	any	`curl -LsSf https://astral.sh/uv/install.sh \| sh`
ffmpeg	any	`brew install ffmpeg` / `apt install ffmpeg`

1. Qdrant

docker run -d --name qdrant -p 6333:6333 -p 6334:6334 qdrant/qdrant

Dashboard: http://localhost:6333/dashboard

2. Backend

cd backend

# Create venv and install dependencies
uv venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
uv pip install -r requirements.txt

# Start (with hot-reload)
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Interactive API docs: http://localhost:8000/docs

3. Frontend

cd frontend
npm install
npm run dev

UI: http://localhost:5173

4. First run: ingest + search

# In a separate terminal, with venv active:
cd backend

# Generate the sample file
python scripts/save_sample_audio.py

# Sample ingest (30 tracks, ~3-5 minutes)
python cli.py ingest --sample

# Full ingest (594 tracks, ~10-15 minutes)
python cli.py ingest --full

# Check status
python cli.py health

Re-running ingest is safe — points are upserted, not duplicated.

Docker Compose

cp .env.example .env
# Fill in API keys in .env (see section above)
# Note: with Docker Compose use QDRANT_HOST=qdrant (not localhost)

docker compose up --build

Services:

The hf_cache volume mounts the HuggingFace cache inside the container, so the dataset is not re-downloaded on every restart.

How It Works (technical)

Why audio embeddings instead of text search?

Most music search systems match on metadata: title, artist, genre tags. That only works if you already know what you are looking for.

This demo does the opposite: it embeds the raw audio signal into a 3072-dimensional vector space using Gemini Embedding 2, a natively multimodal model that maps text, images, audio, and video into a single unified semantic space.

Two audio clips that sound similar — same rhythmic texture, same harmonic mood, same timbre — end up close in this space, regardless of title, artist, or declared genre. You can hum a melody, upload a snippet, or record ambient sound and get musically coherent matches back.

Spotify and YouTube are a second optional enrichment step that fires after the audio match is found.

Full data flow

┌─────────────────────────────────────────────────────────────────┐
│  INGEST (one-time)                                              │
│                                                                 │
│  HuggingFace dataset                                            │
│  atoof/fma-music-descriptions                                   │
│         │                                                       │
│         ▼                                                       │
│  fma_loader.py ──► audio array (numpy)                         │
│         │                                                       │
│         ▼                                                       │
│  audio_utils.py ──► 30s WAV chunks                             │
│         │               (FMA tracks are minutes long;          │
│         │                Gemini hard limit: 80s/file)           │
│         ▼                                                       │
│  embedding_service.py                                           │
│    gemini-embedding-2-preview                                   │
│    Part.from_bytes(wav, "audio/wav")                            │
│         │                                                       │
│         ▼  [3072 floats]                                        │
│  qdrant_service.py ──► upsert point                            │
│    payload: track_id, title, artist,                            │
│             description, tags, genre_tags,                      │
│             chunk_start_sec, chunk_end_sec                      │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  SEARCH (per query)                                             │
│                                                                 │
│  user audio file (upload / microphone)                          │
│         │                                                       │
│         ▼                                                       │
│  audio_utils.ensure_wav() ──► WAV bytes                        │
│         │                                                       │
│         ▼                                                       │
│  embedding_service.embed_audio()                                │
│    → query vector [3072 floats]                                 │
│         │                                                       │
│         ▼                                                       │
│  qdrant_service.search(vector, top_k)                           │
│    → chunk hits with cosine scores                              │
│         │                                                       │
│         ▼                                                       │
│  aggregate by track_id                                          │
│    → top_chunks + top_tracks                                    │
│         │                                                       │
│    ┌────┴────┐                                                  │
│    ▼         ▼                                                  │
│  Spotify   YouTube                                              │
│  search()  search()                                             │
│  title +   title +                                              │
│  artist    artist                                               │
└─────────────────────────────────────────────────────────────────┘

Gemini Embedding 2 — technical details

Parameter	Value
Model ID	`gemini-embedding-2-preview`
Vector dimension	3072 (configurable via `EMBEDDING_DIM`)
Supported audio input	`audio/wav`, `audio/mp3`
Max duration per call	80 seconds
Files per call	1
Python SDK	`google-genai>=1.9.0`
Distance metric in Qdrant	Cosine

# How it is called in embedding_service.py
from google import genai
from google.genai import types

response = client.models.embed_content(
    model="gemini-embedding-2-preview",
    contents=[
        types.Part.from_bytes(data=wav_bytes, mime_type="audio/wav")
    ],
    config=types.EmbedContentConfig(output_dimensionality=3072),
)
vector = response.embeddings[0].values  # list[float], len=3072

Google doc

Dataset

atoof/fma-music-descriptions

594 tracks from the Free Music Archive, CC BY 4.0 license
MP3 audio embedded directly in Parquet — no YouTube downloads, just load_dataset()
Rich metadata per track: title, artist, description (mood/energy/instrumentation), tags (275 unique), genre_tags
Total size ~2.1 GB; in sample mode only the first N tracks are fetched via streaming

Chunking

Many FMA tracks are several minutes long. Since Gemini accepts at most 80 seconds per call, each track is split into CHUNK_DURATION_SEC-second segments (default 30s). Each chunk becomes a separate Qdrant point with chunk_start_sec and chunk_end_sec in its payload. At search time, results are aggregated by track_id, returning both best-chunk and best-track views.

Qdrant document example

Spotify and YouTube enrichment

These are an optional second step that fires after the audio similarity search. We use title + artist from the Qdrant payload to run a text search in each API. This is not audio fingerprinting — it is metadata enrichment layered on top of the audio match.

Project Structure

audio-to-anything-demo/
├── .env.example
├── .gitignore
├── docker-compose.yml
├── README.md
├── samples/                        ← generated by save_sample_audio.py
│   ├── sample_query.wav
│   └── sample_query.json
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── cli.py                      ← CLI: ingest --sample/--full, health
│   ├── scripts/
│   │   └── save_sample_audio.py    ← generates test file from dataset
│   └── app/
│       ├── main.py                 ← FastAPI app, CORS, lifespan
│       ├── config.py               ← pydantic-settings, env vars
│       ├── models.py               ← Pydantic request/response models
│       ├── routers/
│       │   ├── health.py           ← GET /health
│       │   ├── ingest.py           ← POST /ingest, GET /ingest/progress
│       │   ├── search.py           ← POST /search/audio
│       │   ├── spotify.py          ← POST /spotify/search
│       │   └── youtube.py          ← POST /youtube/search
│       ├── services/
│       │   ├── audio_utils.py      ← ensure_wav, chunk_audio, get_duration
│       │   ├── embedding_service.py← Gemini Embedding 2, retry, backoff
│       │   ├── qdrant_service.py   ← collection CRUD, upsert, search
│       │   ├── ingest_service.py   ← orchestration, background thread, progress
│       │   ├── spotify_service.py  ← client credentials, token cache
│       │   └── youtube_service.py  ← Data API v3, Shorts detection
│       ├── dataset_loaders/
│       │   └── fma_loader.py       ← HF streaming, WAV conversion
│       └── tests/
│           ├── test_health.py
│           ├── test_embedding.py
│           └── test_ingest.py
└── frontend/
    ├── Dockerfile
    ├── package.json
    ├── vite.config.js
    ├── index.html
    └── src/
        ├── App.jsx
        ├── App.css
        ├── main.jsx
        ├── api/client.js
        └── components/
            ├── DatasetLoader.jsx   ← sample/full mode, live progress stats
            ├── AudioUploader.jsx   ← drag-and-drop, file input
            ├── Recorder.jsx        ← MediaRecorder API, timer
            ├── SearchResults.jsx   ← tabs: Internal / Spotify / YouTube
            ├── SpotifyResults.jsx  ← album cover, preview audio, Spotify link
            ├── YouTubeResults.jsx  ← video embed, thumbnail, Shorts badge
            ├── ProgressBar.jsx
            └── TryDemoQuery.jsx    ← generates a demo tone in-browser

Environment Variables

Variable	Required	Default	Description
`GOOGLE_API_KEY`	Yes	—	Gemini API key (get one here)
`QDRANT_HOST`	No	`localhost`	Qdrant hostname (`qdrant` with Docker Compose)
`QDRANT_PORT`	No	`6333`	Qdrant port
`QDRANT_COLLECTION`	No	`fma_audio`	Collection name
`EMBEDDING_MODEL`	No	`gemini-embedding-2-preview`	Gemini model ID
`EMBEDDING_DIM`	No	`3072`	Vector dimension (reduce to 768 for speed)
`INGEST_SAMPLE_SIZE`	No	`30`	Tracks in sample mode
`CHUNK_DURATION_SEC`	No	`30`	Seconds per audio chunk
`SPOTIFY_CLIENT_ID`	No	—	Spotify app client ID
`SPOTIFY_CLIENT_SECRET`	No	—	Spotify app client secret
`YOUTUBE_API_KEY`	No	—	YouTube Data API v3 key

API Endpoints

Method	Path	Description
`GET`	`/health`	Backend + Qdrant status
`POST`	`/ingest`	Start ingest `{"sample": true, "sample_size": 30}`
`GET`	`/ingest/progress`	Poll ingest progress
`POST`	`/search/audio`	Upload audio file, returns top-k matches
`POST`	`/spotify/search`	Search Spotify `{"title": "...", "artist": "..."}`
`POST`	`/youtube/search`	Search YouTube `{"title": "...", "artist": "..."}`

Interactive Swagger docs: http://localhost:8000/docs

Tests

cd backend
source .venv/bin/activate
pytest tests/ -v

All tests mock external APIs (Gemini, Qdrant, Spotify, YouTube) — no keys or network required.

PoC Limitations

Gemini Embedding 2 is in public preview (March 2026) — the model ID gemini-embedding-2-preview may change when it reaches GA
Gemini rate limits: ~1,500 req/min on the free tier; full ingest of 594 chunked tracks generates ~1,000+ calls and takes 5–15 minutes
In-memory progress state: restarting the backend resets the progress counter, but Qdrant points persist and upsert is idempotent
Shorts detection: heuristic based on title keywords, not YouTube's official Shorts classification API
Spotify/YouTube enrichment: uses title+artist as a text query, not audio fingerprinting
Long audio: files longer than CHUNK_DURATION_SEC are split into chunks; only the first chunk is used as the search query

Navigation Menu

Gemini Embedding 2 - Demo

Audio-to-Anything: sound → track → Spotify → YouTube

Demo

Table of Contents

Quick Start (5 minutes)

API Key Setup

Google API Key (required — for embeddings)

Spotify (optional — enrichment)

YouTube Data API (optional — enrichment)

Complete `.env` file

Generate a Sample Audio File

Local Development (without Docker)

System requirements

1. Qdrant

2. Backend

3. Frontend

4. First run: ingest + search

Docker Compose

How It Works (technical)

Why audio embeddings instead of text search?

Full data flow

Gemini Embedding 2 — technical details

Dataset

Chunking

Spotify and YouTube enrichment

Project Structure

Environment Variables

API Endpoints

Tests

PoC Limitations

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Gemini Embedding 2 - Demo

Audio-to-Anything: sound → track → Spotify → YouTube

Demo

Table of Contents

Quick Start (5 minutes)

API Key Setup

Google API Key (required — for embeddings)

Spotify (optional — enrichment)

YouTube Data API (optional — enrichment)

Complete .env file

Generate a Sample Audio File

Local Development (without Docker)

System requirements

1. Qdrant

2. Backend

3. Frontend

4. First run: ingest + search

Docker Compose

How It Works (technical)

Why audio embeddings instead of text search?

Full data flow

Gemini Embedding 2 — technical details

Dataset

Chunking

Spotify and YouTube enrichment

Project Structure

Environment Variables

API Endpoints

Tests

PoC Limitations

Complete `.env` file