Skip to content

INDEXING_GUIDE.md

Latest commit

 

History

History
277 lines (208 loc) · 9.66 KB

INDEXING_GUIDE.md

File metadata and controls

277 lines (208 loc) · 9.66 KB
Raw

Music Taster Agent - Indexing Configuration Guide

Overview

This document explains the Azure AI Search indexing setup for the music-taster-agent project, covering the index schema, vector search profiles, skillsets, and PDF ingestion flow.

Index Configuration

File: api/music_taster_agent/config/index.json

The index ks-azureblob-index is configured for knowledge source ks-azureblob.

Number of Indexes Configured

1 index is defined in this file: ks-azureblob-index

Fields Overview

The index contains 6 fields:

Field Type Description Searchable Filterable Sortable
uid String Primary key; uses keyword analyzer for exact matching Yes No Yes
snippet_parent_id String Groups related snippets from same page; filter-only No Yes No
blob_url String Source blob URL for reference; filter-only No Yes No
snippet String Full text content (page or image description); full-text searchable Yes No No
image_snippet_parent_id String Groups snippets from images; filter-only No Yes No
snippet_vector Collection(Single) Vector embeddings (3072 dimensions) for semantic search Yes No No

Vector Search Configuration

Vector Search Profile: ks-azureblob-vector-search-profile

A vector search profile is a combination of 3 properties:

  1. Algorithm – ks-azureblob-vector-search-algorithm
  2. Vectorizer – ks-azureblob-vectorizer
  3. Compression – ks-azureblob-vector-search-scalar-quantization

HNSW Algorithm

HNSW stands for Hierarchical Navigable Small World.

  • Purpose: Graph-based Approximate Nearest Neighbor (ANN) algorithm for efficient vector similarity search.
  • How it works: Builds multi-layer small-world graphs where nearest neighbors can be reached with few hops.
  • Distance metric: Uses cosine similarity (configured in this setup).
  • Tunable parameters:
    • m: 4 – Graph connectivity (edges per node); lower = smaller index, faster insertion; higher = better recall.
    • efConstruction: 400 – Accuracy/speed tradeoff during index build.
    • efSearch: 500 – Recall/latency tradeoff during queries.
  • Use case: Excellent balance of recall and latency for high-dimensional embeddings like the 3072-dimensional vectors in this index.

Vectorizer: Azure OpenAI

  • Service: text-embedding-3-large
  • Endpoint: https://foundry-ge54qtofw3zem.openai.azure.com
  • Deployment: text-embedding-3-large
  • Output dimensions: 3072

Compression: Scalar Quantization

  • Type: int8 scalar quantization
  • Purpose: Reduces storage and memory footprint while maintaining recall.
  • Rescoring: Enabled with 4x oversampling to preserve accuracy.

Similarity Scoring

  • Default: BM25 Similarity (for traditional full-text search on snippet field)

Semantic Configuration

  • Semantic search enabled on snippet field
  • Ranking order: BoostedRerankerScore
  • Uses semantic reranking to improve relevance beyond keyword/vector search

Skillset Configuration

File: api/music_taster_agent/config/skillset.json

The skillset ks-azureblob-skillset handles document enrichment before indexing.

Skills Pipeline

1. SplitSkill – Text Chunking

  • Purpose: Split PDF document content into manageable chunks
  • Split mode: Pages (character-based)
  • Max chunk size: 2,000 characters
  • Overlap: 200 characters (for context continuity)
  • Input: Document text content
  • Output: pages array with chunked text

2. AzureOpenAIEmbeddingSkill – Text Vectorization

  • Purpose: Generate vector embeddings for each text chunk
  • Model: text-embedding-3-large
  • Dimensions: 3,072
  • Context: Applied to each page (/document/pages/*)
  • Output: text_vector for each page

3. GenAISkill – Image Verbalization

  • Purpose: Generate concise text descriptions of images, figures, diagrams, charts
  • Model: GPT-4.1 (Chat Completion)
  • Context: Applied to each extracted image (/document/normalized_images/*)
  • System prompt: "You are tasked with generating concise, accurate descriptions of images, figures, diagrams, or charts in documents."
  • Output: verbalizedImage text description

4. VerbalizedImageAzureOpenAIEmbeddingSkill – Image Vector Embedding

  • Purpose: Vectorize image descriptions
  • Model: text-embedding-3-large
  • Dimensions: 3,072
  • Context: Applied to verbalized images
  • Output: verbalizedImage_vector

PDF Ingestion & Indexing Flow

Step-by-Step Process

1. PDF Blob Ingestion
   ├─ Extract text content
   └─ Extract and normalize images

2. Text Processing
   ├─ SplitSkill: Chunk text into pages (2k chars, 200 overlap)
   └─ AzureOpenAIEmbeddingSkill: Generate text_vector (3072d)

3. Image Processing
   ├─ GenAISkill: Describe images → verbalizedImage
   └─ VerbalizedImageAzureOpenAIEmbeddingSkill: Generate verbalizedImage_vector (3072d)

4. Index Projection
   ├─ Pages selector: Maps page data to index fields
   │  ├─ snippet ← page text
   │  ├─ snippet_vector ← text_vector
   │  ├─ blob_url ← source URL
   │  └─ snippet_parent_id ← parent document ID
   │
   └─ Images selector: Maps image data to index fields
      ├─ snippet ← verbalizedImage description
      ├─ snippet_vector ← verbalizedImage_vector
      ├─ blob_url ← source URL
      └─ image_snippet_parent_id ← parent document ID

5. Index Storage (skipIndexingParentDocuments mode)
   └─ Only child snippets indexed, parent document excluded

Index Projections Configuration

Page Snippets Selector

  • Source context: /document/pages/*
  • Target index: ks-azureblob-index
  • Parent key field: snippet_parent_id
  • Mappings:
    • snippet_vector ← /document/pages/*/text_vector
    • snippet ← /document/pages/* (page text)
    • blob_url ← /document/blob_url

Image Snippets Selector

  • Source context: /document/normalized_images/*
  • Target index: ks-azureblob-index
  • Parent key field: image_snippet_parent_id
  • Mappings:
    • snippet_vector ← /document/normalized_images/*/verbalizedImage_vector
    • snippet ← /document/normalized_images/*/verbalizedImage (description)
    • blob_url ← /document/blob_url

Projection Mode

  • Mode: skipIndexingParentDocuments
  • Effect: Only child snippets (pages and images) are indexed; parent document record is not created

Architecture Diagram

PDF Blob Input
│
├─── Text Extraction
│    ├─ SplitSkill (2k char chunks, 200 overlap)
│    │  └─ Output: /document/pages/*
│    │
│    └─ AzureOpenAIEmbeddingSkill
│       └─ Output: /document/pages/*/text_vector (3072d)
│
└─── Image Extraction
     ├─ GenAISkill (describe image)
     │  └─ Output: /document/normalized_images/*/verbalizedImage
     │
     └─ VerbalizedImageAzureOpenAIEmbeddingSkill
        └─ Output: /document/normalized_images/*/verbalizedImage_vector (3072d)

Index Projections (skipIndexingParentDocuments)
│
├─── Pages Selector
│    └─ Index Fields:
│       ├─ snippet (page text)
│       ├─ snippet_vector (text_vector)
│       ├─ blob_url
│       └─ snippet_parent_id (grouping key)
│
└─── Images Selector
     └─ Index Fields:
        ├─ snippet (verbalizedImage description)
        ├─ snippet_vector (verbalizedImage_vector)
        ├─ blob_url
        └─ image_snippet_parent_id (grouping key)

Search Index: ks-azureblob-index
│
├─ Full-text search: snippet field (BM25 similarity)
├─ Vector search: snippet_vector field (HNSW + Azure OpenAI vectorizer + int8 compression)
├─ Semantic search: snippet field with reranking
└─ Filtering/grouping: blob_url, snippet_parent_id, image_snippet_parent_id

Search Capabilities

1. Full-Text Search

  • Field: snippet
  • Algorithm: BM25 Similarity
  • Use case: Keyword matching on text and verbalized image descriptions

2. Vector/Semantic Search

  • Field: snippet_vector
  • Algorithm: HNSW (cosine similarity)
  • Vectorizer: Azure OpenAI text-embedding-3-large
  • Compression: int8 scalar quantization with rescoring
  • Use case: Semantic similarity search across pages and image descriptions

3. Semantic Reranking

  • Field: snippet
  • Config: BoostedRerankerScore
  • Use case: Improve relevance of BM25 or vector results using semantic understanding

4. Filtering & Grouping

  • By source: blob_url (retrieve results from specific PDFs)
  • By parent document: snippet_parent_id (pages from same document), image_snippet_parent_id (images from same document)

Summary

Component Value
Index Name ks-azureblob-index
Number of Indexes 1
Number of Fields 6
Vector Dimensions 3,072
Vector Algorithm HNSW (cosine similarity)
Vectorizer Azure OpenAI text-embedding-3-large
Compression int8 scalar quantization
Similarity Scoring BM25 + Semantic Reranking
Skillset Name ks-azureblob-skillset
Key Skills SplitSkill, Text Embedding, Image Verbalization, Image Embedding
Projection Mode Skip parent documents (index only snippets)
Supported Content Text (pages) + Images (verbalized)

Files Referenced

  • Index configuration: api/music_taster_agent/config/index.json
  • Skillset configuration: api/music_taster_agent/config/skillset.json