Skip to content

readme.md

Latest commit

 

History

History
258 lines (182 loc) · 10.1 KB

readme.md

File metadata and controls

258 lines (182 loc) · 10.1 KB
Raw

🧠 Semantic Concept Mapper

PythonLicenseStatusBackend

A high-performance hybrid retrieval + multi-stage re-ranking system for semantic concept mapping over ontology databases.

semantic_concept_mapper

🚀 Overview

This project implements a hybrid retrieval system for ontology concept mapping, based on the design outlined in the official design document: Search Design Document.

It combines lexical and semantic search with multi-stage re-ranking to accurately map input terms to ontology concepts stored in a local SQLite database.

This work is designed to function as a standalone component, but is currently integrated into StructSense and developed as part of BrainKB. If you find this work useful or build upon it, please consider citing:

@misc{chhetri2025structsensetaskagnosticagenticframework,
  title        = {STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking},
  author       = {Tek Raj Chhetri and Yibei Chen and Puja Trivedi and Dorota Jarecka and Saif Haobsh and Patrick Ray and Lydia Ng and Satrajit S. Ghosh},
  year         = {2025},
  eprint       = {2507.03674},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2507.03674}
}

✨ Features

  • Hybrid retrieval (BM25 + dense embeddings)
  • Modular re-ranking pipeline
  • FAISS-backed fast vector search
  • Offline index building support
  • API-first design (FastAPI)
  • Scalable batch concept mapping. It supports a batch request with 4000 concept mapping per requests.

🏗️ Architecture

The figure below illustrates the core architecture of the retrieval and ranking pipeline. It supports multiple retrieval strategies—including keyword-based, dense, hybrid, and ensemble methods—as well as different reranking approaches, such as single, dual (an ensemble of two rerankers), and ensemble rerankers. This flexibility allows users to tailor the system to their specific needs.

Based on our evaluation (to be published in a forthcoming paper), a single LLM-based reranker achieves the best performance.

concept_mapping_search-1

⚡ Quick Start

1. Clone the Repository

git clone https://github.com/sensein/search_hybrid.git
cd search_hybrid

2. Download Data

You must download the Ontology DB, indexes, and embeddings and place them into a .ontology/ directory.

The easiest way to do this is using Git (requires Git LFS):

git clone https://huggingface.co/datasets/sensein/ontology-sqlite-vectorstore .ontology

Alternatively, if you have the huggingface_hub Python package installed, you can use the CLI:

hf download sensein/ontology-sqlite-vectorstore --repo-type dataset --local-dir .ontology

Note: The directory does not have to be named .ontology — use any name you like. Just make sure to update the environment variables accordingly to point to the correct location. If the app doesn't find the indexes, it will run the pipeline to generate them. This is very time-consuming and can take days depending on your system.

Once downloaded, unzip the index files from the embeddings/ subdirectory into your cache directory:

unzip .ontology/embeddings/bm25_indexes-20260310T132934Z-3-001.zip -d .ontology/
unzip .ontology/embeddings/embed_indexes.zip -d .ontology/
unzip .ontology/embeddings/ontology_indexes-20260310T133457Z-3-001.zip -d .ontology/

3. Configure Environment

Copy the example environment file:

cp env.example .env

If you downloaded the data into a directory other than .ontology, update CACHE_ROOT in your .env:

CACHE_ROOT=your-directory-name

Also add that directory to .gitignore to avoid accidentally committing large data files, and to ensure Docker does not copy it into the image during docker compose up:

echo "your-directory-name/" >> .gitignore

4. Choose Your Execution Method

Option A: Docker Deployment (Recommended)

This will automatically install dependencies and start the server in a container.

docker compose up --build

Option B: Local Deployment

If you prefer to run the server directly on your machine without Docker, install the dependencies and run the Uvicorn server:

pip install -r requirements.txt
python -m uvicorn main:app --reload --port 8000

⚙️ Configuration

Retrieval

# Retrieval mode — set both weights to select mode:
#   Hybrid (default):  BM25_WEIGHT=0.3  DENSE_WEIGHT=0.7
#   BM25-only:         BM25_WEIGHT=1.0  DENSE_WEIGHT=0.0
#   Dense-only:        BM25_WEIGHT=0.0  DENSE_WEIGHT=1.0
# A weight of 0.0 skips that retriever entirely (no index built, loaded, or queried).
BM25_WEIGHT=0.3
DENSE_WEIGHT=0.7

EMBEDDING_MODEL=BAAI/bge-small-en-v1.5   # Embedding model (fast, biomedical-friendly)

VECTOR_BACKEND=faiss         # faiss (default) | numpy | chroma
EMBED_CACHE_DIR=.ontology/embed_indexes    # Where .npy and FAISS index are stored
BM25_CACHE_DIR=indexes_embedding/bm25_indexes # BM25 index cache directory
CHROMA_DB_PATH=.ontology/chroma_db        # Only used when VECTOR_BACKEND=chroma

Re-ranking

Below are the available reranking configuration options. Note: If you are not using LLM, you do not require Open Router API key.

You can use either single re-ranker or ensemble.

  • Single:
    • llm — OpenRouter LLM scoring only
    • late_interaction — ColBERT-like token interaction, local
    • biomedical — keyword boost, fastest, fully local
  • Dual (two rerankers, weights auto-normalised):
    • llm_late — LLM + late_interaction (quality + semantics)
    • llm_biomedical — LLM + biomedical (quality + domain boost)
    • dual_late — late_interaction + biomedical (fully local, no API key needed)
  • Triple (all three):
    • ensemble — all three (default, best quality, slowest)
# Single reranker:
RERANKER_TYPE=llm              # OpenRouter API only
RERANKER_TYPE=late_interaction # ColBERT-like, local
RERANKER_TYPE=biomedical       # keyword boost, fastest, local

# Dual ensemble (two rerankers, weights auto-normalised):
RERANKER_TYPE=dual_late        # late_interaction + biomedical  ← default (no API key needed)
RERANKER_TYPE=llm_late         # LLM + late_interaction ← requires API key
RERANKER_TYPE=llm_biomedical   # LLM + biomedical ← requires API key

# Triple ensemble:
RERANKER_TYPE=ensemble         # all three (best quality, slowest)

# Weights (used by whichever components are active; inactive components,e.g., weight 0, are ignored)
LLM_WEIGHT=0.5
LATE_INTERACTION_WEIGHT=0.3
BIOMEDICAL_WEIGHT=0.2

LLM Re-ranker (OpenRouter)

OPENROUTER_API_KEY=sk-or-v1-xxxxx   # Required for LLM reranking
OPENROUTER_MODEL=openrouter/anthropic/claude-3.5-sonnet:beta     # Model selection

# Available models (examples): 
#   google/gemini-2.0-flash-001       (fast, high quality)
#   anthropic/claude-3.5-sonnet:beta  (highest quality)
#   meta-llama/llama-2-70b-chat       (open source)
#   mistral/mistral-7b-instruct:free  (free tier)

Late-Interaction Model

LATE_INTERACTION_MODEL=jinaai/jina-colbert-v2

Performance

MAX_CANDIDATES=20            # Candidates retrieved before re-ranking
MAX_RESULTS=5                # Default max results per query
INDEX_CACHE_DIR=.ontology/ontology_indexes
EMBED_CACHE_DIR=.ontology/embed_indexes
DATABASE_PATH=bioportal.db

📡 API Endpoints

Endpoint Method Purpose
/map/concept POST Map a single term; returns retrieval_scores; uses MAX_CANDIDATES=20
/map/search POST Like /map/concept but uses MAX_CANDIDATES=30 (larger pool); designed for context-heavy disambiguation; no retrieval_scores in response
/map/batch POST Map up to 4000 concepts in one call; accepts comma-string, list, or {text, context} objects
/config GET Current server configuration (reranker type, model, backend — no secrets)
/health GET Readiness check
/stats GET DB + index statistics
/ontologies GET List all available ontologies

When to use which mapping endpoint:

  • /map/concept — default for single-term lookups. Use when you have a clean term ("diabetes mellitus", "COPD") and want retrieval scores for debugging.
  • /map/search — use when you have rich context (clinical note snippet, definition, related terms) and disambiguation matters ("cold" → "common cold" vs "cold weather").
  • /map/batch — use when mapping many terms at once to reduce API round trips. Per-concept context is supported via the {text, context} object format. It supports upto 4000 concepts mapping per request.

🏗️ Indexing

Use build_index.py to generate all indexes before starting the server. Note, building indexes is very time consuming task. You can download it from https://huggingface.co/datasets/sensein/ontology-sqlite-vectorstore and place it on .ontology directory.

# Build with default settings (reads from .env)
python build_index.py

# Explicitly choose backend
python build_index.py --backend faiss    # default
python build_index.py --backend numpy    # no faiss-cpu required

# Force rebuild even if caches exist
python build_index.py --force

# Use a non-default database
python build_index.py --db /data/bioportal.db

📄 License

This project is licensed under the Apache 2.0 License.