This document describes the comprehensive Prometheus metrics and Grafana dashboard for monitoring vector search performance across all layers of the read path.
The vector search read path instrumentation provides end-to-end latency, throughput, and error monitoring across:
- HerdDB server query execution
- Index server search operations
- File server reads (disk and S3)
- Client-side read operations
Scope: tablespace_<tablespace>_vector_query_*
| Metric | Type | Description |
|---|---|---|
vector_query_latency |
OpStatsLogger | Query execution latency (p50, p95, p99, p999) |
vector_query_requests |
Counter | Total query requests |
vector_query_errors |
Counter | Failed queries |
Location: herddb-core/src/main/java/herddb/index/vector/VectorIndexManager.java
Scope: indexing_*
| Metric | Type | Description |
|---|---|---|
indexing_search_latency |
OpStatsLogger | Search RPC latency (p50, p95, p99, p999) |
indexing_search_requests |
Counter | Total search requests |
indexing_search_errors |
Counter | Failed searches |
indexing_search_bytes |
Counter | Response size in bytes |
Location: herddb-indexing-service/src/main/java/herddb/indexing/IndexingServiceImpl.java
Scope: rfs_readrange_*
| Metric | Type | Description |
|---|---|---|
rfs_readrange_latency |
OpStatsLogger | Read latency (p50, p95, p99, p999) |
rfs_readrange_requests |
Counter | Total range read requests |
rfs_readrange_bytes |
Counter | Bytes read |
Location: herddb-remote-file-service/src/main/java/herddb/remote/RemoteFileServiceImpl.java (existing)
Scope: rfs_client_read_*
| Metric | Type | Description |
|---|---|---|
rfs_client_read_latency |
OpStatsLogger | Client read latency (p50, p95, p99, p999) |
rfs_client_read_requests |
Counter | Total client read requests |
rfs_client_read_bytes |
Counter | Bytes read |
Location: herddb-remote-file-service/src/main/java/herddb/remote/RemoteRandomAccessReader.java
Scope: rfs_disk_read_*
| Metric | Type | Description |
|---|---|---|
rfs_disk_read_latency |
OpStatsLogger | Disk read latency (p50, p95, p99, p999) |
rfs_disk_read_requests |
Counter | Total disk read requests |
rfs_disk_read_bytes |
Counter | Bytes read from disk |
Location: herddb-remote-file-service/src/main/java/herddb/remote/storage/LocalObjectStorage.java
Scope: rfs_s3_read_*
| Metric | Type | Description |
|---|---|---|
rfs_s3_read_latency |
OpStatsLogger | S3 read latency (p50, p95, p99, p999) |
rfs_s3_read_requests |
Counter | Total S3 read requests |
rfs_s3_read_bytes |
Counter | Bytes read from S3 |
Location: herddb-remote-file-service/src/main/java/herddb/remote/storage/S3ObjectStorage.java
File: herddb-kubernetes/src/main/helm/herddb/monitor/grafana/dashboards/vector-search-latency-dashboard.json
This comprehensive dashboard provides 6 rows of monitoring covering the complete vector search read path:
- Vector Query Execution (HerdDB Server) - Query latency, rate, errors
- Index Server Search (Indexing Service) - Search latency, rate, response throughput
- File Server Client Reads - Client latency, rate, throughput
- File Server Range Reads - Server-side latency, rate, throughput
- File Server Disk Reads - Disk latency, rate, throughput
- File Server S3 Reads - S3 latency, rate, throughput
Each row includes:
- Latency panel with p50, p95, p99, p99.9 quantiles and average
- Rate panel showing operations/second
- Throughput panel showing bytes/second
The metrics can also be integrated into existing dashboards:
- herddb-dashboard.json: Add vector query execution metrics as a new row
- indexing-service-dashboard.json: Add search bytes counter to the "Search RPC" row
- remote-file-service-dashboard.json: Add disk/S3 read metrics as new rows
See the metric names and scopes above to add relevant panels to these dashboards.
Each service exposes metrics on its HTTP metrics endpoint:
# HerdDB Server metrics
curl http://localhost:9845/metrics | grep vector_query
# Indexing Service metrics
curl http://localhost:9851/metrics | grep indexing_search
# File Server metrics
curl http://localhost:9847/metrics | grep rfs_Configure Prometheus to scrape the /metrics endpoints:
scrape_configs:
- job_name: 'herddb-server'
static_configs:
- targets: ['localhost:9845']
- job_name: 'indexing-service'
static_configs:
- targets: ['localhost:9851']
- job_name: 'file-server'
static_configs:
- targets: ['localhost:9847']Open the "HerdDB Vector Search Latency" dashboard in Grafana to view all metrics in one place.
Run the included unit tests:
# Test storage layer metrics (disk/S3 reads)
mvn -pl herddb-remote-file-service -Dtest=RemoteFileServiceStorageMetricsTest test
# Test IndexingService search bytes counter
mvn -pl herddb-indexing-service -Dtest=IndexingServiceSearchBytesTest testAll metric instrumentation is backward compatible:
- StatsLogger parameters are optional (nullable)
- Constructor overloads provided for existing callers
- No breaking changes to public APIs
Storage layer reads (disk and S3) use async APIs (CompletableFuture):
- Metrics are recorded via CompletionHandlers
- Success and failure paths are tracked separately
- Latency includes full async operation time
BookKeeper OpStatsLogger provides:
- p50 (median)
- p95 (90th percentile proxy)
- p99 (99th percentile)
- p999 (99.9th percentile / "pmax" proxy)
Plus computed average: sum(rate(metric_sum[1m])) / sum(rate(metric_count[1m]))
- Add per-tablespace latency metrics
- Add per-vector-index granularity
- Integrate metrics into existing dashboards
- Add alerting rules for latency thresholds
- Add more test coverage for edge cases
- Prometheus Metrics:
/metricsHTTP endpoint - BookKeeper StatsLogger:
org.apache.bookkeeper.stats - Grafana Dashboards:
herddb-kubernetes/src/main/helm/herddb/monitor/grafana/dashboards/