Skip to content

Commit 162a69e

Browse files
Claudelukekim
Claude
authored and
lukekim
committed
docs(enterprise): document cache and Cayenne segment cache metrics
1 parent 74bdab3 commit 162a69e

1 file changed

Lines changed: 50 additions & 0 deletions

File tree

‎enterprise/production/observability.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,56 @@ When a memory pool refuses a reservation, the query fails with the `ResourcesExh
7272
sum(rate(query_failures{err_code="ResourcesExhausted"}[5m])) > 0
7373
```
7474

75+
### Cache metrics
76+
77+
Each cache the runtime maintains exports its own metric family. The SQL results cache uses the `results_` prefix; the search results and embeddings caches use `search_results_` and `embeddings_` and expose the same metric names.
78+
79+
| Metric | Type | Labels | Meaning |
80+
| ------------------------------------------ | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
81+
| `results_cache_requests` | Counter | | Lookups against the cache. |
82+
| `results_cache_hits` | Counter | | Lookups served from the cache. |
83+
| `results_cache_misses` | Counter | | Lookups not served from the cache. |
84+
| `results_cache_hit_ratio` | Gauge | | Hits divided by total requests. |
85+
| `results_cache_items_count` | Gauge | | Entries currently held. |
86+
| `results_cache_size_bytes` | Gauge | | Size of the cache in bytes. |
87+
| `results_cache_max_size_bytes` | Gauge | | Configured `max_size`, in bytes. |
88+
| `results_cache_evictions` | Counter | `reason` | Entries removed from the cache, by cause. |
89+
| `results_cache_stale_rejections` | Counter | | Lookups that found an entry but did not serve it, because a table it read was invalidated first. Also counted as misses. |
90+
| `results_cache_stale_swr_count` | Counter | | Stale-while-revalidate refreshes skipped because a revalidation was already in flight. |
91+
| `results_cache_swr_background_query_count` | Counter | | Background queries started for stale-while-revalidate refreshes. |
92+
93+
The `reason` label on `*_cache_evictions` separates three causes that call for different responses:
94+
95+
| `reason` | Cause | Response |
96+
| ------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------------- |
97+
| `size` | The cache exceeded `max_size` and reclaimed an entry. | Raise `max_size` if the hit ratio is also falling. |
98+
| `expired` | The entry outlived `item_ttl`. | Expected. Raise `item_ttl` only if the data tolerates a longer staleness window. |
99+
| `invalidated` | A refresh or a DML write dropped the entries referencing a table. | Expected on an accelerated dataset with a periodic refresh. |
100+
101+
On an accelerated dataset with a periodic refresh, `invalidated` is normally the dominant cause and would swamp an unlabelled total. Alert on `size` instead, which is the reason that indicates real cache pressure:
102+
103+
```
104+
sum(rate(results_cache_evictions{reason="size"}[5m])) > 0
105+
```
106+
107+
{% hint style="info" %}
108+
Cache counters are published at zero when the runtime starts, so each series exists before it first increments. A series that is absent altogether therefore indicates a scrape or configuration problem rather than an idle cache.
109+
{% endhint %}
110+
111+
### Cayenne segment cache
112+
113+
[Cayenne](https://spiceai.org/docs/components/data-accelerators/cayenne) accelerations read through a segment cache, sized per dataset by the `cayenne_segment_cache_mb` acceleration parameter (default `256`). Every series carries a `dataset` label.
114+
115+
| Metric | Type | Labels | Meaning |
116+
| -------------------------------------- | ------- | --------- | -------------------------------------------------- |
117+
| `cayenne_segment_cache_accesses` | Counter | `dataset` | Segment cache lookups. |
118+
| `cayenne_segment_cache_hits` | Counter | `dataset` | Lookups served from the cache. |
119+
| `cayenne_segment_cache_entries` | Gauge | `dataset` | Approximate number of entries held. |
120+
| `cayenne_segment_cache_weighted_bytes` | Gauge | `dataset` | Approximate size of the live cache, in bytes. |
121+
| `cayenne_segment_cache_capacity_bytes` | Gauge | `dataset` | Configured capacity, in bytes. |
122+
123+
The accesses and hits series are cumulative counters, so read them with `rate()` or `increase()` rather than as instantaneous values. A hit ratio that falls while `cayenne_segment_cache_weighted_bytes` sits at `cayenne_segment_cache_capacity_bytes` means the working set no longer fits; raise `cayenne_segment_cache_mb` for that dataset.
124+
75125
## Grafana dashboard
76126

77127
Spice.ai publishes a maintained Grafana dashboard with the panels operations teams need most often (query rate / latency / errors, acceleration freshness and row counts, executor registration, certificate expiry).

0 commit comments

Comments
 (0)