description	Metrics, logs, traces, and alerts for production Spice.ai Enterprise deployments.
icon	chart-line

Observability

Spice.ai Enterprise exposes a comprehensive set of metrics, structured logs, and OpenTelemetry traces. This page documents how to wire each into a production observability stack and which signals to alert on.

Metrics

The runtime exposes Prometheus metrics on port 9090 at /metrics. The Helm chart and the Kubernetes operator both ship with first-class scrape integration:

Helm chart: set monitoring.podMonitor.enabled: true to deploy a PodMonitor.
Operator: set servicemonitor.enabled: true to deploy a ServiceMonitor for the operator itself.

# values.yaml (Spice Helm chart)
monitoring:
  podMonitor:
    enabled: true
    additionalLabels:
      release: prometheus

Key metrics

Metric names are exported verbatim, without a namespace prefix and without unit or counter suffixes. Durations are milliseconds, and the unit is part of the name.

Metric	Type	Labels	Meaning
`query_duration_ms`	Histogram		End-to-end query latency, in milliseconds.
`query_execution_duration_ms`	Histogram		Query execution latency, excluding planning.
`query_executions`	Counter		Queries executed. The denominator for the error rate alert.
`query_failures`	Counter	`err_code`	Failed queries, by error code.
`dataset_acceleration_refresh_duration_ms`	Histogram	`dataset`	Per-dataset acceleration refresh latency.
`dataset_acceleration_refresh_errors`	Counter	`dataset`	Refresh failures per dataset.
`dataset_acceleration_last_refresh_unix_time_ms`	Gauge	`dataset`	When the last refresh completed, as a Unix timestamp in milliseconds.
`dataset_acceleration_size_bytes`	Gauge	`dataset`	Size of the accelerated table storage.
`http_requests_duration_ms`	Histogram		HTTP API latency.
`flight_request_duration_ms`	Histogram		Arrow Flight RPC latency.
`process_resident_memory_bytes`	Gauge		Resident set size of the `spiced` process.
`query_memory_pool_used_bytes`	Gauge		Memory currently held by the query memory pool.
`cayenne_compaction_memory_pool_bytes`	Gauge		Size of the dedicated compaction memory pool, when one is reserved.
`cayenne_compaction_memory_pool_used_bytes`	Gauge		Memory currently held by the compaction memory pool.
`cayenne_compaction_memory_exhausted_total`	Counter	`table`, `kind`	Compaction passes that could not reserve memory from the compaction pool.
`spiced_cpu_budget_cores`	Gauge	`source`	Cores the runtime sized itself for, and where that value came from. See CPU sizing.
`scheduler_active_executors_count`	Gauge	`node_id`	(`SpicepodCluster`) Executors registered with the scheduler.
`scheduler_job_queue_depth`	Gauge	`node_id`	(`SpicepodCluster`) Queued jobs awaiting scheduling.
`spiceai_operator_reconcile_duration_seconds`	Histogram	`controller`	Operator reconcile loop latency.
`spiceai_operator_cluster_ca_expiry_timestamp_seconds`	Gauge	`cluster`	Cluster CA certificate expiry, as a Unix timestamp in seconds.

The spiced_cpu_budget_cores source label reports how the entitlement was determined: configured, cgroup_quota, affinity, or fallback.

For the full list, query the running runtime: curl localhost:9090/metrics | grep -E '^# HELP'.

{% hint style="info" %} Query metrics are reported regardless of the runtime.task_history setting, so disabling task history does not disable the query counters and histograms above. {% endhint %}

Memory pools

Memory budgets are derived from the running process's own cgroup limit, so a container sees the memory it was allocated rather than the memory of its host.

A dedicated compaction memory pool is reserved only for Cayenne accelerations that can compact into it — file mode with a small-write refresh profile. Other deployments, including refresh_mode: full, keep the whole memory limit available to queries.

Compare query_memory_pool_used_bytes against the memory limit to see query headroom. A rising cayenne_compaction_memory_exhausted_total means compaction cannot reserve the memory its rewrite working set needs; raise the memory limit for the deployment.

When a memory pool refuses a reservation, the query fails with the ResourcesExhausted error code, the HTTP API returns 503 Service Unavailable, and Arrow Flight reports RESOURCE_EXHAUSTED. Both are retriable, so pool exhaustion stays distinguishable from a rejected query, which returns 400. Alert on it through the err_code label:

sum(rate(query_failures{err_code="ResourcesExhausted"}[5m])) > 0

Cache metrics

Each cache the runtime maintains exports its own metric family. The SQL results cache uses the results_ prefix; the search results and embeddings caches use search_results_ and embeddings_ and expose the same metric names.

Metric	Type	Labels	Meaning
`results_cache_requests`	Counter		Lookups against the cache.
`results_cache_hits`	Counter		Lookups served from the cache.
`results_cache_misses`	Counter		Lookups not served from the cache.
`results_cache_hit_ratio`	Gauge		Hits divided by total requests.
`results_cache_items_count`	Gauge		Entries currently held.
`results_cache_size_bytes`	Gauge		Size of the cache in bytes.
`results_cache_max_size_bytes`	Gauge		Configured `max_size`, in bytes.
`results_cache_evictions`	Counter	`reason`	Entries removed from the cache, by cause.
`results_cache_stale_rejections`	Counter		Lookups that found an entry but did not serve it, because a table it read was invalidated first. Also counted as misses.
`results_cache_stale_swr_count`	Counter		Stale-while-revalidate refreshes skipped because a revalidation was already in flight.
`results_cache_swr_background_query_count`	Counter		Background queries started for stale-while-revalidate refreshes.

The reason label on *_cache_evictions separates three causes that call for different responses:

`reason`	Cause	Response
`size`	The cache exceeded `max_size` and reclaimed an entry.	Raise `max_size` if the hit ratio is also falling.
`expired`	The entry outlived `item_ttl`.	Expected. Raise `item_ttl` only if the data tolerates a longer staleness window.
`invalidated`	A refresh or a DML write dropped the entries referencing a table.	Expected on an accelerated dataset with a periodic refresh.

On an accelerated dataset with a periodic refresh, invalidated is normally the dominant cause and would swamp an unlabelled total. Alert on size instead, which is the reason that indicates real cache pressure:

sum(rate(results_cache_evictions{reason="size"}[5m])) > 0

{% hint style="info" %} Cache counters are published at zero when the runtime starts, so each series exists before it first increments. A series that is absent altogether therefore indicates a scrape or configuration problem rather than an idle cache. {% endhint %}

Cayenne segment cache

Cayenne accelerations read through a segment cache, sized per dataset by the cayenne_segment_cache_mb acceleration parameter (default 256). Every series carries a dataset label.

Metric	Type	Labels	Meaning
`cayenne_segment_cache_accesses`	Counter	`dataset`	Segment cache lookups.
`cayenne_segment_cache_hits`	Counter	`dataset`	Lookups served from the cache.
`cayenne_segment_cache_entries`	Gauge	`dataset`	Approximate number of entries held.
`cayenne_segment_cache_weighted_bytes`	Gauge	`dataset`	Approximate size of the live cache, in bytes.
`cayenne_segment_cache_capacity_bytes`	Gauge	`dataset`	Configured capacity, in bytes.

The accesses and hits series are cumulative counters, so read them with rate() or increase() rather than as instantaneous values. A hit ratio that falls while cayenne_segment_cache_weighted_bytes sits at cayenne_segment_cache_capacity_bytes means the working set no longer fits; raise cayenne_segment_cache_mb for that dataset.

Grafana dashboard

Spice.ai publishes a maintained Grafana dashboard with the panels operations teams need most often (query rate / latency / errors, acceleration freshness and row counts, executor registration, certificate expiry).

Import via dashboard ID or copy the JSON from the Spice.ai Grafana dashboard. The dashboard is compatible with Prometheus, Amazon Managed Prometheus, Azure Managed Prometheus, and Google Cloud Managed Service for Prometheus.

Logs

Spice emits structured JSON logs on stdout. The log level is controlled by SPICED_LOG:

Level	Use
`ERROR`	Production default.
`WARN`	Production with elevated visibility.
`INFO`	Default during cutover and incident investigation.
`DEBUG`	Development and targeted debugging only \u2014 not for production.

spec:
  env:
    - name: SPICED_LOG
      value: INFO

Log routing

Destination	Recommended forwarder
CloudWatch Logs	Fluent Bit DaemonSet with the `cloudwatch_logs` output plugin.
Azure Monitor	Container insights.
Google Cloud Logging	The GKE-managed logging agent (default on Autopilot / Standard).
Datadog	Datadog Agent with the Kubernetes integration enabled.
Elastic / OpenSearch	Filebeat or Fluent Bit with the `elasticsearch` / `opensearch` output.
Loki	Grafana Alloy or Promtail.

Always retain query-error and acceleration-refresh-failure log lines for at least 30 days for incident review.

Distributed tracing

When the runtime is started with --otel-endpoint or the SPICED_OTEL_ENDPOINT environment variable, Spice exports OpenTelemetry traces over OTLP/gRPC. Traces cover query parsing, optimization, and execution; for SpicepodCluster, scheduler-to-executor RPCs are linked into a single trace.

spec:
  env:
    - name: SPICED_OTEL_ENDPOINT
      value: http://otel-collector.observability:4317
    - name: SPICED_OTEL_RESOURCE_ATTRIBUTES
      value: service.name=spiceai,deployment.environment=prod

Pair with an OpenTelemetry Collector configured for the organization's tracing backend (Tempo, Jaeger, Datadog, Honeycomb, X-Ray).

Alerts

The following alerts are the minimum recommended set for production. Tune thresholds against the deployment's observed baseline.

groups:
  - name: spiceai
    interval: 30s
    rules:
      - alert: SpiceAIQueryErrorRateHigh
        expr: |
          sum(rate(query_failures[5m]))
            /
          sum(rate(query_executions[5m])) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Query error rate above 5% for 10 minutes."

      - alert: SpiceAIAccelerationRefreshFailing
        expr: |
          increase(dataset_acceleration_refresh_errors[15m]) > 3
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Acceleration refresh failing for {{ $labels.dataset }}."

      - alert: SpiceAIQueryLatencyP95
        expr: |
          histogram_quantile(0.95, sum by (le) (
            rate(query_duration_ms_bucket[5m])
          )) > 2000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 query latency above 2s."

      - alert: SpiceAIExecutorMissing
        expr: |
          scheduler_active_executors_count < 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Scheduler {{ $labels.node_id }} has fewer than 2 executors registered."

      - alert: SpiceAIClusterCAExpiringSoon
        expr: |
          spiceai_operator_cluster_ca_expiry_timestamp_seconds - time() < 7 * 24 * 3600
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.cluster }} CA certificate expires in less than 7 days."

The latency threshold is in milliseconds, matching the query_duration_ms histogram.

{% hint style="info" %} spiceai_operator_cluster_ca_expiry_timestamp_seconds is emitted by the Kubernetes Operator, not the runtime, so this alert requires the operator to be scraped as well. See Operator Metrics. {% endhint %}

Crashloop protection is not exposed as a metric. The operator records it on the resource as a pause reason and emits Kubernetes events \u2014 see Crashloop Protection. Alert on container restarts using the kube_pod_container_status_restarts_total series from kube-state-metrics.

Health endpoints

Endpoint	Purpose
`GET /health`	Liveness. Returns `200 OK` when the process is responsive. Use for container liveness probes.
`GET /v1/ready`	Readiness. Returns `200 OK` only once datasets and accelerations have completed initial load. Use for container readiness probes and load balancer health checks.
`GET /metrics`	Prometheus metrics on port `9090`. Should never be exposed externally.

The Spice Helm chart and the operator configure liveness and readiness probes against these endpoints by default. Tune the probe parameters on heavy initial-load workloads via probes:

spec:
  probes:
    readiness:
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 6
    liveness:
      initialDelaySeconds: 60
      periodSeconds: 30
      failureThreshold: 5

Gating readiness on executor availability (scheduler role)

In distributed (SpicepodCluster) deployments a scheduler can finish loading its own datasets and accelerations before enough executors have connected to actually serve queries. To keep a scheduler out of rotation until the cluster has capacity, GET /v1/ready accepts two optional executor gates. They apply only to the scheduler role; supplying a non-zero value on a non-scheduler node returns 400.

Query parameter	Description
`min_ready_executors`	Minimum number of currently-ready executors required for the probe to succeed. "Ready" means the scheduler holds a live FlightSQL client for the executor (i.e. it can route queries to it). `0` disables the gate.
`min_ready_executors_percent`	Minimum percentage (`0`–`100`) of ready executors relative to the executors currently registered (control stream open). `0` disables the gate. Values above `100` return `400`.
`verbose`	When `true`, the response body becomes a multi-line diagnostic listing the result of each gate. The HTTP status code is unchanged — useful for `kubectl describe` and `curl` debugging.

When both gates are supplied, both must pass for the probe to return 200 OK; otherwise the endpoint returns 503 (gate not yet satisfied) or 400 (invalid parameter, e.g. a percentage outside 0–100, or a non-zero gate requested outside scheduler role). Datasets and accelerations must still have completed their initial load regardless of executor gating.

Pass the gates as query parameters on the readiness probe path:

readinessProbe:
  httpGet:
    path: /v1/ready?min_ready_executors=3&min_ready_executors_percent=80
    port: 8090

Navigation Menu

Uh oh!

Observability

Metrics

Key metrics

Memory pools

Cache metrics

Cayenne segment cache

Grafana dashboard

Logs

Log routing

Distributed tracing

Alerts

Health endpoints

Gating readiness on executor availability (scheduler role)

Uh oh!

FilesExpand file tree

observability.md

Latest commit

History

observability.md

File metadata and controls

Observability

Metrics

Key metrics

Memory pools

Cache metrics

Cayenne segment cache

Grafana dashboard

Logs

Log routing

Distributed tracing

Alerts

Health endpoints

Gating readiness on executor availability (scheduler role)