feat(helm)!: Add Presto query engine support to the Helm chart.

docs/src/user-docs/guides-k8s-deployment.md

@@ -298,11 +298,82 @@ helm template clp . -f custom-values.yaml
 
 ::::
 
+### Using Presto as the query engine
+
+To use [Presto][presto-guide] as the query engine instead of the default clp-s query pipeline, set
+`query_engine` to `"presto"` and configure the Presto-specific settings:
+
+```{code-block} yaml
+:caption: presto-values.yaml
+
+image:
+  prestoCoordinator:
+    repository: "ghcr.io/y-scope/presto/coordinator"
+    tag: "dev"
+  prestoWorker:
+    repository: "ghcr.io/y-scope/presto/prestissimo-worker"
+    tag: "dev"
+
+prestoWorker:
+  # See below "Worker scheduling" for more details on configuring Presto scheduling
+  replicas: 2
+
+clpConfig:
+  package:
+    storage_engine: "clp-s"
+    query_engine: "presto"
+
+  # Disable results cache retention since the Presto integration doesn't yet support garbage
+  # collection of search results.
+  results_cache:
+    retention_period: null
+
+  presto:
+    coordinator:
+      logging_level: "INFO"
+      query_max_memory_gb: 1
+      query_max_memory_per_node_gb: 1
+    worker:
+      query_memory_gb: 4
+      system_memory_gb: 8
+    # Split filter config for the Presto CLP connector. For each dataset you want to query, add a
+    # filter entry. Replace <dataset> with the dataset name (use "default" if you didn't specify one
+    # when compressing) and <timestamp-key> with the timestamp key used during compression.
+    # See https://docs.yscope.com/presto/connector/clp.html#split-filter-config-file
+    split_filter:
+      clp.default.<dataset>:
+        - columnName: "<timestamp-key>"
+          customOptions:
+            rangeMapping:
+              lowerBound: "begin_timestamp"
+              upperBound: "end_timestamp"
+          required: false
+```
+
+Install with the Presto values:
+
+```bash
+helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f presto-values.yaml
+```
+
+:::{note}
+When `query_engine` is set to `"presto"`, the chart deploys a Presto coordinator and Presto
+worker(s) instead of the query scheduler, query workers, reducers, and results cache.
+:::
+
+For more details on querying logs through Presto, see the [Using Presto][presto-guide] guide.
+
 ### Worker scheduling
 
 You can control where workers are scheduled using standard Kubernetes scheduling primitives
 (`nodeSelector`, `affinity`, `tolerations`, `topologySpreadConstraints`).
 
+:::{note}
+When using Presto as the query engine, use `prestoWorker:` instead of `queryWorker:` and `reducer:`
+to configure Presto worker scheduling. The `prestoWorker:` key supports the same `scheduling:`
+options.
+:::
+
 #### Dedicated node pools
 
 To run compression workers, query workers, and reducers in separate node pools:
@@ -588,6 +659,7 @@ To tear down a `kubeadm` cluster:
 * [External database setup][external-db-guide]: Using external MariaDB and MongoDB
 * [Using object storage][s3-storage]: Configuring S3 storage
 * [Configuring retention periods][retention-guide]: Setting up data retention policies
+* [Using Presto][presto-guide]: Distributed SQL queries on compressed logs
 
 [aks]: https://azure.microsoft.com/en-us/products/kubernetes-service
 [api-server]: guides-using-the-api-server.md
@@ -605,6 +677,7 @@ To tear down a `kubeadm` cluster:
 [kubeadm]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
 [kubectl]: https://kubernetes.io/docs/tasks/tools/
 [logging-infra-issue]: https://github.com/y-scope/clp/issues/1760
+[presto-guide]: guides-using-presto.md
 [quick-start]: quick-start/index.md
 [retention-guide]: guides-retention.md
 [rfc-1918]: https://datatracker.ietf.org/doc/html/rfc1918#section-3

docs/src/user-docs/guides-using-presto.md

@@ -14,17 +14,89 @@ maintained in a [fork][yscope-presto] of the Presto project. At some point, thes
 been merged into the main Presto repository so that you can use official Presto releases with CLP.
 :::
 
-## Requirements
+## Deployment options
+
+CLP supports Presto through two deployment methods:
+
+* **[Kubernetes (Helm)](#kubernetes-helm)**: Presto is deployed as part of the CLP Helm chart. This
+  is the simplest option if you are already using the [Kubernetes deployment][k8s-deployment].
+* **[Docker Compose](#docker-compose)**: Presto is deployed separately using Docker Compose alongside
+  a CLP package installation.
+
+## Kubernetes (Helm)
+
+When deploying CLP on Kubernetes using Helm, Presto can be enabled by setting the `query_engine` to
+`"presto"` in your Helm values.
+
+### Requirements
+
+* A running CLP Kubernetes deployment (see the [Kubernetes deployment guide][k8s-deployment])
+
+### Set up
+
+1. Create a values file to enable Presto:
+
+   ```{code-block} yaml
+   :caption: presto-values.yaml
+
+   clpConfig:
+     package:
+       query_engine: "presto"
+
+     # Disable results cache retention since the Presto integration doesn't yet support
+     # garbage collection of search results.
+     results_cache:
+       retention_period: null
+
+     presto:
+       # Split filter config for the Presto CLP connector. For each dataset, add a filter entry.
+       # Replace <dataset> with the dataset name (use "default" if you didn't specify one when
+       # compressing) and <timestamp-key> with the timestamp key used during compression.
+       # See https://docs.yscope.com/presto/connector/clp.html#split-filter-config-file
+       split_filter:
+         clp.default.<dataset>:
+           - columnName: "<timestamp-key>"
+             customOptions:
+               rangeMapping:
+                 lowerBound: "begin_timestamp"
+                 upperBound: "end_timestamp"
+             required: false
+   ```
+
+2. Install (or upgrade) the Helm chart with the Presto values:
+
+   ```bash
+   helm install clp clp/clp DOCS_VAR_HELM_VERSION_FLAG -f presto-values.yaml
+   ```
+
+3. Verify that the Presto coordinator and worker pods are running:
+
+   ```bash
+   kubectl get pods -l "app.kubernetes.io/component in (presto-coordinator, presto-worker)"
+   ```
+
+Once the pods are ready, you can [query your logs through Presto](#querying-your-logs-through-presto)
+using CLP's Web UI.
+
+:::{note}
+When using Kubernetes, Presto worker scheduling can be configured using the `prestoWorker.scheduling`
+key in Helm values. See the [worker scheduling][k8s-scheduling] section of the Kubernetes deployment
+guide for details.
+:::
+
+## Docker Compose
+
+### Requirements
 
 * [CLP][clp-releases] (clp-json) v0.5.0 or higher
 * [Docker] v28 or higher
 * [Docker Compose][docker-compose] v2.20.2 or higher
 * Python
 * python3-venv (for the version of Python installed)
 
-## Set up
+### Set up
 
-Using Presto with CLP requires:
+Using Presto with CLP via Docker Compose requires:
 
 * [Setting up CLP](#setting-up-clp) and compressing some logs.
 * [Setting up Presto](#setting-up-presto) to query CLP's metadata database and archives.
@@ -227,6 +299,8 @@ These limitations will be addressed in a future release of the Presto integratio
 [clp-releases]: https://github.com/y-scope/clp/releases
 [docker-compose]: https://docs.docker.com/compose/install/
 [Docker]: https://docs.docker.com/engine/install/
+[k8s-deployment]: guides-k8s-deployment.md
+[k8s-scheduling]: guides-k8s-deployment.md#worker-scheduling
 [postgresql]: https://zenodo.org/records/10516401
 [Presto]: https://prestodb.io/
 [y-scope/presto#8]: https://github.com/y-scope/presto/issues/8

tools/deployment/package-helm/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: "v2"
 name: "clp"
-version: "0.1.4-dev.2"
+version: "0.1.4-dev.4"
 description: "A Helm chart for CLP's (Compressed Log Processor) package deployment"
 type: "application"
 appVersion: "0.9.1-dev"

tools/deployment/package-helm/templates/api-server-deployment.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.clpConfig.api_server }}
+{{- if and .Values.clpConfig.api_server (ne .Values.clpConfig.package.query_engine "presto") }}
 apiVersion: "apps/v1"
 kind: "Deployment"
 metadata:

tools/deployment/package-helm/templates/api-server-logs-pv.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.clpConfig.api_server }}
+{{- if and .Values.clpConfig.api_server (ne .Values.clpConfig.package.query_engine "presto") }}
 {{- include "clp.createStaticPv" (dict
   "root" .
   "component_category" "api-server"

tools/deployment/package-helm/templates/api-server-logs-pvc.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.clpConfig.api_server }}
+{{- if and .Values.clpConfig.api_server (ne .Values.clpConfig.package.query_engine "presto") }}
 {{- include "clp.createPvc" (dict
   "root" .
   "component_category" "api-server"

tools/deployment/package-helm/templates/api-server-service.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.clpConfig.api_server }}
+{{- if and .Values.clpConfig.api_server (ne .Values.clpConfig.package.query_engine "presto") }}
 apiVersion: "v1"
 kind: "Service"
 metadata:

tools/deployment/package-helm/templates/configmap.yaml

@@ -219,6 +219,13 @@ data:
     {{- else }}
     mcp_server: null
     {{- end }}
+    {{- if eq .Values.clpConfig.package.query_engine "presto" }}
+    presto:
+      host: "{{ include "clp.fullname" . }}-presto-coordinator"
+      port: 8080
+    {{- else }}
+    presto: null
+    {{- end }}
 
   mysql-logging.cnf: |
     [mysqld]
@@ -321,6 +328,91 @@ data:
         {{ .Values.clpConfig.archive_output.target_segment_size | int }},
       "ClpQueryEngine": {{ .Values.clpConfig.package.query_engine | quote }},
       "ClpStorageEngine": {{ .Values.clpConfig.package.storage_engine | quote }},
+      {{- if eq .Values.clpConfig.package.query_engine "presto" }}
+      "PrestoHost": "{{ include "clp.fullname" . }}-presto-coordinator",
+      "PrestoPort": 8080
+      {{- else }}
       "PrestoHost": null,
       "PrestoPort": null
+      {{- end }}
     }
+
+{{- if eq .Values.clpConfig.package.query_engine "presto" }}
+{{- with .Values.clpConfig.presto }}
+  presto-coordinator-catalog-clp.properties: |
+    connector.name=clp
+    clp.metadata-provider-type=mysql
+    clp.metadata-db-url=jdbc:mysql://{{ include "clp.fullname" $ }}-database:3306
+    clp.metadata-db-name={{ $.Values.clpConfig.database.names.clp }}
+    clp.metadata-db-user={{ $.Values.credentials.database.username }}
+    clp.metadata-db-password={{ $.Values.credentials.database.password }}
+    clp.metadata-table-prefix=clp_
+    clp.split-provider-type=mysql
+    clp.split-filter-provider-type=mysql
+    clp.split-filter-config=/opt/presto-server/etc/split-filter.json
+
+  presto-coordinator-config-config.properties: |
+    coordinator=true
+    node-scheduler.include-coordinator=false
+    http-server.http.port=8080
+    query.max-memory={{ .coordinator.query_max_memory_gb }}GB
+    query.max-memory-per-node={{ .coordinator.query_max_memory_per_node_gb }}GB
+    discovery-server.enabled=true
+    discovery.uri=http://{{ include "clp.fullname" $ }}-presto-coordinator:8080
+    optimizer.optimize-hash-generation=false
+    regex-library=RE2J
+    use-alternative-function-signatures=true
+    inline-sql-functions=false
+    nested-data-serialization-enabled=false
+    native-execution-enabled=true
+
+  presto-coordinator-config-jvm.config: |
+    -server
+    -Xmx4G
+    -XX:+UseG1GC
+    -XX:G1HeapRegionSize=32M
+    -XX:+UseGCOverheadLimit
+    -XX:+ExplicitGCInvokesConcurrent
+    -XX:+HeapDumpOnOutOfMemoryError
+    -XX:+ExitOnOutOfMemoryError
+    -Djdk.attach.allowAttachSelf=true
+
+  presto-coordinator-config-log.properties: |
+    com.facebook.presto={{ .coordinator.logging_level }}
+
+  presto-coordinator-config-node.properties: |
+    node.environment=production
+    node.id=presto-coordinator
+
+  presto-coordinator-config-split-filter.json: |
+    {{ .split_filter | toJson }}
+
+  presto-worker-catalog-clp.properties: |
+    connector.name=clp
+    {{- with $.Values.clpConfig.archive_output.storage }}
+    {{- if eq .type "s3" }}
+    clp.storage-type=s3
+    clp.s3-auth-provider=clp_package
+    clp.s3-access-key-id={{ .s3_config.aws_authentication.credentials.access_key_id }}
+    clp.s3-end-point=https://{{ .s3_config.bucket }}.s3.{{ .s3_config.region_code }}.amazonaws.com/
+    clp.s3-secret-access-key={{ .s3_config.aws_authentication.credentials.secret_access_key }}
+    {{- end }}{{/* if eq .type "s3" */}}
+    {{- end }}{{/* with $.Values.clpConfig.archive_output.storage */}}
+
+  presto-worker-config-config.properties: |
+    discovery.uri=http://{{ include "clp.fullname" $ }}-presto-coordinator:8080
+    http-server.http.port=8080
+    query-memory-gb={{ .worker.query_memory_gb }}
+    shutdown-onset-sec=1
+    system-memory-gb={{ .worker.system_memory_gb }}
+    register-test-functions=false
+    runtime-metrics-collection-enabled=false
+
+  presto-worker-config-node.properties: |
+    node.environment=production
+    node.location=worker-location
+
+  presto-worker-config-velox.properties: |
+    mutable-config=true
+{{- end }}{{/* with .Values.clpConfig.presto */}}
+{{- end }}{{/* if eq .Values.clpConfig.package.query_engine "presto" */}}