Offload leaf search work to AWS Lambda functions (quickwit-oss#6157)

* Offload leaf search work to AWS Lambda functions

The goal is to handle traffic spikes gracefully without provisioning additional searcher nodes: when the local search queue is saturated, overflow splits are transparently routed to Lambda for processing.

The offloading decision happens **on the leaf side**, inside the `SearchPermitProvider`. The permit provider already manages a bounded queue of pending split search tasks (gated by memory budget and download slots). When a leaf search request arrives, the provider checks the current queue depth against a configurable `offload_threshold`. If granting permits for all requested splits would exceed this threshold, only enough splits to fill up to the threshold are processed locally — the rest are marked for offloading.

The offloaded splits are batched (up to `max_splits_per_invocation` splits per batch, balanced by document count) and sent to Lambda in parallel. Each Lambda invocation runs the same leaf search code path and **returns per-split results individually**. This is important: the per-split responses are fed back into the `IncrementalCollector` and populate the **partial result cache**, so subsequent queries hitting the same splits benefit from cached results regardless of whether the split was searched locally or on Lambda.

Depending on the configuration, the Lambda function code can be **deployed automatically** at startup. The `quickwit-lambda-client` crate embeds a compressed Lambda binary at compile time. When `auto_deploy` is configured, Quickwit will:
1. Check if a published Lambda version matching the current binary already exists (identified by a description tag `quickwit:{version}-{hash}`)
2. Create or update the function and publish a new version if needed
3. Garbage-collect old versions (keeping the current one + 5 most recent)

This ensures the Lambda function always matches the running Quickwit version without any external deployment tooling. Manual deployment is also supported for users who prefer to manage Lambda functions through Terraform or other IaC tools.

Lambda offloading is opt-in. Add a `lambda` section under `searcher` in the node configuration:

```yaml
searcher:
  lambda:
    offload_threshold: 100     # queue depth before offloading kicks in (0 = always offload)
    max_splits_per_invocation: 10
    auto_deploy:
      execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role
      memory_size: 5 GiB
      invocation_timeout_secs: 15
```

- **`quickwit-lambda-client`**: Handles Lambda invocation (with metrics) and auto-deployment logic. Embeds the Lambda binary at build time.
- **`quickwit-lambda-server`**: The Lambda function handler itself — receives a `LeafSearchRequest`, runs `multi_index_leaf_search`, and returns per-split `LeafSearchResponse`s.

- **`quickwit-search`**: New `LambdaLeafSearchInvoker` trait; `SearchPermitProvider` gains `get_permits_with_offload` to split work between local and offloaded; `leaf.rs` orchestrates local and Lambda tasks in parallel.
- **`quickwit-config`**: New `LambdaConfig` and `LambdaDeployConfig` structs under `SearcherConfig`.
- **`quickwit-serve`**: Initializes the Lambda invoker at startup when configured.
- **`quickwit-proto`**: New `LeafSearchResponses` wrapper message for batched per-split responses.

Author: fulmicoton-dd

.github/actions/cargo-build-macos-binary/action.yml

@@ -60,7 +60,7 @@ runs:
         path: ./${{ env.ASSET_FULL_NAME }}.tar.gz
         retention-days: 3
     - name: Deploy archive to GitHub release
-      uses: quickwit-inc/upload-to-github-release@v1
+      uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190
       env:
         GITHUB_TOKEN: ${{ inputs.token }}
       with:

.github/actions/cross-build-binary/action.yml

@@ -56,7 +56,7 @@ runs:
         path: ./${{ env.ASSET_FULL_NAME }}.tar.gz
         retention-days: 3
     - name: Upload archive
-      uses: quickwit-inc/upload-to-github-release@v1
+      uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190
       env:
         GITHUB_TOKEN: ${{ inputs.token }}
       with:

.github/workflows/publish_lambda.yaml

@@ -0,0 +1,78 @@
+# This workflow creates a new release for a quickwit search aws lambda.
+# The artifact is a zip file containing a binary for ARM 64,
+# ready to be deployed by the deployer.
+#
+# See quickwit-lambda-client/README.md
+name: Release Lambda binary
+
+on:
+  push:
+    tags:
+      - 'lambda-*'
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version tag (e.g., v0.8.0)'
+        required: false
+        default: 'dev'
+
+permissions:
+  contents: read
+
+jobs:
+  build-lambda:
+    name: Build Lambda ARM64
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      actions: write
+    steps:
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+
+      - name: Set version
+        run: |
+          if [ "${{ github.ref_type }}" = "tag" ]; then
+            # Extract version from tag (e.g., lambda-v0.8.0 -> v0.8.0)
+            echo "ASSET_VERSION=${GITHUB_REF_NAME#lambda-}" >> $GITHUB_ENV
+          elif [ -n "${{ github.event.inputs.version }}" ] && [ "${{ github.event.inputs.version }}" != "dev" ]; then
+            echo "ASSET_VERSION=${{ github.event.inputs.version }}" >> $GITHUB_ENV
+          else
+            echo "ASSET_VERSION=dev-$(git rev-parse --short HEAD)" >> $GITHUB_ENV
+          fi
+
+      - name: Install rustup
+        run: curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none -y
+
+      - name: Install cross
+        run: cargo install cross
+
+      - name: Retrieve and export commit date, hash, and tags
+        run: |
+          echo "QW_COMMIT_DATE=$(TZ=UTC0 git log -1 --format=%cd --date=format-local:%Y-%m-%dT%H:%M:%SZ)" >> $GITHUB_ENV
+          echo "QW_COMMIT_HASH=$(git rev-parse HEAD)" >> $GITHUB_ENV
+          echo "QW_COMMIT_TAGS=$(git tag --points-at HEAD | tr '\n' ',')" >> $GITHUB_ENV
+
+      - name: Build Lambda binary
+        run: cross build --release --features lambda-release --target aarch64-unknown-linux-gnu -p quickwit-lambda-server --bin quickwit-aws-lambda-leaf-search
+        env:
+          QW_COMMIT_DATE: ${{ env.QW_COMMIT_DATE }}
+          QW_COMMIT_HASH: ${{ env.QW_COMMIT_HASH }}
+          QW_COMMIT_TAGS: ${{ env.QW_COMMIT_TAGS }}
+        working-directory: ./quickwit
+
+      - name: Create Lambda zip
+        run: |
+          cd quickwit/target/aarch64-unknown-linux-gnu/release
+          cp quickwit-aws-lambda-leaf-search bootstrap
+          zip quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip bootstrap
+          mv quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip ../../../../
+
+      - name: Upload to GitHub release
+        uses: quickwit-inc/upload-to-github-release@9b2c40fba23bf8dea05b7d2eece24cbc95d4a190
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          file: quickwit-aws-lambda-${{ env.ASSET_VERSION }}-aarch64.zip
+          overwrite: true
+          draft: true
+          tag_name: ${{ env.ASSET_VERSION }}

LICENSE-3rdparty.csv

@@ -21,13 +21,16 @@ arrayvec,https://github.com/bluss/arrayvec,MIT OR Apache-2.0,bluss
 assert-json-diff,https://github.com/davidpdrsn/assert-json-diff,MIT,David Pedersen <david.pdrsn@gmail.com>
 async-compression,https://github.com/Nullus157/async-compression,MIT OR Apache-2.0,"Wim Looman <wim@nemo157.com>, Allen Bui <fairingrey@gmail.com>"
 async-speed-limit,https://github.com/tikv/async-speed-limit,MIT OR Apache-2.0,The TiKV Project Developers
+async-stream,https://github.com/tokio-rs/async-stream,MIT,Carl Lerche <me@carllerche.com>
+async-stream-impl,https://github.com/tokio-rs/async-stream,MIT,Carl Lerche <me@carllerche.com>
 async-trait,https://github.com/dtolnay/async-trait,MIT OR Apache-2.0,David Tolnay <dtolnay@gmail.com>
 atomic-waker,https://github.com/smol-rs/atomic-waker,Apache-2.0 OR MIT,"Stjepan Glavina <stjepang@gmail.com>, Contributors to futures-rs"
 aws-config,https://github.com/smithy-lang/smithy-rs,Apache-2.0,"AWS Rust SDK Team <aws-sdk-rust@amazon.com>, Russell Cohen <rcoh@amazon.com>"
 aws-credential-types,https://github.com/smithy-lang/smithy-rs,Apache-2.0,AWS Rust SDK Team <aws-sdk-rust@amazon.com>
 aws-lc-rs,https://github.com/aws/aws-lc-rs,ISC AND (Apache-2.0 OR ISC),AWS-LibCrypto
 aws-lc-sys,https://github.com/aws/aws-lc-rs,ISC AND (Apache-2.0 OR ISC) AND OpenSSL,AWS-LC
 aws-runtime,https://github.com/smithy-lang/smithy-rs,Apache-2.0,AWS Rust SDK Team <aws-sdk-rust@amazon.com>
+aws-sdk-lambda,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,"AWS Rust SDK Team <aws-sdk-rust@amazon.com>, Russell Cohen <rcoh@amazon.com>"
 aws-sdk-s3,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,"AWS Rust SDK Team <aws-sdk-rust@amazon.com>, Russell Cohen <rcoh@amazon.com>"
 aws-sdk-sso,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,"AWS Rust SDK Team <aws-sdk-rust@amazon.com>, Russell Cohen <rcoh@amazon.com>"
 aws-sdk-ssooidc,https://github.com/awslabs/aws-sdk-rust,Apache-2.0,"AWS Rust SDK Team <aws-sdk-rust@amazon.com>, Russell Cohen <rcoh@amazon.com>"
@@ -222,6 +225,8 @@ itoa,https://github.com/dtolnay/itoa,MIT OR Apache-2.0,David Tolnay <dtolnay@gma
 jobserver,https://github.com/rust-lang/jobserver-rs,MIT OR Apache-2.0,Alex Crichton <alex@alexcrichton.com>
 js-sys,https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys,MIT OR Apache-2.0,The wasm-bindgen Developers
 json_comments,https://github.com/tmccombs/json-comments-rs,Apache-2.0,Thayne McCombs <astrothayne@gmail.com>
+lambda_runtime,https://github.com/awslabs/aws-lambda-rust-runtime,Apache-2.0,"David Calavera <dcalaver@amazon.com>, Harold Sun <sunhua@amazon.com>"
+lambda_runtime_api_client,https://github.com/awslabs/aws-lambda-rust-runtime,Apache-2.0,"David Calavera <dcalaver@amazon.com>, Harold Sun <sunhua@amazon.com>"
 lazy_static,https://github.com/rust-lang-nursery/lazy-static.rs,MIT OR Apache-2.0,Marvin Löbel <loebel.marvin@gmail.com>
 levenshtein_automata,https://github.com/tantivy-search/levenshtein-automata,MIT,Paul Masurel <paul.masurel@gmail.com>
 libc,https://github.com/rust-lang/libc,MIT OR Apache-2.0,The Rust Project Developers
@@ -397,6 +402,7 @@ serde_core,https://github.com/serde-rs/serde,MIT OR Apache-2.0,"Erick Tryzelaar
 serde_derive,https://github.com/serde-rs/serde,MIT OR Apache-2.0,"Erick Tryzelaar <erick.tryzelaar@gmail.com>, David Tolnay <dtolnay@gmail.com>"
 serde_json,https://github.com/serde-rs/json,MIT OR Apache-2.0,"Erick Tryzelaar <erick.tryzelaar@gmail.com>, David Tolnay <dtolnay@gmail.com>"
 serde_json_borrow,https://github.com/PSeitz/serde_json_borrow,MIT,Pascal Seitz <pascal.seitz@gmail.com>
+serde_path_to_error,https://github.com/dtolnay/path-to-error,MIT OR Apache-2.0,David Tolnay <dtolnay@gmail.com>
 serde_qs,https://github.com/samscott89/serde_qs,MIT OR Apache-2.0,Sam Scott <sam@osohq.com>
 serde_spanned,https://github.com/toml-rs/toml,MIT OR Apache-2.0,The serde_spanned Authors
 serde_urlencoded,https://github.com/nox/serde_urlencoded,MIT OR Apache-2.0,Anthony Ramine <n.oxyde@gmail.com>
@@ -497,9 +503,11 @@ unicode-width,https://github.com/unicode-rs/unicode-width,MIT OR Apache-2.0,"kwa
 unit-prefix,https://codeberg.org/commons-rs/unit-prefix,MIT,"Fabio Valentini <decathorpe@gmail.com>, Benjamin Sago <ogham@bsago.me>"
 unsafe-libyaml,https://github.com/dtolnay/unsafe-libyaml,MIT,David Tolnay <dtolnay@gmail.com>
 untrusted,https://github.com/briansmith/untrusted,ISC,Brian Smith <brian@briansmith.org>
+ureq-proto,https://github.com/algesten/ureq-proto,MIT OR Apache-2.0,Martin Algesten <martin@algesten.se>
 url,https://github.com/servo/rust-url,MIT OR Apache-2.0,The rust-url developers
 urlencoding,https://github.com/kornelski/rust_urlencoding,MIT,"Kornel <kornel@geekhood.net>, Bertram Truong <b@bertramtruong.com>"
 username,https://pijul.org/darcs/user,MIT OR Apache-2.0,Pierre-Étienne Meunier <pierre-etienne.meunier@aalto.fi>
+utf-8,https://github.com/SimonSapin/rust-utf8,MIT OR Apache-2.0,Simon Sapin <simon.sapin@exyr.org>
 utf8-ranges,https://github.com/BurntSushi/utf8-ranges,Unlicense OR MIT,Andrew Gallant <jamslam@gmail.com>
 utf8_iter,https://github.com/hsivonen/utf8_iter,Apache-2.0 OR MIT,Henri Sivonen <hsivonen@hsivonen.fi>
 utf8parse,https://github.com/alacritty/vte,Apache-2.0 OR MIT,"Joe Wilm <joe@jwilm.com>, Christian Duerr <contact@christianduerr.com>"

docs/configuration/lambda-config.md

@@ -0,0 +1,240 @@
+---
+title: Lambda configuration
+sidebar_position: 6
+---
+
+Quickwit supports offloading leaf search operations to AWS Lambda for horizontal scaling. When the local search queue becomes saturated, overflow splits are automatically sent to Lambda functions for processing.
+
+:::note
+Lambda offloading is currently only supported on AWS.
+:::
+
+## How it works
+
+Lambda offloading is **only active when a `lambda` configuration section is present** under `searcher` in your node configuration. When configured:
+
+1. Quickwit monitors the local search queue depth
+2. When pending searches exceed the `offload_threshold`, new splits are sent to Lambda instead of being queued locally
+3. Lambda returns per-split search results that are cached and merged with local results
+
+This allows Quickwit to handle traffic spikes without provisioning additional searcher nodes.
+
+## Startup validation
+
+When a `lambda` configuration is defined, Quickwit performs a **dry run invocation** at startup to verify that:
+- The Lambda function exists
+- The function version matches the embedded binary
+- The invoker has permission to call the function
+
+If this validation fails, **Quickwit will fail to start**. This ensures that Lambda offloading works correctly before the node begins serving traffic.
+
+## Configuration
+
+Add a `lambda` section under `searcher` in your node configuration:
+
+```yaml
+searcher:
+  lambda:
+    offload_threshold: 100
+    auto_deploy:
+      execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role
+      memory_size: 5 GiB
+      invocation_timeout_secs: 15
+```
+
+### Lambda configuration options
+
+| Property | Description | Default value |
+| --- | --- | --- |
+| `function_name` | Name of the AWS Lambda function to invoke. | `quickwit-lambda-search` |
+| `max_splits_per_invocation` | Maximum number of splits to send in a single Lambda invocation. Must be at least 1. | `10` |
+| `offload_threshold` | Number of pending local searches before offloading to Lambda. A value of `0` offloads everything to Lambda. | `100` |
+| `auto_deploy` | Auto-deployment configuration. If set, Quickwit automatically deploys or updates the Lambda function at startup. | (none) |
+
+### Auto-deploy configuration options
+
+| Property | Description | Default value |
+| --- | --- | --- |
+| `execution_role_arn` | **Required.** IAM role ARN for the Lambda function's execution role. | |
+| `memory_size` | Memory allocated to the Lambda function. More memory provides more CPU. | `5 GiB` |
+| `invocation_timeout_secs` | Timeout for Lambda invocations in seconds. | `15` |
+
+## Deployment options
+
+### Automatic deployment (recommended)
+
+With `auto_deploy` configured, Quickwit automatically:
+1. Creates the Lambda function if it doesn't exist
+2. Updates the function code if the embedded binary has changed
+3. Publishes a new version with a unique identifier
+4. Garbage collects old versions (keeps current + 5 most recent)
+
+This is the recommended approach as it ensures the Lambda function always matches the Quickwit binary version.
+
+### Manual deployment
+
+You can deploy the Lambda function manually without `auto_deploy`:
+1. Download the Lambda zip from [GitHub releases](https://github.com/quickwit-oss/quickwit/releases)
+2. Create or update the Lambda function using AWS CLI, Terraform, or the AWS Console
+3. Publish a version with description format `quickwit_{version}_{sha256}_{timeout}_{deploy_config}"` (e.g., `quickwit_0_8_0_fa940f44_5120_60s_6c3b2`)
+
+The description must match the format Quickwit expects, or it won't find the function version.
+
+## IAM permissions
+
+### Permissions for the Quickwit node
+
+The IAM role or user running Quickwit needs the following permissions to invoke Lambda:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "lambda:InvokeFunction"
+      ],
+      "Resource": "arn:aws:lambda:*:*:function:quickwit-lambda-search:*"
+    }
+  ]
+}
+```
+
+If using `auto_deploy`, additional permissions are required for deployment:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "lambda:CreateFunction",
+        "lambda:GetFunction",
+        "lambda:UpdateFunctionCode",
+        "lambda:PublishVersion",
+        "lambda:ListVersionsByFunction",
+        "lambda:DeleteFunction"
+      ],
+      "Resource": "arn:aws:lambda:*:*:function:quickwit-lambda-search"
+    },
+    {
+      "Effect": "Allow",
+      "Action": "iam:PassRole",
+      "Resource": "arn:aws:iam::*:role/quickwit-lambda-role",
+      "Condition": {
+        "StringEquals": {
+          "iam:PassedToService": "lambda.amazonaws.com"
+        }
+      }
+    }
+  ]
+}
+```
+
+### Lambda execution role
+
+The Lambda function requires an execution role with S3 read access to your index data.
+
+Example policy:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": "s3:GetObject",
+      "Resource": "arn:aws:s3:::your-index-bucket/*"
+    }
+  ]
+}
+```
+
+The execution role must also have a trust policy allowing Lambda to assume it:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Principal": {
+        "Service": "lambda.amazonaws.com"
+      },
+      "Action": "sts:AssumeRole"
+    }
+  ]
+}
+```
+
+## CloudWatch logging
+
+The Lambda function emits structured logs (JSON) to stdout. To have these logs captured by CloudWatch, add the following iam permissions to the Lambda execution role:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "logs:CreateLogGroup",
+        "logs:CreateLogStream",
+        "logs:PutLogEvents"
+      ],
+      "Resource": "arn:aws:logs:*:*:*"
+    }
+  ]
+}
+```
+
+No additional configuration is needed on the Quickwit side.
+
+## Versioning
+
+Quickwit uses content-based versioning for Lambda:
+- A SHA256 hash of the Lambda binary is computed at build time
+- This hash is embedded in the Lambda function description as `quickwit:{version}-{sha256_short}`
+- When Quickwit starts, it searches for a version matching this description
+- Different Quickwit builds with the same Lambda binary share the same Lambda version
+- Updating the Lambda binary automatically triggers a new deployment
+
+## Example configuration
+
+
+Minimal configuration (with auto-deployment):
+
+```yaml
+searcher:
+  lambda:
+    auto_deploy:
+      execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role
+```
+
+
+Full configuration (auto-deployment):
+
+```yaml
+searcher:
+  lambda:
+    function_name: quickwit-lambda-search
+    max_splits_per_invocation: 10
+    offload_threshold: 10
+    auto_deploy:
+      execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role
+      memory_size: 5 GiB
+      invocation_timeout_secs: 15
+```
+
+Aggressive offloading (send everything to Lambda):
+
+```yaml
+searcher:
+  lambda:
+    function_name: quickwit-lambda-search
+    offload_threshold: 0
+    auto_deploy:
+      execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role
+```