Skip to content

Commit fd14933

Browse files
feat: add support for DuckDB (googleapis#879)
Fixes googleapis#861 This PR adds support for DuckDB which is a free, open-source, embedded, in-process, relational database management system (RDBMS) designed for analytical processing (OLAP) --------- Co-authored-by: Averi Kitsch <akitsch@google.com>
1 parent c1305b5 commit fd14933

File tree

13 files changed

+912
-21
lines changed

13 files changed

+912
-21
lines changed

.ci/integration.cloudbuild.yaml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -487,6 +487,24 @@ steps:
487487
looker \
488488
looker
489489
490+
- id: "duckdb"
491+
name: golang:1
492+
waitFor: ["compile-test-binary"]
493+
entrypoint: /bin/bash
494+
env:
495+
- "GOPATH=/gopath"
496+
- "SERVICE_ACCOUNT_EMAIL=$SERVICE_ACCOUNT_EMAIL"
497+
volumes:
498+
- name: "go"
499+
path: "/gopath"
500+
secretEnv: ["CLIENT_ID"]
501+
args:
502+
- -c
503+
- |
504+
.ci/test_with_coverage.sh \
505+
"DuckDB" \
506+
duckdb \
507+
duckdb
490508
491509
492510
- id: "alloydbwaitforoperation"

cmd/root.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ import (
5353
_ "github.com/googleapis/genai-toolbox/internal/tools/couchbase"
5454
_ "github.com/googleapis/genai-toolbox/internal/tools/dataplex/dataplexsearchentries"
5555
_ "github.com/googleapis/genai-toolbox/internal/tools/dgraph"
56+
_ "github.com/googleapis/genai-toolbox/internal/tools/duckdbsql"
5657
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoredeletedocuments"
5758
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoregetdocuments"
5859
_ "github.com/googleapis/genai-toolbox/internal/tools/firestore/firestoregetrules"
@@ -107,6 +108,7 @@ import (
107108
_ "github.com/googleapis/genai-toolbox/internal/sources/couchbase"
108109
_ "github.com/googleapis/genai-toolbox/internal/sources/dataplex"
109110
_ "github.com/googleapis/genai-toolbox/internal/sources/dgraph"
111+
_ "github.com/googleapis/genai-toolbox/internal/sources/duckdb"
110112
_ "github.com/googleapis/genai-toolbox/internal/sources/firestore"
111113
_ "github.com/googleapis/genai-toolbox/internal/sources/http"
112114
_ "github.com/googleapis/genai-toolbox/internal/sources/looker"
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: DuckDB
3+
linkTitle: DuckDB
4+
type: docs
5+
weight: 1
6+
description: >
7+
DuckDB is an in-process SQL OLAP database management system designed for analytical query processing.
8+
---
9+
10+
## About
11+
12+
[DuckDB](https://duckdb.org/) is an embedded analytical database management system that runs in-process with the client application. It is optimized for analytical workloads, providing high performance for complex queries with minimal setup.
13+
14+
DuckDB has the following notable characteristics:
15+
16+
- In-process, serverless database engine
17+
- Supports complex SQL queries for analytical processing
18+
- Can operate on in-memory or persistent storage
19+
- Zero-configuration - no external dependencies or server setup required
20+
- Highly optimized for columnar data storage and query execution
21+
22+
For more details, refer to the [DuckDB Documentation](https://duckdb.org/).
23+
24+
## Available Tools
25+
- [`duckdb-sql`](../tools/duckdb/duckdb-sql.md)
26+
Execute pre-defined prepared SQL queries in DuckDB.
27+
28+
## Requirements
29+
30+
### Database File
31+
32+
To use DuckDB, you can either:
33+
34+
- Specify a file path for a persistent database stored on the filesystem
35+
- Omit the file path to use an in-memory database
36+
37+
## Example
38+
39+
For a persistent DuckDB database:
40+
41+
```yaml
42+
sources:
43+
my-duckdb:
44+
kind: "duckdb"
45+
dbFilePath: "/path/to/database.db"
46+
configuration:
47+
memory_limit: "2GB"
48+
threads: "4"
49+
```
50+
51+
For an in-memory DuckDB database:
52+
53+
```yaml
54+
sources:
55+
my-duckdb-memory:
56+
name: "my-duckdb-memory"
57+
kind: "duckdb"
58+
```
59+
60+
## Reference
61+
62+
### Configuration Fields
63+
64+
| **field** | **type** | **required** | **description** |
65+
|-------------------|:-----------------:|:------------:|---------------------------------------------------------------------------------|
66+
| kind | string | true | Must be "duckdb". |
67+
| dbFilePath | string | false | Path to the DuckDB database file. Omit for an in-memory database. |
68+
| configuration | map[string]string | false | Additional DuckDB configuration options (e.g., `memory_limit`, `threads`). |
69+
70+
For a complete list of available configuration options, refer to the [DuckDB Configuration Documentation](https://duckdb.org/docs/stable/configuration/overview.html#local-configuration-options).
71+
72+
73+
For more details on the Go implementation, see the [go-duckdb package documentation](https://pkg.go.dev/github.com/scottlepp/go-duckdb#section-readme).
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: "DuckDB"
3+
type: docs
4+
weight: 1
5+
description: >
6+
Tools that work with DuckDB Sources.
7+
---
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
---
2+
title: "duckdb-sql"
3+
type: docs
4+
weight: 1
5+
description: >
6+
Execute SQL statements against a DuckDB database using the DuckDB SQL tools configuration.
7+
aliases:
8+
- /resources/tools/duckdb-sql
9+
---
10+
11+
## About
12+
13+
A `duckdb-sql` tool executes a pre-defined SQL statement against a [DuckDB](https://duckdb.org/) database. It is compatible with any DuckDB source configuration as defined in the [DuckDB source documentation](../../sources/duckdb.md).
14+
15+
The specified SQL statement is executed as a prepared statement, and parameters are inserted according to their position: e.g., `$1` is the first parameter, `$2` is the second, and so on. If template parameters are included, they are resolved before execution of the prepared statement.
16+
17+
DuckDB's SQL dialect closely follows the conventions of the PostgreSQL dialect, with a few exceptions listed in the [DuckDB PostgreSQL Compatibility documentation](https://duckdb.org/docs/stable/sql/dialect/postgresql_compatibility.html). For an introduction to DuckDB's SQL dialect, refer to the [DuckDB SQL Introduction](https://duckdb.org/docs/stable/sql/introduction).
18+
19+
### Concepts
20+
21+
DuckDB is a relational database management system (RDBMS). Data is stored in relations (tables), where each table is a named collection of rows. Each row in a table has the same set of named columns, each with a specific data type. Tables are stored within schemas, and a collection of schemas constitutes the entire database.
22+
23+
For more details, see the [DuckDB SQL Introduction](https://duckdb.org/docs/stable/sql/introduction).
24+
25+
## Example
26+
27+
> **Note:** This tool uses parameterized queries to prevent SQL injections. Query parameters can be used as substitutes for arbitrary expressions but cannot be used for identifiers, column names, table names, or other parts of the query.
28+
29+
```yaml
30+
tools:
31+
search-users:
32+
kind: duckdb-sql
33+
source: my-duckdb
34+
description: Search users by name and age
35+
statement: SELECT * FROM users WHERE name LIKE $1 AND age >= $2
36+
parameters:
37+
- name: name
38+
type: string
39+
description: The name to search for
40+
- name: min_age
41+
type: integer
42+
description: Minimum age
43+
```
44+
45+
## Example with Template Parameters
46+
47+
> **Note:** Template parameters allow direct modifications to the SQL statement, including identifiers, column names, and table names, which makes them more vulnerable to SQL injections. Using basic parameters (see above) is recommended for performance and safety. For more details, see the [templateParameters](../#template-parameters) section.
48+
49+
```yaml
50+
tools:
51+
list_table:
52+
kind: duckdb-sql
53+
source: my-duckdb
54+
statement: |
55+
SELECT * FROM {{.tableName}};
56+
description: |
57+
Use this tool to list all information from a specific table.
58+
Example:
59+
{{
60+
"tableName": "flights",
61+
}}
62+
templateParameters:
63+
- name: tableName
64+
type: string
65+
description: Table to select from
66+
```
67+
68+
## Reference
69+
70+
### Configuration Fields
71+
72+
| **field** | **type** | **required** | **description** |
73+
|--------------------|:-------------------------------:|:------------:|--------------------------------------------------------------------------------------------------------------------------------------------|
74+
| kind | string | true | Must be "duckdb-sql". |
75+
| source | string | true | Name of the DuckDB source configuration (see [DuckDB source documentation](../../sources/duckdb.md)). |
76+
| description | string | true | Description of the tool that is passed to the LLM. |
77+
| statement | string | true | The SQL statement to execute. |
78+
| authRequired | []string | false | List of authentication requirements for the tool (if any). |
79+
| parameters | [parameters](../#specifying-parameters) | false | List of parameters that will be inserted into the SQL statement |
80+
| templateParameters | [templateParameters](../#template-parameters) | false | List of template parameters that will be inserted into the SQL statement before executing the prepared statement. |

go.mod

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
module github.com/googleapis/genai-toolbox
22

3-
go 1.23.8
3+
go 1.24
44

55
toolchain go1.24.5
66

@@ -48,7 +48,17 @@ require (
4848
modernc.org/sqlite v1.38.0
4949
)
5050

51-
require golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
51+
require (
52+
github.com/duckdb/duckdb-go-bindings v0.1.17 // indirect
53+
github.com/duckdb/duckdb-go-bindings/darwin-amd64 v0.1.12 // indirect
54+
github.com/duckdb/duckdb-go-bindings/darwin-arm64 v0.1.12 // indirect
55+
github.com/duckdb/duckdb-go-bindings/linux-amd64 v0.1.12 // indirect
56+
github.com/duckdb/duckdb-go-bindings/linux-arm64 v0.1.12 // indirect
57+
github.com/duckdb/duckdb-go-bindings/windows-amd64 v0.1.12 // indirect
58+
github.com/marcboeker/go-duckdb/arrowmapping v0.0.10 // indirect
59+
github.com/marcboeker/go-duckdb/mapping v0.0.11 // indirect
60+
golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
61+
)
5262

5363
require (
5464
cel.dev/expr v0.23.0 // indirect
@@ -66,6 +76,7 @@ require (
6676
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.27.0 // indirect
6777
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.53.0 // indirect
6878
github.com/ajg/form v1.5.1 // indirect
79+
github.com/apache/arrow-go/v18 v18.1.0 // indirect
6980
github.com/apache/arrow/go/v15 v15.0.2 // indirect
7081
github.com/cenkalti/backoff/v5 v5.0.2 // indirect
7182
github.com/cespare/xxhash/v2 v2.3.0 // indirect
@@ -86,12 +97,13 @@ require (
8697
github.com/go-logr/stdr v1.2.2 // indirect
8798
github.com/go-playground/locales v0.14.1 // indirect
8899
github.com/go-playground/universal-translator v0.18.1 // indirect
89-
github.com/goccy/go-json v0.10.2 // indirect
100+
github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
101+
github.com/goccy/go-json v0.10.5 // indirect
90102
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
91103
github.com/golang-sql/sqlexp v0.1.0 // indirect
92104
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
93105
github.com/golang/snappy v0.0.4 // indirect
94-
github.com/google/flatbuffers v23.5.26+incompatible // indirect
106+
github.com/google/flatbuffers v25.1.24+incompatible // indirect
95107
github.com/google/s2a-go v0.1.9 // indirect
96108
github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
97109
github.com/googleapis/gax-go/v2 v2.15.0 // indirect
@@ -102,15 +114,16 @@ require (
102114
github.com/jackc/pgpassfile v1.0.0 // indirect
103115
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
104116
github.com/jackc/puddle/v2 v2.2.2 // indirect
105-
github.com/klauspost/compress v1.16.7 // indirect
106-
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
117+
github.com/klauspost/compress v1.17.11 // indirect
118+
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
107119
github.com/leodido/go-urn v1.4.0 // indirect
120+
github.com/marcboeker/go-duckdb/v2 v2.3.3
108121
github.com/mattn/go-isatty v0.0.20 // indirect
109122
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
110123
github.com/modern-go/reflect2 v1.0.2 // indirect
111124
github.com/montanaflynn/stats v0.7.1 // indirect
112125
github.com/ncruces/go-strftime v0.1.9 // indirect
113-
github.com/pierrec/lz4/v4 v4.1.18 // indirect
126+
github.com/pierrec/lz4/v4 v4.1.22 // indirect
114127
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
115128
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
116129
github.com/spf13/pflag v1.0.6 // indirect

0 commit comments

Comments
 (0)