This repository is a VGI worker written in Go. DuckDB starts it (or connects to it over HTTP) and calls its functions from SQL; data moves as Apache Arrow batches. Background reading, in order of usefulness:
- https://query.farm/vgi/llms.txt — index of all VGI docs, written for agents
- https://query.farm/vgi/docs/go/ — the Go SDK (vgi-go)
- https://query.farm/vgi/building-a-worker — concepts: catalogs, schemas, function kinds
make check passes. It runs formatting and static checks, unit tests, the SQL
tests over the stdio, launch and http transports, and the metadata quality check
(vgi-lint, at the fail_on level in vgi-lint.toml). Don't hand work back until it passes. If a
check is wrong, fix the check and say why. Never delete a test to get green.
Faster loops while iterating: make test-unit, make test-sql, make lint-explain.
cmd/vgi-steampipe-worker/ # main.go: the worker binary (flags -> serve.Main; --plugins defaults)
internal/steampipe/
worker.go # assembles the catalog and its handlers
catalog.go # ATTACH options, attach state, per-plugin schemas via catalog_contents
plugin_ref.go # [schema=]ref parsing (OCI refs and local paths)
oci.go # OCI fetch: Bearer handshake, digest verification, cache
host.go # go-plugin process pool, SetAllConnectionConfigs, GetSchema
scan.go # scan(schema, table): Execute streams -> Arrow batches
quals.go # key columns -> quals and required filters
filter_eval.go # exact re-application of pushed filters
config.go # HCL/JSON connection config, merge of ATTACH config + secret
types.go # Steampipe column types -> Arrow
info.go # main.plugin / plugin_table / plugin_key_column tables
docs.go, plugin_docs.go, meta.go # discovery metadata
version.go # the worker version (checked against release tags)
internal/serve/ # --http / --unix / stdio and logging flags, shared with recipes
test/fixture-plugin/ # offline Steampipe plugin built with the real SDK (own go.mod)
test/mock_registry/ # offline OCI registry serving the fixture plugin image
test/data/ # CSV files for the published turbot/csv plugin test
test/sql/ # sqllogictest files run through DuckDB
examples/ # recipes (one package main, served by bin/recipes <name>)
ci/ # integration harness used by `make test-sql` and CI
scripts/ # vgi_template.py (init / template-diff)
vgi-agent-tests.yaml # private graders for the agent test tasks
- Create
internal/steampipe/<name>.go. Copy the closest example:greet.gofor a scalar,sequence.gofor a generator,fetch_items.gofor an API. - Declare arguments as a struct with
vgi:"..."tags. Usepos=Nfor positional arguments anddefault=V(nopos) for named ones; addge=,le=,pattern=orchoices='a,b'constraints. Scalar column arguments are*array.String(etc.) fields withconst=false.doc=must be the last key in the tag. - Implement
Name(),Metadata()(withDescription,Tags: withExamples(objectTags(...), examples)andExamples), thenOnBindTyped/ProcessTyped(scalar, wrap withvgi.AsScalarFunction[Args]) orArgumentSpecs/OnBind/NewState/Process(table, wrap withvgi.AsTableFunction[State]). - Register it in
NewWorkerininternal/steampipe/worker.go, and add its examples toallExamples.
Then, for every new function:
- Metadata (
internal/steampipe/meta.goand each function'sMetadata()): title, category from the schema's category list, keywords,doc_llm(for agents: what it does, arguments, when to use it) anddoc_md(for humans). Describe each argument, and for table functions describe the result columns. Add 1–2exampleswith fully qualified, analytical SQL (steampipe.main.fn(...)): project, filter or aggregate rather thanSELECT *. - Executable examples: if the function is deterministic and needs no network,
add an
executable_examplesentry withORDER BYand anexpected_result. - SQL test: add cases to
test/sql/steampipe.test, or a newtest/sql/<name>.testwith the same header (require vgi,require-env STEAMPIPE_WORKER, thenATTACH). Avoid expected-error text that depends on SDK wording; use<REGEX>:.*for errors you don't control. - Unit test for any non-trivial logic.
- Agent task: add a
{name, prompt}to the catalog's agent test tasks, and its grader (reference_sqlorcheck_sql) tovgi-agent-tests.yaml. - Run
make check.
make lint-explain runs vgi-lint against
the built worker with the settings in vgi-lint.toml. The linter isn't pinned, so new
rules arrive on their own. When it reports findings:
- Run
make lint-explain LINT_ARGS="--format agent". Findings are grouped by rule: each rule's fix is stated once, followed by every object it applies to. - For a rule you don't understand, run
uvx --prerelease=allow --from 'vgi-lint-check>=2,<3' vgi-lint explain VGI123. For tag names and value shapes, readvgi-lint specrather than guessing. - Fix errors first, then warnings, then info. Fix the metadata in the worker, never the linter's settings.
- If a finding is genuinely wrong for this worker, waive it in
vgi-lint.tomlwith akindand areason(see the comments there). Never add an unexplainedignore.
The check executes examples (assurance level L2). make lint-ai adds L3: an agent must
solve the tasks in vgi-agent-tests.yaml using only the metadata, and the docs are
reviewed. It's optional, so run it before publishing.
| Need | Kind |
|---|---|
| one value per input row | scalar |
| rows from arguments (API, file, generator) | table function |
| transform the rows of a subquery | table in-out function |
| combine many rows into one value | aggregate |
| a fixed dataset referenced by name | catalog table |
| write data out to a file or service | COPY … TO copy function |
| accept INSERT / UPDATE / DELETE | writable catalog table: vgi.CatalogTable InsertFunction / UpdateFunction / DeleteFunction |
Recipes for most kinds are in examples/. Writable tables have no recipe:
A writable table is a vgi.CatalogTable that names a table in-out function per operation in
InsertFunction, UpdateFunction and DeleteFunction, with the richest result each one
produces in WriteResultModes. A write function declares vgi.WriteOptionsArgSpec(), reads
the requested mode with vgi.ParseWriteOptions, and binds and emits
vgi.WriteResultSchema / vgi.WriteResultBatch. UPDATE and DELETE need a vgi.RowIDField
column emitted by a scan that declares projection pushdown. Transactions:
vgi.WithSupportsTransactions (or AttachDecision.SupportsTransactions per ATTACH) and
vgi.WithTransactionHooks. Since VGI protocol 3.3 a table's scan and write functions are
table-only: hidden from duckdb_functions() and not callable from SQL. Set ExposeScan: true
on the table to keep the scan callable as an ordinary table function too (write functions never
are). The write helpers and transaction hooks need vgi-go v0.42.0 or newer: if go.mod has an
older v0.42.x, run go get github.com/Query-farm/vgi-go@v0.42 (the newest v0.42.x; every
v0.42.x speaks the same protocol). See
Writable catalog tables and
examples/writable_tx.
- Never write to stdout. stdout is the Arrow protocol channel, and one stray print
corrupts the stream. Log to stderr. Use
log/slog(the SDK configures it to write to stderr);--debugor--log-level debugturns it up. - Scalars have no named arguments. For an optional scalar argument, add an overload
(a second function with the same SQL name and one more argument). Named arguments
(
step := 2) work on table functions only. - Declared constraints are enforced at bind.
choices,ge,leandpatternon a constant argument are published so DuckDB and agents can discover them, and the SDK (vgi-go) rejects a violating value when the query is bound, before your code runs, for positional and named arguments alike.patternis an unanchored search (JSON Schema semantics) in every SDK: write^https?://, nothttps?://. Constraints cover constant arguments only: validate the values of a columnar scalar argument in the compute body. - Reserved argument names. DuckDB keywords such as
interval,end,from,orderandtablebreakname := valuesyntax. Pick another name (start_date,end_date). - State must be serializable. Table-function state (the
SinTypedTableFunc[S]) is gob-encoded into the HTTP continuation token between batches, because each batch can be served by a different request or process. Keep the cursor in exported fields (gob silently drops unexported ones, so the scan would restart from row 0), keep it small, and never put secrets in it. - Examples must be deterministic. Add
ORDER BY, and nonow(), random values or network. Examples that hit a live API go inexamples, notexecutable_examples. - Credentials. Read them from a DuckDB secret first (
CREATE SECRET (TYPE steampipe_api, …)), then from an environment variable. Never put them in logs, examples or the repo, and keep them out of table-function state unless the state rule above says the SDK encrypts it. Workers that need real credentials setexecute: falseon the metadata check. - Errors. Raise errors with a clear message. For HTTP failures, include the status code. DuckDB shows the message to the user.
- Rate limits. Retry 429 and 5xx with backoff, honour
Retry-After, and set timeouts on every request. - Version pins. Don't bump the vgi-go minor version by hand. A minor release can
change the wire protocol, which must match the DuckDB extension exactly. Template
updates (
make template-diff) carry tested bumps.
- Scalars receive whole Arrow columns.
vgi.MapColumn/vgi.MapAllColumnsbuild the output column in one pass and keep NULLs as NULLs. - Table functions: implement
vgi.TypedTableFunc[S]and wrap it withvgi.AsTableFunction[S].Sis the per-scan cursor. Over HTTP the SDK gob-encodes it into the continuation token after every batch, so every field that must survive must be exported. An unexported field is silently reset on the next request. Copy the arguments intoSinNewStateinstead of re-parsing them inProcess. - Emit with
out.EmitArrays(cols, rows)in output-schema order,Release()the arrays you built, and callout.Finish()(not an error) when done. - Use
column(name, type, comment, nullable)frommeta.gofor output fields. The comment becomes the column description in DuckDB and agent tools. - No cgo by default (vgi-go 0.40+). The worker builds as a static ~40 MB binary
with
CGO_ENABLED=0. Evaluating pushed filters inside the worker (Evaluate, orAutoApplyFilters) needsimport _ "github.com/Query-farm/vgi-go/vgi/exprduckdb", which embeds DuckDB through cgo: then setCGO_ENABLED=1(Makefile, CI, Dockerfile,cgo: "1"in release.yml) and expect ~100 MB binaries. Translating filters to SQL yourself, asexamples/pushdown_sqlite.godoes, needs neither. - Table-function state fields must be exported. vgi-go 0.40+ refuses unexported fields at registration (they would be lost between HTTP batches).
- The
replaceingo.modis required. Go doesn't inherit a dependency'sreplacedirectives, and vgi-go needs a patchedgithub.com/google/flatbufferson every transport (dictionary fields with Arrow metadata otherwise produce IPC schemas strict readers reject, until google/flatbuffers#9414 ships). Keep the line, and update it when a template update changes it. The SDK's secondreplace, an Arrow fork, is needed only for the optional shared-memory transport (VGI_RPC_SHM_SIZE_BYTES), so this template leaves it out. Add it from the vgi-go README if you enable SHM. - Recipes: all six are implemented in Go (see
examples/). - Dependencies go in
go.mod(go get, thengo mod tidy). Keep them few: every dependency ships to every user of the worker. - Gates:
gofmt,go vet,golangci-lint(config in.golangci.yml),go test.
- "Add a scalar
slugify(text VARCHAR) -> VARCHARwith full metadata and tests." - "Replace
fetch_itemswith a table function for . Map its JSON to typed columns, read the token from asteampipe_apisecret, and add mock-server tests." - "Add a catalog table
countries(code, name) with a primary key, followingexamples/catalog_tables."