Skip to content

Latest commit

 

History

History
162 lines (116 loc) · 6.8 KB

README.md

File metadata and controls

162 lines (116 loc) · 6.8 KB

Vector Gateway Interface (VGI)

A Query.Farm VGI worker for DuckDB.

Scan Data & Files against YARA Rules in DuckDB

vgi-yara · a Query.Farm VGI worker · powered by yara-x

A VGI worker (Rust, a compiled binary) that brings YARA malware scanning to DuckDB / SQL over Apache Arrow. DuckDB launches the worker and talks to it over Arrow IPC; the functions appear under the catalog yara, schema main. This is a defensive security tool: it scans data/files against YARA rules for malware detection.

Rule compilation and scanning are powered by yara-x, VirusTotal's official pure-Rust rewrite of YARA — no native libyara/C dependency.

LOAD vgi;
ATTACH 'yara' (TYPE vgi, LOCATION 'github-auto://Query-farm/vgi-yara@v0.4.4');
SET search_path = 'yara.main';

-- Per-row predicate over a column of blobs/files.
SELECT path
FROM files
WHERE yara_matches(content, 'rule eicar { strings: $a = "EICAR" condition: $a }');

-- First matching rule / how many rules matched.
SELECT yara_first_rule(content, $rules)  FROM files;   -- VARCHAR (NULL if none)
SELECT yara_match_count(content, $rules) FROM files;   -- INT

-- Validate a ruleset compiles.
SELECT yara_check('rule r { condition: true }');       -- → true

-- Fan one constant blob into its matches (table functions).
SELECT * FROM yara_scan(read_blob('sample.bin'), $rules);
-- rule | namespace | tags
SELECT * FROM yara_string_matches(read_blob('sample.bin'), $rules);
-- rule | identifier | offset | matched

Quick start

Every GitHub release ships a prebuilt worker binary. The vgi extension downloads the one for your platform, checks its SHA-256 and caches it, so there is nothing to install besides DuckDB:

INSTALL vgi FROM community;
LOAD vgi;

ATTACH 'yara' AS yara (TYPE vgi, LOCATION 'github-auto://Query-farm/vgi-yara@v0.4.4');

-- Validate that a YARA ruleset compiles before scanning with it.
SELECT yara.main.yara_check('rule demo { strings: $a = "malware" condition: $a }') AS compiles;

-- Predicate: does the text match any rule in a ruleset?
SELECT yara.main.yara_matches('this file contains malware', 'rule demo { strings: $a = "malware" condition: $a }') AS hit;

Prebuilt binaries cover macOS (Apple silicon), Linux (x86-64 and arm64). github-auto:// needs an explicit release tag; change @v0.4.4 to move to a newer release.

Other ways to run it

  • Docker (the image bundles the worker; needs a container runtime):

    ATTACH 'yara' AS yara (TYPE vgi, LOCATION 'docker://ghcr.io/query-farm/vgi-yara:0.4.4');
  • Manual download. Download the archive for your platform from the releases page, verify it against its .sha256 file (a cosign .cosign.bundle is published alongside), unpack it, and set LOCATION to the path of the extracted worker executable.

Build from source

To build the worker yourself, see Building & testing.

Functions

Scalar

Function Returns Description
yara_matches(data, rules) BOOLEAN Does data match any rule?
yara_first_rule(data, rules) VARCHAR Identifier of the first matching rule (NULL if none).
yara_match_count(data, rules) INT Number of matching rules.
yara_check(rules) BOOLEAN Do the rules compile? (validation; never errors).

data is a BLOB or VARCHAR (the bytes/text to scan); rules is a YARA rule source string.

The worker build version is published as the catalog's implementation_version (read it from vgi_catalogs()), not as a scalar function.

Table

Function Columns Description
yara_scan(data, rules) rule VARCHAR, namespace VARCHAR, tags VARCHAR[] One row per matching rule.
yara_string_matches(data, rules) rule VARCHAR, identifier VARCHAR, "offset" BIGINT, matched VARCHAR One row per pattern (string) hit.
yara_modules module VARCHAR, description VARCHAR The YARA-X modules compiled into this worker that a rule may import (browsable — SELECT * FROM yara.main.yara_modules).

DuckDB table functions take constant arguments (no subqueries), so the data and rules passed to yara_scan / yara_string_matches must be constant-foldable expressions (literals, read_blob('…'), etc.). matched is the matched bytes rendered as UTF-8 text when printable, else a lowercase hex string.

Behavior & robustness

The scanned data is untrusted — by definition it may be live malware:

  • A malformed, truncated, binary, or hostile blob never crashes the worker. Scanning is total: it yields no matches (false / NULL / 0 / no rows), never an error. A bad blob beside a good one still produces the good one's matches.
  • Scanned data is bounded (64 MiB): an oversized blob is truncated to the cap before scanning so it cannot exhaust memory.
  • NULL input → NULL output / no rows.
  • An invalid rule source (a user mistake) surfaces a clear DuckDB error from the scan functions, carrying the compiler diagnostic. yara_check instead returns false for a non-compiling source (it is the "does it compile?" predicate).

Building & testing

cargo build --release                                    # build the worker
cargo test --workspace --all-features                    # unit + integration tests
cargo clippy --all-targets --all-features -- -D warnings # lint
make test-sql                                            # DuckDB SQL end-to-end

make test-sql builds the release worker, points VGI_YARA_WORKER at it, and runs the haybarn-unittest sqllogictest suite under test/sql/. The worker speaks VGI protocol vgi.v3 / 3.2, so it needs haybarn 2.0. Install the runner once with uv tool install --prerelease allow 'haybarn-unittest>=2.0.0rc11,<2.1'.

Licensing

  • This worker: MIT — see LICENSE.
  • yara-x (the scanning engine): BSD-3-Clause.
  • vgi / vgi-rpc (the worker SDK) and arrow-*: Apache-2.0.

Authorship & License

Written by Query.Farm.

Copyright 2026 Query Farm LLC - https://query.farm

Executable notebooks

See the executed SQL notebook for module discovery, rule validation, preserved sample coverage, matching rules and byte-level evidence. It demonstrates all seven public SQL objects with SVG charts, a parameter experiment and an Open in Colab link.