Skip to content

Commit 63e8046

Browse files
committed
feat: generic jq support (gojq) — POST /jq, jq-update, response transforms
One shared evaluation core (new jqeval package): compile cache, timeout, size caps, exactly-one-output; disabled by default via the JQ config section. - PATCH ?jq= atomic read-modify-write (FOR UPDATE, all-or-nothing, RLS/triggers unchanged), program alternatively in the body as application/vnd.smoothdb.jq - jq=/jq_args= response transforms on table reads and RPC - POST /jq batch eval + parse_only validation
1 parent e7c96a0 commit 63e8046

17 files changed

Lines changed: 1770 additions & 48 deletions

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
11
# Change Log
22

3+
## 0.8.0 - 2026-07-07
4+
5+
### Added
6+
* Generic jq support via [gojq](https://github.com/itchyny/gojq), disabled by default (`JQ.Enabled`). Evaluation is bounded: no I/O, per-evaluation timeout (`JQ.Timeout`), program size cap (`JQ.MaxProgramBytes`), compiled-program LRU cache (`JQ.CacheEntries`), and an exactly-one-output convention. Three surfaces over one shared core (new `jqeval` package):
7+
* **jq updates**: `PATCH /table?filters&jq=<program>` performs an atomic read-modify-write of the matched rows — rows are selected `FOR UPDATE` inside the request transaction, each row is fed to the program, and its object output becomes that row's `UPDATE ... SET`. All-or-nothing, capped by `JQ.MaxUpdateRows`; RLS and triggers apply unchanged. The program can also be sent as the raw request body with `Content-Type: application/vnd.smoothdb.jq`
8+
* **Response transforms**: `jq=` on table reads and RPC calls transforms the JSON response body; `Content-Range`/count headers reflect the pre-transform result set
9+
* **`POST /jq`**: standalone batch evaluation endpoint with per-item errors and `parse_only` compile-checking for authoring-time validation
10+
* `jq_args=` (URL-encoded JSON object) binds values as jq variables on both updates and transforms
11+
312
## 0.7.2 - 2026-06-30
413

514
### Added

README.md

Lines changed: 85 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Please create issues to let me know your priorities.
2121

2222
## About this project
2323

24-
SmoothDB is not *vibe-coded*. It was started in 2022 as a solid, versatile middleware meant to serve as a reliable building block for data-driven applications. Since 2025 we have been pairing that foundation with strong LLMs to harden, review, and extend the code aiming to follow the high bar set by PostgREST in correctness, safety, and API fidelity.
24+
SmoothDB is not *vibe-coded*. It was started in 2022 as a solid, versatile middleware meant to serve as a reliable building block for data-driven applications. Since 2025 we have been pairing that foundation with strong LLMs to harden, review, and extend the code - aiming to follow the high bar set by PostgREST in correctness, safety, and API fidelity.
2525

2626
## Getting started
2727

@@ -407,7 +407,7 @@ GET /api/testdb/employees?id=start.1&manager_id=recurse.3 HTTP/1.1
407407

408408
Returns row `id=1` and its descendants up to 3 levels deep, following `manager_id → id`. Use `recurse.all` (or bare `recurse`) for unlimited depth (capped by `MaxRecursiveDepth`), and `after` instead of `start` to exclude the seed row.
409409

410-
**Result vs. traversal filters.** A plain filter restricts the *result*: the whole subtree is walked, then non-matching rows are dropped. Prefix a filter with `walk.` to prune the *traversal* instead a non-matching node, and everything beyond it, is skipped:
410+
**Result vs. traversal filters.** A plain filter restricts the *result*: the whole subtree is walked, then non-matching rows are dropped. Prefix a filter with `walk.` to prune the *traversal* instead - a non-matching node, and everything beyond it, is skipped:
411411

412412
```http
413413
GET /api/testdb/employees?id=start.1&manager_id=recurse.all&is_active=is.true HTTP/1.1 # keep only active rows
@@ -429,6 +429,84 @@ GET /api/testdb/documents?id=after.1&id=recurse.all&relationships=via!both(src_i
429429

430430
Embedding is not supported together with `via` traversal.
431431

432+
### jq Support
433+
434+
> [!NOTE]
435+
> This is a SmoothDB extension to PostgREST syntax.
436+
437+
SmoothDB can evaluate [jq](https://jqlang.github.io/jq/) programs server-side, via [gojq](https://github.com/itchyny/gojq). The feature is **disabled by default**: set `JQ.Enabled` to `true` in the configuration to use it. Evaluation is strictly bounded - programs have no I/O, run under a timeout (`JQ.Timeout`, default 250ms) and a size cap (`JQ.MaxProgramBytes`), and must produce **exactly one** output value (wrap streams in an array: `[.items[] | ...]`).
438+
439+
There are three surfaces, sharing the same evaluation core:
440+
441+
#### jq updates
442+
443+
`PATCH` with a `jq=` query parameter (and an empty body) performs an atomic read-modify-write of the matched rows, with no client round trip:
444+
445+
```http
446+
PATCH /api/testdb/products?id=eq.42&jq={"counter": (.counter + 1)} HTTP/1.1
447+
```
448+
449+
Within the request transaction, the matched rows are selected `FOR UPDATE`; each row (a JSON object with all the columns visible to the role) is fed to the program, whose output must be a JSON object of columns to update - it becomes that row's `UPDATE ... SET` (an empty object `{}` leaves the row untouched). Any error - parse, evaluation, non-object output, unknown column - aborts the whole request: all rows or none. Row level security and triggers apply as in a normal update. At most `JQ.MaxUpdateRows` (default 1000) rows can be updated in one request.
450+
451+
`Prefer: return=representation` returns the resulting rows (all visible columns). Arguments can be passed with `jq_args=` (a URL-encoded JSON object) and are available as jq variables:
452+
453+
```http
454+
PATCH /api/testdb/products?id=eq.42&jq={"stock": (.stock - $n)}&jq_args={"n": 3} HTTP/1.1
455+
```
456+
457+
For longer programs, the raw program text can be sent as the request body with `Content-Type: application/vnd.smoothdb.jq` instead of the `jq=` parameter — no URL encoding, newlines and `#` comments allowed (`jq_args=` stays in the query string):
458+
459+
```http
460+
PATCH /api/testdb/products?id=eq.42&jq_args={"n": 3} HTTP/1.1
461+
Content-Type: application/vnd.smoothdb.jq
462+
463+
# restock and log
464+
{
465+
"stock": (.stock + $n),
466+
"history": (.history + [{restocked: $n}])
467+
}
468+
```
469+
470+
#### Response transforms
471+
472+
`jq=` (with optional `jq_args=`) on table reads and on function calls (`GET /rpc/fn`, `POST /rpc/fn`) transforms the JSON response body before it is returned:
473+
474+
```http
475+
GET /api/testdb/products?category=eq.tools&jq=map(.name) HTTP/1.1
476+
GET /api/testdb/products?jq={total: (map(.price) | add)} HTTP/1.1
477+
```
478+
479+
The transform applies after the query: filters, `select`, `order`, `limit` and the `Content-Range`/count headers all reflect the pre-transform result set. Only JSON content types can be transformed (`Accept: text/csv` with `jq=` is an error).
480+
481+
#### POST /jq
482+
483+
A standalone endpoint (behind the normal authentication) evaluates a batch of programs against provided inputs - useful for testing programs and for authoring-time validation with `parse_only`:
484+
485+
```http
486+
POST /jq HTTP/1.1
487+
488+
{
489+
"parse_only": false,
490+
"evals": [
491+
{"program": ".a + $delta", "input": {"a": 1}, "args": {"delta": 41}}
492+
]
493+
}
494+
```
495+
496+
The response is a `200` array with one item per evaluation: `{"output": ...}` or `{"error": "..."}` - errors are reported per item. With `"parse_only": true` each program is only compile-checked (no input needed). The endpoint is not registered when `JQ.Enabled` is false (404).
497+
498+
#### Configuration
499+
500+
```json
501+
{
502+
"JQ": {
503+
"Enabled": true
504+
}
505+
}
506+
```
507+
508+
See the [configuration table](#configuration-file) for `JQ.Timeout`, `JQ.MaxProgramBytes`, `JQ.MaxUpdateRows` and `JQ.CacheEntries`.
509+
432510
## Example for using SmoothDB in your application
433511

434512
You can embed SmoothDB functionalities in your backend app with relative ease.
@@ -693,6 +771,11 @@ The configuration file *config.jsonc* (JSON with Comments) is created automatica
693771
| Database.TransactionMode | General transaction mode for operations: "none", "commit", "rollback" | "none" |
694772
| Database.AggregatesEnabled | Enable aggregate functions | true |
695773
| Database.MaxRecursiveDepth | Maximum recursive query depth; 0 disables recursive queries | 100 |
774+
| JQ.Enabled | Enable jq evaluation: /jq route, jq= query parameter | false |
775+
| JQ.Timeout | Timeout in milliseconds for a single jq evaluation | 250 |
776+
| JQ.MaxProgramBytes | Maximum size in bytes for a jq program or its arguments | 4096 |
777+
| JQ.MaxUpdateRows | Maximum number of rows updatable with a single jq update | 1000 |
778+
| JQ.CacheEntries | Size of the compiled jq program cache | 256 |
696779
| Logging.Level | Log level: trace, debug, info, warn, error, fatal, panic | "info" |
697780
| Logging.FileLogging | Enable logging to file | true |
698781
| Logging.FilePath | File path for file-based logging | "./smoothdb.log" |

api/jq.go

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
package api
2+
3+
import (
4+
"bytes"
5+
"context"
6+
"encoding/json"
7+
"fmt"
8+
"io"
9+
"mime"
10+
"net/http"
11+
12+
"github.com/sted/heligo"
13+
"github.com/sted/smoothdb/database"
14+
"github.com/sted/smoothdb/jqeval"
15+
)
16+
17+
// maxJQBatchEvals caps the number of evaluations in a single POST /jq call
18+
const maxJQBatchEvals = 200
19+
20+
type jqEvalItem struct {
21+
Program string `json:"program"`
22+
Input json.RawMessage `json:"input"`
23+
Args map[string]any `json:"args"`
24+
}
25+
26+
type jqBatchRequest struct {
27+
ParseOnly bool `json:"parse_only"`
28+
Evals []jqEvalItem `json:"evals"`
29+
}
30+
31+
type jqOutputItem struct {
32+
Output json.RawMessage `json:"output"`
33+
}
34+
35+
type jqErrorItem struct {
36+
Error string `json:"error"`
37+
}
38+
39+
// InitJQRoute registers the standalone jq evaluation endpoint.
40+
// It is called only when jq evaluation is enabled, so the route 404s otherwise.
41+
func InitJQRoute(apiHelper Helper) {
42+
router := apiHelper.GetRouter()
43+
jq := router.Group("/jq", apiHelper.MiddlewareDBE())
44+
45+
// POST /jq: batch evaluation of jq programs, or compile-only validation
46+
// with parse_only. Errors are reported per item; 400 is reserved for a
47+
// malformed envelope.
48+
jq.Handle("POST", "", func(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
49+
var req jqBatchRequest
50+
err := r.ReadJSON(&req)
51+
if err != nil {
52+
return WriteBadRequest(w, err)
53+
}
54+
if len(req.Evals) > maxJQBatchEvals {
55+
return WriteBadRequest(w, fmt.Errorf("too many evals in a single call (max %d)", maxJQBatchEvals))
56+
}
57+
results := make([]any, 0, len(req.Evals))
58+
for _, item := range req.Evals {
59+
results = append(results, jqEvalOne(c, &item, req.ParseOnly))
60+
}
61+
return heligo.WriteJSON(w, http.StatusOK, results)
62+
})
63+
}
64+
65+
// jqContentType is the media type for a raw jq program in a request body.
66+
// There is no registered media type for jq, so we use the vendor tree,
67+
// following the application/vnd.pgrst.* precedent.
68+
const jqContentType = "application/vnd.smoothdb.jq"
69+
70+
// hasJQBody reports whether the request carries a raw jq program in its body
71+
func hasJQBody(r heligo.Request) bool {
72+
ct, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
73+
return err == nil && ct == jqContentType
74+
}
75+
76+
// jqUpdateHandler handles a jq-update PATCH /{table}?{filters}: an atomic
77+
// read-modify-write of the matched rows, driven by a jq program given either
78+
// in the jq= query parameter (with an empty body) or as the raw request body
79+
// with Content-Type: application/jq.
80+
func jqUpdateHandler(c context.Context, w http.ResponseWriter, r heligo.Request, sourcename string) (int, error) {
81+
body, err := io.ReadAll(r.Body)
82+
if err != nil {
83+
return WriteBadRequest(w, err)
84+
}
85+
options := database.GetQueryOptions(c)
86+
if hasJQBody(r) {
87+
if options.JQ != "" {
88+
return WriteBadRequest(w, fmt.Errorf("the jq program must be given either in the jq= parameter or in the request body, not both"))
89+
}
90+
options.JQ = string(body)
91+
} else if len(bytes.TrimSpace(body)) != 0 {
92+
return WriteBadRequest(w, fmt.Errorf("a request body and the jq= parameter cannot be used together"))
93+
}
94+
data, count, err := database.UpdateRecordsWithJQ(c, sourcename, r.URL.Query())
95+
if err == nil {
96+
SetResponseHeaders(c, w, r, count)
97+
if data == nil {
98+
return heligo.WriteHeader(w, http.StatusNoContent)
99+
} else {
100+
return WriteContent(c, w, http.StatusOK, data)
101+
}
102+
} else {
103+
return WriteError(w, err)
104+
}
105+
}
106+
107+
func jqEvalOne(ctx context.Context, item *jqEvalItem, parseOnly bool) any {
108+
if parseOnly {
109+
if err := jqeval.Parse(item.Program, item.Args); err != nil {
110+
return jqErrorItem{err.Error()}
111+
}
112+
return struct{}{}
113+
}
114+
var input any
115+
if len(item.Input) != 0 {
116+
var err error
117+
input, err = jqeval.Unmarshal(item.Input)
118+
if err != nil {
119+
return jqErrorItem{err.Error()}
120+
}
121+
}
122+
output, err := jqeval.Eval(ctx, item.Program, input, item.Args)
123+
if err != nil {
124+
return jqErrorItem{err.Error()}
125+
}
126+
encoded, err := jqeval.Marshal(output)
127+
if err != nil {
128+
return jqErrorItem{err.Error()}
129+
}
130+
return jqOutputItem{encoded}
131+
}

api/sources.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ func InitSourcesRouter(apiHelper Helper) {
2626
api.Handle("GET", "/:sourcename", func(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
2727
sourcename := r.Param("sourcename")
2828
json, count, err := database.GetRecords(c, sourcename, r.URL.Query())
29+
if err == nil {
30+
json, err = database.JQTransformResponse(c, json)
31+
}
2932
if err == nil {
3033
status := SetResponseHeaders(c, w, r, count)
3134
if status >= http.StatusBadRequest {
@@ -61,6 +64,9 @@ func InitSourcesRouter(apiHelper Helper) {
6164

6265
api.Handle("PATCH", "/:sourcename", func(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
6366
sourcename := r.Param("sourcename")
67+
if database.GetQueryOptions(c).JQ != "" || hasJQBody(r) {
68+
return jqUpdateHandler(c, w, r, sourcename)
69+
}
6470
records, status, err := ReadRequest(c, w, r)
6571
if err != nil || status != 0 {
6672
return status, err
@@ -98,6 +104,9 @@ func InitSourcesRouter(apiHelper Helper) {
98104
api.Handle("GET", "/rpc/:fname", func(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
99105
fname := r.Param("fname")
100106
json, count, err := database.ExecFunction(c, fname, nil, r.URL.Query(), true)
107+
if err == nil {
108+
json, err = database.JQTransformResponse(c, json)
109+
}
101110
if err == nil {
102111
status := SetResponseHeaders(c, w, r, count)
103112
if status >= http.StatusBadRequest {
@@ -119,6 +128,9 @@ func InitSourcesRouter(apiHelper Helper) {
119128
return status, err
120129
}
121130
data, count, err := database.ExecFunction(c, fname, records[0], r.URL.Query(), false)
131+
if err == nil {
132+
data, err = database.JQTransformResponse(c, data)
133+
}
122134
if err == nil {
123135
SetResponseHeaders(c, w, r, count)
124136
if data == nil {

api/utils.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import (
2020
"github.com/jackc/pgx/v5/pgconn"
2121
"github.com/sted/heligo"
2222
"github.com/sted/smoothdb/database"
23+
"github.com/sted/smoothdb/jqeval"
2324
)
2425

2526
var verboseErrors = true
@@ -107,7 +108,7 @@ func WriteServerError(w http.ResponseWriter, err error) (int, error) {
107108
func WriteError(w http.ResponseWriter, err error) (int, error) {
108109
var status int
109110
switch err.(type) {
110-
case *database.ParseError, *database.BuildError:
111+
case *database.ParseError, *database.BuildError, *jqeval.Error:
111112
return WriteBadRequest(w, err)
112113
case *database.SerializeError, *database.ContentTypeError:
113114
status = http.StatusNotAcceptable

0 commit comments

Comments
 (0)