Skip to content

Commit 713d717

Browse files
benaiadbenaiad
committed
docs: rewrite README and remove research directory
1 parent f448b5f commit 713d717

5 files changed

Lines changed: 90 additions & 931 deletions

File tree

‎README.md‎

Lines changed: 90 additions & 127 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,19 @@
11
# sqlize
22

3-
SQL interface for REST APIs.
3+
An SQL layer for REST API endpoints.
44

5-
Point sqlize at an OpenAPI spec and query any REST API using SQL. Path parameters become `WHERE` clauses, query parameters become filters, response fields become columns. One tool, any API.
5+
Query GitHub issues, Stripe customers, GitLab pipelines — any REST API that has an OpenAPI spec — using plain SQL. Pagination, auth, filter pushdown, and cross-API JOINs are handled automatically.
66

77
![sqlize demo](vhs-demo/demo.gif)
88

9-
## How it works
10-
11-
```sh
12-
export SQLIZE_BEARER_ENV_VAR=GITHUB_TOKEN
13-
sqlize --spec specs/github-minimal.json --format toon
9+
```sql
10+
SELECT number, title, state
11+
FROM issues
12+
WHERE owner = 'rust-lang' AND repo = 'rust' AND state = 'open'
13+
LIMIT 5;
1414
```
1515

16-
```sql
17-
sqlize> SELECT number, title, state FROM issues
18-
> WHERE owner = 'rust-lang' AND repo = 'rust' AND state = 'open'
19-
> LIMIT 5;
16+
```
2017
[5]{number,title,state}:
2118
154162,"(EXPERIMENT) Replace zero-deps nodes with a singleton",open
2219
154161,On E0277 tweak help when single type impls traits,open
@@ -25,25 +22,13 @@ sqlize> SELECT number, title, state FROM issues
2522
154157,Enforce deterministic signed zero behavior in float min/max and clamp,open
2623
```
2724

28-
Same tool, different API — Stripe:
29-
30-
```sh
31-
export SQLIZE_BEARER_ENV_VAR=STRIPE_TEST_API_KEY
32-
sqlize --spec specs/stripe-minimal.json
33-
```
25+
## Why SQL
3426

35-
```sql
36-
sqlize> SELECT email, name FROM customers;
37-
╭──────────────────────┬────────────────╮
38-
│ email │ name │
39-
├──────────────────────┼────────────────┤
40-
│ sp@summerproject.com │ Summer Project │
41-
╰──────────────────────┴────────────────╯
42-
```
27+
REST APIs are imperative — you need to know the endpoint, the parameters, the pagination scheme, the response shape. SQL is declarative — you say what you want and the engine figures out how to get it.
4328

44-
Powered by [Apache DataFusion](https://datafusion.apache.org/). Supports `SELECT`, `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`, `GROUP BY`, `HAVING`, `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `JOIN`, subqueries, CTEs, `UNION`/`INTERSECT`, `CASE`, `CAST`, and more. Read-only — no INSERT/UPDATE/DELETE.
29+
The mapping is natural: endpoints become tables, parameters become columns, and the query planner translates SQL into API calls. `WHERE owner = 'rust-lang'` becomes a path parameter in the URL. `WHERE state = 'open'` becomes `?state=open` in the query string. `ORDER BY`, `GROUP BY`, `LIMIT` are applied locally by the engine after fetching.
4530

46-
Results are returned in [TOON](https://github.com/toon-format/toon) (compact, token-oriented encoding, 40-50% smaller than JSON), JSON, or as a table.
31+
Powered by [Apache DataFusion](https://datafusion.apache.org/). Supports `SELECT`, `WHERE`, `ORDER BY`, `LIMIT`, `OFFSET`, `GROUP BY`, `HAVING`, `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `JOIN`, subqueries, CTEs, `UNION`/`INTERSECT`, `CASE`, `CAST`, and more. Read-only.
4732

4833
## Quickstart
4934

@@ -58,95 +43,64 @@ powershell -ExecutionPolicy Bypass -c "irm https://github.com/Benaiad/sqlize/rel
5843
cargo install sqlize
5944
```
6045

61-
Curated minimal specs ship with the repo:
62-
63-
| Spec | Tables | Auth | Notes |
64-
|------|--------|------|-------|
65-
| `specs/github-minimal.json` | 9 | Bearer token | Issues, PRs, commits, releases, repos |
66-
| `specs/gitlab-minimal.json` | 5 | Bearer token | Projects, issues, MRs, pipelines, members |
67-
| `specs/stripe-minimal.json` | 5 | Bearer token | Customers, charges, subscriptions, invoices, products |
68-
69-
Set your API token:
46+
### Query GitHub
7047

7148
```sh
72-
# Option 1: set the token directly
73-
export SQLIZE_BEARER_TOKEN=ghp_...
74-
75-
# Option 2: point to an existing env var (e.g., GITHUB_TOKEN)
49+
export GITHUB_TOKEN=ghp_...
7650
export SQLIZE_BEARER_ENV_VAR=GITHUB_TOKEN
51+
sqlize --spec specs/github-minimal.json
7752
```
7853

79-
### CLI
80-
81-
Single-shot commands for scripts and agents:
82-
83-
```sh
84-
sqlize --spec specs/github-minimal.json query "SELECT number, title FROM issues WHERE owner = 'rust-lang' AND repo = 'rust' LIMIT 5"
85-
sqlize --spec specs/github-minimal.json explain "SELECT ..."
86-
sqlize --spec specs/github-minimal.json schema issues
54+
```sql
55+
sqlize> SHOW TABLES;
56+
sqlize> DESCRIBE issues;
57+
sqlize> SELECT number, title FROM issues WHERE owner = 'rust-lang' AND repo = 'rust' LIMIT 5;
8758
```
8859

89-
Output is JSON by default, `--format toon` for compact output.
90-
91-
### Interactive REPL
60+
### Query Stripe
9261

9362
```sh
94-
sqlize --spec specs/github-minimal.json
63+
export STRIPE_API_KEY=sk_test_...
64+
export SQLIZE_BEARER_ENV_VAR=STRIPE_API_KEY
65+
sqlize --spec specs/stripe-minimal.json
9566
```
9667

68+
```sql
69+
sqlize> SELECT email, name FROM customers;
9770
```
98-
sqlize> SHOW TABLES
99-
sqlize> DESCRIBE issues
100-
sqlize> SELECT number, title FROM issues WHERE owner = 'rust-lang' AND repo = 'rust' LIMIT 5;
101-
```
102-
103-
Tab completion, SQL syntax highlighting, multiline input, persistent history.
10471

105-
With full OpenAPI specs, use `--tags` to filter endpoints by their OpenAPI [tag](https://swagger.io/docs/specification/v3_0/grouping-operations-with-tags/):
72+
### Query GitLab
10673

10774
```sh
108-
curl -L -o specs/github.json \
109-
https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json
75+
export GITLAB_TOKEN=glpat-...
76+
export SQLIZE_BEARER_ENV_VAR=GITLAB_TOKEN
77+
sqlize --spec specs/gitlab-minimal.json
78+
```
11079

111-
sqlize --spec specs/github.json --tags repos,issues
80+
```sql
81+
sqlize> SELECT title, state FROM issues WHERE id = '12345' LIMIT 5;
11282
```
11383

114-
### MCP server
84+
Curated minimal specs ship with the repo:
11585

116-
sqlize also runs as an MCP server, giving AI agents SQL access to APIs through three tools:
86+
| Spec | Tables | Notes |
87+
|------|--------|-------|
88+
| `specs/github-minimal.json` | 9 | Issues, PRs, commits, releases, repos |
89+
| `specs/gitlab-minimal.json` | 5 | Projects, issues, MRs, pipelines, members |
90+
| `specs/stripe-minimal.json` | 5 | Customers, charges, subscriptions, invoices, products |
11791

118-
- **`get_schema`** — returns `CREATE TABLE` DDL for table discovery
119-
- **`query`** — executes read-only SQL, returns TOON
120-
- **`explain`** — shows the execution plan without running it
92+
Any REST API with an OpenAPI spec works — these are just the ones we've tested.
12193

122-
```sh
123-
claude mcp add \
124-
--transport stdio \
125-
--env SQLIZE_SPEC_PATH=/path/to/specs/github-minimal.json \
126-
--env SQLIZE_BEARER_ENV_VAR=GITHUB_TOKEN \
127-
--scope user \
128-
sqlize-github -- sqlize mcp
129-
```
130-
131-
## Multi-spec federation
94+
## Cross-API JOINs
13295

133-
Query multiple APIs in a single session. Each `--spec` registers a named schema:
96+
Query multiple APIs in a single session:
13497

13598
```sh
13699
sqlize \
137100
--spec github:specs/github-minimal.json \
138101
--spec stripe:specs/stripe-minimal.json
139102
```
140103

141-
Use qualified table names to query across APIs:
142-
143-
```sql
144-
sqlize> SELECT name, stargazers_count FROM github.orgs_repos WHERE org = 'openclaw' LIMIT 5;
145-
sqlize> SELECT email, name FROM stripe.customers LIMIT 5;
146-
```
147-
148-
JOINs across APIs work too:
149-
150104
```sql
151105
SELECT c.name, c.email, k.commit_message
152106
FROM stripe.customers c
@@ -155,86 +109,95 @@ WHERE k.owner = 'openclaw' AND k.repo = 'openclaw'
155109
LIMIT 5;
156110
```
157111

112+
Each spec registers a named schema. Use qualified names (`github.issues`, `stripe.customers`) to query across APIs.
113+
158114
### Per-spec auth
159115

160-
Each spec resolves its token independently. Set per-spec env vars using the spec name (uppercased):
116+
Each spec resolves its token independently:
161117

162118
```sh
163-
# Per-spec tokens
164-
export SQLIZE_BEARER_TOKEN_GITHUB=ghp_...
165-
export SQLIZE_BEARER_TOKEN_STRIPE=sk_test_...
166-
167-
# Or per-spec env var indirection
168119
export SQLIZE_BEARER_ENV_VAR_GITHUB=GITHUB_TOKEN
169-
export SQLIZE_BEARER_ENV_VAR_STRIPE=STRIPE_TEST_API_KEY
120+
export SQLIZE_BEARER_ENV_VAR_STRIPE=STRIPE_API_KEY
170121
```
171122

172123
Falls back to `SQLIZE_BEARER_TOKEN` / `SQLIZE_BEARER_ENV_VAR` when no per-spec var is set.
173124

174-
### Schema name resolution
125+
## MCP server
175126

176-
The schema name is derived from the `--spec` flag:
127+
sqlize runs as an MCP server, giving AI agents SQL access to REST APIs through three tools:
177128

178-
| Flag | Schema name |
179-
|------|-------------|
180-
| `--spec github:specs/github.json` | `github` |
181-
| `--spec specs/github-minimal.json` | `github` (auto-derived, `-minimal` stripped) |
182-
| `--spec specs/stripe-minimal.json` | `stripe` |
129+
- **`get_schema`** — `CREATE TABLE` DDL for table discovery
130+
- **`query`** — executes SQL, returns results in TOON
131+
- **`explain`** — shows the execution plan without running it
132+
133+
Three tools. Not 25 per service. Adding more APIs adds tables, not tools — the context window cost stays flat.
134+
135+
```sh
136+
claude mcp add \
137+
--transport stdio \
138+
--env SQLIZE_SPEC_PATH=/path/to/specs/github-minimal.json \
139+
--env SQLIZE_BEARER_ENV_VAR=GITHUB_TOKEN \
140+
--scope user \
141+
sqlize-github -- sqlize mcp
142+
```
183143

184-
With a single `--spec`, bare table names work without a schema prefix.
144+
Results are returned in [TOON](https://github.com/toon-format/toon) format by default — 40-50% smaller than JSON, designed for LLM consumption.
185145

186-
## Aggregations
146+
## CLI reference
187147

188-
DataFusion provides full aggregate support:
148+
Single-shot commands for scripts and automation:
189149

190-
```sql
191-
SELECT language, COUNT(*) as count
192-
FROM orgs_repos
193-
WHERE org = 'openclaw'
194-
GROUP BY language
195-
ORDER BY count DESC;
150+
```sh
151+
sqlize --spec specs/github-minimal.json query "SELECT number, title FROM issues WHERE owner = 'rust-lang' AND repo = 'rust' LIMIT 5"
152+
sqlize --spec specs/github-minimal.json explain "SELECT ..."
153+
sqlize --spec specs/github-minimal.json schema issues
196154
```
197155

156+
Output is JSON by default, `--format toon` for compact output, `--format table` for human-readable tables.
157+
198158
## How queries map to API calls
199159

200160
```sql
201-
sqlize> EXPLAIN SELECT number, title FROM issues
202-
> WHERE owner = 'openclaw' AND repo = 'openclaw' AND state = 'open'
203-
> ORDER BY created_at DESC
204-
> LIMIT 10;
161+
EXPLAIN SELECT number, title FROM issues
162+
WHERE owner = 'openclaw' AND repo = 'openclaw' AND state = 'open'
163+
ORDER BY created_at DESC
164+
LIMIT 10;
205165
```
206166

207-
`WHERE` conditions on path parameters (`owner`, `repo`) are substituted into the URL. Query parameters (`state`) are pushed to the API as `?key=value`. Everything else (`ORDER BY`, `LIMIT`, `GROUP BY`, `JOIN`) is applied locally by DataFusion after the fetch.
167+
- `owner`, `repo` — path parameters, substituted into the URL
168+
- `state` — query parameter, pushed to the API as `?state=open`
169+
- `ORDER BY`, `LIMIT` — applied locally by DataFusion after the fetch
208170

209171
Path parameters are required — omitting them fails at query planning, before any HTTP call is made.
210172

211173
## Pagination
212174

213-
sqlize paginates automatically using the standard `Link` header (`rel="next"`) or common response body fields (`next`, `next_url`). This works with GitHub, GitLab, Stripe, and most REST APIs without configuration.
175+
sqlize paginates automatically using the standard `Link` header (`rel="next"`) or common response body fields (`next`, `next_url`). Works with GitHub, GitLab, Stripe, and most REST APIs without configuration.
214176

215177
Each table scan fetches pages lazily until one of these limits is reached:
216178

217-
- **SQL `LIMIT`** — when DataFusion can push it down (simple queries), only the needed pages are fetched
218-
- **`max_rows`** (default 1000) — caps total rows per table scan when no SQL `LIMIT` applies (e.g. JOINs, aggregations, or queries without `LIMIT`)
219-
220-
The current page always completes, so actual row count may slightly exceed the cap.
179+
- **SQL `LIMIT`** — only the needed pages are fetched
180+
- **`max_rows`** (default 1000) — caps total rows per table scan when no SQL `LIMIT` applies
221181

222182
Override the default:
223183

224184
```sh
225-
# CLI flag (one-off)
226185
sqlize --spec specs/github.json --max-rows 5000
227-
228-
# Environment variable (persistent)
186+
# or
229187
export SQLIZE_MAX_ROWS=5000
230188
```
231189

232-
## Why SQL
190+
## Bring your own API
191+
192+
sqlize works with any REST API that has an OpenAPI 3.x spec. For large specs, use `--tags` to filter endpoints:
233193

234-
REST APIs are imperative — you need to know the endpoint, the parameters, the pagination scheme, the response shape. SQL is declarative — you say what you want and the engine figures out how to get it. The mapping is natural: endpoints become tables, parameters become columns, and the query planner translates SQL into API calls.
194+
```sh
195+
curl -L -o specs/github.json \
196+
https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json
235197

236-
Research and competitive analysis in [`research/`](research/).
198+
sqlize --spec specs/github.json --tags repos,issues
199+
```
237200

238201
## Status
239202

240-
Research prototype. The core pipeline works end-to-end against live APIs. Not production-hardened.
203+
Early development — APIs may change.

0 commit comments

Comments
 (0)