Skip to content

Commit 89060fc

Browse files
J0hnG4ltJ0hnG4lt
authored
Merge pull request #12 from J0hnG4lt/feat/add-gizmo-microservice-demo
feat: Add GizmoSQL microservice and end-to-end testing infrastructure
2 parents 485709b + d714431 commit 89060fc

File tree

10 files changed

+2586
-81
lines changed

10 files changed

+2586
-81
lines changed

‎.claude/commands/e2e-test.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
---
2+
description: Run end-to-end tests for the Metabase Arrow Flight SQL driver. Starts all services from scratch, runs setup, and validates the dashboard works.
3+
allowed-tools: Bash, Read, Write, Glob, Grep
4+
---
5+
6+
# End-to-End Test for Metabase Arrow Flight SQL Driver
7+
8+
Run a complete end-to-end test of the driver by starting all services fresh and validating functionality.
9+
10+
## Pre-requisites
11+
12+
- Podman must be installed and running
13+
- Python 3 must be available
14+
15+
## Test Steps
16+
17+
### 1. Clean up existing environment
18+
19+
```bash
20+
cd $PROJECT_ROOT
21+
podman compose down -v
22+
rm -f .env
23+
```
24+
25+
This stops all containers, removes volumes, and deletes any stale `.env` file for a fresh start.
26+
27+
**Important**: The `.env` file contains API keys from previous runs. If not deleted, the setup script will try to use the old (now invalid) key and fail with 401 errors.
28+
29+
### 2. Start all services
30+
31+
```bash
32+
podman compose up -d
33+
```
34+
35+
This starts:
36+
- **builder**: Compiles the driver JAR using Leiningen
37+
- **postgres**: Metabase application database
38+
- **spiced**: Spice.ai Flight SQL server (port 50051)
39+
- **gizmosql**: GizmoSQL Flight SQL server (port 31337)
40+
- **metabase**: Metabase BI tool (port 3000)
41+
42+
### 3. Wait for Metabase to be ready
43+
44+
```bash
45+
for i in {1..60}; do
46+
if podman exec metabase curl -s -f "http://localhost:3000/api/health" >/dev/null 2>&1; then
47+
echo "Metabase is ready!"
48+
break
49+
fi
50+
echo "Waiting... attempt $i"
51+
sleep 5
52+
done
53+
```
54+
55+
### 4. Run the setup script
56+
57+
```bash
58+
python scripts/metabase_setup.py
59+
```
60+
61+
This script:
62+
- Performs initial Metabase setup (creates admin user)
63+
- Creates API key and saves to `.env`
64+
- Creates GizmoSQL and Spice database connections
65+
- Syncs database schemas
66+
- Creates a comprehensive test dashboard with 32 cards and 5 field filters
67+
68+
### 5. Validate the dashboard
69+
70+
Open in browser: http://localhost:3000/dashboard/2
71+
72+
Or test via API (read API key from .env and use it directly):
73+
```bash
74+
# Read the API key from .env file
75+
API_KEY=$(grep METABASE_API_KEY .env | cut -d'=' -f2)
76+
77+
# Validate dashboard exists with expected cards and parameters
78+
podman exec metabase curl -s -H "x-api-key: $API_KEY" \
79+
"http://localhost:3000/api/dashboard/2" | python -c "
80+
import sys,json
81+
d=json.load(sys.stdin)
82+
print(f'Dashboard: {d.get(\"name\")}')
83+
print(f'Cards: {len(d.get(\"dashcards\",[]))}')
84+
print(f'Parameters: {len(d.get(\"parameters\",[]))}')
85+
"
86+
```
87+
88+
**Expected output:**
89+
```
90+
Dashboard: FlightSQL Driver Test Dashboard v2
91+
Cards: 32
92+
Parameters: 5
93+
```
94+
95+
### 6. Test a filtered query
96+
97+
```bash
98+
API_KEY=$(grep METABASE_API_KEY .env | cut -d'=' -f2)
99+
100+
podman exec metabase curl -s -H "x-api-key: $API_KEY" \
101+
-H "Content-Type: application/json" -X POST \
102+
"http://localhost:3000/api/card/44/query" \
103+
-d '{"parameters":[{"type":"string/=","value":["Delivered"],"target":["dimension",["template-tag","status"]]}]}' \
104+
| python -c "
105+
import sys,json
106+
d=json.load(sys.stdin)
107+
print(f'Status: {d.get(\"status\")}')
108+
print(f'Rows: {len(d.get(\"data\",{}).get(\"rows\",[]))}')
109+
"
110+
```
111+
112+
**Expected output:**
113+
```
114+
Status: completed
115+
Rows: 9
116+
```
117+
118+
### 7. Verify connection pool health
119+
120+
```bash
121+
podman logs metabase 2>&1 | grep -i "hash.*changed" | tail -5
122+
```
123+
124+
**Expected**: No output (no "Hash of database details changed" warnings). If you see these warnings, the connection pooling fix in the driver may have regressed.
125+
126+
## Expected Results
127+
128+
| Check | Expected |
129+
|-------|----------|
130+
| Dashboard cards | 32 |
131+
| Dashboard parameters | 5 (Order Status, Country, Department, Marketing Channel, Date Range) |
132+
| Filtered query status | `completed` |
133+
| Filtered query rows | 9 (for status=Delivered) |
134+
| Connection pool warnings | None ("Hash of database details changed" should not appear) |
135+
| GizmoSQL tables synced | 13 tables in memory catalog (analytics, hr, sales schemas) |
136+
| Spice tables synced | 1 table |
137+
138+
## Troubleshooting
139+
140+
### Volume removal error on Windows
141+
When running `podman compose down -v`, you may see an error like:
142+
```
143+
error during connect: Delete ".../volumes/...": EOF
144+
```
145+
This is a known issue on Windows with podman. The containers are still removed successfully - you can safely ignore this error.
146+
147+
### Check Metabase logs
148+
```bash
149+
podman logs metabase 2>&1 | tail -100
150+
```
151+
152+
### Check for connection pool issues
153+
```bash
154+
podman logs metabase 2>&1 | grep -i "hash.*changed\|connections:"
155+
```
156+
157+
### Check GizmoSQL logs
158+
```bash
159+
podman logs gizmosql 2>&1 | tail -50
160+
```
161+
162+
### Restart Metabase after driver changes
163+
```bash
164+
podman compose down metabase builder
165+
podman compose up -d builder
166+
# Wait for JAR build (~30 seconds)
167+
podman compose up -d metabase
168+
```
169+
170+
## Credentials
171+
172+
- **Metabase URL**: http://localhost:3000
173+
- **Admin Email**: admin@metabase.local
174+
- **Admin Password**: Metabase123!
175+
- **GizmoSQL**: gizmosql:31337 (user: gizmosql, pass: gizmosql_password)
176+
- **Spice**: spiced-container:50051 (token: 1234567890)

‎.claude/settings.local.json‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
{
2+
"permissions": {
3+
"allow": [
4+
"WebFetch(domain:github.com)",
5+
"Bash(dir \"C:\\\\Users\\\\DEEPGAMING\\\\workspace\\\\repos\\\\metabase-flightsql-driver\\\\data\")",
6+
"Bash(podman compose:*)",
7+
"Bash(podman machine start:*)",
8+
"Bash(podman logs:*)",
9+
"Bash(gh api:*)",
10+
"Bash(curl:*)",
11+
"Bash(python:*)",
12+
"WebFetch(domain:www.metabase.com)",
13+
"Bash(gh search code:*)",
14+
"Bash(podman network inspect:*)",
15+
"Bash(podman exec metabase curl:*)",
16+
"Bash(podman exec metabase timeout 5 bash -c \"echo > /dev/tcp/spiced-container/50051\")",
17+
"Bash(podman ps:*)",
18+
"Bash(start:*)",
19+
"WebSearch",
20+
"Bash(podman exec metabase sh -c \"tail -200 /var/log/metabase.log 2>/dev/null || journalctl -u metabase --no-pager -n 100 2>/dev/null || cat /tmp/*.log 2>/dev/null || echo 'Checking stdout logs...'\" 2>&1:*)",
21+
"Bash(for i in {1..30})",
22+
"Bash(do if podman exec metabase curl -s -f \"http://localhost:3000/api/health\")",
23+
"Bash(then echo \"Metabase is ready!\")",
24+
"Bash(break)",
25+
"Bash(fi)",
26+
"Bash(echo:*)",
27+
"Bash(done)",
28+
"Bash(for i in {1..60})",
29+
"Bash(if [ $i -eq 60 ])",
30+
"Bash(then echo \"Timeout\")",
31+
"Bash(cat:*)",
32+
"Bash(gh pr diff:*)",
33+
"Bash(then echo \"\")",
34+
"Skill(e2e-test)"
35+
]
36+
}
37+
}

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,6 @@ node_modules
1717
.clj-kondo/
1818
.cpcache/
1919
.lsp/
20+
21+
# Environment files (auto-generated by setup script)
22+
.env

0 commit comments

Comments
 (0)