Skip to content

Commit f10437f

Browse files
canyugsclaude
andcommitted
refactor: port tabbed Playground/Dashboard UI from zeabur-rag
- Merge Chat, Report, Learn into tabbed index.html - Add Inspect, Triage, Edit tabs to Playground - Simplify nav to [Playground | Dashboard] pill - Move Settings/Refresh to tab bar as icon-only buttons - Add tooltip help on stat cards (Triage, Dashboard, Query Signals) - Add source color legend in Query Signals - Fix formatAuditChanges to read old_value/new_value - Rename "Similarity" to "Score" with mode-aware tooltips - Redirect /report and /learn to /?tab=xxx - Remove standalone learn.html and report.html - Add zeabur-rag-feedback skill - Sync skill doc fixes (RRF vs cosine explanations) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 52e235c commit f10437f

11 files changed

Lines changed: 1087 additions & 633 deletions

File tree

skills/zeabur-rag-curate/SKILL.md

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Auth: `Authorization: Bearer $RAG_API_KEY` — **admin scope required**
1515
## Loop
1616

1717
1. **Call `zeabur-rag-triage`** to get the pending items list.
18-
2. **Present results** as a numbered list grouped by category (🔴 Reports, 🟡 Unverified, 🟠 Low-similarity, 🔵 Negative feedback). Show counts.
18+
2. **Present results** as a numbered list grouped by category (🔴 Reports, 🟡 Unverified, 🟠 Low-score, 🔵 Negative feedback, ⚪ Needs-frontmatter). Show counts.
1919
3. **If all categories are empty:** tell the user "The knowledge base has no pending items — looks healthy ✅" and stop.
2020
4. **Wait for user** to pick an item by number or description.
2121
5. **Run the decision tree** for that item type (see below).
@@ -51,7 +51,7 @@ Auth: `Authorization: Bearer $RAG_API_KEY` — **admin scope required**
5151
| Correct but needs polish | `zeabur-rag-edit` to fix → then verify |
5252
| Low quality or wrong | Reject (see Inline commands) |
5353

54-
### 🟠 Low-similarity signal
54+
### 🟠 Low-score signal
5555

5656
1. Call `zeabur-rag-search` with the original query to confirm the gap.
5757
2. Classify:
@@ -75,6 +75,30 @@ Auth: `Authorization: Bearer $RAG_API_KEY` — **admin scope required**
7575
| Answer was right but user query was ambiguous | Dismiss — no action |
7676
| No chunks to inspect (null top_chunk_ids) | Treat as knowledge gap → `zeabur-rag-learn` |
7777

78+
### ⚪ Needs-frontmatter
79+
80+
A chunk was ingested from a source file that has no frontmatter. Its ID is a
81+
temporary path-hash (`MAN-tmp-*`), and it is excluded from some reorg-safety
82+
guarantees. The fix is to add frontmatter to the source file in the origin
83+
repo (e.g. `zebra-manual`), then re-ingest.
84+
85+
1. Call `zeabur-rag-inspect` to show the chunk's `url` and content.
86+
2. Tell the user which file to edit (the `url` field points at the repo-relative
87+
path, e.g. `docs/customer-support/xxx.md`).
88+
3. Direct them to the backfill tool in the origin repo. For `zebra-manual`, that
89+
tool lives in `zebra-manual/scripts/` and walks through one file at a time
90+
with LLM-drafted frontmatter + human confirmation.
91+
4. After the user finishes adding frontmatter, tell them to re-run the pipeline:
92+
`bun run pipeline --adapter zebra-manual --input /path/to/zebra-manual/docs`
93+
5. The adapter's reverse-lookup will automatically delete the old `tmp-*` chunks
94+
and insert the new permanent-ID chunks in place. No manual DB surgery.
95+
96+
| Classification | Action |
97+
|---------------|--------|
98+
| File is real and worth keeping | User adds frontmatter in origin repo → re-ingest |
99+
| File is not worth keeping in KB | Reject the chunk (see Inline commands); user also deletes the source file |
100+
| File is a duplicate of another chunk | Reject this chunk; point user at the canonical one |
101+
78102
## Inline commands
79103

80104
These actions don't have dedicated skills. Execute them as direct API calls:
@@ -86,7 +110,7 @@ curl -s -X POST "$ZEABUR_RAG_URL/api/admin/learned/<id>/verify" \
86110
-H "Authorization: Bearer $RAG_API_KEY"
87111
```
88112

89-
Returns `{"success": true}`. The chunk now ranks at full weight in search (unverified `learned` chunks are downranked to ×0.7 similarity).
113+
Returns `{"success": true}`. The chunk now ranks at full weight in search (unverified `learned` chunks are downranked to ×0.7 score).
90114

91115
### Reject a learned chunk (soft delete)
92116

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
name: zeabur-rag-feedback
3+
description: Send feedback on RAG search results — thumbs-up (+1) or thumbs-down (-1) to indicate whether the answer was helpful. Use after calling zeabur-rag-search when you can evaluate the quality of the results. Positive feedback reinforces good results; negative feedback flags poor matches for review. Requires write:feedback scope.
4+
---
5+
6+
# RAG Feedback
7+
8+
Send feedback on search results to improve knowledge base quality over time.
9+
10+
Base URL: `$ZEABUR_RAG_URL`
11+
Auth: `Authorization: Bearer $RAG_API_KEY`
12+
13+
## API
14+
15+
```bash
16+
curl -s -X POST "$ZEABUR_RAG_URL/api/feedback" \
17+
-H "Authorization: Bearer $RAG_API_KEY" \
18+
-H "Content-Type: application/json" \
19+
-d '{
20+
"query": "how to deploy to zeabur",
21+
"answer": "the generated answer text",
22+
"chunks": ["DOC-1234", "SUP-5678"],
23+
"score": 1,
24+
"signal_id": "optional-signal-id",
25+
"comment": "optional explanation"
26+
}'
27+
```
28+
29+
| Field | Required | Description |
30+
|-------|----------|-------------|
31+
| `query` | Yes | The original search query |
32+
| `answer` | No | The generated RAG answer |
33+
| `chunks` | No | Array of chunk IDs that were returned |
34+
| `score` | Yes | `1` (helpful) or `-1` (not helpful) |
35+
| `signal_id` | No | Signal ID from the search response — links feedback to the specific query signal |
36+
| `comment` | No | Free-text explanation of why the result was good or bad |
37+
38+
## When to send feedback
39+
40+
| Situation | Score |
41+
|-----------|-------|
42+
| Answer correctly addressed the question | `1` |
43+
| Retrieved chunks were relevant and accurate | `1` |
44+
| Answer was wrong, incomplete, or misleading | `-1` |
45+
| Retrieved chunks were irrelevant to the query | `-1` |
46+
| No useful results were returned | `-1` |
47+
48+
## Notes
49+
50+
- If `signal_id` is provided, the feedback score is written back to the `rag_query_signals` table, linking it to the original query for analytics.
51+
- Negative feedback surfaces in the Triage review queue for admin follow-up.
52+
- Feedback is also appended to `data/user-feedback.jsonl` as a backup log.

skills/zeabur-rag-inspect/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ curl -s "$ZEABUR_RAG_URL/api/admin/chunks/<chunk_id>" \
4242
{ "id": "...", "type": "outdated", "query": "...", "detail": "...", "status": "open", "created_at": "..." }
4343
],
4444
"signals": [
45-
{ "id": "...", "query": "...", "mode": "hybrid", "top_similarity": 0.45, "feedback_score": null, "created_at": "..." }
45+
{ "id": "...", "query": "...", "mode": "hybrid", "top_similarity": 0.012, "feedback_score": null, "created_at": "..." }
4646
],
4747
"audit_log": [
4848
{ "id": "...", "chunk_id": "SUP-1234", "action": "edit", "old_value": {}, "new_value": {}, "created_at": "..." }

skills/zeabur-rag-search/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ curl -s -X POST "$ZEABUR_RAG_URL/api/query" \
3434

3535
**Always pass `client: "claude-code"` when calling from Claude Code** so queries are trackable in the dashboard.
3636

37-
Response includes `chunks[]` (each with `id`, `title`, `answer`, `tags`, `similarity`, `source`, `verified`) and `signal_id` for linking feedback.
37+
Response includes `chunks[]` (each with `id`, `title`, `answer`, `tags`, `similarity` (relevance score), `source`, `verified`) and `signal_id` for linking feedback. Note: `similarity` is an RRF score in hybrid mode (~0–0.02) or cosine similarity in semantic mode (0–1). The scale depends on the search mode — do not compare values across modes.
3838

3939
## Tips
4040

skills/zeabur-rag-triage/SKILL.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
---
22
name: zeabur-rag-triage
3-
description: List pending knowledge base maintenance items — open reports, unverified learned chunks, low-similarity queries, and negative feedback. Use when checking KB health, reviewing the maintenance queue, or starting a curation session. Triggers on "review queue", "what needs attention in the KB", "pending reports", "unverified chunks", "failed queries", "KB health check". Requires admin scope.
3+
description: List pending knowledge base maintenance items — open reports, unverified learned chunks, low-score queries, and negative feedback. Use when checking KB health, reviewing the maintenance queue, or starting a curation session. Triggers on "review queue", "what needs attention in the KB", "pending reports", "unverified chunks", "failed queries", "KB health check". Requires admin scope.
44
---
55

66
# RAG — Triage
77

8-
Show what needs attention in the knowledge base. Returns 4 categories of pending items in a single call.
8+
Show what needs attention in the knowledge base. Returns 5 categories of pending items in a single call.
99

1010
Base URL: `$ZEABUR_RAG_URL`
1111
Auth: `Authorization: Bearer $RAG_API_KEY`**admin scope required**
@@ -20,17 +20,18 @@ curl -s "$ZEABUR_RAG_URL/api/admin/triage?days=14&similarity_threshold=0.4&limit
2020
| Param | Default | Description |
2121
|-------|---------|-------------|
2222
| `days` | `14` | Time window for signals (reports and learned have no time window) |
23-
| `similarity_threshold` | `0.4` | Queries with `top_similarity` below this are flagged |
23+
| `similarity_threshold` | `0.4` | Queries with `top_similarity` below this are flagged. Note: in hybrid mode (RRF), scores are ~0–0.02 so the default 0.4 catches all hybrid queries. Adjust to ~0.005 for meaningful hybrid-mode filtering. |
2424
| `limit` | `20` | Max items per category |
2525

2626
## Response
2727

28-
Returns JSON with 4 arrays + `counts` (true totals, not truncated) + `has_more` (boolean per category):
28+
Returns JSON with 5 arrays + `counts` (true totals, not truncated) + `has_more` (boolean per category):
2929

3030
- **`open_reports`** — reports with `status='open'`, no time window. Fields: `id`, `type`, `chunk_id`, `query`, `detail`, `created_at`.
3131
- **`unverified_learned`** — learned chunks with `status='unverified'`, no time window. Fields: `id`, `title`, `tags`, `created_at`, `age_days`.
32-
- **`low_similarity_signals`** — queries where the best result scored below the threshold, within the time window, deduped by query text. Fields: `id`, `query`, `top_similarity`, `top_chunk_ids`, `client`, `created_at`.
32+
- **`low_similarity_signals`** — queries where the top score fell below the threshold, within the time window, deduped by query text. The score is RRF in hybrid mode (~0–0.02) or cosine in semantic mode (0–1). Fields: `id`, `query`, `top_similarity`, `top_chunk_ids`, `client`, `created_at`.
3333
- **`negative_feedback_signals`** — queries that got negative user feedback, within the time window. Fields: `id`, `query`, `feedback_score`, `feedback_comment`, `top_chunk_ids`, `created_at`.
34+
- **`needs_frontmatter`** — chunks whose tags include `"needs-frontmatter"`, meaning they were ingested from a source file that lacks proper frontmatter and got a temporary ID. Fields: `id`, `title`, `tags`, `url`, `source`, `created_at`.
3435

3536
If a category query fails, its array is empty and a `<category>_error` field appears with the error message.
3637

src/server.ts

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1182,9 +1182,13 @@ const server = serve({
11821182
"/": "ui/index.html",
11831183
"/index.html": "ui/index.html",
11841184
"/dashboard": "ui/dashboard.html",
1185-
"/report": "ui/report.html",
1186-
"/learn": "ui/learn.html",
11871185
};
1186+
// Redirect old standalone pages to tabbed UI
1187+
if (url.pathname === "/report" || url.pathname === "/learn") {
1188+
const tab = url.pathname.slice(1);
1189+
const qs = url.search ? "&" + url.searchParams.toString() : "";
1190+
return Response.redirect(new URL("/?tab=" + tab + qs, url.origin), 302);
1191+
}
11881192
if (url.pathname in uiRoutes) {
11891193
if (!checkBasicAuth(req)) return unauthorizedResponse();
11901194
return new Response(Bun.file(uiRoutes[url.pathname]), {

0 commit comments

Comments
 (0)