Skip to content

Commit 42ae7f8

Browse files
committed
add rule agent
1 parent 4e316ea commit 42ae7f8

7 files changed

Lines changed: 314 additions & 9 deletions

File tree

.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ dist/
66
*.log
77

88

9-
TEST.md
109

1110
# Test fixtures intentionally verify ignored paths.
1211
!packages/cli/test/fixtures/nextjs-project/.env

AGENTS.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,31 @@
11
# DevMap — Agent
22

3+
When working in this repository:
4+
5+
- Create a branch before making changes unless the user explicitly asks to work on the current branch.
6+
- Use branch names with the `codex/` prefix, for example `codex/fix-analyzer`.
7+
- Keep changes scoped to the user request.
8+
- Do not revert unrelated user changes.
9+
- Prefer opening a pull request instead of pushing directly to `main`, unless the user explicitly asks to push to `main`.
10+
11+
## Commits
12+
13+
- Commit AI-authored work as:
14+
15+
```bash
16+
git -c user.name="devmap-agent" -c user.email="238585242+devmap-agent@users.noreply.github.com" commit -m "Clear commit message"
17+
```
18+
19+
- Do not change global or repository Git config for the author identity unless the user explicitly asks.
20+
- If the user wants their own account as the main author, keep their author identity and add this trailer to the commit message:
21+
22+
```text
23+
Co-authored-by: devmap-agent <238585242+devmap-agent@users.noreply.github.com>
24+
```
25+
26+
- Keep commit messages short, specific, and focused on the user-visible fix or feature.
27+
28+
329
read PRD and all documentation first before hands-on.
430

531
## Source of Truth
@@ -8,6 +34,9 @@ The PRD is the source of truth for product behavior, requirements, and prioritie
834

935
If there is any conflict between the PRD and other documents (README, docs, comments, plans, or specifications), follow the PRD.
1036

37+
38+
39+
1140
## Relevant Skills
1241

1342
ECC skills are stored in `.agents/skills/`.

docs/for-me-personal/DEBUG.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -558,6 +558,54 @@ cache yang sama. Smoke test distribusi harus deterministik dan berurutan.
558558

559559
---
560560

561+
## 11. Pertanyaan NextAuth Jatuh ke Fallback Critical File
562+
563+
**Tanggal:** 2026-06-11
564+
565+
**Status:** Selesai.
566+
567+
### Gejala
568+
569+
Pada benchmark Context Builder, pertanyaan:
570+
571+
```text
572+
Explain the NextAuth configuration
573+
```
574+
575+
menempatkan `app/api/session/route.ts` di posisi pertama, bukan `lib/auth.ts`.
576+
File yang benar masih masuk top-3, tetapi top-1 accuracy hanya 19/20.
577+
578+
### Akar Masalah
579+
580+
Keyword tokenizer menghasilkan `nextauth`, sedangkan concept alias auth hanya
581+
memiliki `auth` dan tidak memiliki bentuk nama provider tersebut. Karena tidak
582+
ada direct match, ranking jatuh ke fallback critical file.
583+
584+
### Solusi
585+
586+
Tambahkan `nextauth` ke concept alias authentication.
587+
588+
### Verifikasi
589+
590+
```powershell
591+
pnpm --filter devmap exec tsx --test test/context-builder-eval.test.ts
592+
```
593+
594+
Hasil:
595+
596+
```text
597+
Context Builder top-1 accuracy: 20/20
598+
Context Builder top-3 recall: 20/20
599+
```
600+
601+
### Pelajaran
602+
603+
Concept alias perlu mencakup nama provider populer, bukan hanya nama concern
604+
generik. Eval bilingual membantu menemukan gap yang tidak terlihat dari unit
605+
test satu pertanyaan.
606+
607+
---
608+
561609
## Checklist Saat Menambahkan Debug Baru
562610

563611
Tambahkan catatan baru ketika:

docs/for-me-personal/PROGRESS.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,11 @@ Terakhir diperbarui: 2026-06-11
2828
- File besar memakai relevant line window.
2929
- Path traversal dan symlink escape di luar project root ditolak.
3030
- `devmap ask` sudah memakai Context Builder secara lokal.
31-
- Automated test saat ini berjumlah 20 dan seluruhnya lulus.
31+
- Benchmark 20 pertanyaan mencakup auth, database, route session, halaman,
32+
layout, payment, dan entry point dalam Bahasa Indonesia dan English.
33+
- Hasil benchmark saat ini: top-1 accuracy 20/20 dan top-3 recall 20/20.
34+
- Alias `nextauth` ditambahkan setelah eval menemukan satu fallback ranking yang
35+
salah.
3236
- Context Builder sudah siap digunakan sebagai input AI layer.
3337

3438
### AI Ask
@@ -73,7 +77,7 @@ Terakhir diperbarui: 2026-06-11
7377
- Availability selected model ikut diperiksa.
7478
- Snapshot dibedakan menjadi valid, missing, corrupt, dan unsupported schema.
7579
- API key dan raw stack trace tidak pernah ditampilkan.
76-
- Automated test saat ini berjumlah 33 dan seluruhnya lulus.
80+
- Automated test saat ini berjumlah 34 dan seluruhnya lulus.
7781

7882
### Distribusi npm
7983

@@ -113,7 +117,7 @@ Terakhir diperbarui: 2026-06-11
113117

114118
1. Pastikan rerun GitHub Actions hijau pada seluruh 9 kombinasi.
115119
2. Lakukan manual verification Groq pada project nyata.
116-
3. Benchmark Context Builder dengan minimal 20 pertanyaan.
120+
3. Uji `init`, `analyze`, `ask`, dan `doctor` dari tarball pada project nyata.
117121
4. Tambahkan streaming output untuk jawaban AI jika waktu MVP masih tersedia.
118122

119123
## Status Saat Ini

docs/for-me-personal/TEST.md

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -78,8 +78,8 @@ Saat ini test mencakup:
7878
Hasil minimum yang diharapkan:
7979

8080
```text
81-
tests 33
82-
pass 33
81+
tests 34
82+
pass 34
8383
fail 0
8484
```
8585

@@ -120,8 +120,8 @@ npx -p node@20 node packages\cli\node_modules\tsx\dist\cli.mjs packages\cli\test
120120
Hasil minimum yang diharapkan untuk keduanya:
121121

122122
```text
123-
tests 33
124-
pass 33
123+
tests 34
124+
pass 34
125125
fail 0
126126
```
127127

@@ -149,6 +149,36 @@ Jalankan:
149149
pnpm test:cli
150150
```
151151

152+
## Benchmark Context Builder
153+
154+
Benchmark deterministic berada di:
155+
156+
```text
157+
packages/cli/test/context-builder-eval.test.ts
158+
```
159+
160+
Jalankan benchmark saja:
161+
162+
```powershell
163+
pnpm --filter devmap exec tsx --test test/context-builder-eval.test.ts
164+
```
165+
166+
Benchmark terdiri dari 20 pertanyaan Bahasa Indonesia dan English untuk:
167+
168+
- authentication dan NextAuth;
169+
- database dan Prisma;
170+
- API session;
171+
- page dan layout Next.js;
172+
- payment dan Stripe;
173+
- entry point Express.
174+
175+
Target minimum saat ini:
176+
177+
```text
178+
Context Builder top-1 accuracy: 20/20
179+
Context Builder top-3 recall: 20/20
180+
```
181+
152182
## Testing AI Ask dengan Groq
153183

154184
Testing manual ini memakai API key dan dapat menggunakan quota Groq.

packages/cli/src/ai/contextBuilder.ts

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,18 @@ const STOP_WORDS = new Set([
2828
]);
2929

3030
const CONCEPT_ALIASES: Record<string, string[]> = {
31-
auth: ["auth", "authentication", "autentikasi", "login", "session", "sesi", "token", "jwt", "middleware"],
31+
auth: [
32+
"auth",
33+
"authentication",
34+
"autentikasi",
35+
"login",
36+
"session",
37+
"sesi",
38+
"token",
39+
"jwt",
40+
"middleware",
41+
"nextauth"
42+
],
3243
database: ["database", "db", "data", "prisma", "drizzle", "mongoose", "supabase"],
3344
payment: ["payment", "payments", "pembayaran", "stripe", "midtrans", "checkout"],
3445
route: ["route", "routes", "routing", "api", "endpoint"],
Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
import assert from "node:assert/strict";
2+
import { dirname, join } from "node:path";
3+
import test from "node:test";
4+
import { fileURLToPath } from "node:url";
5+
import { buildQuestionContext } from "../src/ai/contextBuilder.js";
6+
import { createProjectMap } from "../src/analyzers/projectMap.js";
7+
8+
const testDirectory = dirname(fileURLToPath(import.meta.url));
9+
const fixtures = {
10+
nextjs: join(testDirectory, "fixtures", "nextjs-project"),
11+
express: join(testDirectory, "fixtures", "express-project")
12+
} as const;
13+
14+
type ContextEvalCase = {
15+
name: string;
16+
fixture: keyof typeof fixtures;
17+
question: string;
18+
expectedFile: string;
19+
};
20+
21+
const evalCases: ContextEvalCase[] = [
22+
{
23+
name: "authentication architecture",
24+
fixture: "nextjs",
25+
question: "How does authentication work?",
26+
expectedFile: "lib/auth.ts"
27+
},
28+
{
29+
name: "Indonesian authentication",
30+
fixture: "nextjs",
31+
question: "Bagaimana autentikasi pengguna bekerja?",
32+
expectedFile: "lib/auth.ts"
33+
},
34+
{
35+
name: "session implementation",
36+
fixture: "nextjs",
37+
question: "Where is the user session implemented?",
38+
expectedFile: "lib/auth.ts"
39+
},
40+
{
41+
name: "login handling",
42+
fixture: "nextjs",
43+
question: "File mana yang menangani login?",
44+
expectedFile: "lib/auth.ts"
45+
},
46+
{
47+
name: "NextAuth configuration",
48+
fixture: "nextjs",
49+
question: "Explain the NextAuth configuration",
50+
expectedFile: "lib/auth.ts"
51+
},
52+
{
53+
name: "database client",
54+
fixture: "nextjs",
55+
question: "Where is the database client initialized?",
56+
expectedFile: "lib/db.ts"
57+
},
58+
{
59+
name: "Indonesian database connection",
60+
fixture: "nextjs",
61+
question: "Di mana koneksi database dibuat?",
62+
expectedFile: "lib/db.ts"
63+
},
64+
{
65+
name: "Prisma initialization",
66+
fixture: "nextjs",
67+
question: "Which file initializes Prisma?",
68+
expectedFile: "lib/db.ts"
69+
},
70+
{
71+
name: "database access module",
72+
fixture: "nextjs",
73+
question: "Show the main db module",
74+
expectedFile: "lib/db.ts"
75+
},
76+
{
77+
name: "session API endpoint",
78+
fixture: "nextjs",
79+
question: "Where is the session API endpoint?",
80+
expectedFile: "app/api/session/route.ts"
81+
},
82+
{
83+
name: "Indonesian session route",
84+
fixture: "nextjs",
85+
question: "Route API mana yang mengembalikan sesi?",
86+
expectedFile: "app/api/session/route.ts"
87+
},
88+
{
89+
name: "GET session handler",
90+
fixture: "nextjs",
91+
question: "Which GET endpoint handles session requests?",
92+
expectedFile: "app/api/session/route.ts"
93+
},
94+
{
95+
name: "home page",
96+
fixture: "nextjs",
97+
question: "Where is the main home page?",
98+
expectedFile: "app/page.tsx"
99+
},
100+
{
101+
name: "Indonesian main page",
102+
fixture: "nextjs",
103+
question: "File mana untuk page utama?",
104+
expectedFile: "app/page.tsx"
105+
},
106+
{
107+
name: "root layout",
108+
fixture: "nextjs",
109+
question: "Where is the root layout component?",
110+
expectedFile: "app/layout.tsx"
111+
},
112+
{
113+
name: "payment implementation",
114+
fixture: "express",
115+
question: "How are payments implemented?",
116+
expectedFile: "src/routes/payments.ts"
117+
},
118+
{
119+
name: "Indonesian payment",
120+
fixture: "express",
121+
question: "Di mana pembayaran diproses?",
122+
expectedFile: "src/routes/payments.ts"
123+
},
124+
{
125+
name: "Stripe setup",
126+
fixture: "express",
127+
question: "Which file configures Stripe?",
128+
expectedFile: "src/routes/payments.ts"
129+
},
130+
{
131+
name: "Express entry point",
132+
fixture: "express",
133+
question: "Where does the Express server start?",
134+
expectedFile: "src/server.ts"
135+
},
136+
{
137+
name: "Indonesian server entry",
138+
fixture: "express",
139+
question: "File mana yang menjalankan server utama?",
140+
expectedFile: "src/server.ts"
141+
}
142+
];
143+
144+
test("context builder benchmark keeps relevant files in the top results", async (testContext) => {
145+
const snapshots = {
146+
nextjs: await createProjectMap(fixtures.nextjs),
147+
express: await createProjectMap(fixtures.express)
148+
};
149+
const failures: string[] = [];
150+
const topOneMisses: string[] = [];
151+
let topOneHits = 0;
152+
153+
for (const evalCase of evalCases) {
154+
const context = await buildQuestionContext(
155+
fixtures[evalCase.fixture],
156+
snapshots[evalCase.fixture],
157+
evalCase.question
158+
);
159+
const rankedPaths = context.files.map((file) => file.path);
160+
const expectedRank = rankedPaths.indexOf(evalCase.expectedFile);
161+
162+
if (expectedRank === 0) {
163+
topOneHits += 1;
164+
} else {
165+
topOneMisses.push(
166+
`${evalCase.name}: expected ${evalCase.expectedFile}, got ${rankedPaths[0] ?? "none"}`
167+
);
168+
}
169+
170+
if (expectedRank < 0 || expectedRank >= 3) {
171+
failures.push(
172+
`${evalCase.name}: expected ${evalCase.expectedFile}, got ${rankedPaths.join(", ")}`
173+
);
174+
}
175+
}
176+
177+
assert.deepEqual(failures, []);
178+
assert.equal(topOneHits, evalCases.length);
179+
testContext.diagnostic(`Context Builder top-1 accuracy: ${topOneHits}/20`);
180+
testContext.diagnostic("Context Builder top-3 recall: 20/20");
181+
for (const miss of topOneMisses) {
182+
testContext.diagnostic(`Top-1 miss: ${miss}`);
183+
}
184+
});

0 commit comments

Comments
 (0)