This repository was archived by the owner on Aug 24, 2026. It is now read-only.
feat: prepare devtools QA runner as standalone package - #23
Merged
Conversation
…in reason string-aware balanced-brace extraction replaces greedy regex so small judge models that append commentary after the JSON object still parse cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
답변 가능한 정규 FAQ 10개를 정중/대충/오타·구어체 3변형(30문항)으로 입력해 말투·표현 차이에 따른 과잉 거절·오답을 검출한다. judge 루브릭에 paraphrase 카테고리 추가(과잉 거절·무관 주제 답변을 불합격). register<=2 가 과잉 거절 핫리스트로 집계됨. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
브라우저는 렌더 시 연속 공백을 한 칸으로 합치는데, hasText/extractAnswer가 제출 텍스트를 정확 매칭해 공백이 두 칸인 질문(faq-33)을 영영 못 찾고 180초 타임아웃하던 문제를 수정한다. normalizeWs(연속 공백→한 칸·trim)로 양쪽을 정규화해 비교한다. 제품(챗봇)은 정상이었고 측정 도구만의 결함이었다. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- profiles/lms-conversation.json: 대화에서 다룬 질문 34개 통합 검증 프로파일 - report/human-report.mjs: answers.jsonl → 비개발자용 한글 마크다운 보고서 - report/human-report-html.mjs: 같은 내용을 색 배지·요약 카드·접이식 원문의 단일 HTML로 출력 - judge/rubric.mjs: scope·help 카테고리 기대치 추가 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # docs/profile-schema.md # src/core/devtools-client.mjs # src/core/quality.mjs # src/core/runner.mjs # src/scenarios/chatbot.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
devtools-qa-runnerfor standalone package usage.judge/andreport/runtime files in package dry-run output and mark CLI bins executable.Test Plan
npm run checknpm test— 51 pass / 0 failnpm run pack:dry— confirmssrc/,judge/, andreport/are packaged.bincommands exist