test: tagging eval_set_v2 평가 인프라 + v2 좁힘 제거 (v3 준비)#14
Conversation
gyesswhat
added
fix
|
Warning Review limit reached
Your plan currently allows 1 review/hour. Refill in 49 minutes and 33 seconds. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more review capacity refills, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughv2_postscore 후보 검증 로직을 카테고리별 화이트리스트에서 전역 ALL_TAGS 기반으로 단순화합니다. 이를 검증하는 평가 프레임워크 및 48케이스 벤치마크 결과, 교차 카테고리 후보 수용을 검증하는 회귀 테스트를 추가합니다. Changesv2_postscore 개선 및 벤치마크 평가
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 📊 복잡도 분석증가 요인:
감소 요인:
파일 다양성: 7개 파일 (코어 로직 1 + 테스트 2 + 평가 5), 함수·클래스 추가 3개, 의존성 재구성 필요 Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
tests/test_tagging.py (1)
173-195: 💤 Low value🔧 (선택) 실제 태그로 경계값 테스트 명확성 향상
8-tag 케이스(Line 178)가
"#A"~"#H"fake tags를 사용하여 count violation(8 > 7)과 pool violation을 동시에 트리거합니다. Corrective retry 메커니즘 검증에는 충분하지만, 테스트 의도를 더 명확히 하려면 실제 태그 8개를 사용하는 것도 고려할 수 있습니다(예: PROBLEM_SOLVING 카테고리에서 8개 선택).현재 구현도 동작에는 문제없으나, "count violation 특화" 테스트임을 강조하려면 실제 태그가 조금 더 명확합니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_tagging.py` around lines 173 - 195, The 8-tag case in test_v1_tagging_count_violation_then_recovers_via_corrective uses synthetic tags ("`#A`".."`#H`") which can trigger both count and pool violations; change the third param value in the bad_response parametrize list to a JSON string containing eight real tags from the PROBLEM_SOLVING category (instead of "`#A`".."`#H`") so the test isolates a count-violation and still allows the corrective retry flow via _anthropic_msg and _HAPPY_LLM_RAW to be exercised.tests/conftest.py (1)
25-38: 💤 Low value📌 TAGGING_VARIANT 고정 — CI/로컬 일관성 vs v1 테스트 커버리지
TAGGING_VARIANT=v2_postscore고정은 CI/로컬 환경 간 테스트 결과 불일치를 막는 좋은 선택입니다(도크스트링 Lines 29-31에 명시된 대로).다만 이로 인해
v1_baseline이 CI에서 테스트되지 않게 됩니다. PR objectives가 "v3 진입 전 v2 정합성 수정"이므로 v2 집중은 의도된 것으로 보이지만, 만약 v1이 여전히 프로덕션에서 사용 중이거나 롤백 대상이라면 별도 테스트 전략(예: variant별 분리된 test suite 또는 matrix CI)이 필요할 수 있습니다.현재 PR 범위에서는 문제없으나, v1 deprecation timeline을 고려한 테스트 전략 검토를 권장합니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/conftest.py` around lines 25 - 38, The fixture _clean_settings currently force-sets TAGGING_VARIANT="v2_postscore", which prevents running tests for v1_baseline; change the fixture to set TAGGING_VARIANT from a configurable test option or env var (e.g. prefer a pytest CLI option like --tagging-variant or respect an env var TEST_TAGGING_VARIANT) with a default of "v2_postscore", so CI can run a matrix (or local dev can override) while retaining get_settings.cache_clear calls before and after; update the test configuration to add pytest_addoption to expose --tagging-variant (or document TEST_TAGGING_VARIANT) and have _clean_settings read that value instead of hardcoding "v2_postscore".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@experiments/tagging/eval_v3_tiebreak.py`:
- Line 572: The print call using Path.relative_to(Path.cwd()) can raise
ValueError when the current working directory differs; update the print at the
end of the block (the f"\nRaw saved → ..." statement) to use the existing helper
_rel() instead of calling relative_to directly so the path is safely rendered;
locate the print in eval_v3_tiebreak.py and replace the relative_to(Path.cwd())
usage with _rel(saved_json) (or the appropriate variable passed to _rel) to
avoid the exception.
- Around line 161-163: The code currently assigns the raw LLM exception string
to llm_err (except LLMError as e -> llm_err = f"{type(e).__name__}: {e}"), which
may leak vendor org/request_id info; replace this by creating/using a sanitizer
(e.g. sanitize_llm_error or mask_sensitive) that returns only the exception
class plus a short, masked summary (remove UUIDs, tokens like org_ or
request_id, or truncate to ~100–200 chars), and set llm_err =
sanitize_llm_error(e) instead of storing the raw exception; ensure the helper is
used wherever LLMError is caught so raw vendor responses are never written to
results.
- Around line 106-107: The code currently reads and writes files without
specifying encoding which makes it locale-dependent; update _load_fixtures to
call path.read_text(encoding="utf-8") and similarly ensure any file read/write
operations in the block referenced around lines 517-529 use explicit
encoding="utf-8" (e.g., Path.read_text(encoding="utf-8") and open(..., "w"/"r",
encoding="utf-8")) so fixtures and Korean text are correctly handled across
environments.
---
Nitpick comments:
In `@tests/conftest.py`:
- Around line 25-38: The fixture _clean_settings currently force-sets
TAGGING_VARIANT="v2_postscore", which prevents running tests for v1_baseline;
change the fixture to set TAGGING_VARIANT from a configurable test option or env
var (e.g. prefer a pytest CLI option like --tagging-variant or respect an env
var TEST_TAGGING_VARIANT) with a default of "v2_postscore", so CI can run a
matrix (or local dev can override) while retaining get_settings.cache_clear
calls before and after; update the test configuration to add pytest_addoption to
expose --tagging-variant (or document TEST_TAGGING_VARIANT) and have
_clean_settings read that value instead of hardcoding "v2_postscore".
In `@tests/test_tagging.py`:
- Around line 173-195: The 8-tag case in
test_v1_tagging_count_violation_then_recovers_via_corrective uses synthetic tags
("`#A`".."`#H`") which can trigger both count and pool violations; change the third
param value in the bad_response parametrize list to a JSON string containing
eight real tags from the PROBLEM_SOLVING category (instead of "`#A`".."`#H`") so the
test isolates a count-violation and still allows the corrective retry flow via
_anthropic_msg and _HAPPY_LLM_RAW to be exercised.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 73fc339b-9d57-4b57-90f7-26cc8628ca86
📒 Files selected for processing (9)
app/services/tagging/v2_postscore.pyexperiments/tagging/2026-05-22_v2_postscore.mdexperiments/tagging/_template.mdexperiments/tagging/eval_v3_tiebreak.pyexperiments/tagging/fixtures/eval_set_v2.jsonlexperiments/tagging/results/2026-05-22T151749Z_v2_postscore.jsonexperiments/tagging/results/2026-05-22T160708Z_v2_postscore.jsontests/conftest.pytests/test_tagging.py
1 participant
개요
v3 코드 변경 진입 전 평가 인프라 정리 + v2 정합성 fix.
새 평가셋 (test_case_v2.md 의 48케이스) 으로 must/accept/source/직군모호 분리 metric 까지
갖추고, v2 의 카테고리 좁힘이 새 평가셋과 충돌하는 문제 해소.
fixture accept 확장 + rate limit 회피 sleep 까지 포함.
변경 사항
세부 내용
7개 commit, 크게 3 갈래:
A. 새 평가 인프라
2 + accept 35/길이/모호 그룹별 집계)
B. v2 정합성 fix
C. 측정 baseline (v2 2회)
must 65.6% · accept 32.1% · 풀외 19.4%
결정 과정 (왜 이 순서로 갔는지)
원래 plan 은 v3 알고리즘 변경 (점수 격차 + 후보 갯수 + tie-break) 으로 바로 가려
했는데, 다음 두 번 우회:
metric "단일 expected" 가 측정 변동성 컸음)
vs 새 fixture 의 cross-category accept 충돌)
합리적이라 accept 에 흡수, LLM_ERR 4건은 분당 토큰 rate limit 이었음을 raw 분석
으로 확인)
v2 측정 변화 (3회 비교)
참고사항
측정 방법
# .env 의 TAGGING_VARIANT 가 측정 대상 (v1_baseline / v2_postscore / v3_*) uv run python experiments/tagging/eval_v3_tiebreak.py주의사항
v3 진입 시 권장 순서
v2 측정 md (`experiments/tagging/2026-05-22_v2_postscore.md`) §결정 1~6 참고.
요약:
7 → 710 + 점수 격차 H/M/L=3/2/1 → 5/2/1 + tie-breakscore - α·index)
변경 안 한 것
로 측정하려면 같은 좁힘 제거 또는 별도 trial 필요
테스트
롤백/리스크
리스크 포인트
롤백
관련 이슈
Refs #7 (이미 PR #10 에서 closed). 후속 v3 작업은 별도 issue 권장.