feat: 세부 역량 태그 추출 v1_baseline 구현 + v2_postscore variant 검증 (#7)#10
Merged
Conversation
gyesswhat
added
feat
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (4)
📝 WalkthroughWalkthroughSTAR 회고(직군·주요역량·S/T/A/R)로부터 도메인 화이트리스트 기반 세부 태그를 추출하는 스키마·프롬프트·정적데이터·서비스(v1/v2)·안정/실험 API·테스트·실험 평가를 통합 추가했습니다. /api/tagging은 더미 안정 경로, /v1/tagging은 LLM 호출 실험 경로입니다. 세부 역량 태그 추출(Tagging) 기능
Sequence Diagram(s) sequenceDiagram
participant Client
participant PublicRouter as /api/tagging
participant V1Router as /v1/tagging
participant Service as tagging.run()
participant LLM as LLMClient
Client->>PublicRouter: POST TaggingRequest (validated)
PublicRouter-->>Client: fixed TaggingResponse (dummy)
Client->>V1Router: POST TaggingRequest (X-Internal-Token)
V1Router->>Service: tagging.run(req, llm, model)
Service->>LLM: 1st LLM call (system+user)
LLM-->>Service: assistant response (text)
Service->>Service: strip/parse/validate
alt validation fail (1st)
Service->>LLM: corrective 2nd call
LLM-->>Service: assistant response
Service->>Service: parse/validate
end
Service-->>V1Router: TaggingResponse or raise TaggingValidationError/LLMError
V1Router-->>Client: HTTP 200 / 422 / 502 / 503
🎯 3 (Moderate) | ⏱️ ~20 minutes
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (3)
tests/test_tagging.py (2)
226-233: ⚡ Quick win
dependency_overrides.clear()대신 대상 키만 정리해 주세요 🧹전체 clear는 다른 fixture/테스트가 세팅한 override까지 지워서 순서 의존 flaky를 만들 수 있습니다. 여기서는
get_llm_client만 제거하는 방식이 안전합니다.🔧 제안 패치
- finally: - app.dependency_overrides.clear() + finally: + app.dependency_overrides.pop(get_llm_client, None)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_tagging.py` around lines 226 - 233, The test currently calls app.dependency_overrides.clear() which removes all overrides and can break other tests; instead remove only the get_llm_client override by deleting or popping that specific key from app.dependency_overrides (referencing get_llm_client and _RaisingLLM in the test block) so the cleanup is targeted and other fixtures remain intact.
83-87: ⚡ Quick win
/api/tagging인증 실패 케이스도 같이 고정해두는 게 좋아요 ✅안정 경로는 “토큰 보호”가 핵심 계약인데 현재 해피 패스만 검증합니다. 401/403 케이스를 추가해 두면 추후 cutover 때 보안 회귀를 빨리 잡을 수 있습니다.
➕ 테스트 추가 예시
def test_api_tagging_dummy_returns_fixed_response(client: TestClient, token: str) -> None: @@ assert r.json() == {"primaryCategory": "PROBLEM_SOLVING", "detailTags": ["`#문제해결`"]} +def test_api_tagging_missing_token_returns_401(client: TestClient) -> None: + r = client.post("/api/tagging", json=_PAYLOAD) + assert r.status_code == 401 + +def test_api_tagging_wrong_token_returns_403(client: TestClient) -> None: + r = client.post("/api/tagging", json=_PAYLOAD, headers={"X-Internal-Token": "wrong"}) + assert r.status_code == 403As per coding guidelines
tests/**: "해피 패스뿐 아니라 경계값·실패 케이스도 커버하는지 확인".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_tagging.py` around lines 83 - 87, Add negative auth tests for the tagging endpoint: extend the test suite around test_api_tagging_dummy_returns_fixed_response by adding at least two new tests that call POST /api/tagging using TestClient and assert proper failure status codes and responses when the X-Internal-Token header is missing and when an invalid token is provided; reference the existing test function name test_api_tagging_dummy_returns_fixed_response and the endpoint "/api/tagging" to locate where to add tests, and assert expected 401/403 status codes (and any standard error body your app returns) to ensure authentication failure behavior is covered.app/services/tagging/data.py (1)
167-256: ⚡ Quick win🧪 태그/가중치 정합성 가드를 추가하면 드리프트를 바로 잡을 수 있어요.
현재 누락 매핑은
tag_score()에서0으로 조용히 처리되어 품질 저하가 숨어버릴 수 있습니다. 모듈 로드 시점에 테이블 정합성을 검증하면 회귀를 빨리 탐지할 수 있습니다.제안 수정안
WEIGHTS_BY_TAG: dict[str, dict[JobRole, TagWeight]] = { @@ } """태그 × 직군 → 가중치 (tags.txt 26.05.01 표 그대로). 총 66 entry.""" # noqa: RUF001 +_missing = ALL_TAGS - WEIGHTS_BY_TAG.keys() +_extra = WEIGHTS_BY_TAG.keys() - ALL_TAGS +if _missing or _extra: + raise ValueError( + f"WEIGHTS_BY_TAG mismatch: missing={sorted(_missing)}, extra={sorted(_extra)}" + ) + +for _tag, _role_weights in WEIGHTS_BY_TAG.items(): + if set(_role_weights.keys()) != set(JobRole): + raise ValueError(f"Role weights incomplete for tag {_tag}: {_role_weights.keys()}") + def tag_score(tag: str, role: JobRole) -> int:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/services/tagging/data.py` around lines 167 - 256, Add a module-load validation that checks WEIGHTS_BY_TAG covers all expected JobRole keys and that each mapped TagWeight exists in _WEIGHT_SCORE; if any tag is missing a role mapping or uses an invalid weight, raise an explicit exception (or log and raise) so the mistake is caught at import time rather than silently returning 0 in tag_score. Implement this check in the same module (run once at import), iterating over WEIGHTS_BY_TAG items, verifying for each tag that all JobRole enum members are present and that weights are keys in _WEIGHT_SCORE, and surface a clear error referencing the failing tag and role so regressions are detected early.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/api/v1/tagging.py`:
- Around line 47-49: Replace the assert-based dependency check for
request.state.llm_client with an explicit runtime exception: verify that client
= request.state.llm_client is an instance of LLMClient and if not raise a clear
exception (e.g., RuntimeError or TypeError) with the existing message so the
check works in optimized Python and yields a useful stack trace; update the code
around the client retrieval (the client variable, the isinstance(...) check and
the return) to use this explicit raise.
In `@app/core/config.py`:
- Around line 69-76: The tagging_variant Field currently accepts any string and
only raises RuntimeError later; add a startup fail-fast validation in the
settings class that owns tagging_variant (the Pydantic SettingsModel containing
the tagging_variant attribute) to verify the corresponding prompt and service
files exist (app/prompts/tagging/{tagging_variant}.md and
app/services/tagging/{tagging_variant}.py) and raise a clear ValidationError on
import if missing; implement this via a Pydantic validator/root_validator or
post_init hook for the settings class so invalid TAGGING_VARIANT values fail
fast during app boot rather than at request time.
In `@app/services/tagging/v1_baseline.py`:
- Around line 114-117: The current validation only checks against ALL_TAGS
(variable ALL_TAGS and list comprehension producing out_of_pool) so tags from a
different primaryCategory can slip through; update validate logic where
out_of_pool is computed to also check that each tag belongs to the allowed set
for the provided primaryCategory (use the primaryCategory input and the
category-to-tag mapping you have—e.g., a dict like CATEGORY_TAGS or method
get_allowed_tags_for_category) and raise TaggingValidationError listing tags
that are outside the selected primaryCategory as well as those not in ALL_TAGS
so the error message and thrown condition reflect both global and
category-specific violations.
- Around line 49-60: _render_prompt currently does chained .replace() calls that
let user text containing tokens like "{result}" or braces corrupt subsequent
replacements and enables prompt-injection; change to a single-pass, safe
substitution: introduce a sanitizer (e.g., sanitize_user_input) that escapes
braces/placeholder-like substrings in req.situation_task, req.action,
req.result, then build a single mapping dict of placeholders to sanitized values
and render _PROMPT_TEMPLATE in one pass (use string.Template or a single regex
replace over placeholders) instead of repeated .replace(); ensure
JOB_ROLE_LABELS_KO and PRIMARY_CATEGORY_LABELS_KO are used via the same mapping
and that placeholder names in _PROMPT_TEMPLATE are unique/unambiguous to avoid
collisions.
In `@app/services/tagging/v2_postscore.py`:
- Around line 64-71: _render_prompt currently inserts user fields directly into
_PROMPT_TEMPLATE using chained .replace calls which allows prompt-injection or
accidental re-replacement when user text contains placeholders; change it to
perform a single-pass templating (e.g., use Python str.format / Template with
named placeholders on _PROMPT_TEMPLATE) and ensure user-provided values from
TaggingRequest (situation_task, action, result) are boundary-escaped or
sanitized (e.g., wrap/quote values or escape placeholder tokens) before
formatting; keep lookup of JOB_ROLE_LABELS_KO[req.job_role] and
PRIMARY_CATEGORY_LABELS_KO[req.selected_competency] but do not allow user text
to contain template markers that could be re-interpreted by the templating step.
---
Nitpick comments:
In `@app/services/tagging/data.py`:
- Around line 167-256: Add a module-load validation that checks WEIGHTS_BY_TAG
covers all expected JobRole keys and that each mapped TagWeight exists in
_WEIGHT_SCORE; if any tag is missing a role mapping or uses an invalid weight,
raise an explicit exception (or log and raise) so the mistake is caught at
import time rather than silently returning 0 in tag_score. Implement this check
in the same module (run once at import), iterating over WEIGHTS_BY_TAG items,
verifying for each tag that all JobRole enum members are present and that
weights are keys in _WEIGHT_SCORE, and surface a clear error referencing the
failing tag and role so regressions are detected early.
In `@tests/test_tagging.py`:
- Around line 226-233: The test currently calls app.dependency_overrides.clear()
which removes all overrides and can break other tests; instead remove only the
get_llm_client override by deleting or popping that specific key from
app.dependency_overrides (referencing get_llm_client and _RaisingLLM in the test
block) so the cleanup is targeted and other fixtures remain intact.
- Around line 83-87: Add negative auth tests for the tagging endpoint: extend
the test suite around test_api_tagging_dummy_returns_fixed_response by adding at
least two new tests that call POST /api/tagging using TestClient and assert
proper failure status codes and responses when the X-Internal-Token header is
missing and when an invalid token is provided; reference the existing test
function name test_api_tagging_dummy_returns_fixed_response and the endpoint
"/api/tagging" to locate where to add tests, and assert expected 401/403 status
codes (and any standard error body your app returns) to ensure authentication
failure behavior is covered.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b4f97a40-e10c-4409-afc9-b7c988081f4c
⛔ Files ignored due to path filters (2)
.env.exampleis excluded by!.env.example.gitignoreis excluded by!.gitignore
📒 Files selected for processing (23)
app/api/tagging_public.pyapp/api/v1/__init__.pyapp/api/v1/tagging.pyapp/core/config.pyapp/main.pyapp/prompts/tagging/v1_baseline.mdapp/prompts/tagging/v2_postscore.mdapp/schemas/common.pyapp/schemas/tagging.pyapp/services/tagging/__init__.pyapp/services/tagging/data.pyapp/services/tagging/exceptions.pyapp/services/tagging/v1_baseline.pyapp/services/tagging/v2_postscore.pyexperiments/tagging/2026-05-17_v1_baseline.mdexperiments/tagging/2026-05-18_v2_postscore.mdexperiments/tagging/_template.mdexperiments/tagging/eval_v1_baseline.pyexperiments/tagging/fixtures/eval_set_v1.jsonlexperiments/tagging/results/.gitkeepexperiments/tagging/results/2026-05-17T160028Z_v1_baseline.jsonexperiments/tagging/results/2026-05-18T121137Z_v2_postscore.jsontests/test_tagging.py
hyhy-j
approved these changes
May 18, 2026
hyhy-j
left a comment
hyhy-j
left a comment
Contributor
There was a problem hiding this comment.
오오 일단 더 수정할 건 없는 것 같습니다!
인수인계 정리 감사합니당 👍
저희가 동시에 작업하느라 공동 파일에 충돌이 나서! 그것만 해결하고 머지하면 될 것 같습니다!
gyesswhat
force-pushed
the
feat/#7-tagging-v1
branch
from
May 18, 2026 16:25
22a813d to
3d4459e
Compare
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
요약
변경 사항
세부 작업
1) 도메인·계약
JobRole/PrimaryCategoryenum + 한글 라벨 매핑 (app/schemas/common.py)TaggingRequest/TaggingResponse(camelCase alias, 300자 가드)ALL_TAGS·TAGS_BY_CATEGORY·TAG_TO_CATEGORY)2) 프롬프트 / 설정
app/prompts/tagging/v1_baseline.md— zero-shot 분류기 (66 태그풀 + High/Mid/Low 가중치 표 + JSON 출력 강제)Settings.tagging_variant(env:TAGGING_VARIANT) — 활성 variant 토글, defaultv1_baseline3) 서비스 / 라우터
app/services/tagging/v1_baseline.py—run()+ corrective 재시도 + 검증 (풀·갯수·중복·JSON)app/services/tagging/__init__.py—tagging_variantlazy resolve dispatchPOST /v1/tagging— 실제 LLM 호출,X-Internal-Token보호, LLM/검증 예외 → 502/503/422 매핑POST /api/tagging— Spring 합의 안정 경로, 현재 고정 더미 응답 (TODO: cutover 시 service 호출로 교체)4) 평가 인프라
experiments/tagging/fixtures/eval_set_v1.jsonl—.claude/test_case.txt10케이스 (직군 3종 × 5대 역량)experiments/tagging/eval_v1_baseline.py— 활성 variant 측정 (corrective 포함), raw JSON dump + .md 자동 생성experiments/tagging/_template.md—<!-- AUTO:START/END METRICS -->마커 박힌 템플릿. 재측정 시 메트릭만 갱신, 사람이 적은 "관찰/결정" 보존experiments/tagging/2026-05-17_v1_baseline.md5) v2_postscore variant
v1 측정에서 드러난 한계를 종합 해소한 두 번째 variant. 변경 세 가지:
\1로 1·2·3개 백틱 모두 strip (v1 에서 Haiku 의 single backtick 응답이 재시도 6/10 발동시킨 단일 원인 해결)tag_score()로 직군 가중치 점수 매겨 상위 3개로 자른다 (deterministic 후처리)app/services/tagging/data.py에TagWeightenum +WEIGHTS_BY_TAG(66 entry) +tag_score()함수 추가experiments/tagging/2026-05-18_v2_postscore.mdv1 vs v2 측정 비교 (10케이스 실측)
활성 default 는 아직 v1_baseline
이 PR 에서는 v2 측정 결과만 기록하고 default 전환은 별도 PR (관련 issue 의 결정란 참고).
운영 트래픽 영향 0 —
/api/tagging은 더미 응답 그대로.테스트
LLM 실측 평가 (수동, Anthropic API 키 필요)
experiments/tagging/results/<UTC_ts>_<variant>.jsonraw 저장experiments/tagging/<date>_<variant>.md자동 생성 (메트릭 마커 사이만 채워짐, "요약/관찰/결정" 은 개발자가 작성)스크린샷/로그
experiments/tagging/2026-05-17_v1_baseline.mdexperiments/tagging/2026-05-18_v2_postscore.md(요약·관찰·결정 채워져 있음)experiments/tagging/results/*.json롤백/리스크
리스크 포인트
/api/tagging은 현재는 고정 더미 응답 (#문제해결1개)./v1/tagging만 실 LLM 호출. 활성 variant 는TAGGING_VARIANTenv 따라감 (defaultv1_baseline).롤백 방법
✅ 인수인계 관련 메모!
app/prompts/tagging/v{N}_{strategy}.md작성app/services/tagging/v{N}_{strategy}.py에 동일 시그니처run()작성app/services/tagging/__init__.py::_resolve분기 한 줄 추가.env의TAGGING_VARIANT=v{N}_{strategy}. prod 는 SSM/groute/ai/TAGGING_VARIANT.uv run python experiments/tagging/eval_v1_baseline.py실행. raw JSON + .md 자동 생성..md의 "요약/관찰/결정" 채워서 commit. 같은 날 같은 variant 재실행 = 메트릭만 자동 갱신 (인사이트 보존).7 → 710 확장 — 정답 누락 회수위 1·2·3 묶음을 다음 measurement round 에서 같이 시도 권장. 그 뒤 default 전환 +
/api/taggingcutover.관련 이슈
Closes #7
Summary by CodeRabbit
새로운 기능
문서
테스트