feat: inspect and bound GEval retrieval context prompts#2742
Open
RitwijParmar wants to merge 6 commits into
Open
feat: inspect and bound GEval retrieval context prompts#2742RitwijParmar wants to merge 6 commits into
RitwijParmar wants to merge 6 commits into
Conversation
|
@RitwijParmar is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
Signed-off-by: Ritwij Aryan Parmar <ritwij.aryan.parmar@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1764.
This adds an opt-in retrieval context budget for GEval, a no-LLM inspection path for large RAG judge prompts, and an evidence coverage report so users can see whether prompt compaction removed terms the judge still needs.
What changed
max_retrieval_context_tokensto GEval. Default behavior is unchanged unless this option is set.retrieval_contextvalues are compacted before they enter the judge prompt.RetrievedContextData, keeps head and tail evidence from visible chunks, and inserts explicit omission markers.get_retrieval_context_budget_report(test_case).get_retrieval_context_evidence_coverage(test_case)for the coverage ratio, covered terms, missing terms, and warning message.preview_evaluation_prompt(test_case)so users can inspect the exact bounded judge prompt in CI or while tuning RAG metrics without calling the evaluation model.Why this matters
Large RAG contexts can make custom GEval prompts expensive and hard to trust. Clipping blindly is risky because it can hide the evidence that should support or refute the answer. This PR gives users a bounded prompt and a diagnostic report that says what survived the budget and what evidence terms disappeared.
Validation
uv run --isolated --python 3.11 --with-editable . python -m pytest tests/test_metrics/test_g_eval_utils.py tests/test_metrics/test_g_eval_prompt_budget.py -qpython3 -m black --check deepeval/metrics/g_eval/g_eval.py deepeval/metrics/g_eval/utils.py deepeval/metrics/g_eval/__init__.py tests/test_metrics/test_g_eval_utils.py tests/test_metrics/test_g_eval_prompt_budget.pyuvx ruff check deepeval/metrics/g_eval/g_eval.py deepeval/metrics/g_eval/utils.py deepeval/metrics/g_eval/__init__.py tests/test_metrics/test_g_eval_utils.py tests/test_metrics/test_g_eval_prompt_budget.pypython3 -m compileall -q deepeval/metrics/g_eval tests/test_metrics/test_g_eval_prompt_budget.py tests/test_metrics/test_g_eval_utils.pygit diff --checkNote on current CI