Skip to content

[codex] Add contextual precision grouping#2729

Open
cat0825 wants to merge 1 commit into
confident-ai:mainfrom
cat0825:codex/deepeval-2594-contextual-precision-grouping
Open

[codex] Add contextual precision grouping#2729
cat0825 wants to merge 1 commit into
confident-ai:mainfrom
cat0825:codex/deepeval-2594-contextual-precision-grouping

Conversation

@cat0825

@cat0825 cat0825 commented Jun 8, 2026

Copy link
Copy Markdown

Summary

  • Add an optional group_by hook to ContextualPrecisionMetric for scoring overlapping retrieval chunks as retrieval units.
  • Preserve current per-context scoring when group_by is not provided.
  • Keep ungrouped contexts as singleton units when group_by returns None.
  • Use raw context text for verdict generation while allowing group_by to inspect the original retrieval context item, including RetrievedContextData.source.

Closes #2594

Context

PR #2659 currently remains open but conflicting. This PR applies the same current-main direction as a focused opt-in grouping implementation with deterministic tests.

Validation

  • poetry run pytest tests/test_metrics/test_contextual_precision_grouping.py -q
  • OPENAI_API_KEY= poetry run pytest tests/test_metrics/test_contextual_precision_metric.py tests/test_metrics/test_contextual_precision_grouping.py -q
  • poetry run black --check deepeval/metrics/contextual_precision/contextual_precision.py tests/test_metrics/test_contextual_precision_grouping.py
  • poetry run ruff check deepeval/metrics/contextual_precision/contextual_precision.py tests/test_metrics/test_contextual_precision_grouping.py

Note: running the existing OpenAI-backed contextual precision tests with the local OPENAI_API_KEY present fails in this environment before metric execution because the machine has a SOCKS proxy configured and socksio is not installed. The command above clears OPENAI_API_KEY so those integration-style tests follow the repository skip rule while the new deterministic tests run.

@vercel

vercel Bot commented Jun 8, 2026

Copy link
Copy Markdown

@cat0825 is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@cat0825

cat0825 commented Jun 9, 2026

Copy link
Copy Markdown
Author

Update: I opened #2737 as the source-first follow-up to #2594/#2669.

I am treating this draft as superseded rather than continuing the group_by API direction. The new PR keeps the RetrievedContextData.source approach discussed by maintainers, adds source-grouping parity for TurnContextualPrecisionMetric, and includes deterministic tests/docs around source granularity and first-seen rank behavior.

@cat0825 cat0825 marked this pull request as ready for review June 11, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Contextual Precision over-penalizes overlapping chunks in financial-document RAG

1 participant