Skip to content

[codex] Add source grouping parity for contextual precision#2737

Open
cat0825 wants to merge 2 commits into
confident-ai:mainfrom
cat0825:codex/deepeval-2594-source-grouping-followup
Open

[codex] Add source grouping parity for contextual precision#2737
cat0825 wants to merge 2 commits into
confident-ai:mainfrom
cat0825:codex/deepeval-2594-source-grouping-followup

Conversation

@cat0825

@cat0825 cat0825 commented Jun 9, 2026

Copy link
Copy Markdown

Summary

  • share the source-based retrieval-context grouping helper used by contextual precision metrics
  • apply RetrievedContextData.source grouping to TurnContextualPrecisionMetric before LLM verdict generation, matching the single-turn ContextualPrecisionMetric behavior
  • add deterministic source-grouping tests and document source granularity/rank semantics

Why

This is a focused follow-up to #2594 and #2669. The main branch already supports source-based grouping for single-turn contextual precision, but turn contextual precision accepted RetrievedContextData without grouping same-source contexts before verdict generation.

This PR keeps the schema/source-first direction discussed by maintainers, avoids adding a new group_by public API, and supersedes the draft direction from #2729 with tests and docs around the merged source behavior.

Notes

  • Plain string retrieval contexts remain independent nodes.
  • RetrievedContextData entries with the same source are grouped at the first position where that source appears, preserving best/first-seen rank behavior for fused retrieval lists.
  • source can be a section/span ID for section-level scoring or a chunk hash for exact duplicate/cross-arm hybrid search deduplication.

Validation

  • PYTHONPATH=. poetry run pytest tests/test_metrics/test_contextual_precision_source_grouping.py -q
  • PYTHONPATH=. poetry run pytest tests/test_core/test_imports.py tests/test_core/test_datasets/test_dataset.py -q
  • OPENAI_API_KEY= PYTHONPATH=. poetry run pytest tests/test_metrics/test_contextual_precision_metric.py tests/test_metrics/test_turn_contextual_precision_metric.py -q
  • PYTHONPATH=. poetry run black --check deepeval/metrics/utils.py deepeval/metrics/contextual_precision/contextual_precision.py deepeval/metrics/turn_contextual_precision/turn_contextual_precision.py tests/test_metrics/test_contextual_precision_source_grouping.py
  • PYTHONPATH=. poetry run ruff check deepeval/metrics/utils.py deepeval/metrics/contextual_precision/contextual_precision.py deepeval/metrics/turn_contextual_precision/turn_contextual_precision.py tests/test_metrics/test_contextual_precision_source_grouping.py

Refs #2594

@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown

@cat0825 is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@cat0825

cat0825 commented Jun 9, 2026

Copy link
Copy Markdown
Author

CI note: the lint (ubuntu-latest) job is failing on the repository-wide Black check because unrelated existing files would be reformatted. The touched Python files in this PR are formatted; local targeted checks passed:

  • PYTHONPATH=. poetry run black --check deepeval/metrics/utils.py deepeval/metrics/contextual_precision/contextual_precision.py deepeval/metrics/turn_contextual_precision/turn_contextual_precision.py tests/test_metrics/test_contextual_precision_source_grouping.py
  • PYTHONPATH=. poetry run ruff check deepeval/metrics/utils.py deepeval/metrics/contextual_precision/contextual_precision.py deepeval/metrics/turn_contextual_precision/turn_contextual_precision.py tests/test_metrics/test_contextual_precision_source_grouping.py

I am not formatting the unrelated 58 files in this PR to keep the contribution focused on #2594/#2669.

@cat0825 cat0825 marked this pull request as ready for review June 11, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant