minor bug fixes

[omri374]

This pull request updates the evaluation logic for span matching, focusing on improving the accuracy of character-based IoU calculations and adjusting the threshold for matches. It also adds new tests to ensure correct behavior, especially for split entities. The most important changes are grouped below:

Evaluation Logic Improvements

• Changed the default iou_threshold in SpanEvaluator from 0.9 to 0.75 to allow more flexibility in span matching.
• Fixed an off-by-one error in character span calculation within _calculate_combined_iou by removing the unnecessary + 1 in the range calls for both annotation and prediction spans. This ensures that character indices are properly exclusive at the end, improving the accuracy of IoU calculations. [1] [2]

Testing Enhancements

• Added two new tests in test_span_evaluator.py:
  • test_calculate_combined_iou_char_based_split_entity verifies that split predictions covering an annotation result in an IoU of 1.0, confirming the off-by-one bug is fixed.
  • test_calculate_combined_iou_char_based_exact_match checks that exact matches yield an IoU of 1.0, ensuring no regression in the basic matching logic.

[Copilot]

Pull request overview

This pull request addresses character-based IoU calculation issues in span matching evaluation by fixing an off-by-one error and adjusting the matching threshold for more flexible span matching.

• Fixed off-by-one error in character span calculation by removing unnecessary + 1 in range calls for both annotation and prediction end indices
• Reduced the default IoU threshold from 0.9 to 0.75 to allow more flexibility in span matching
• Added comprehensive tests to verify the fix for split entity scenarios and exact matches

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
presidio_evaluator/evaluation/span_evaluator.py	Updated IoU threshold default to 0.75 and fixed off-by-one error in character-based IoU calculations by making end indices properly exclusive
tests/test_span_evaluator.py	Added two new test cases to verify character-based IoU calculations work correctly for split entities and exact matches
presidio_evaluator/evaluation/scorers.py	Removed obsolete scoring pipeline code that is no longer needed

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The documentation comment at line 35 states the default iou_threshold is 0.5, but the actual default has been changed to 0.75. The documentation should be updated to reflect this change:

:param iou_threshold: Minimum Intersection over Union (IoU) threshold for considering spans as matching.
                    Value between 0 and 1, where higher values require more overlap (default: 0.75)