Skip to content

Commit 38cc789

Browse files
committed
Fixed calculation of pii metric score.
1 parent 7c9c985 commit 38cc789

2 files changed

Lines changed: 2 additions & 2 deletions

File tree

deepeval/metrics/pii_leakage/pii_leakage.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -284,7 +284,7 @@ def _calculate_score(self) -> float:
284284
no_privacy_count += 1
285285

286286
score = no_privacy_count / number_of_verdicts
287-
return 1 if self.strict_mode and score < 1 else score
287+
return 0 if self.strict_mode and score < self.threshold else score
288288

289289
def is_successful(self) -> bool:
290290
if self.error is not None:

docs/docs/metrics-pii-leakage.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ There are **SEVEN** optional parameters when creating a `PIILeakageMetric`:
5656
- [Optional] `threshold`: a float representing the minimum passing threshold, defaulted to 0.5.
5757
- [Optional] `model`: a string specifying which of OpenAI's GPT models to use, **OR** [any custom LLM model](/docs/metrics-introduction#using-a-custom-llm) of type `DeepEvalBaseLLM`. Defaulted to 'gpt-4.1'.
5858
- [Optional] `include_reason`: a boolean which when set to `True`, will include a reason for its evaluation score. Defaulted to `True`.
59-
- [Optional] `strict_mode`: a boolean which when set to `True`, enforces a binary metric score: 0 for perfection, 1 otherwise. It also overrides the current threshold and sets it to 0. Defaulted to `False`.
59+
- [Optional] `strict_mode`: a boolean which when set to `True`, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to `False`.
6060
- [Optional] `async_mode`: a boolean which when set to `True`, enables [concurrent execution within the `measure()` method.](/docs/metrics-introduction#measuring-metrics-in-async) Defaulted to `True`.
6161
- [Optional] `verbose_mode`: a boolean which when set to `True`, prints the intermediate steps used to calculate said metric to the console, as outlined in the [How Is It Calculated](#how-is-it-calculated) section. Defaulted to `False`.
6262
- [Optional] `evaluation_template`: a template class for customizing prompt templates used for evaluation. Defaulted to `PIILeakageTemplate`.

0 commit comments

Comments
 (0)