Surfaced while reviewing #27; the author flagged it there.
BaseScorer.__init__ uses None as the sentinel for "keep the class attribute":
if category is not None:
self.category = category
LLMJudgeScorer.__init__ defaults category to "" and passes it through, so the sentinel never fires and every judge subclass loses its declared category when constructed directly:
>>> FactualityJudge(api_key="x").category
''
>>> GroundednessScorer(api_key="x").category
''
All ten judge scorers are affected. It doesn't bite via the registry because resolve_scorers passes category explicitly, but anything constructed by hand (additional_scorers, notebooks, tests) ends up grouped under the scorer name in reports, and category-based policy expectations don't match.
Fix is a one-liner in rai_toolkit/scorers/llm_judges.py: change the default to category: str | None = None. Add a test asserting FactualityJudge(api_key="x").category == "MIT-3.1" and one for a subclass constructed with an explicit override.
Surfaced while reviewing #27; the author flagged it there.
BaseScorer.__init__usesNoneas the sentinel for "keep the class attribute":LLMJudgeScorer.__init__defaultscategoryto""and passes it through, so the sentinel never fires and every judge subclass loses its declared category when constructed directly:All ten judge scorers are affected. It doesn't bite via the registry because
resolve_scorerspassescategoryexplicitly, but anything constructed by hand (additional_scorers, notebooks, tests) ends up grouped under the scorer name in reports, and category-based policy expectations don't match.Fix is a one-liner in
rai_toolkit/scorers/llm_judges.py: change the default tocategory: str | None = None. Add a test assertingFactualityJudge(api_key="x").category == "MIT-3.1"and one for a subclass constructed with an explicit override.