Skip to content

LLMJudgeScorer.__init__ wipes the class-level category when constructed directly #28

Description

@knisar

Surfaced while reviewing #27; the author flagged it there.

BaseScorer.__init__ uses None as the sentinel for "keep the class attribute":

if category is not None:
    self.category = category

LLMJudgeScorer.__init__ defaults category to "" and passes it through, so the sentinel never fires and every judge subclass loses its declared category when constructed directly:

>>> FactualityJudge(api_key="x").category
''
>>> GroundednessScorer(api_key="x").category
''

All ten judge scorers are affected. It doesn't bite via the registry because resolve_scorers passes category explicitly, but anything constructed by hand (additional_scorers, notebooks, tests) ends up grouped under the scorer name in reports, and category-based policy expectations don't match.

Fix is a one-liner in rai_toolkit/scorers/llm_judges.py: change the default to category: str | None = None. Add a test asserting FactualityJudge(api_key="x").category == "MIT-3.1" and one for a subclass constructed with an explicit override.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions