Skip to content

rubric10-semantic: add conditional applicability to match rubric20 (core datasheets penalized for by-design omissions) #157

Description

@realmarcin

Context

Follow-up from review of #154 (rubric-review branch).

Problem

#154 added conditional applicability ("Always report results… but only score if identified elsewhere") only to d4d-rubric20-semantic.md. d4d-rubric10-semantic.md was left with strict binary present/absent scoring.

This asymmetry directly hurts D4D-core evaluation. D4D_Core.yaml intentionally excludes fields:

"Fields not in the exchange layer mapping (e.g., file_collections, relationships, splits, consent workflow) are intentionally excluded; use the full data_sheets_schema.yaml for those."

So rubric10-semantic will penalize claudecode_agent_core datasheets (the new *_core_evaluation.json runs) for absences that are by design — while rubric20-semantic now degrades gracefully for the same datasheets.

Proposed fix

Either:

  • (a) Add the same conditional-applicability treatment to rubric10-semantic sub-elements (report but only score when the relevant context is present elsewhere in the datasheet), or
  • (b) Add an explicit "absent-by-design ≠ fail when evaluating a D4D-core datasheet" stance, with a documented list of core-excluded fields that should not be penalized.

Depends on / coordinates with the N/A-encoding convention (companion issue) so rubric10 scores stay comparable.

Scope

  • .claude/agents/d4d-rubric10-semantic.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions