Skip to content

rubric20-semantic: circular applicability gates can suppress the fields they're meant to score #161

Description

@realmarcin

Context

Found by adversarial review of PR #154 (rubric-review). Blocking. Not covered by the existing follow-ups (#155#160).

Problem

PR #154 replaces rubric20's hard-coded Applies to: lists with a conditional gate: "Always report results… but only score if <X> is identified elsewhere." For several questions the only place <X> can be identified is the question's own scoring fields, creating a circular gate that can suppress exactly the items it is meant to score.

Examples in .claude/agents/d4d-rubric20-semantic.md:

  • Q8 Ethical and Privacy Declarations — gate (line 209): "only score if human subjects or governance restrictions are identified elsewhere." Q8's own fields (line ~199): ethical_reviews, human_subject_research, is_deidentified, participant_privacy, … i.e. the "elsewhere" evidence is Q8's evidence.
  • Q9 Access Requirements and Governance (line 225) — same governance gate over its own governance fields.
  • Q11 Tool and Software Transparency (line 259) — "only score if software tools were identified elsewhere" over its own software fields.

The v2.1 source rubric instead uses explicit applicability lists (e.g. data/rubric/rubric20.txt:282, :303).

Impact

  • A genuine ethics/governance/software gap can be silently excluded from scoring because the evaluator finds no "elsewhere" signal — most likely on sparse D4D-core files.
  • "Identified elsewhere" is undefined, so the decision depends on evaluator guesswork → non-determinism (these agents claim temp=0.0 reproducibility).

Recommendation

Define per-question applicability as explicit predicates over named fields (a deterministic pre-pass), require the evaluator to emit an applicability_status + applicability_evidence before scoring, and forbid using a question's own absence as evidence to skip itself. (Pairs with #155's N/A encoding.)

Evidence

  • .claude/agents/d4d-rubric20-semantic.md:197-259
  • data/rubric/rubric20.txt:282,303

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions