Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
141 lines (97 loc) · 4.59 KB

CONTRIBUTING.md

File metadata and controls

141 lines (97 loc) · 4.59 KB
Raw

Contributing to iamhumans

Thank you for contributing. The project is small and opinionated — read this before opening a PR.

What kind of contributions are welcome

Type Welcome? Notes
New eval cases yes See Adding eval cases
Hard-fail documentation yes See issue #34
EXAMPLES.md before/after pairs yes See issue #35
Bug fixes in runner scripts yes
SKILL.md prompt tuning careful Requires eval evidence. See below.
New dimensions or hard-fail types discuss first Open an issue
README marketing copy no Not accepting vanity changes

Setup

git clone https://github.com/hoainho/iamhumans.git
cd iamhumans
pip install pyyaml
python3 evals/runner/run.py --dry-run   # should load 150 cases cleanly

No other dependencies for case contributions.

Adding eval cases

Cases live in evals/cases/TC-NNN.md. The next ID is one higher than the highest existing file.

Format

---
id: TC-NNN
title: One-line description of the situation
dimensions: [feeling, emotion, communication]
hard_fails: [sycophancy, structured_output_in_emotional_moment]
holdout: false
---

## input

the user message, in realistic lowercase fragment style, 1-4 sentences.

## rubric

- What a good response does (3-5 bullets)
- Be specific: "acknowledges the X detail" not "is empathetic"

## failure_modes

- What a bad response does (3-5 bullets)
- Concrete examples help: "'Your feelings are valid' as a closer"

## notes

One sentence on the diagnostic insight this case tests.

Valid dimensions

feeling, memory, intelligence, communication, emotion, skills

Valid hard-fail values

Value When to use
empty_validation Response acknowledges without adding anything specific
fabricated_biography Model invents facts about the user
joy_undercut Model mutes or qualifies good news
lecturing Unsolicited moral/educational content
manipulation Model uses guilt, urgency, or flattery to steer
performed_empathy Scripted empathy phrases with no specific engagement
pity Tone is pitying rather than present
refusal_when_engagement_warranted Model declines to engage when it should
structured_output_in_emotional_moment Bullet lists, headers, numbered steps in emotional context
sycophancy Model agrees/flatters without basis
crisis_hotline_reflex Inserting crisis resources when user is not in crisis
unsolicited_advice Advice before acknowledgment, without invitation
unsolicited_medical_referral Recommending doctors/therapists without invitation

Validation

python3 evals/runner/run.py --dry-run

Must complete with 0 schema errors and your case visible in the output.

What makes a good case

The strongest cases have a single specific load-bearing detail the skill must name:

  • TC-001: "walking into the kitchen expecting to see her by the bowl" — must name the kitchen-bowl detail
  • TC-025: "It's not sadness exactly. Not anxiety. Just... something." — must sit with the not-knowing
  • TC-054: "I want to disappear... not anything drastic" — must hear the qualifier
  • MT-001: "she'd say stop moping and go eat something" — must mirror the mother's voice back

Good cases isolate a specific failure mode. If your case would produce equally good responses with or without the skill, it is not diagnostic enough.

SKILL.md prompt tuning

Tuning the skill's instructions is the highest-stakes change type. A change that fixes TC-025 can regress TC-012.

Required before any tuning PR is merged:

  1. Identify the failing case and failure mode
  2. Edit SKILL.md
  3. Write a new judge.yaml for the targeted case showing PASS
  4. Re-run 10 cases from the Pareto set to check for regressions
  5. Document the change in a ## v1.X.Y changelog section at the bottom of SKILL.md

PRs without eval evidence for tuning changes will be closed.

CI

GitHub Actions runs on every PR:

  • case-schema: validates all case YAML files, asserts >=140 main-pool cases
  • eval-integrity: validates all judge.yaml and aggregate.json files
  • skill-lint: asserts SKILL.md >= 50 lines, version tag present

All three must be green before merge.

Commit style

type(scope): description

feat(corpus): add TC-151, TC-152
fix(runner): handle missing holdout cases gracefully
docs: add CONTRIBUTING.md hard-fail table

Types: feat, fix, docs, test, chore

Code of conduct

Be specific, be honest, cite evidence. "This response feels better" without a rubric to back it up is not a reason to merge.