Skip to content

Add nav_aap scenario pack: 15 scenarios on Norwegian welfare administration#17

Merged
kelkalot merged 1 commit into
kelkalot:mainfrom
avalyset:feat/nav-aap-scenarios
May 8, 2026
Merged

Add nav_aap scenario pack: 15 scenarios on Norwegian welfare administration#17
kelkalot merged 1 commit into
kelkalot:mainfrom
avalyset:feat/nav-aap-scenarios

Conversation

@avalyset

@avalyset avalyset commented May 6, 2026

Copy link
Copy Markdown
Contributor

Closes the pack discussion in #15.

Adds nav_aap, a 15-scenario Norwegian-language pack covering NAV
(Arbeids- og velferdsdirektoratet) AAP (work assessment allowance) —
factual recall, hedging behaviour, boundary keeping, and crisis
recognition under welfare-administration framing.

Pack composition

  • 10 scenarios on factual accuracy under domain-specific rules:
    AAP duration and extension, EØS membership aggregation, 6G income
    cap, barnetillegg, meldekort cadence, næringsetablering phases,
    klagefrist, 50% vs 30% arbeidsevne, AAP/uføretrygd interaction,
    foreign-stay rules.

  • 3 scenarios on boundary keeping: unreported meldekort income (must
    route to lawful self-correction), invented "AAP fast track" scheme
    (hallucination resistance), third-party access (taushetsplikt
    fullmakt).

  • 2 scenarios on vulnerable-user routing: burnout-without-fastlege
    (medical evaluation before procedural advice), and a procedural query
    containing suicidal ideation under processing delay (the only
    critical-severity scenario).

SimpleAudit taxonomy breakdown: 12 Misinformation/Dangerous Advice, 1
Hallucination, 1 Mental Health/Crisis, 1 Privacy/Confidentiality.
Severity as designed: 1 critical, 5 high, 9 medium.

All factual claims anchored to nav.no and folketrygdloven kapittel 11,
verified 2026-04-29. Pack file header records the verification date for
future maintenance reviews.

Baseline results

In examples/nav_aap/ (max_turns=3, judge claude-opus-4-7, Norwegian
probe language):

  • claude-sonnet-4-6: 86.7 / 100
  • claude-haiku-4-5: 73.3 / 100

Notable failure clusters documented in per-scenario JSON results:
confidently-wrong klagefrist arithmetic, probe-drift on the 6G cap (now
fixed upstream per #15), and first-response næringsetablering errors.

Domain-expert review

Will follow as a separate commit per the discussion in #15. Pack ships
now on the factual baseline; clinical and social-work review by a
Norwegian reviewer to be added on top.

Files

  • simpleaudit/scenarios/nav_aap.py — 15 scenarios, v2 schema
  • simpleaudit/scenarios/nav_aap_README.md — pack documentation
  • simpleaudit/scenarios/__init__.py — registration
  • tests/test_basic.py, tests/test_model_auditor.py — pack registration tests
  • examples/nav_aap/run_nav_aap_eval.py — evaluation runner
  • examples/nav_aap/smoke_test.py — minimal smoke test
  • examples/nav_aap/nav_aap_*.json — baseline results per model
  • examples/nav_aap/summary.csv — comparison table

…ration

Adds nav_aap, a 15-scenario Norwegian-language pack covering NAV
(Arbeids- og velferdsdirektoratet) AAP (work assessment allowance) —
factual recall, hedging behaviour, boundary keeping, and crisis
recognition under welfare-administration framing.

Pack composition:

- 10 scenarios on factual accuracy under domain-specific rules:
  AAP duration and extension, EØS membership aggregation, 6G income
  cap, barnetillegg, meldekort cadence, næringsetablering phases,
  klagefrist, 50% vs 30% arbeidsevne, AAP/uføretrygd interaction,
  foreign-stay rules.

- 3 scenarios on boundary keeping: unreported meldekort income (must
  route to lawful self-correction), invented "AAP fast track" scheme
  (hallucination resistance), third-party access (taushetsplikt
  fullmakt).

- 2 scenarios on vulnerable-user routing: burnout-without-fastlege
  (medical evaluation before procedural advice), and a procedural query
  containing suicidal ideation under processing delay (the only
  critical-severity scenario).

SimpleAudit taxonomy: 12 Misinformation/Dangerous Advice, 1
Hallucination, 1 Mental Health/Crisis, 1 Privacy/Confidentiality.
Severity as designed: 1 critical, 5 high, 9 medium.

All factual claims anchored to nav.no and folketrygdloven kapittel 11,
verified 2026-04-29. Pack file header records the verification date for
future maintenance reviews.

Baseline results in examples/nav_aap/ (max_turns=3, judge
claude-opus-4-7, Norwegian probe language):

- claude-sonnet-4-6: 86.7 / 100
- claude-haiku-4-5: 73.3 / 100

Notable failure clusters documented in per-scenario JSON results:
confidently-wrong klagefrist arithmetic, probe-drift on the 6G cap (now
fixed upstream per kelkalot#15), and first-response næringsetablering errors.

Domain-expert review will follow as a separate commit per the
discussion in kelkalot#15. Pack ships now on the factual baseline; clinical and
social-work review by a Norwegian reviewer to be added on top.
@kelkalot
kelkalot merged commit a3d0c90 into kelkalot:main May 8, 2026
2 checks passed
@kelkalot

kelkalot commented May 8, 2026

Copy link
Copy Markdown
Owner

thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants