Skip to content

chore: prepare Korean BullshitBench for public release - #8

Merged
wo0jun merged 2 commits into
mainfrom
chore/public-release-packaging
Aug 8, 2026
Merged

chore: prepare Korean BullshitBench for public release#8
wo0jun merged 2 commits into
mainfrom
chore/public-release-packaging

Conversation

@wo0jun

@wo0jun wo0jun commented Aug 8, 2026

Copy link
Copy Markdown
Member

Summary

  • reposition ko_native_v0_1 as the primary benchmark and translated_v2 as a comparison baseline
  • document the methodology, 0-2 evaluation protocol, control-question policy, and mapping to the original BullshitBench taxonomy
  • publish the 48 native verification sources as an ID-linked sidecar
  • add contribution, citation, changelog, and machine-readable benchmark metadata
  • strengthen dataset validation and run it automatically in GitHub Actions

Why

The repository still described most native rows as placeholders even though all 48 questions were complete. It also lacked the documentation and automated release checks needed for public use. This change brings the repository description, metadata, and validation contract in line with the current dataset.

Impact

  • benchmark questions are unchanged
  • ko_native_v0_1 is documented as 48 complete questions across all 6 domain by 8 technique combinations
  • verification sources remain outside model and judge inputs
  • current scoring remains a 0-2 nonsense-detection rubric, with future control accuracy reported separately

Validation

  • python3 scripts/validate.py
  • python3 -m py_compile scripts/build_release.py scripts/validate.py
  • YAML parsing for benchmark.yaml, CITATION.cff, and the validation workflow
  • JSON parsing for every release JSONL file
  • git diff --check HEAD^ HEAD

@wo0jun
wo0jun marked this pull request as ready for review August 8, 2026 16:19
@wo0jun
wo0jun merged commit 48a650c into main Aug 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant