Add jfinqa: Japanese Financial Numerical Reasoning QA (1000 questions)

[ajtgjmdjp]

Summary

• Add jfinqa, a benchmark of 1000 questions for evaluating LLMs on numerical reasoning over Japanese corporate financial statements
• Three subtasks: numerical reasoning (550), consistency checking (200), temporal reasoning (250)
• Questions require multi-step arithmetic (1–5 steps) over tables extracted from real EDINET filings, spanning 68 companies across J-GAAP, IFRS, and US-GAAP
• Zero-shot, generate_until format with numerical matching (1% tolerance)

Task structure

Group	Tasks
jfinqa	jfinqa_numerical, jfinqa_consistency, jfinqa_temporal

Baseline results (zero-shot, temperature=0, 1000 questions)

Model	Overall	NR	CC	TR
GPT-4o	86.8%	79.6%	93.5%	97.2%
Gemini 2.0 Flash	77.3%	78.7%	82.5%	70.0%
GPT-4o-mini	69.0%	83.6%	86.0%	23.2%

Links

• Dataset: https://huggingface.co/datasets/ajtgjmdjp/jfinqa
• Code & paper: https://github.com/ajtgjmdjp/jfinqa
• License: Apache-2.0

Checklist

• Is the task an existing benchmark in the literature?
• Have you verified the samples from the dataset are correct?
• Is the dataset publicly available?
• Does the task have a dedicated README.md?
• Have you cited the original paper?
• No external dependencies (utils.py uses only Python stdlib)

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.

[ajtgjmdjp]

Hi — just a friendly ping. Happy to address any feedback.

Quick summary: this adds jfinqa, a Japanese financial numerical reasoning QA benchmark (1,000 questions from 68 companies). It includes 4 baselines (GPT-4o 87.0%, Gemini 2.0 Flash 80.4%, GPT-4o-mini 67.7%, Qwen2.5-3B 39.6%) and is published on both PyPI and HuggingFace.