Skip to content

datasets.md

Latest commit

 

History

History
432 lines (355 loc) · 23.4 KB

datasets.md

File metadata and controls

432 lines (355 loc) · 23.4 KB
Raw

Quranic QA Benchmarking: Dataset Survey

A curated, non-redundant survey of datasets and benchmarks relevant to Quranic and Islamic QA research. Compiled from literature review covering 2014–2025.

Contents

  1. Quranic QA Datasets (Arabic)
  2. Broader Islamic QA Datasets
  3. General Arabic QA Datasets (used in Quranic QA research)
  4. Non-Arabic / Multilingual Islamic QA Datasets
  5. Cross-Cutting Observations

1. Quranic QA Datasets (Arabic)

AyaTEC

Field Details
Full Name AyaTEC — A Fully Reusable Test Collection for Arabic QA over the Qur'an
Year 2020 (v1.2 used in 2023 shared task)
Authors Malhas and Elsayed
Dataset GitLab — Quran-QA · Task site
Paper ACM TALLIP
Size 207 questions · 1,762 verse-based answers (v1.2: 199 Q · 1,132 gold passage IDs)
Language Arabic (MSA)
Format Verse-based QA; three answer types: single, multi, zero-answer
Categories 11 topic categories spanning the Holy Qur'an
Tasks Addressed Ad hoc Quranic search · Verse-based QA · Factoid and non-factoid questions
Annotations Exhaustive — Islamic scholars annotated all directly answering verses
Disadvantages Skewed topic distribution · Small size limits model training · Does not test free generation or hallucination · No evaluation of morpho-syntactic complexity

QRCD v1.1 — Qur'anic Reading Comprehension Dataset

Field Details
Full Name Qur'anic Reading Comprehension Dataset v1.1
Year 2022
Authors Malhas and Elsayed
Dataset GitLab — Quran-QA · GitHub
Paper ACL Anthology · arXiv
Size 1,093 question-passage (QP) pairs · 1,289 question-passage-answer (QPA) triplets
Language Arabic
Format SQuAD v1.1-style extractive MRC; span-extraction
Categories Inherits AyaTEC's 11 Quranic topic categories
Tasks Addressed Machine Reading Comprehension · Answer span extraction · Used in Qur'an QA 2022 Shared Task (13 teams, 30 runs)
Disadvantages All questions are answerable (no unanswerable cases) · Imbalanced topic distribution · Does not evaluate generative hallucinations

QRCD v1.2 — Qur'anic Reading Comprehension Dataset (Extended)

Field Details
Full Name Qur'anic Reading Comprehension Dataset v1.2
Year 2023
Authors Malhas et al.
Dataset GitLab — Quran-QA 2023 · HuggingFace · GitHub (mirror)
Paper ACL Anthology
Size 1,155 QP pairs · 1,399 QPA triplets · ~15% zero-answer questions
Split 70% train / 10% dev / 20% test
Language Arabic
Format SQuAD v2.0-style MRC with unanswerable questions
Categories 11 Quranic topic areas + zero-answer augmentation
Tasks Addressed Task A — Passage Retrieval over QPC (1,266 passages) · Task B — MRC / span detection · Qur'an QA 2023 Shared Task
Disadvantages Still small for LLM fine-tuning · Weak retrieval performance reported by participating teams · Imbalanced training data · Does not test generative hallucinations

Al-Bayan

Field Details
Full Name Al-Bayan — Arabic QA Dataset for the Holy Quran
Year 2014
Authors Abdelnasser et al.
Dataset GitHub
Paper ACL Anthology — ANLP@EMNLP 2014
Size Not reported
Language Arabic
Format Retrieval-based QA over Quran + Tafseer books
Categories Quran text + Tafseer (interpretation)
Tasks Addressed QA → retrieve relevant verses → extract passage from Quran and Tafseer; reported 85% accuracy at top-3 results
Disadvantages Early foundational work; limited scope · No size reported · Predates modern transformer-based evaluation norms

AQQAC — Annotated Corpus of Arabic Al-Qur'an QA

Field Details
Full Name Annotated Corpus of Arabic Al-Qur'an Question and Answer
Year 2018
Authors Alqahtani and Atwell
Dataset University of Leeds Research Data Repository (DOI: 10.5518/356)
Paper Semantic Scholar
Size Not reported
Language Arabic
Format Ontology-based QA corpus
Categories Arabic Quranic domain (ontology-driven)
Tasks Addressed Ontology-based Quranic QA evaluation
Disadvantages No size details; early ontology-based approach now superseded by neural methods

QUQA — Quranic Understanding Question Answering

Field Details
Full Name QUQA — Quranic Understanding Question Answering Dataset
Year 2023
Authors Alnefaie et al.
Dataset GitHub — HAQA-and-QUQA
Paper ACL Anthology — RANLP 2023
Size Not reported
Language Arabic
Format Question-passage pairs from the Qur'an
Categories Wide thematic and linguistic diversity from Quranic text
Tasks Addressed Supervised QA over Quranic passages · Semantic and interpretative understanding
Disadvantages Size unknown · Used as supplementary data, not standalone benchmark · Limited documentation

Aljaji et al. 2025 Benchmark

Field Details
Full Name Benchmarking Generative AI on Quranic Knowledge (Quranic MCQ + Verse Identification)
Year 2025
Authors Aljaji et al.
Dataset Not public
Paper OpenReview / NeurIPS 2025
Size Not reported
Language Arabic
Format Multiple-choice questions + verse identification tasks
Categories Quranic content organized by Bloom's taxonomy cognitive levels and familiarity/perplexity levels
Tasks Addressed MCQ answering · Verse identification · Cognitive difficulty stratification
Disadvantages Dataset not released · Does not evaluate generative hallucinations · Does not test complex morpho-syntactic-semantic queries

2. Broader Islamic QA Datasets

Comprehensive Islamic QA Dataset (Qamar et al., 2024)

Field Details
Full Name Benchmark Dataset with Larger Context for Non-Factoid QA over Islamic Text
Year 2024
Authors Qamar, Latif, and Latif
Source IslamQA.org (90,000+ QA pairs)
Dataset HuggingFace — LCQA-Islamic
Paper arXiv 2409.09844
Size 73,000+ question-answer pairs
Language Arabic
Format Non-factoid QA with large contextual passages
Categories Quranic Tafsir (commentary) + Ahadith (Hadith literature)
Tasks Addressed Complex non-factoid QA · Interpretive queries over Tafsir and Hadith
Disadvantages Automatic metrics (ROUGE) agree with expert scholars only 11–20% of the time · No established evaluation protocol for Islamic systems · Limited to textual QA without multimodal content

Quran Tafseer Dataset

Field Details
Full Name Quran Tafseer Dataset
Year ~2023
Authors Community releases (MohamedRashad; tarteel-ai)
Dataset HuggingFace — MohamedRashad/Quran-Tafseer · HuggingFace — tarteel-ai/quran-tafsir
Paper No dedicated paper; used in RAG studies (e.g., arXiv 2412.11431)
Size 84 Tafseer books (MohamedRashad release)
Language Arabic
Format Semantically rich Tafseer text passages
Categories Quranic interpretation / commentary
Tasks Addressed Deep contextual inference and reasoning · Used as knowledge base in RAG pipeline evaluation
Disadvantages Community release with no peer-reviewed paper · Construction methodology undocumented

Hajj-FQA

Field Details
Full Name Hajj-FQA — Fatwa-based QA Benchmark for Hajj
Year 2025
Authors Aleid and Azmi
Dataset Not public (contact authors)
Paper Springer — Journal of KSU-CIS 2025
Size 2,826 QA pairs extracted from 800 expert-annotated fatwas
Language Arabic
Format Extractive QA over fatwa texts
Categories Hajj pilgrimage · Islamic jurisprudence
Tasks Addressed QA benchmarking over religious legal rulings (fatwas)
Disadvantages Narrow domain (Hajj only) · No cross-domain evaluation · Does not generalize to other areas of Fiqh or Quran

FiqhQA Benchmark

Field Details
Full Name FiqhQA — Sacred or Synthetic? Evaluating LLM Reliability for Religious Questions
Year 2025
Authors Atif et al.
Dataset Not public
Paper arXiv 2508.08287 · AAAI AIES 2025
Size 960 QA pairs
Source Kuwaiti Fiqh Encyclopedia
Language Arabic and English (bilingual)
Format Multiple-choice QA
Categories Islamic jurisprudence (Fiqh al-'Ibādāt — acts of worship) across four Sunni schools: Hanafi, Maliki, Shafi'i, Hanbali
Tasks Addressed Evaluating LLMs on Islamic jurisprudence in bilingual setting
Disadvantages Limited to four Sunni schools only (no Shi'i jurisprudence) · Single subdomain (acts of worship) · No complex legal reasoning tasks · Small size

QIAS 2025 — Islamic Inheritance Law Shared Task

Field Details
Full Name QIAS 2025 — ArabicNLP 2025 Shared Task on Islamic Inheritance Law
Year 2025
Authors ArabicNLP 2025 Shared Task organizers
Dataset CodaBench · GitHub
Paper ACL Anthology — ArabicNLP 2025
Size Derived from 32,000 fatwas; exact QA count not reported
Language Arabic
Format Multiple-choice QA across three difficulty levels
Categories Islamic inheritance law ('ilm al-mawārīth)
Tasks Addressed MCQ QA over inheritance law · Expert-verified reasoning across difficulty tiers
Disadvantages Single subdomain (inheritance law) · Does not generalize to other Islamic or Quranic content

MizanQA

Field Details
Full Name MizanQA — Moroccan Islamic Jurisprudence QA Dataset
Year 2025
Authors Adlbh et al.
Dataset HuggingFace — adlbh/MizanQA-v0
Paper arXiv 2508.16357
Size ~1,700 multiple-choice questions
Language Arabic (with Moroccan/French-influenced conventions)
Format Multiple-choice QA
Categories Maliki jurisprudence · Moroccan Islamic legal conventions
Tasks Addressed MCQ QA over Maliki Fiqh
Disadvantages Single school (Maliki) only · Emphasizes factual recall over complex reasoning · Moroccan-specific conventions reduce generalizability · No cross-school comparative evaluation

3. General Arabic QA Datasets (used in Quranic QA research)

These are not Quranic datasets but are frequently used as supplementary training data or baselines in Quranic QA papers.

ARCD — Arabic Reading Comprehension Dataset

Field Details
Full Name Arabic Reading Comprehension Dataset
Year 2019
Authors Mozannar et al.
Dataset HuggingFace — arcd · GitHub — SOQAL
Paper ACL Anthology — WANLP 2019 · arXiv 1906.05394
Size 1,395 questions (crowdworker-annotated over Arabic Wikipedia)
Language Arabic (MSA)
Format Extractive MRC (SQuAD-style)
Categories Arabic Wikipedia; general domain
Tasks Addressed Open-domain factual Arabic QA · MRC benchmarking
Disadvantages General purpose only · No religious or Quranic domain knowledge · Not suitable without adaptation for Islamic QA

Arabic-SQuAD (ArSQuAD)

Field Details
Full Name Arabic adaptation of Stanford QA Dataset (machine-translated)
Year ~2019
Authors Various (machine translation of SQuAD)
Dataset HuggingFace — i0xs0/Arabic-SQuAD · GitHub — Arabic_QA_Datasets
Paper ACL Anthology — WANLP 2019 · arXiv 1906.05394
Size ~87,000+ QA pairs via machine translation
Language Arabic (machine-translated from English)
Format Extractive QA
Categories Arabic Wikipedia / general domain
Tasks Addressed Extractive Arabic QA · MRC benchmarking
Disadvantages Machine-translation quality issues explicitly documented in literature · Not suitable for specialized domains without quality filtering · No Islamic/Quranic adaptation

ArabicaQA

Field Details
Full Name ArabicaQA
Year 2024
Authors Mozannar et al.
Dataset GitHub — DataScienceUIBK/ArabicaQA
Paper arXiv 2403.17848 · ACM SIGIR 2024
Size 89,000+ answerable + 3,700 unanswerable questions
Language Arabic
Format Extractive QA including unanswerable questions (SQuAD v2.0-style)
Categories Arabic Wikipedia; general domain
Tasks Addressed Arabic QA including unanswerable question detection
Disadvantages General domain; Wikipedia-derived; not applicable to Islamic/Quranic content without adaptation

TyDi QA (Arabic Subset)

Field Details
Full Name TyDi QA — Typologically Diverse Question Answering
Year 2020
Authors Clark et al.
Dataset GitHub — google-research-datasets/tydiqa · HuggingFace
Paper arXiv 2003.05002
Size ~200,000 QA pairs total; Arabic subset not separately reported
Language Arabic (and 10 other typologically diverse languages)
Format Information-seeking QA (passage retrieval + answer span)
Categories Multilingual; general domain
Tasks Addressed General Arabic QA; used as supplementary training resource alongside Quranic datasets
Disadvantages General multilingual benchmark; no religious or Quranic content

4. Non-Arabic / Multilingual Islamic QA Datasets

Indonesian Quranic QA Corpus

Field Details
Full Name Indonesian Translation of Holy Qur'an QA Corpus (series)
Years Gusmita et al. (2014) → Sukmana et al. (2016) → Putra et al. (2016)
Dataset Not public
Paper Semantic Scholar — Gusmita 2014
Size ~222 ontology concepts (77 Person, 24 Location, 6 Time); two test QA sets
Language Indonesian
Format Rule-based / semantic ontology-based factoid QA
Categories Indonesian Quranic translation; named entities: Person, Location, Time
Tasks Addressed Factoid QA (Who, When, Where) over Indonesian Quranic translation
Disadvantages Covers only three factoid question types · Limited to Indonesian translation · Does not address Arabic source text · Early rule-based and ontology-based approaches

QASiNa

Field Details
Full Name QASiNa — Religious Domain QA Using Sirah Nabawiyah
Year 2023
Authors Rizqullah et al.
Dataset GitHub — rizquuula/QASiNa · HuggingFace — SEACrowd/qasina
Paper arXiv 2310.08102
Size Not reported
Language Indonesian
Format Extractive QA
Categories Sirah Nabawiyah (biography of the Prophet Muhammad)
Tasks Addressed QA in Islamic biographical domain · Evaluation of multilingual transformers (XLM-R, mBERT, IndoBERT) vs. LLMs (GPT-3.5, GPT-4)
Disadvantages Narrow domain (Sirah only) · Indonesian language limits Arabic-focused applicability · Does not cover Quran or Fiqh

IslamicPCQA

Field Details
Full Name IslamicPCQA — Persian Complex QA in the Islamic Domain
Year 2023
Authors Ghafouri et al.
Dataset Not public
Paper arXiv 2304.11664 · IEEE Xplore
Size 12,282 question-answer pairs
Source Nine Islamic encyclopedias
Language Persian (Farsi)
Format Multi-hop complex QA (modeled after HotpotQA)
Categories Islamic domain broadly; requires multi-step reasoning across documents
Tasks Addressed Multi-hop complex QA · Multi-step reasoning across Islamic texts
Disadvantages Persian only — not applicable for Arabic Quranic research · First of its kind in Persian; breadth across Islamic subdisciplines unknown

5. Cross-Cutting Observations

Key Limitations Across All Datasets

Issue Description
Small size The most universal limitation. Virtually all Quranic QA datasets are too small (< 10,000 QA pairs) for modern LLM fine-tuning.
Automatic metrics unreliable Qamar et al. (2024) found ROUGE scores agreed with Islamic scholars only 11–20% of the time — standard NLP evaluation is fundamentally misaligned with Islamic domain expertise.
No hallucination testing No existing Quranic/Islamic QA dataset is designed to specifically detect or measure LLM hallucinations on religious content.
No morpho-syntactic-semantic evaluation Complex Arabic linguistic queries combining morphology + syntax + semantics are not addressed by any existing benchmark.
Imbalanced topic coverage AyaTEC and its derivatives have skewed distributions across Quranic topics.
Licensing undocumented Licensing terms are almost entirely absent across reviewed datasets.
No generative QA evaluation Most benchmarks focus on extractive span detection or MCQ; free-form generative answer evaluation is absent.
Single-domain silos Most Islamic QA datasets (Hajj-FQA, FiqhQA, QIAS, MizanQA) are narrowly scoped to one subdomain.

Availability Summary

Dataset Dataset Link Paper
AyaTEC / QRCD v1.1 gitlab.com/bigirqu/quranqa ACM TALLIP / ACL
QRCD v1.2 gitlab.com/bigirqu/quran-qa-2023 · HF ACL
Al-Bayan GitHub ACL
AQQAC Leeds Repository Semantic Scholar
QUQA GitHub ACL/RANLP 2023
Aljaji 2025 Not public OpenReview/NeurIPS 2025
Comprehensive Islamic QA HuggingFace arXiv
Quran Tafseer Dataset HF-MohamedRashad · HF-tarteel —
Hajj-FQA Not public Springer
FiqhQA Not public arXiv
QIAS 2025 CodaBench · GitHub ACL
MizanQA HuggingFace arXiv
ARCD HF · GitHub ACL
Arabic-SQuAD HuggingFace ACL
ArabicaQA GitHub arXiv
TyDi QA HF · GitHub arXiv
Indonesian Quranic QA Not public Semantic Scholar
QASiNa GitHub · HF arXiv
IslamicPCQA Not public arXiv · IEEE

Compiled from: "Quranic QA Benchmarking Resources and Datasets: Characteristics, Limitations, and Availability" (PDF) and existing benchmarks review (DOCX), March 2026.