Date: 2026-05-12 Researcher: Jessica Irwin (Gru.ai) Source: Databricks Blog | Paper (arXiv) | GitHub
OfficeQA is Databricks' open-source benchmark for evaluating AI agents on grounded reasoning — answering questions that require searching, extracting, and analyzing information from complex real-world document collections.
Unlike academic benchmarks (ARC-AGI, MMLU, Humanity's Last Exam) that test abstract reasoning or parametric knowledge, OfficeQA focuses on economically valuable enterprise tasks: answering precise questions grounded in proprietary, unstructured data.
- Corpus: ~89,000 pages of U.S. Treasury Bulletins (1939–present)
- Document types: Scanned physical PDFs (pre-1996) + digital-native PDFs
- Data density: 26+ million numerical values across the corpus
- Questions: 246 total (133 Pro + 113 Easy)
- License: Apache 2.0 (code) / CC-BY-SA 4.0 (data)
| Tier | Count | Description |
|---|---|---|
| OfficeQA Pro | 133 | Hard questions — the primary benchmark for frontier models |
| OfficeQA Full | 246 | Includes 113 easier questions for system optimization |
"Easy" questions are those answered correctly by both leading frontier agents (GPT-5.1 and Claude Opus 4.5 at time of initial release). Pro questions require more complex multi-step reasoning.
- 62% require analytical reasoning beyond basic arithmetic
- 22% involve internet search for external values
- 11% require data from 3+ separate bulletins
- 3% require visual reasoning over charts/figures
- All questions have single, unambiguous correct answers
- Questions span 1–20+ pages of source material
- Annotators generate questions by referencing corpus documents
- A second annotator independently reproduces the answer using the same documents
- AI agent conflict resolution through a three-reviewer system
- Questions answerable by parametric knowledge alone are filtered out
- Difficulty classification based on frontier agent performance
Databricks partnered with SuperAnnotate for annotation infrastructure.
Deterministic exact-match evaluation with configurable tolerance:
Absolute Relative Error (%) = |predicted - actual| / |actual| × 100
| Tolerance | Description |
|---|---|
| 0.0% | Exact match (default for reporting) |
| 0.1% | Minor rounding differences |
| 1.0% | Small numerical variations |
| 5.0% | Approximate answers |
Non-numerical answers use fuzzy text matching with punctuation normalization.
| Configuration | Claude Opus 4.6 | GPT-5.4 | Gemini 3.1 Pro |
|---|---|---|---|
| Prompt Only (no docs) | 2.26% | 0.75% | 2.26% |
| With Web Search | 3.01% | 11.28% | 3.01% |
| Oracle PDF Pages (raw) | 36.09% | 56.39% | 51.13% |
| Oracle Parsed Pages | 57.14% | 65.41% | 56.39% |
Key insight: Without document access, frontier models score ~2%. Even the best parametric knowledge cannot answer grounded questions.
| Agent Framework | Model | Format | Accuracy |
|---|---|---|---|
| Claude Agent SDK | Opus 4.6 | Raw PDF | 48.12% |
| Claude Agent SDK | Opus 4.6 | Parsed | 54.14% |
| Codex CLI | GPT-5.4 | Raw PDF | 36.09% |
| Codex CLI | GPT-5.4 | Parsed | 56.39% |
| Gemini CLI | Gemini 3.1 | Raw PDF | 18.05% |
| Gemini CLI | Gemini 3.1 | Parsed | 29.32% |
Best agent result: Claude Opus 4.6 with parsed corpus = 54.14% (full corpus), 66.92% (oracle pages)
Databricks' ai_parse_document tool dramatically improves results:
- Claude Opus 4.5: +30.2 percentage points with parsing
- GPT-5.1: +9.1 percentage points with parsing
- Average relative performance gain: 16.1%
| Setting | Accuracy | Avg. Latency |
|---|---|---|
| Oracle Pages (Human) | 51.1% | 19.2 min |
| Oracle Pages (Agent) | 71.1–72.2% | 2.1–5.3 min |
| Full Corpus (Human) | 34.6% | 31.4 min |
| Full Corpus (Agent) | 50.0–56.7% | 3.5–15.4 min |
Agents consistently outperform humans in both speed and accuracy on this benchmark, though humans excel at identifying layout-based formatting issues in raw PDFs.
Agents prematurely converge on the first numerically plausible value instead of identifying the most recent revision. Treasury Bulletins frequently revise earlier figures, and agents fail to verify temporal currency, leading to cascading errors.
Even state-of-the-art document parsing introduces errors:
- Misread numbers (OCR artifacts)
- Corrupted text from scanning
- Misaligned table topology (columns/rows shifted)
- These errors propagate through downstream calculations
"Agent systems lack the granular resolution to correctly interpret dense financial charts and struggle to map visual trend lines with high precision."
Only 3% of questions require visual reasoning, but agents fail disproportionately on these.
Agents retrieve contextually misaligned alternatives, apply mathematically valid but contextually wrong formulas, and ignore explicit calculation requirements stated in the question.
- 23.6 minutes average per question over full PDF corpus
- 3.9 minutes with pre-parsed documents
- Enterprise use cases demand faster turnaround
| Dimension | OfficeQA | GDPVal |
|---|---|---|
| Creator | Databricks | OpenAI |
| Focus | Grounded reasoning over documents | Economically valuable work product |
| Scope | Deep (single domain, financial docs) | Broad (44 occupations, 9 industries) |
| Tasks | 246 questions with deterministic answers | 1,320 tasks requiring multimodal deliverables |
| Corpus | ~89K pages of Treasury Bulletins | Various professional scenarios |
| Evaluation | Deterministic exact-match with tolerance | Human expert pairwise comparison |
| Output | Numerical/text answers | PDFs, spreadsheets, presentations, etc. |
| Question | "Can AI accurately reason over enterprise data?" | "Can AI replace a knowledge worker?" |
| Dimension | OfficeQA | ARC-AGI-2 / HLE |
|---|---|---|
| Task type | Document grounded QA | Abstract reasoning / expert knowledge |
| Data dependency | Requires external corpus | Self-contained |
| Enterprise relevance | High (directly mimics real work) | Low (academic/theoretical) |
| Evaluation | Numerical accuracy | Pattern matching / expert verification |
Bottom line: GDPVal asks "can AI replace a knowledge worker?" (breadth). OfficeQA asks "can AI accurately answer questions grounded in real documents?" (depth). They are complementary.
Use freely available, information-dense collections:
- SEC EDGAR filings (10-K, 10-Q, proxy statements)
- Patent databases (USPTO full-text)
- Government reports (GAO, CBO, census)
- Academic papers (arXiv, PubMed)
- Legal case files (PACER, state courts)
- Municipal budgets and city planning documents
- Use a frontier LLM to generate candidate questions from documents
- Have the LLM attempt to answer them
- Human annotators verify/correct answers (~3-5x cheaper than full human authoring)
- Filter questions solvable by parametric knowledge alone
Partner with companies for anonymized internal docs:
- HR policy manuals + compliance questions
- Technical documentation + troubleshooting scenarios
- Financial reports + analytical queries
- Contract repositories + legal reasoning tasks
Extend beyond PDFs:
- Spreadsheets with formulas and pivot tables
- Slide decks with embedded charts
- Email threads with attachments
- Database exports (CSV/JSON)
- Mixed-format document packages
Using platforms like Scale AI or Surge AI:
- Recruit domain experts (accountants, lawyers, analysts)
- Provide a document corpus
- Have them write questions they'd actually ask in their job
- Cross-validate with a second expert
- Filter by difficulty using AI agent baselines
| Method | Cost per Question | 250 Questions |
|---|---|---|
| Full human annotation | $50–100 | $12.5K–25K |
| Synthetic + human verify | $15–30 | $3.75K–7.5K |
| Domain expert crowdsource | $30–60 | $7.5K–15K |
- Questions must require the corpus (filter out parametric knowledge)
- Answers should be deterministic and verifiable
- Include multi-step reasoning (not just lookup)
- Mix difficulty levels based on agent performance
- Use strict tolerance evaluation (0.0% error default)
- Include both scanned and digital documents
- Cross-annotator verification is essential
OfficeQA highlights critical gaps in AI agent capabilities for enterprise document understanding:
- Document parsing quality is a major differentiator (+16-30pp accuracy gains)
- Agent frameworks matter — Claude Agent SDK consistently outperforms alternatives
- Real enterprise work requires grounding in proprietary data, not parametric knowledge
- Building domain-specific benchmarks could differentiate Gru.ai in vertical markets
- The Grounded Reasoning Cup (Spring 2026) presents an opportunity for visibility
- Evaluate Gru.ai agent performance on OfficeQA Pro
- Build a domain-specific benchmark for a target vertical
- Investigate document parsing improvements for enterprise use cases
- Consider entering the Databricks Grounded Reasoning Cup
- Databricks Blog: Introducing OfficeQA (Dec 2025)
- arXiv Paper: OfficeQA Pro (Mar 2026)
- GitHub: databricks/officeqa
- SuperAnnotate Case Study: How SuperAnnotate Helped Build OfficeQA (Feb 2026)
- HuggingFace Papers: OfficeQA Pro (Mar 2026)