FAIR Workbench

Finance Agent Integrity & Reliability Workbench is a local, privacy-preserving prototype for evaluating finance-agent outputs before they become trusted workflow artifacts.

It is not a chatbot and not a static profile page. It is a technical product demo for turning finance-agent responses into:

• structured prompts
• claim-level audits
• deterministic formula checks
• JSON/schema and source-policy checks
• evaluator notes
• follow-up prompts
• human-review triggers
• reusable regression cases

The core thesis:

Evaluation should not end as a checklist. It should become infrastructure.

Why I Built It

Finance agents can produce outputs that look polished while still mixing calculation bases, inventing unsupported claims, drifting from source policy, or producing inconsistent tables and JSON.

FAIR treats the agent output as an artifact that must be inspected, reconciled, and converted into future safeguards.

Workflow

The public demo follows a production-style agent reliability pattern:

1. Privacy Intake: keep the demo local and separate synthetic data from private files.
2. Task Classification: identify whether the case is lender review, cash-flow planning, source comparison, rubric evaluation, or a custom finance workflow.
3. Workflow Contract: generate source, evidence, formula, schema, and review expectations before evaluating the answer.
4. Artifact Parsing: extract sections, claims, numbers, JSON blocks, tables, source terms, visual terms, and self-audit language.
5. Dynamic Fact + Formula Registry: let the user define facts and formulas instead of hardcoding one case.
6. Deterministic Validation: evaluate formulas, compare output numbers to known facts/formulas, and detect basis drift.
7. Issue Taxonomy: classify arithmetic mismatch, definition drift, source-policy risk, schema risk, visual-semantics risk, false self-audit, and missing uncertainty labels.
8. Human Review Routing: mark high-risk failures for human review instead of silently accepting the output.
9. Regression Builder: export failure tags and must-pass conditions so repeated failures can become eval fixtures.

What Makes It Agentic

The demo is intentionally workflow-first rather than autonomy-first. It borrows from strong production agent patterns:

• use simple, inspectable workflows before adding autonomy
• keep state and traces explicit
• use typed tools for deterministic operations
• add guardrails around inputs, outputs, schemas, and tool boundaries
• preserve a human-in-the-loop path for high-risk decisions
• convert recurring failures into regression tests and CI checks

Documentation

• Architecture
• Demo guide
• Release process
• Application answer
• Privacy boundary
• Security notes

Demo Features

• Paste a full model response, multi-turn conversation, table, JSON, or OCR text.
• Upload a screenshot preview without sending it anywhere.
• Edit facts and formulas dynamically.
• Generate a high-quality evaluation prompt.
• Parse claims and high-risk terms.
• Audit numbers against facts and formulas.
• Generate an evaluator note and follow-up prompt.
• Export regression JSON, promptfoo-style YAML, and JSONL-style eval cases.
• Save/load local cases in localStorage.

Privacy

This repository is designed to be public-safe:

• No backend.
• No model API key.
• No upload endpoint.
• No telemetry.
• No real customer files.
• No private employer data.
• No private PDFs or personal documents.
• All included scenarios are synthetic.

See PRIVACY.md for the public-data boundary.

Tech Stack

• React
• TypeScript
• Vite
• mathjs
• lucide-react

Run Locally

npm install
npm run dev

Build:

npm run build

Preview the production build:

npm run preview

Run browser-based verification against a local URL:

npm run verify:cdp -- http://127.0.0.1:4173

The verification script saves screenshots under artifacts/screenshots/, which is intentionally ignored by Git.

Deploy to GitHub Pages

This repo includes .github/workflows/pages.yml.

1. Push the repo to GitHub.
2. In GitHub, open Settings -> Pages.
3. Set source to GitHub Actions.
4. Push to main.

GitHub Actions will build the Vite app and publish dist.

Public References

This project is inspired by public writing and documentation from:

• Anthropic, Building effective agents
• OpenAI, Agents SDK guide
• LangChain/LangGraph, Human-in-the-loop
• Promptfoo, LLM evals and red teaming
• Langfuse, Observability overview
• Guardrails AI, Structured data generation
• OpenAI, Evals
• Cohere, Tool use, Structured outputs, and Rerank

Status

Public prototype. The current version is a local demo and architecture artifact, not a production financial decision system.