Proposal: EvalPort reporter for portable evaluation ResultSets

[adhabnr-ux]

TL;DR

deepfabric.evaluation already produces a well-typed, structured evaluation result (EvaluationResult → EvaluationMetrics + list[SampleEvaluation]), and the reporter layer (BaseReporter → FileReporter / CloudReporter) is already the seam where output format is decided. I'd like to propose adding one more reporter that serializes to EvalPort — an open, framework-agnostic JSON format for eval TestCase / Grader / EvalSuite / ResultSet documents — so a DeepFabric evaluation run can be consumed by other tools (dashboards, benchmark aggregators, CI gates) without them needing to understand DeepFabric's internal schema.

I'm not affiliated with DeepFabric — just proposing this as someone maintaining the interop spec, based on actually reading deepfabric/evaluation/. Posting here rather than opening a PR straight away, per CONTRIBUTING's "discuss before changing" guidance.

What I looked at

Specifically, in deepfabric/evaluation/:

• evaluator.py — Evaluator, EvaluatorConfig, EvaluationResult
• metrics.py — EvaluationMetrics, SampleEvaluation, compute_metrics()
• parser.py — GroundTruth, ExpectedToolCall (tool_name + parameters)
• evaluators/base.py, evaluators/builtin/tool_calling.py — BaseEvaluator, ToolCallingEvaluator, EvaluatorResult
• reporters/base.py, reporters/file_reporter.py — BaseReporter, FileReporter (writes result.model_dump_json(indent=2))

The mapping

EvaluationResult already lines up closely with EvalPort's ResultSet document:

DeepFabric	EvalPort
SampleEvaluation.sample_id	Result.test_case_id (stringified)
SampleEvaluation.query	(source TestCase.input, carried through)
SampleEvaluation.predicted_answer / predicted_tool + predicted_parameters	Result.actual_output
SampleEvaluation.tool_selection_correct	one GraderResult (score 1.0/0.0, passed)
SampleEvaluation.parameters_correct	another GraderResult
SampleEvaluation.execution_valid	another GraderResult
SampleEvaluation.error	Result.error.message
EvaluationMetrics.tool_selection_accuracy / .parameter_accuracy / .execution_success_rate	ResultSet.summary.by_grader.<id>.avg_score
EvaluationMetrics.overall_score	ResultSet.summary.avg_score
EvaluationMetrics.samples_evaluated / samples_processed / processing_errors	ResultSet.summary.total / derived passed/failed

A sketch of the reporter, following the exact shape of FileReporter/CloudReporter:

# deepfabric/evaluation/reporters/openeval_reporter.py
from __future__ import annotations
from datetime import datetime, timezone
from pathlib import Path
from typing import TYPE_CHECKING, Any

from .base import BaseReporter

if TYPE_CHECKING:
    from ..evaluator import EvaluationResult
    from ..metrics import SampleEvaluation

_GRADER_SPECS = (
    ("gr_tool_selection", "tool_selection_correct", "tool_selection_accuracy"),
    ("gr_parameter_accuracy", "parameters_correct", "parameter_accuracy"),
    ("gr_execution_valid", "execution_valid", "execution_success_rate"),
)


def _sample_to_result(s: "SampleEvaluation") -> dict[str, Any]:
    grader_results = [
        {
            "grader_id": grader_id,
            "type": "custom",
            "score": 1.0 if getattr(s, field) else 0.0,
            "passed": bool(getattr(s, field)),
        }
        for grader_id, field, _ in _GRADER_SPECS
    ]
    result: dict[str, Any] = {
        "test_case_id": str(s.sample_id),
        "actual_output": s.predicted_answer
        or (s.predicted_tool and f"{s.predicted_tool}({s.predicted_parameters})")
        or "",
        "grader_results": grader_results,
        "passed": s.execution_valid,
        "metadata": {
            "deepfabric": {
                "query": s.query,
                "available_tools": s.available_tools,
                "predicted_tool": s.predicted_tool,
                "predicted_parameters": s.predicted_parameters,
                # ExpectedToolCall.parameters has no first-class EvalPort field
                # (TestCase.expected_tools is name-only) -- preserved here.
                "expected_tool": s.expected_tool,
                "expected_parameters": s.expected_parameters,
            }
        },
    }
    if s.error:
        result["error"] = {"message": s.error, "type": "runner_error"}
    return result


class OpenEvalReporter(BaseReporter):
    """Writes evaluation results as an EvalPort ResultSet JSON document."""

    def __init__(self, config: dict | None = None):
        super().__init__(config)
        self.output_path = (config or {}).get("path")

    def report(self, result: "EvaluationResult") -> None:
        m = result.metrics
        by_grader = {
            grader_id: {"avg_score": round(getattr(m, metric_attr), 4)}
            for grader_id, _, metric_attr in _GRADER_SPECS
        }
        resultset = {
            "version": "1.0.0",
            "suite_id": Path(result.config.dataset_path or "deepfabric_eval").stem,
            "run_id": f"run_{datetime.now(timezone.utc):%Y%m%dT%H%M%SZ}",
            "started_at": datetime.now(timezone.utc).isoformat(),
            "runner": {"name": "deepfabric", "version": None},
            "results": [_sample_to_result(s) for s in result.predictions],
            "summary": {
                "total": m.samples_evaluated,
                "passed": sum(1 for s in result.predictions if s.execution_valid),
                "failed": m.samples_evaluated
                - sum(1 for s in result.predictions if s.execution_valid),
                "skipped": 0,
                "pass_rate": (
                    sum(1 for s in result.predictions if s.execution_valid)
                    / m.samples_evaluated
                    if m.samples_evaluated
                    else 0
                ),
                "avg_score": round(m.overall_score, 4),
                "by_grader": by_grader,
            },
            "metadata": {"openeval": {"source": "deepfabric"}},
        }
        if self.output_path:
            path = Path(self.output_path)
            path.parent.mkdir(parents=True, exist_ok=True)
            path.write_text(__import__("json").dumps(resultset, indent=2))

    def get_name(self) -> str:
        return "openeval"

Wiring it into Evaluator._initialize_reporters() would be a one-line addition next to the existing "file" / "cloud" branches (reporters=["openeval"] or reporters={"openeval": {"path": "..."}} in EvaluatorConfig) — no change needed to evaluator.py's evaluation loop itself, since report()/report_sample() is exactly the seam BaseReporter already defines.

Honest gaps / open questions

1. ExpectedToolCall.parameters has no first-class EvalPort home. EvalPort's TestCase.expected_tools is an array of tool names only (agent-eval field, no per-tool parameter schema) — there's no TestCase.expected_parameters in the spec at all. So round-tripping DeepFabric's rich per-tool ground truth (GroundTruth.expected_tools: list[ExpectedToolCall], each with tool_name + parameters) into a spec-clean TestCase isn't fully lossless; it has to live under metadata, same as the opik-adapter's approach to Opik's schema-less dataset items.
2. This sketch is results-out only. It converts EvaluationResult → ResultSet. A full round-trip would also need a Conversation → EvalSuite/TestCase exporter on the dataset-generation side (deepfabric/schemas.py), which I haven't audited in depth — happy to look at that as a follow-up if there's interest.
3. Score semantics differ. EvaluationMetrics.overall_score is a fixed-weight composite (40% tool selection / 35% params / 25% execution, per compute_overall_score), while EvalPort's summary.avg_score is conventionally a flat mean over grader scores. The sketch above reports overall_score directly rather than recomputing a flat mean, which is defensible but worth being explicit about in any real implementation.
4. Unlike evaluators (which go through EvaluatorRegistry), reporters in _initialize_reporters() are matched by hardcoded string branches rather than a registry — adding "openeval" fits the existing pattern fine, just noting it's not as pluggable as the evaluator side.

Why bother

If this is interesting, I'd be glad to either open a PR with the OpenEvalReporter above (tests included) directly against deepfabric/evaluation/reporters/, or — if you'd rather keep DeepFabric's reporter surface minimal — ship it as a standalone deepfabric-openeval-adapter package the way I did for Opik, AutoGen, and CrewAI: zero changes to DeepFabric core, built only against the public EvaluationResult/SampleEvaluation surface. Either way, no obligation — flagging it because the data model here is already unusually clean for this kind of conversion, and I wanted to check appetite before writing code.

Full EvalPort spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

— Sahi, independent contributor (not affiliated with this project)