Add trulens-openeval: to_openeval()/from_openeval() for EvalPort interchange

[adhabnr-ux]

Closes #2680.

This is the minimal to_openeval()/from_openeval() module discussed and approved in that thread — a standalone module in src/, not a separate package, per @sfc-gh-jreini's scope call ("a to_openeval()/from_openeval() module in src/ is the right scope to start").

What it does

Converts between TruLens Run records and EvalPort (Apache 2.0), the open interchange format for portable LLM evaluation test cases, graders, suites, and results.

• to_openeval(records_df, ...) — takes the DataFrame Run.get_records()/get_record_details() returns (record_id, input, output, latency, + one column per feedback score) and produces an EvalPort ResultSet. Each feedback column becomes its own GraderResult ("Context Relevance" → grader_id: "context_relevance", type: "custom"). It correctly excludes TruLens's <name>_calls companion columns (per-call detail, not a score) and clamps scores into EvalPort's required [0, 1] range.
• from_openeval(suite) — returns (input_df, dataset_spec), where dataset_spec maps EvalPort's input/expected_output/id onto TruLens's actual reserved dataset-spec fields (input, ground_truth_output, input_id) — verified against trulens.core.run.validate_dataset_spec, so it's ready to hand straight to RunConfig(dataset_spec=...).

Why it converts DataFrames, not Run/RunConfig objects directly

trulens.core.run.Run needs a live RunDaoBase + TruSession + app instance just to construct — there's no way to unit test against it without standing up a full session and database (confirmed by reading trulens/core/run.py). Run.get_records()/get_record_details() (and Run.start(input_df=...)) already hand back/accept plain DataFrames, and that's the real portable surface, so this module converts at that boundary instead — the same reason opik-openeval-adapter and ragas-openeval-adapter convert at their SDKs' plain-data boundaries.

Testing

17 tests in tests/unit/test_openeval.py, all passing locally against the real trulens-core package (not a mock) and the real evalport-sdk validator (openeval.validate.validate_result_set()/validate_suite(), from PyPI). Covers: basic conversion + real-spec validation, _calls companion-column exclusion, score clamping to [0,1], custom pass thresholds, both latency units, run-id defaulting, explicit timestamps, empty/malformed-input errors, summary pass/fail counts matching actual results, and a full suite → input_df → simulated run → ResultSet round trip validated end-to-end against the real spec. from_openeval's dataset_spec output is additionally asserted against TruLens's own validate_dataset_spec().

Packaging

Mirrors src/hotspots's existing layout exactly: a poetry-managed trulens-openeval package under src/openeval/, importable as trulens.openeval, with tests living centrally under the repo's own tests/unit/ per that same convention.

Lossiness, documented honestly

Every field this module explicitly maps round-trips cleanly. TruLens-specific semantics that don't map onto EvalPort's schema (span attributes, cost, per-call args) are preserved under metadata["trulens"] rather than dropped — a different EvalPort-speaking tool won't know how to interpret that namespaced data, but nothing TruLens-specific is silently lost on a TruLens round trip. Full writeup in src/openeval/README.md's "What round-trips losslessly, and what doesn't" section.

---

One logistics note: this was staged at adhabnr-ux/evalport/pending-contributions/trulens-openeval while I didn't have a fork of this repo set up yet — that staging copy is now redundant and I'll remove it once this PR is up. The code here is byte-identical to what was reviewable there.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[adhabnr-ux]

Fixed the CI failures from the last run, in two commits:

1. Lint & format (pre-commit) — applied ruff/ruff-format's exact reformatting (isort single-import-per-line, exploded a couple of multi-element dict/list literals and one except (X) as e: past this repo's line-length config). No behavioral change; re-ran all 17 tests locally afterward against the real trulens-core/evalport-sdk — still passing.

2. PRBranchProtect default/py310/py311/py313-static — root cause was tests/unit/test_openeval.py living in the root test suite (run by every CI job via make test-unit), while trulens-openeval (and its evalport-sdk dependency) was never wired into [tool.poetry.group.required.dependencies] the way trulens-core/trulens-feedback/every other src/* package with root-level tests already is. So the "basic" env never installed it: ModuleNotFoundError: No module named 'openeval' during collection, which aborted the whole job after only 10 tests.

   I pushed the pyproject.toml fix (adds trulens-openeval = { path = "src/openeval", develop = true } to the required group). poetry.lock still needs a matching poetry lock run to add the two new entries — I did this in a clean local clone of this branch and confirmed:

   • the diff touches only those two new package entries (trulens-openeval, evalport-sdk) — zero version changes to anything else already pinned
   • poetry check --lock passes
   • poetry install --only required && pytest tests/unit/test_openeval.py (the exact env-tests-basic + test-unit path CI runs) installs cleanly and all 17 tests pass, where collection previously failed

   I don't have a way to push the resulting ~1MB poetry.lock through my current tooling in one piece, so I left it out rather than fabricate or truncate it. Running poetry lock (not --regenerate) after pulling the current head of this branch will reproduce that same two-entry diff — or let me know if you'd like me to paste the diff here instead.

One unrelated CI failure to flag so it isn't mistaken for something this PR caused: PRBranchProtect default's basic test run also failed on TestOtelAsyncConcurrency.test_cancelled_finalization_failure_preserves_cancelled_error (an AssertionError about a missing ERROR-level log in trulens.core.otel.instrument) — that's in tests/unit/test_otel_async_concurrency.py, nothing this PR touches, and looks like a pre-existing flake/timing issue in the OTEL test suite.

[adhabnr-ux]

Update: poetry.lock is pushed (d0de212) — the ModuleNotFoundError: No module named 'openeval' from before is resolved.

Checked the fresh CI run (build 16188) directly against the Azure Pipelines logs to confirm:

• Run basic unit Test Suite (the "required" group, where trulens-openeval now lives) — succeeds on both py310-static and py313-static. tests/unit/test_openeval.py collects and all 17 tests pass on both:

  tests/unit/test_openeval.py .................
  

• The two red checks (PRBranchProtect py310-static, py313-static) are still failing, but for two causes unrelated to this PR, both isolated to the optional/snowflake test suites, not the required suite test_openeval.py lives in:
  1. tests/unit/test_otel_async_concurrency.py::TestOtelAsyncConcurrency — the same 2-3 pre-existing flaky tests I flagged in my last comment (test_cancelled_finalization_failure_preserves_cancelled_error and friends).
  2. A segfault (Segmentation fault (core dumped)) during pytest collection of tests/unit/test_feedback.py, via the trulens.apps.langchain.tru_chain import chain, on both the optional and snowflake suites, on both Python versions. That's a native-extension crash (numpy/scipy/sklearn/pyarrow are all in the loaded-extensions dump right before it) triggered while importing LangChain app code — nothing in this PR touches src/apps/langchain or any native dependency, so this looks like a separate pre-existing environment issue, not something introduced here.

Happy to open separate issues for both if that's useful, or if either turns out to not be pre-existing I'll take another look — just didn't want CI staying red to read as "the openeval change is still broken" when the actual required-suite tests it lives in are green on every job that's finished so far.

[adhabnr-ux]

Dug into the 3 failing checks (PR Validation Eval, py310-static, py313-static) via the Azure Pipelines logs — wanted to post the diagnosis rather than leave it unexplained:

tests/unit/test_openeval.py passes cleanly everywhere it runs — 17/17 (all dots, zero failures) in every single job: py310, py311, and py313's "basic" suite all show tests/unit/test_openeval.py ................. with no failures.

The actual failures are two pre-existing issues unrelated to this PR:

1. test_otel_async_concurrency.py::TestOtelAsyncConcurrency::test_cancelled_* — flaky/racy across every Python version in this run (3 of its subtests failed on py310, 2 on py313, 1 on py311; py312/"default" passed). This is an async-cancellation timing test with zero relation to trulens.openeval.
2. A segfault during dependency import in the "optional"/"snowflake" suites on py310 and py313 only (Fatal Python error: Segmentation fault, repeated across make test-unit-optional/test-unit-snowflake), happening during collection of the heavy ML dependency stack (scipy/sklearn/PIL/tornado/google._upb._message) — before test_openeval.py is even reached. py311 and py312/"default" don't hit this.

Both look like pre-existing environment/version flakiness in the broader suite, not anything introduced by src/openeval. Happy to help chase either down further if useful, but wanted to surface the diagnosis rather than let the red X sit unexplained. Let me know if you'd like anything else from my side to get this mergeable.

[adhabnr-ux]

The three failing checks (PR Validation Eval, py313-static, py310-static) all fail on the same two tests: test_cancelled_finalization_failure_preserves_cancelled_error and test_cancelled_instrumented_task_records_function_metadata, both in tests/unit/test_otel_async_concurrency.py.

This PR doesn't touch that file, or anything in trulens.core.otel.instrument — the diff is scoped to src/openeval/ (new module) plus pyproject.toml/poetry.lock (adding the two new deps, verified no other package versions moved in d0de212's commit message). The 17 tests for this PR's own module (tests/unit/test_openeval.py) aren't among the failures. Given async cancellation-timing tests are exactly the kind of thing that's sensitive to CI runner load, and #2621/b50359f shows task-cancellation-finalization in otel/instrument was actively being reworked on main just a few days before this branch's base commit, this looks like it could be flakiness in that test rather than something this PR caused — but I don't have visibility into whether it's been flaky before, so flagging rather than asserting.

Happy to push an empty commit to re-trigger a fresh run if that's useful, or if there's a known re-run mechanism for the Azure Pipelines checks here, let me know and I'll use it. Let me know if you'd like anything else from me on this one.

[joshreini1]

Solid implementation of EvalPort interchange adapter with 17 comprehensive tests validating against real spec validators. Code correctly handles edge cases (score clamping, missing values, _calls companion exclusion), follows project conventions (mirrors trulens-hotspots package layout, proper dependency wiring), and integrates cleanly with TruLens's validate_dataset_spec. Failing CI checks are pre-existing async concurrency flakes in test_otel_async_concurrency.py unrelated to this PR. Ready to merge.

[adhabnr-ux]

Thank you for the review and approval, @joshreini1 — glad the module holds up.

Since your approval says "ready to merge" but mergeable_state is still blocked (required-status-check gate, not a merge conflict — mergeable: true) on the same three checks we already traced to the pre-existing test_otel_async_concurrency.py flakiness and, on two of the three jobs, the unrelated LangChain-import segfault in the optional/snowflake suites — this is genuinely stuck on branch protection now, not on anything left for me to fix in the diff.

Two ways I can think of to get it across the line, whichever's easiest on your end:

1. An admin merge override, if branch protection allows bypassing required checks for a maintainer.
2. I close and reopen the PR to force a fresh set of check runs, in case either flake happens not to reproduce on a clean run (this doesn't touch the commit, so d0de212 stays as-is and there's nothing new to re-review).

Happy to do (2) right now if that's useful, or to wait if you'd rather handle the merge a different way. Let me know and I'll follow whichever path you prefer.

[adhabnr-ux]

No word back after a couple of days, so I went ahead with option (2) from above: closed and reopened to force a fresh check run (same commit, d0de212, nothing to re-review). Will check back on whether the flaky checks clear this time; if they don't, an admin merge override still seems like the right call given the approval already on record.

[adhabnr-ux]

Thanks for the review and approval, @sfc-gh-jreini!

I see one check failing — "PR Validation Eval" — and the PR shows as blocked from merging. Before I dig further: is that an internal validation eval that needs repo secrets/credentials that aren't available to a fork PR (so it'd need a maintainer-side re-run or override), or is it flagging something in the actual diff I should fix? Happy to push a follow-up commit if it's the latter — just want to make sure I'm not missing something real on my end before assuming it's an environment issue.

[adhabnr-ux]

Thanks for merging main in — still seeing 2 failing checks though ("PR Validation Eval" and "PR Validation Eval (PRBranchProtect default)"), same as before. Wanted to flag in case the merge was meant to clear it and didn't, rather than assume it's expected. If this needs a maintainer-side re-run, or credentials/secrets that aren't available to a fork PR, no action needed from me — just let me know either way so I'm not silently blocking on something only you can unblock.

[adhabnr-ux]

Thanks for the approval! Looks like the two failing "PR Validation Eval" checks are unrelated to this PR — both failures are in async/OTel instrumentation tests (test_cancelled_finalization_failure_preserves_cancelled_error, test_cancelled_instrumented_task_restores_context_and_finishes_span, test_cancelled_instrumented_task_records_function_metadata, all about asyncio.CancelledError handling during task cancellation). This PR only touches src/openeval/ (the new to_openeval()/from_openeval() module) and its own tests — nothing in the instrumentation/OTel code path. Happy to rebase onto latest main if that's easier than a rerun, just let me know.

[adhabnr-ux]

Checking back in — this is still sitting blocked purely on PR Validation Eval / PR Validation Eval (PRBranchProtect default), which @joshreini1 already flagged as "pre-existing async concurrency flakes in test_otel_async_concurrency.py unrelated to this PR" when approving. 7/9 checks pass; nothing in this diff touches OTel or async concurrency (it's a standalone src/openeval/ module + tests). No action needed from me here — just flagging in case a manual re-run or an admin merge override is the easiest way to get it unblocked on your end, since I don't have permissions to re-run checks on this repo myself. Happy to rebase onto latest main if that'd help instead.

[joshreini1]

Hey @adhabnr-ux - I'll fix the merge conflict shortly.

One new thought that occurs to me, it would be more discoverable and intuitive for the user if this package was named trulens-connectors-openeval instead of just trulens-openeval. agree?

[adhabnr-ux]

Agreed — trulens-connectors-openeval is the better name. It's more discoverable (groups naturally with anything else under a connectors-* naming pattern, if that becomes a convention here) and it's more accurate too: this module is a connector to an external interchange format, not a TruLens-owned "openeval" concept, so the name shouldn't read like it might be. I'll push a commit renaming the package directory (src/openeval/ → wherever the trulens-connectors-openeval convention puts it), the pyproject.toml package name, and the import path, and re-verify all 17 tests pass under the new name before pushing. Let me know if there's a specific directory/import convention you'd rather I match, otherwise I'll follow whatever src/connectors/ (if that's the established pattern) or the closest existing analog does.

[adhabnr-ux]

Quick update on the rename: I built out trulens-connectors-openeval under src/connectors/openeval/ (mirroring src/connectors/snowflake's exact layout) and re-verified all 17 tests passing under the new import path, installed via poetry install --only required against the real trulens-core and evalport-sdk.

I then hit a tooling limitation getting poetry.lock (1MB+, needs updating for the renamed path dependency) reliably transported into this branch from my environment, and rather than risk pushing a corrupted lock file, I reverted the branch back to the current trulens-openeval / src/openeval/ state you see now -- so nothing here is left broken. I'll get the rename up properly as a follow-up once I've sorted out a reliable way to update the lock file; will comment again once that commit is up. Sorry for the noise in the commit history in the meantime.

[adhabnr-ux]

Rename update: the code side is done and verified, but I'm handing off the last mechanical step rather than risk a bad push.

Done and verified (staged at adhabnr-ux:openeval-adapter-rename-staged, not on this PR's branch yet — see why below):

• src/openeval/ → src/connectors/openeval/ (mirrors src/connectors/snowflake's layout exactly)
• Package name trulens-openeval → trulens-connectors-openeval
• Import path trulens.openeval → trulens.connectors.openeval
• Root pyproject.toml's required-group path dependency, the module README, and tests/unit/test_openeval.py's two import lines all updated to match
• All 17 tests in tests/unit/test_openeval.py re-run and passing under the new name/path, poetry install --only required against the real trulens-core and evalport-sdk (not mocks)
• poetry check --lock passes locally against the regenerated poetry.lock — diff is clean: only the trulens-connectors-openeval package block moves to its new alphabetical position + path, and the top-level content-hash changes. No other package versions move.

What I didn't push to this PR's branch: poetry.lock itself. My tooling can reliably push small text files (verified byte-for-byte against my local build via blob SHA before and after each push above) but can't reliably transport this repo's ~1MB poetry.lock without real risk of silently corrupting an unrelated package's hash entry somewhere in those other ~13,900 lines — and pushing the pyproject.toml change without a matching lock file would immediately break poetry install --only required for the entire required suite (the same ModuleNotFoundError collection failure from earlier in this thread, but this time for every test, not just this module's). Rather than land that, I staged everything else on a side branch and I'm asking for a hand on just this one file.

Two easy ways to finish it, whichever's less friction on your end:

1. Pull openeval-adapter-rename-staged, run poetry lock (should reproduce exactly the diff described above — happy to be proven wrong if your resolver picks something different), then merge that onto openeval-adapter and this PR's branch.
2. Or just apply the pyproject.toml/src/ changes from that branch onto openeval-adapter yourself however's easiest, and run poetry lock as the last step either way.

Sorry for another round of noise on this one — the substance was ready days ago; it's specifically this one large generated file that my current environment can't move safely. Let me know if there's anything else you'd like from me here, otherwise I'll leave this with you.