Skip to content

Feat/extended evaluation#22

Merged
dawidlinek merged 5 commits into
mainfrom
feat/extended-evaluation
Apr 9, 2026
Merged

Feat/extended evaluation#22
dawidlinek merged 5 commits into
mainfrom
feat/extended-evaluation

Conversation

@dawidlinek

Copy link
Copy Markdown
Owner

No description provided.

@dawidlinek dawidlinek self-assigned this Apr 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds extended evaluation and export capabilities to epftoolbox2 by introducing additional built-in evaluators (including a cross-model metric) and a new wide-format CSV exporter, with accompanying report/plumbing and documentation updates.

Changes:

  • Extend the evaluator interface to support compute(df, **kwargs) and pass cross-model context (model_dfs) through EvaluationReport.
  • Add RMSEEvaluator and rMAEEvaluator (relative MAE vs. a benchmark model) and register them for YAML pipeline serialization.
  • Add CsvExporter (wide-format detailed results) and update docs/reference materials accordingly.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
llms.txt Updates evaluator/exporter reference docs and examples (new kwargs interface, new evaluators/exporter).
epftoolbox2/results/report.py Passes model_dfs into evaluators; refactors grouped aggregation to support cross-model metrics; adds source_data storage.
epftoolbox2/pipelines/model_pipeline.py Registers new components and passes source_data into EvaluationReport.
epftoolbox2/exporters/csv.py New CsvExporter that writes a wide CSV and can join extra columns from source data.
epftoolbox2/exporters/init.py Exports CsvExporter from package.
epftoolbox2/evaluators/rmse.py New RMSE metric implementation.
epftoolbox2/evaluators/rmae.py New relative MAE metric requiring cross-model context.
epftoolbox2/evaluators/mae.py Updates MAE to accept **kwargs per new evaluator interface.
epftoolbox2/evaluators/base.py Updates abstract evaluator interface to compute(df, **kwargs).
epftoolbox2/evaluators/init.py Exports new evaluators from package.
docs/src/content/docs/reference/extending.md Updates custom evaluator examples and documents the new evaluator interface/kwargs.
docs/src/content/docs/reference/api.md Updates API reference to include new evaluators/exporter and new evaluator signature.
docs/src/content/docs/exporters/csv.md New documentation page for CsvExporter.
docs/src/content/docs/evaluators/rmse.md New documentation page for RMSEEvaluator.
docs/src/content/docs/evaluators/rmae.md New documentation page for rMAEEvaluator.
docs/src/content/docs/evaluators/mae.md Adds “See Also” links to new evaluators and extending docs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread epftoolbox2/exporters/csv.py
Comment thread epftoolbox2/exporters/csv.py
Comment thread epftoolbox2/evaluators/rmae.py
Comment thread epftoolbox2/results/report.py
Comment thread epftoolbox2/results/report.py
Comment thread epftoolbox2/results/report.py
Comment thread epftoolbox2/evaluators/rmse.py
Comment thread epftoolbox2/evaluators/rmae.py
Comment thread epftoolbox2/exporters/csv.py
Comment thread epftoolbox2/pipelines/model_pipeline.py
@dawidlinek
dawidlinek merged commit 90f43e4 into main Apr 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants