Create misfit export tab in Manage Experiments #12566

SAKavli · 2026-01-05T12:02:24Z

Issue
Resolves #12432

PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

When there are user facing changes: Updated documentation
New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
Large PR: Prepare changes in small commits for more convenient review
Bug fix: Add regression test for the bug
Bug fix: Add backport label to latest release (format: 'backport release-branch-name')

codecov-commenter · 2026-01-05T12:51:50Z

Codecov Report

❌ Patch coverage is 81.03448% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.51%. Comparing base (b8a7508) to head (968f480).
⚠️ Report is 57 commits behind head on main.

Files with missing lines	Patch %	Lines
...ui/tools/manage_experiments/storage_info_widget.py	73.80%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12566      +/-   ##
==========================================
- Coverage   90.57%   90.51%   -0.06%     
==========================================
  Files         435      435              
  Lines       29914    30023     +109     
==========================================
+ Hits        27095    27176      +81     
- Misses       2819     2847      +28

Flag	Coverage Δ
cli-tests	`37.35% <0.00%> (+0.02%)`	⬆️
gui-tests	`68.59% <81.03%> (-0.05%)`	⬇️
performance-and-unit-tests	`73.87% <18.96%> (-0.06%)`	⬇️
test	`38.05% <0.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codspeed-hq · 2026-01-06T10:18:28Z

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing SAKavli:reuse-gui-components-to-export-misfit (968f480) with main (ef47dad)}

Summary

✅ 22 untouched benchmarks

src/ert/gui/tools/manage_experiments/export_dialog.py

Copilot

Pull request overview

This PR adds a new "Misfit" tab to the Manage Experiments dialog, allowing users to view and export misfit data alongside existing parameter data. The implementation refactors the existing ExportParametersDialog into a base ExportDialog class with two concrete implementations: ExportParametersDialog and ExportMisfitDialog.

Key changes:

Extracted base ExportDialog class to share common export dialog functionality
Implemented ExportMisfitDialog for exporting misfit data to CSV
Added Misfit tab to the Manage Experiments UI with table view and export button
Refactored tests to work with the new class hierarchy

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

File	Description
src/ert/gui/tools/manage_experiments/export_parameters_dialog.py	Refactored to create base ExportDialog class and added ExportMisfitDialog subclass; enhanced file path validation
src/ert/gui/tools/manage_experiments/storage_info_widget.py	Added Misfit tab with table and export button; refactored frame creation into reusable helper method; implemented misfit data loading
tests/ert/ui_tests/gui/test_export_parameters_dialog.py	Refactored validation tests to use base ExportDialog class; split parametrized test into separate focused tests; added test for misfit export dialog

Comments suppressed due to low confidence (1)

src/ert/gui/tools/manage_experiments/export_parameters_dialog.py:82

The error message mentions "ensemble parameters" specifically, but this base class is now used for both parameters and misfit exports. The error message should be more generic (e.g., "Error exporting data") or the subclass should be able to customize the error message.

            self._export_text_area.insertHtml(
                "<span style='color: red;'>"
                f"Error exporting ensemble parameters: {e!s}"
                "</span><br>"

Copilot · 2026-01-06T12:00:53Z

src/ert/gui/tools/manage_experiments/export_dialog.py

+        df = LibresFacade.load_all_misfit_data(self._ensemble)
+        realization_column = pl.Series(df.index)
+        df = pl.from_pandas(df)
+        df.insert_column(0, realization_column)


This data transformation logic (loading misfit data, extracting index as realization column, converting from pandas to polars, and inserting the column) is duplicated in storage_info_widget.py's get_misfit_df method (lines 446-452). Consider extracting this into a shared helper method or static method on ExportMisfitDialog to eliminate duplication and ensure consistency.

This will be a task for #12579
I could create a public method, but that would cause cirular imports. I could create a common endpoint somewhere, but that would be within the scope of the other issue.

src/ert/gui/tools/manage_experiments/storage_info_widget.py

Copilot · 2026-01-06T12:00:54Z

src/ert/gui/tools/manage_experiments/export_dialog.py

+        df = pl.from_pandas(df)
+        df.insert_column(0, realization_column)
+
+        df.write_csv(output_file)


The misfit export doesn't specify float_precision when calling write_csv, while ExportParametersDialog uses float_precision=6 (line 152). For consistency with the CSV export workflow (csv_export.py line 147) and parameter exports, consider adding float_precision=6 here as well.

Suggested change

df.write_csv(output_file)

df.write_csv(output_file, float_precision=6)

src/ert/gui/tools/manage_experiments/export_dialog.py

tests/ert/ui_tests/gui/test_export_parameters_dialog.py

Copilot · 2026-01-06T12:00:55Z

tests/ert/ui_tests/gui/test_export_parameters_dialog.py

+def test_that_export_misfit_dialogue_exports_misfit_data_from_ensemble(
+    qtbot, snake_oil_storage: Storage
+):
+    ensemble = next(snake_oil_storage.ensembles)
+    dialog = ExportMisfitDialog(ensemble)
+
+    with patch.object(
+        pl.DataFrame, "write_csv", new_callable=MagicMock
+    ) as mock_write_csv:
+        dialog.export()
+
+        assert mock_write_csv.called


This test only verifies that write_csv was called but doesn't validate what data was actually exported. The test should verify that the exported misfit data matches the expected format and content - at minimum, check the DataFrame structure (columns, index) and ideally compare it against LibresFacade.load_all_misfit_data output. Consider following the pattern in test_that_export_writes_to_file_and_check_the_file_content which validates the actual exported file content.

I had trouble checking what the mock was called upon, I only got to extract the filename. I will have another go at this one.

I have remade the solution, including a commit for passing DataFrames to the Export Classes instead of Ensembles, making testing a lot easier.

Copilot · 2026-01-06T12:00:55Z

src/ert/gui/tools/manage_experiments/export_dialog.py

+        if path.is_file():
+            _set_invalid(tooltip_text=f"'{path!s}' is an existing file.")
            return


This validation prevents exporting to an existing file, which may be overly restrictive. Users typically expect to be able to overwrite files when exporting data. Consider either removing this check or prompting the user for confirmation instead of completely blocking the export. If this behavior is intentional, it should be tested to ensure it works as expected.

I have removed this now.

Copilot · 2026-01-06T12:00:56Z

src/ert/gui/tools/manage_experiments/export_dialog.py

+        output_file: str = self._file_path_edit.text().strip()
+
+        df = LibresFacade.load_all_misfit_data(self._ensemble)
+        realization_column = pl.Series(df.index)


When creating the realization_column Series from df.index, the Series doesn't preserve the index name "Realization" set in LibresFacade.load_all_misfit_data. Consider using pl.Series(name="Realization", values=df.index) or preserving the original index name with pl.Series(name=df.index.name, values=df.index) to maintain consistency with the source data structure.

Suggested change

realization_column = pl.Series(df.index)

realization_column = pl.Series(name=df.index.name, values=df.index)

Not True I believe

SAKavli · 2026-01-06T12:06:58Z

Seems like there is a bunch to improve here. Will get started with it =)

SAKavli · 2026-01-06T15:05:36Z

src/ert/gui/tools/manage_experiments/export_parameters_dialog.py

        try:
+            output_file: str = self._file_path_edit.text().strip()
            self._export_button.setEnabled(False)
            parameters_df = self._ensemble.load_scalar_keys(transformed=True)


This bit will be generalized in a later commit

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Copilot · 2026-01-07T12:59:33Z

src/ert/gui/tools/manage_experiments/export_dialog.py

+    def export_action(self) -> None:
+        raise NotImplementedError("Subclasses must implement this method.")
+


The export_action method is defined but never called anywhere in the codebase. Since ExportDialog directly performs exports via the export method and is instantiated directly (not subclassed), this method appears to be unused dead code and should be removed.

Suggested change

def export_action(self) -> None:

raise NotImplementedError("Subclasses must implement this method.")

Good catch, fixed!

src/ert/gui/tools/manage_experiments/storage_info_widget.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (1)

src/ert/gui/tools/manage_experiments/export_dialog.py:117

The validation logic treats "." as an invalid path (equivalent to empty string), but this is inconsistent. In Unix/Linux systems, "." represents the current directory and Path(".") would return True for is_dir(), so it should be caught by the directory check on line 119 instead. The special case for "." on line 115 is redundant and could be removed for consistency.

Copilot · 2026-01-07T13:22:06Z

tests/ert/ui_tests/gui/test_export_dialog.py

+        "valid-export",
+    ],
+)
+def test_that_file_path_validation_succeeds_on_valid_paths(


The test name doesn't follow the specification style required by the coding guidelines. Test names should avoid vague terms like "succeeds". Instead, describe the specific behavior being verified.

Suggested rename: test_that_valid_file_paths_enable_export_button_and_clear_error_state

src/ert/gui/tools/manage_experiments/storage_info_widget.py

Copilot · 2026-01-07T13:22:07Z

tests/ert/ui_tests/gui/test_export_dialog.py

+from ert.gui.tools.manage_experiments.export_dialog import ExportDialog
+
+
+def test_that_export_writes_to_file_and_check_the_file_content(qtbot, change_to_tmpdir):


The test name doesn't follow the specification style required by the coding guidelines. Test names should start with test_that_ and clearly state the expected behavior without vague terms. The current name includes "and check" which is redundant and doesn't describe what correctness criterion is being tested.

Suggested rename: test_that_exported_csv_contains_the_provided_dataframe_values

Copilot · 2026-01-07T13:22:07Z

tests/ert/ui_tests/gui/test_export_dialog.py

+    assert_invalidation_in_dialog(dialog, expected_error="No filename provided")
+
+
+def test_that_file_path_validation_fails_on_non_existing_path(qtbot):


The test name doesn't follow the specification style required by the coding guidelines. Test names should describe the expected behavior or invariant, not just state what fails. The phrase "validation fails" is a vague judgment rather than describing the concrete behavior being tested.

Suggested rename: test_that_non_existent_directory_path_shows_invalid_file_path_error

Copilot · 2026-01-07T13:22:07Z

tests/ert/ui_tests/gui/test_export_dialog.py

+
+
+@patch("polars.DataFrame.write_csv")
+def test_that_export_failure_is_handled_correctly(patched_write_csv, qtbot):


The test name doesn't follow the specification style required by the coding guidelines. Test names should avoid vague terms like "correctly" and "handled". Instead, describe the specific behavior being verified.

Suggested rename: test_that_export_shows_error_message_when_csv_write_raises_exception

src/ert/gui/tools/manage_experiments/storage_info_widget.py

Copilot · 2026-01-07T13:22:08Z

tests/ert/ui_tests/gui/test_export_dialog.py

+    assert_invalidation_in_dialog(dialog, expected_error="Invalid file path")
+
+
+def test_that_file_path_validation_fails_on_existing_directories(qtbot):


The test name doesn't follow the specification style required by the coding guidelines. Test names should describe the expected behavior or invariant, not just state what fails. The phrase "validation fails" is a vague judgment rather than describing the concrete behavior being tested.

Suggested rename: test_that_existing_directory_path_shows_directory_error_message

- Split up cases to easier know which states are valid and not - Improve test names to follow our CONTRIBUTING guidelines

tests/ert/ui_tests/gui/test_export_parameters_dialog.py

frode-aarstad · 2026-01-08T12:36:20Z

tests/ert/ui_tests/gui/test_export_dialog.py

+
+def test_that_file_path_is_invalidated_given_empty_path(qtbot):
+    parameter_df = DataFrame({"a": 1, "b": 2})
+    dialog = ExportDialog(parameter_df)


Might be better for readability if you create an empty dataframe when the data in it is not needed

Agreed, I can change that!

frode-aarstad

Very good 🚀

By not needing the storage fixture anymore, the tests are reduced from taking 1 sec to 0.15 sec.

Extract creation of export table tab to class method

SAKavli added the release-notes:new-feature Automatically categorise as new feature in release notes label Jan 5, 2026

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch 2 times, most recently from 9848742 to f95a859 Compare January 5, 2026 12:39

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch 7 times, most recently from b7e0d2a to 9380f18 Compare January 6, 2026 09:40

SAKavli changed the title ~~Create misfit tab in Manage Experiments~~ Create misfit export tab in Manage Experiments Jan 6, 2026

frode-aarstad requested a review from Copilot January 6, 2026 11:56

Copilot started reviewing on behalf of frode-aarstad January 6, 2026 11:56 View session

frode-aarstad reviewed Jan 6, 2026

View reviewed changes

src/ert/gui/tools/manage_experiments/export_dialog.py Show resolved Hide resolved

Copilot AI reviewed Jan 6, 2026

View reviewed changes

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch 6 times, most recently from 2422596 to 4875143 Compare January 6, 2026 15:04

SAKavli commented Jan 6, 2026

View reviewed changes

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch 4 times, most recently from 5da75e9 to e5c78de Compare January 7, 2026 08:04

SAKavli requested a review from Copilot January 7, 2026 12:58

Copilot AI reviewed Jan 7, 2026

View reviewed changes

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch from e5c78de to 7fe4b58 Compare January 7, 2026 13:18

SAKavli requested a review from Copilot January 7, 2026 13:18

Copilot started reviewing on behalf of SAKavli January 7, 2026 13:18 View session

Copilot AI reviewed Jan 7, 2026

View reviewed changes

SAKavli added 4 commits January 7, 2026 14:27

Split up export parameters dialog tests

2d7c7ae

- Split up cases to easier know which states are valid and not - Improve test names to follow our CONTRIBUTING guidelines

Improve tooltip for invalid paths

798f699

Generalize export dialog

588a3ba

Rename to export_dialog

802a604

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch 2 times, most recently from b7eb95a to c83a521 Compare January 7, 2026 13:39

frode-aarstad reviewed Jan 8, 2026

View reviewed changes

tests/ert/ui_tests/gui/test_export_parameters_dialog.py Show resolved Hide resolved

frode-aarstad reviewed Jan 8, 2026

View reviewed changes

frode-aarstad approved these changes Jan 8, 2026

View reviewed changes

SAKavli added 2 commits January 8, 2026 13:49

Change ExportClass input to DataFrames

6f600f7

By not needing the storage fixture anymore, the tests are reduced from taking 1 sec to 0.15 sec.

Add export misfit tab to manage experiments

968f480

Extract creation of export table tab to class method

SAKavli force-pushed the reuse-gui-components-to-export-misfit branch from c73aa41 to 968f480 Compare January 8, 2026 12:50

SAKavli merged commit c7dae2d into equinor:main Jan 8, 2026
35 checks passed

	df.write_csv(output_file)
	df.write_csv(output_file, float_precision=6)

	realization_column = pl.Series(df.index)
	realization_column = pl.Series(name=df.index.name, values=df.index)

		def export_action(self) -> None:
		raise NotImplementedError("Subclasses must implement this method.")

		from ert.gui.tools.manage_experiments.export_dialog import ExportDialog


		def test_that_export_writes_to_file_and_check_the_file_content(qtbot, change_to_tmpdir):

		assert_invalidation_in_dialog(dialog, expected_error="No filename provided")


		def test_that_file_path_validation_fails_on_non_existing_path(qtbot):



		@patch("polars.DataFrame.write_csv")
		def test_that_export_failure_is_handled_correctly(patched_write_csv, qtbot):

Create misfit export tab in Manage Experiments #12566

Create misfit export tab in Manage Experiments #12566

Conversation

SAKavli commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

When applicable

Uh oh!

codecov-commenter commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq bot commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Summary

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SAKavli commented Jan 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

SAKavli commented Jan 5, 2026 •

edited

Loading

codecov-commenter commented Jan 5, 2026 •

edited

Loading

codspeed-hq bot commented Jan 6, 2026 •

edited

Loading