Remove LocalEnsemble.load_all_gen_kw_data() by jonathan-eq · Pull Request #12015 · equinor/ert

jonathan-eq · 2025-10-07T13:23:34Z

Issue
Resolves #11939

Approach
This commit removes the function, and replaces it with LocalEnsemble.load_scalars() as it gradually moves from pandas towards polars.

(Screenshot of new behavior in GUI if applicable)

PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

When there are user facing changes: Updated documentation
New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
Large PR: Prepare changes in small commits for more convenient review
Bug fix: Add regression test for the bug
Bug fix: Add backport label to latest release (format: 'backport release-branch-name')

jonathan-eq · 2025-10-07T13:24:29Z

+            ensemble_data = ensemble.load_scalars().to_pandas().set_index("realization")
+            ensemble_data.columns.name = None
+            ensemble_data.index.name = "Realization"
+            ensemble_data = ensemble_data.sort_index(axis=1)


This is more or less the same as what was in load_all_gen_kw_data(), but it will be removed as we rewrite csv_export to use polars instead of pandas. We should do this gradually and in increments.

@xjules
I will create an issue for this rewrite pandas -> polars, but it won't be done in this PR.

codecov-commenter · 2025-10-07T13:33:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.72%. Comparing base (1e382c8) to head (56a1a39).
⚠️ Report is 962 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12015      +/-   ##
==========================================
- Coverage   90.73%   90.72%   -0.01%     
==========================================
  Files         432      432              
  Lines       28911    28902       -9     
==========================================
- Hits        26231    26222       -9     
  Misses       2680     2680

Flag	Coverage Δ
cli-tests	`38.85% <0.00%> (-0.08%)`	⬇️
gui-tests	`69.07% <100.00%> (-0.01%)`	⬇️
performance-and-unit-tests	`74.02% <100.00%> (-0.01%)`	⬇️
test	`40.04% <0.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codspeed-hq · 2025-10-07T14:16:05Z

CodSpeed Performance Report

Merging #12015 will not alter performance

_{Comparing jonathan-eq:improve-plotter (56a1a39) with main (1e382c8)}

Summary

✅ 22 untouched

xjules · 2025-10-08T06:50:50Z

        for iter_nr in range(2):
            ensemble = experiment.get_ensemble_by_name(f"iter-{iter_nr}")
-            data.append(ensemble.load_all_gen_kw_data())
+            ensemble_data = ensemble.load_scalars().to_pandas().set_index("realization")


do we need to have pandas here? Ie. maybe just update snapshot?

xjules · 2025-10-08T06:52:44Z

            assert (
-                np.linalg.det(posterior.cov().to_numpy())
-                <= np.linalg.det(prior.cov().to_numpy()) + 0.001
+                np.linalg.det(posterior.to_pandas().cov().to_numpy())


you do have pl.cov https://docs.pola.rs/api/python/dev/reference/expressions/api/polars.cov.html

Yes, but not for all columns.

~~posterior.cov().to_numpy()~~
cov_matrix = np.cov(posterior.to_numpy(), rowvar=False)

xjules · 2025-10-08T06:55:32Z


        for i in range(ensemble.ensemble_size):
            assert (
                f",{name},{gen_kw_data.iloc[i]['COEFFS:a']:.6f},{gen_kw_data.iloc[i]['COEFFS:b']:.6f},{gen_kw_data.iloc[i]['COEFFS:c']:.6f},{misfit_data.iloc[i]['MISFIT:POLY_OBS']:.6f},{misfit_data.iloc[i]['MISFIT:TOTAL']:.6f}"


you can do directly row in pl.Dataframe

row_gen = gen_kw_data.row(i, named=True) {row_gen['COEFFS:a']:.6f}

xjules · 2025-10-08T06:59:41Z

    # generalized variance for the parameters.
    assert (
        0
        < np.linalg.det(df_posterior.cov().to_numpy())


the same as above pl.cov ....

xjules · 2025-10-08T07:00:06Z

+        data = ensemble.load_scalars().to_pandas().set_index("realization")
+        data.columns.name = None
+        data.index.name = "Realization"
+        data = data.sort_index(axis=1)


Can we update snapshot instead?

That will change behavior. I don't think we want to do that in this PR.

Not sure that I understand. What do you mean by behaviour?

If we have to update the snapshot, the behavior/format has been altered. I don't think that should be done in this refactor-pr

This commit removes the function, and replaces it with `LocalEnsemble.load_scalars()` as it gradually moves from pandas towards polars.

xjules

Nice job @jonathan-eq ! Can you create an issue to remove pandas from the tests further on?

jonathan-eq · 2025-10-10T05:45:33Z

#12043

jonathan-eq · 2025-10-10T05:46:48Z

Blocked by equinor/semeio#772

jonathan-eq commented Oct 7, 2025

View reviewed changes

jonathan-eq force-pushed the improve-plotter branch 2 times, most recently from 59fb512 to 0acac2f Compare October 7, 2025 13:48

jonathan-eq self-assigned this Oct 7, 2025

jonathan-eq added the release-notes:maintenance Automatically categorise as maintenance change in release notes label Oct 7, 2025

jonathan-eq force-pushed the improve-plotter branch from 0acac2f to cc0f87a Compare October 8, 2025 05:54

xjules reviewed Oct 8, 2025

View reviewed changes

jonathan-eq force-pushed the improve-plotter branch from cc0f87a to 9157fc6 Compare October 8, 2025 07:47

Remove LocalEnsemble.load_all_gen_kw_data()

56a1a39

This commit removes the function, and replaces it with `LocalEnsemble.load_scalars()` as it gradually moves from pandas towards polars.

jonathan-eq force-pushed the improve-plotter branch from 9157fc6 to 56a1a39 Compare October 9, 2025 05:32

xjules approved these changes Oct 9, 2025

View reviewed changes

jonathan-eq mentioned this pull request Oct 10, 2025

Remove polars_dataframe.to_pandas() from tests #12043

Closed

4 tasks

jonathan-eq added the blocked label Oct 10, 2025

jonathan-eq removed the blocked label Oct 10, 2025

jonathan-eq merged commit 3a7b4e4 into equinor:main Oct 10, 2025
34 of 35 checks passed

jonathan-eq deleted the improve-plotter branch February 25, 2026 07:40

Uh oh!

Conversation

jonathan-eq commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

When applicable

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #12015 will not alter performance

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xjules Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xjules Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xjules left a comment

Choose a reason for hiding this comment

Uh oh!

jonathan-eq commented Oct 10, 2025

Uh oh!

jonathan-eq commented Oct 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jonathan-eq commented Oct 7, 2025 •

edited

Loading

codecov-commenter commented Oct 7, 2025 •

edited

Loading

codspeed-hq Bot commented Oct 7, 2025 •

edited

Loading

xjules Oct 8, 2025 •

edited

Loading

xjules Oct 8, 2025 •

edited

Loading