feat: Add pandas Series support to Parameter.magnitude #71

euronion · 2025-10-23T20:40:27Z

Careful - this PR was fully prepared using GH Copilot, including the very verbose description. I didn't review it yet.

Low priority

This PR adds support for pandas Series and lists as magnitude values in the Parameter class, enabling users to work with time series data directly while maintaining full backward compatibility.

Key Changes

Updated Parameter.magnitude to accept int | float | list[float] | pd.Series
Added element-wise arithmetic operations for series data
Enhanced conversion methods for series support
Preserved pandas Series indices through operations
Added 16 comprehensive tests covering series functionality
Updated documentation with series examples

Testing

All 64 tests pass (48 existing + 16 new), ensuring no regression while adding new functionality.

Usage Example

import pandas as pd
costs = pd.Series([1000, 800, 600], index=[2020, 2025, 2030])
param = Parameter(magnitude=costs, units="USD_2020/kW")
doubled = param * 2  # Element-wise operation

What's Included in the PR:

Modified Files:
- src/technologydata/parameter.py - Core implementation
- docs/user_guide/parameter.md - Updated documentation
New Files:
- test/test_parameter_series.py - Comprehensive test suite
Commit: 2d8e469 with detailed commit message

The branch is ready and all tests pass. Just create the PR manually through the GitHub interface! 🚀### What's Included in the PR:

Modified Files:
- src/technologydata/parameter.py - Core implementation
- docs/user_guide/parameter.md - Updated documentation
New Files:
- test/test_parameter_series.py - Comprehensive test suite
Commit: 2d8e469 with detailed commit message

The branch is ready and all tests pass. Just create the PR manually through the GitHub interface! 🚀

- Update Parameter.magnitude type to accept int | float | list[float] | pd.Series - Add helper methods for series/scalar detection and conversion - Update all arithmetic operations (__add__, __sub__, __mul__, __truediv__, __pow__) for element-wise series operations - Update conversion methods (to, change_currency, change_heating_value) to handle series data - Preserve pandas Series indices through all operations - Add comprehensive test suite (16 new tests) covering series functionality - Update documentation with series usage examples and limitations - Maintain full backward compatibility with existing scalar usage - Add proper Pydantic configuration for pandas Series validation All existing tests (48) continue to pass, ensuring no regression. Total test coverage: 64 tests with 88% coverage of parameter.py.

for more information, see https://pre-commit.ci

…on/technology-data into feature/parameter-series-support

euronion

Some comments @finozzifa to think about.

src/technologydata/parameter.py

euronion · 2025-10-30T14:57:13Z

src/technologydata/parameter.py

+
    magnitude: Annotated[
-        int | float, Field(description="The numerical value of the parameter.")
+        int | float | pd.Series,


Can we get pydantic to validate that the pd.Series is a series with only numerical values?
We don't want something like this:

pd.Series( ["a", "b", "c"], index=[0, 1, 2] )

Also we want to allow any numericals, not just float, but also e.g. int, and dare I say also complex?

(Maybe not directly, but you can add custom validators for pydantic that get triggered whenever a value is assigned. Maybe we need to use that. But checking inside the __init__ is not great.)

uuuh this is a tough (but interesting) request I fear. I did try to implement but could not make it work. Hence I inclined to the validation in the constructor. Let us chat about it :)

No worries, there is a package for that: pandera -> Making pandas and pydantic happy together.
Details for DataFrame here, looks like it works similarly for Series (but the docs for Series are not as nice):

https://pandera.readthedocs.io/en/stable/pydantic_integration.html

euronion · 2025-10-30T15:10:44Z

src/technologydata/parameter.py

+            if self._is_magnitude_series():
+                # For series data, create a quantity with array magnitude
+                magnitude_array = self._get_magnitude_as_series().to_numpy()
+                self._pint_quantity = technologydata.ureg.Quantity(
+                    magnitude_array, self.units
+                )
+            else:
+                self._pint_quantity = technologydata.ureg.Quantity(
+                    self.magnitude, self.units
+                )


Internally we could always use an array, even if it is just a scalar? That'd allow you to get rid of a couple of if statements

src/technologydata/parameter.py

euronion · 2025-10-30T15:21:32Z

src/technologydata/parameter.py

+        # Handle series vs scalar magnitude
+        if self._is_magnitude_series():
+            magnitude_series = self._get_magnitude_as_series()
+            new_magnitude = magnitude_series * multiplier
+        else:
+            new_magnitude = self.magnitude * multiplier


If pd.Series is always a numerical, shouldn't mypy then not accept the multiplication and the if-else block be obsolete?

euronion · 2025-10-30T15:24:46Z

src/technologydata/parameter.py

-            magnitude=new_quantity.magnitude,
-            units=new_quantity.units,
+            magnitude=new_magnitude,
+            units=str(new_quantity.units),


why is this necessary?

you mean the new_magnitude? If so, this was therefore before I started to modify the code

If you also think it is not necessary feel free to remove it. I had not checked the code beforehand ;)

euronion · 2025-10-30T15:29:01Z

src/technologydata/parameter.py

        This method checks for parameter compatibility before performing the addition.
        The resulting Parameter retains the carrier, heating value, and combines provenance,
        notes, and sources from both operands.
+        When adding series parameters, the operation is performed element-wise.


Do we need this special behaviour? Can't we just do self_series + other_series and let pandas handle it? (scalar + Series = Series with scalar applied to each element; Series + Series = element-wise addition on aligned indices or NaN on indices that are only present in one Series).

Then we would have default pandas behaviour + code should be simpler?

Same for the other operations :)

Co-authored-by: Johannes HAMPP <[email protected]>

…/open-energy-transition/technology-data into feature/parameter-series-support

euronion · 2025-11-04T10:36:10Z

On hold for now, because more complicated than anticipated.

euronion requested a review from finozzifa October 23, 2025 20:40

[pre-commit.ci] auto fixes from pre-commit.com hooks

d2acc93

for more information, see https://pre-commit.ci

euronion added the priority::low label Oct 23, 2025

finozzifa added 7 commits October 28, 2025 15:33

code: start to address mypy comments

0b9ce29

Merge branch 'prototype-2' of https://github.com/open-energy-transiti…

7ab56c2

…on/technology-data into feature/parameter-series-support

code: update code to account for mypy suggestions

1901f00

code: introduce safe_multiply

d859c0d

code: fix mypy comments

a2877ac

code: remove safe methods

dd653b9

add new unit test

a1b6d16

euronion commented Oct 30, 2025

View reviewed changes

finozzifa and others added 5 commits October 31, 2025 18:43

Update src/technologydata/parameter.py

373d7cd

Co-authored-by: Johannes HAMPP <[email protected]>

merge from prototype-2

eb67def

Merge branch 'feature/parameter-series-support' of https://github.com…

477f891

…/open-energy-transition/technology-data into feature/parameter-series-support

doc: remove all examples

0616298

code: update test_parameter_series.py

4965a37

feat: Add pandas Series support to Parameter.magnitude #71

Are you sure you want to change the base?

feat: Add pandas Series support to Parameter.magnitude #71

Uh oh!

Conversation

euronion commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Changes

Testing

Usage Example

What's Included in the PR:

Uh oh!

euronion left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

finozzifa Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

euronion commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

euronion commented Oct 23, 2025 •

edited

Loading

finozzifa Nov 3, 2025 •

edited

Loading