Need a way to define new columns through functions from python

[acampove]

Feature description

Hi,

We need something like:

variable_computer = VariableComputer()
rdf = rdf.Define('var', variable_computer )

where variable_computer is an instance of a python class that operates on each entry of the dataframe and returns a float. That float is var. E.g.

class VariableComputer:
    def __call__(self, entry):
        return entry.x + entry.y

i imagine that should be doable. In pandas I can do:

import pandas as pd

# Create a sample DataFrame
df = pd.DataFrame({
    "x": [i * 0.1 for i in range(10)],  # x = 0.0, 0.1, ..., 0.9
    "y": [i * 0.2 for i in range(10)]   # y = 0.0, 0.2, ..., 1.8
})

# Define a function with complex logic
def custom_logic(row):
    return row["x"]**2 + row["y"]**2 > 1.0  # Example: Outside the unit circle

# Apply the function row-wise
df["is_outside_circle"] = df.apply(custom_logic, axis=1)

# Display the DataFrame
print(df)

the logic can be a function too, as in pandas. In general, could you try to follow pandas design and check what they do? Currently the only reasons why I use ROOT dataframes are:

• I can do multithreading. Which libraries like this let me do in pandas anyway.
• I can read ROOT files directly. Which I might be able to get around with by using uproot.

Alternatives considered

No response

Additional context

No response

[vepadulano]

Dear @acampove ,

Thank you for reaching out and for your useful feedback!

As you just discovered yourself, RDataFrame does not yet support the ability to use a generic Python callable as the input operation to a node of a computation graph. There are of course ways to use Python callables, but with some requirements, be it that they either are Python proxies to C++ callables, or that they are Python callables that can be compiled to native code via https://numba.pydata.org/. If you are interested in these applications, there is documentation and examples so let me know if you need more guidance.

If you want to be able to use a generic Python callable, including and most importantly classes and functions that access generic Python resources, that is currently not possible. It might be possible in the future, and getting more knowledge from expert users such as yourself as to where and why you would be using these generic Python callables would actually help us in better developing this feature, once we get to a place where it can be done.

Just to give you a hint at an obstacle towards this goal, the implicit multithreading capabilities of RDataFrame that you are praising in your comment are simply not achievable in Python today. Parallel libraries, including this one you pointed at, do not parallelise via threads but via processes. This has an inherent overhead (also described in the documentation of that library, but that is just an example and everything in Python works the same way). So overcoming this limitation of the Python language is unfortunately something out of scope for the ROOT project. That is why we offer the performance of native threads via the C++ processing. But, as you describe, it comes at the cost of not offering 100% of the Python features, due to this important Python limitation.

[acampove]

Dear @vepadulano

Thanks for your reply. The numba approach seems to work in simple cases, but If I do:

import ROOT

def fun(x, y):
    return x*y
 
@ROOT.Numba.Declare(['float', 'int'], 'float')
def pypow(x, y):
    val = fun(x,y) 
    return val
 
rdf = ROOT.RDataFrame(4)
rdf = rdf.Define('x', '(float)rdfentry_')
rdf = rdf.Define('x_pow3', 'Numba::pypow(x, 3)')
data= rdf.AsNumpy()

ROOT tells me that it Failed to jit. Can you reproduce this? In reality, what we have is not a product, we have a function that does a lot of work behind the scenes, but that work does not need to be known by ROOT. ROOT only needs to know that there is a function that takes two floats and returns a float.

[acampove]

Parallel libraries, including this one you pointed at, do not parallelise via threads but via processes.

If it's parallel it's parallel, we just want to run it faster. I do not really know what the difference is and as long as the parallelism makes the process faster by using all our cores, that's good enough for us.

This has an inherent overhead (also described in the documentation of that library, but that is just an example and everything in Python works the same way).

I have 16 cores in my machine, if the overhead makes things 20% slower, I think it's still worth it. I might even buy a machine with 32 cores if I need to. The biggest overhead for me and most of us is having to write the code itself.

FYI, I am trying to use something like:

# ---------------------------------------------------
@Numba.Declare(['float', 'float', 'int'], 'float')
def x_from_rowcol(row : float, col : float, area : int) -> float:
    '''
    Will return the X coordinate for a row column and area (region in ECAL) combination
    '''
    x, _ = ctran.from_id_to_xy(row=row, col=col, area=area)

    return x
# ---------------------------------------------------
@Numba.Declare(['float', 'float', 'int'], 'float')
def y_from_rowcol(row : float, col : float, area : int) -> float:
    '''
    Will return the Y coordinate for a row column and area (region in ECAL) combination
    '''
    _, y = ctran.from_id_to_xy(row=row, col=col, area=area)

    return y

but that does not work. from_id_to_xy is implemented, tested and validated. I just want to plug it in my code. I do not really mind an overhead of 10% 20%, even 50%. But If I have to figure out how to rewrite the whole thing to make it work from c++, that's going to take days, that matters to the user.