-
Notifications
You must be signed in to change notification settings - Fork 261
Clarification of the "control flow" page in the "DataOps" User Guide section #2010
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,8 +2,11 @@ | |
|
|
||
| .. _user_guide_data_ops_control_flow: | ||
|
|
||
| Control flow in DataOps: eager and deferred evaluation | ||
| ====================================================== | ||
| Running complex operations on DataOps variables: deferred evaluation | ||
| ==================================================================== | ||
|
|
||
| Why some operations cannot be passed straight to DataOps objects | ||
| ---------------------------------------------------------------- | ||
|
|
||
| DataOps represent computations that have not been executed yet, and will | ||
| only be triggered when we call :meth:`.skb.eval() <DataOp.skb.eval>`, or when we | ||
|
|
@@ -31,27 +34,17 @@ Traceback (most recent call last): | |
| TypeError: This object is a DataOp that will be evaluated later, when your learner runs. So it is not possible to eagerly iterate over it now. | ||
|
|
||
| We get an error because the ``for`` statement tries to iterate immediately | ||
| over the columns. However, ``orders.columns`` is not an actual list of | ||
| columns: it is a skrub DataOp that will produce a list of columns, later, | ||
| over the columns. This is the way any computation on any variable is usually run, | ||
| referred to here as *eager* evaluation. However, ``orders.columns`` is not an actual | ||
| list of columns: it is a skrub DataOp that will produce a list of columns, later, | ||
| when we run the computation. | ||
|
|
||
| This remains true even if we have provided a value for ``orders`` and we can | ||
| see a result for that value: | ||
|
Comment on lines
-38
to
-39
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this still need to be stated somewhere because it can be confusing: what do you mean it will be computed later I see it right here in the repr |
||
|
|
||
| >>> orders.columns | ||
| <GetAttr 'columns'> | ||
| Result: | ||
| ――――――― | ||
| Index(['item', 'price', 'qty'], dtype=...) | ||
|
|
||
| The "result" we see is an *example* result that the computation produces for the | ||
| data we provided. But we want to fit our pipeline and apply it to different | ||
| datasets, for which it will return a new object every time. So even if we see a | ||
| preview of the output on the data we provided, ``orders.columns`` still | ||
| represents a future computation that remains to be evaluated. | ||
| Eager vs. deferred | ||
| ------------------ | ||
|
|
||
| Therefore, we must delay the execution of the ``for`` statement until the computation | ||
| actually runs and ``orders.columns`` has been evaluated. | ||
| To circumvent this issue, we must delay the execution of the ``for`` statement until | ||
| the computation actually runs and ``orders.columns`` has been evaluated, hence the designation of | ||
| *deferred* computation rather than *eager*. | ||
|
|
||
| We can achieve this by defining a function that contains the control flow logic | ||
| we need, and decorating it with :func:`deferred`. This decorator defers the execution | ||
|
|
@@ -85,8 +78,19 @@ so it is possible to use standard Python control flow statements such as | |
| ``if``, ``for``, and it is possible to treat the inputs as if they were | ||
| regular objects (e.g., a Pandas DataFrame or Series). | ||
|
|
||
| When the first argument to our function is a skrub DataOp, rather than | ||
| applying ``deferred`` and calling the function as shown above we can use | ||
|
|
||
| .. warning:: | ||
| DataOps are evaluated *lazily* (we are building a pipeline, not immediately | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am not sure I understand what this section is saying |
||
| computing a single result), similarly to the transformers in a scikit-learn | ||
| pipeline where each transformer computes a new result without modifying its input. | ||
| As a result, any transformation that we apply *must not modify its | ||
| input in-place*, but leave it unchanged and return a new value. | ||
|
|
||
| Alternate notations | ||
| ------------------- | ||
|
|
||
| A function can be marked as deferred using the :func:`deferred` decorator, but | ||
| also applied directly to a skrub DataOp using | ||
| :meth:`.skb.apply_func() <DataOp.skb.apply_func>`: | ||
|
|
||
| >>> def with_upper_columns(df): | ||
|
|
@@ -103,6 +107,13 @@ Result: | |
| 2 pen 1.5 2 | ||
| 3 fork 2.2 4 | ||
|
|
||
| It is also possible to create a specific deferred function from a preexisting eager | ||
| one: for instance, if we need to call module-level functions from a library. For | ||
| example, to delay the loading of a CSV file, we could write something like: | ||
|
|
||
| >>> csv_path = skrub.var("csv_path") | ||
| >>> data = skrub.deferred(pd.read_csv)(csv_path) | ||
|
|
||
| Unpacking multiple outputs from deferred functions | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wonder if after editing the rest of this section, the part about unpacking should be put into a drop down as additional explanation, something like "note about unpacking" |
||
| -------------------------------------------------- | ||
|
|
||
|
|
@@ -126,37 +137,6 @@ Instead, keep the result as a single DataOp and index into it: | |
| >>> left = res[0] | ||
| >>> right = res[1] | ||
|
|
||
| :func:`deferred` is useful not only for our own functions, but also when we | ||
| need to call module-level functions from a library. For example, to delay the | ||
| loading of a CSV file, we could write something like: | ||
|
|
||
| >>> csv_path = skrub.var("csv_path") | ||
| >>> data = skrub.deferred(pd.read_csv)(csv_path) | ||
|
|
||
| or, with ``apply_func``: | ||
|
|
||
| >>> data = csv_path.skb.apply_func(pd.read_csv) | ||
|
|
||
| Another consequence of the fact that DataOps are evaluated lazily (we are | ||
| building a pipeline, not immediately computing a single result), any | ||
| transformation that we apply must not modify its input, but leave it unchanged | ||
| and return a new value. | ||
|
|
||
| Consider the transformers in a scikit-learn pipeline: each computes a new | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. from oral discussions I remember this comparison can help understand why each node must return a new value instead of modifying the input |
||
| result without modifying its input. | ||
|
|
||
| >>> orders['total'] = orders['price'] * orders['qty'] | ||
| Traceback (most recent call last): | ||
| ... | ||
| TypeError: Do not modify a DataOp in-place. Instead, use a function that returns a new value. This is necessary to allow chaining several steps in a sequence of transformations. | ||
| For example if df is a pandas DataFrame: | ||
| df = df.assign(new_col=...) instead of df['new_col'] = ... | ||
|
|
||
| Note the suggestion in the error message: using :meth:`pandas.DataFrame.assign`. | ||
| When we do need assignments or in-place transformations, we can put them in a | ||
| :func:`deferred` function. But we should make a (shallow) copy of the inputs and | ||
| return a new value. | ||
|
|
||
| Finally, there are other situations where using :func:`deferred` can be helpful: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the "Finally..." paragraph should be moved at the end of the previous section, possibly with the examples |
||
|
|
||
| - When we have many nodes in our graph and want to collapse a sequence of steps into | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if this section could benefit from a small example, something like
to show that the result of the evaluation depends on what's in side the variable