Skip to content

Commit 094211f

Browse files
GaelVaroquauxGaelVaroquaux
authored
DOC: iterate on selectors doc: (#1745)
1 parent 77aad07 commit 094211f

3 files changed

Lines changed: 170 additions & 150 deletions

File tree

‎doc/modules/multi_column_operations/advanced_selectors.rst‎

Lines changed: 5 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.. currentmodule :: skrub.selectors
1+
.. currentmodule:: skrub.selectors
22

33
.. |ApplyToCols| replace:: :class:`~skrub.ApplyToCols`
44
.. |StandardScaler| replace:: :class:`~sklearn.preprocessing.StandardScaler`
@@ -7,8 +7,8 @@
77

88
.. _user_guide_advanced_selectors:
99

10-
Advanced selectors: |filter| and |filter_names|
11-
-------------------------------------------
10+
|filter| and |filter_names| to select with user-defined criteria
11+
-----------------------------------------------------------------
1212

1313
:func:`filter` and :func:`filter_names` allow
1414
selecting columns based on arbitrary user-defined criteria. These are also used to
@@ -49,64 +49,9 @@ lambda or local functions and thus ensure the selector is picklable.
4949
0 297.0 210.0
5050
1 420.0 297.0
5151

52-
Combining selectors with other skrub transformers
53-
-------------------------------------------------
54-
Skrub transformers are designed to be used in conjunction with other transformers
55-
that operate on columns to improve their versatility.
5652

57-
For example, we can drop columns that have more unique values than a certain amount
58-
by combining :func:`cardinality_below` with :class:`skrub.DropCols`.
59-
We first select the columns that have more than 3 unique values, then we invert the
60-
selector and finally transform the dataframe.
61-
62-
>>> df = pd.DataFrame({
63-
... "not a lot": [1, 1, 1, 2, 2],
64-
... "too_many": [1,2,3,4,5]})
65-
66-
>>> from skrub import DropCols
67-
>>> DropCols(cols=~s.cardinality_below(3)).fit_transform(df)
68-
not a lot
69-
0 1
70-
1 1
71-
2 1
72-
3 2
73-
4 2
74-
75-
Selectors can be used in conjunction with |ApplyToCols| to transform columns
76-
based on specific requirements.
77-
78-
Consider the following example:
79-
80-
>>> import pandas as pd
81-
>>> data = {
82-
... "subject": ["Math", "English", "History", "Science", "Art"],
83-
... "grade": [5, 4, 3, 4, 3]
84-
... }
85-
>>> df = pd.DataFrame(data)
86-
>>> df
87-
subject grade
88-
0 Math 5
89-
1 English 4
90-
2 History 3
91-
3 Science 4
92-
4 Art 3
93-
94-
We might want to apply the |StandardScaler| only to the numeric column. We can
95-
do this like this:
96-
97-
>>> from skrub import ApplyToCols
98-
>>> from sklearn.preprocessing import StandardScaler
99-
>>> ApplyToCols(StandardScaler(), cols=s.numeric()).fit_transform(df)
100-
subject grade
101-
0 Math 1.603567
102-
1 English 0.267261
103-
2 History -1.069045
104-
3 Science 0.267261
105-
4 Art -1.069045
106-
107-
108-
Custom criteria in :func:`filter`, example of selecting columns with outliers
109-
------------------------------------------------------------------------------
53+
Example of custom criteria in :func:`filter`: selecting columns with outliers
54+
.............................................................................
11055

11156
The :func:`filter` selector can be used to select columns based on custom
11257
criteria. For example, we can define a function that checks if a column contains

‎doc/modules/multi_column_operations/selectors.rst‎

Lines changed: 162 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
.. _user_guide_selectors:
22

3-
Skrub Selectors: helpers for selecting columns in a dataframe
4-
=============================================================
3+
Skrub Selectors, for selecting columns in a dataframe
4+
=====================================================
55

66
In skrub, a selector represents a column selection rule, such as "all columns
77
that have numeric data types, except the column ``'User ID'``".
@@ -15,120 +15,192 @@ Selectors have two main benefits:
1515
instantiate a :class:`~skrub.SelectCols` that selects all columns except those with
1616
the suffix 'ID' if the data on which it will be fitted is not yet available.
1717

18+
Introduction to selectors
19+
------------------------------
1820

1921
Here is an example dataframe. Note that selectors support both Pandas and Polars
20-
dataframes:
21-
22-
>>> import pandas as pd
23-
>>> df = pd.DataFrame(
24-
... {
25-
... "height_mm": [297.0, 420.0],
26-
... "width_mm": [210.0, 297.0],
27-
... "kind": ["A4", "A3"],
28-
... "ID": [4, 3],
29-
... }
30-
... )
22+
dataframes::
23+
24+
>>> import pandas as pd
25+
>>> df = pd.DataFrame(
26+
... {
27+
... "height_mm": [297.0, 420.0],
28+
... "width_mm": [210.0, 297.0],
29+
... "kind": ["A4", "A3"],
30+
... "ID": [4, 3],
31+
... }
32+
... )
3133

3234
:func:`~skrub.selectors.cols` is a simple kind of selector which selects a fixed list of
33-
column names:
35+
column names::
36+
37+
>>> from skrub import selectors as s
38+
>>> mm_cols = s.cols('height_mm', 'width_mm')
39+
>>> mm_cols
40+
cols('height_mm', 'width_mm')
41+
42+
Using selectors:
43+
44+
* **select function** The above selector can be passed to the :func:`~skrub.selectors.select`
45+
function::
46+
47+
>>> s.select(df, mm_cols)
48+
height_mm width_mm
49+
0 297.0 210.0
50+
1 420.0 297.0
51+
52+
* **transformers**: various transformers in skrub use selectors to select and transform columns
53+
in a scikit-learn pipeline: :class:`~skrub.ApplyToCols`, :class:`~skrub.ApplyToFrame`,
54+
:class:`~skrub.DropCols`, :class:`~skrub.SelectCols`, as
55+
:ref:`detailed below <selectors_and_transformer>`_.
56+
57+
* **DataOps** selectors can be passed to
58+
:ref:`skrub DataOps <user_guide_data_ops_index>` when applying an
59+
estimator with the :func:`skrub.DataOp.skb.apply` function::
60+
61+
>>> import skrub
62+
>>> from sklearn.preprocessing import StandardScaler
63+
>>> skrub.X(df).skb.apply(StandardScaler(), cols=mm_cols)
64+
<Apply StandardScaler>
65+
Result:
66+
―――――――
67+
kind ID height_mm width_mm
68+
0 A4 4 -1.0 -1.0
69+
1 A3 3 1.0 1.0
3470

35-
>>> from skrub import selectors as s
36-
>>> mm_cols = s.cols('height_mm', 'width_mm')
37-
>>> mm_cols
38-
cols('height_mm', 'width_mm')
71+
Type of selectors
72+
-----------------
3973

40-
This selector can then be passed to a :func:`~skrub.selectors.select` function:
74+
:func:`~skrub.selectors.all` is another simple selector, especially useful for default
75+
arguments since it keeps all columns::
4176

42-
>>> s.select(df, mm_cols)
43-
height_mm width_mm
44-
0 297.0 210.0
45-
1 420.0 297.0
77+
>>> from skrub import SelectCols
78+
>>> SelectCols(cols=s.all()).fit_transform(df)
79+
height_mm width_mm kind ID
80+
0 297.0 210.0 A4 4
81+
1 420.0 297.0 A3 3
4682

47-
It can also be passed to :class:`~skrub.SelectCols` or :class:`~skrub.DropCols`
48-
to be embedded in scikit-learn pipelines:
83+
Selectors can be combined with operators, for example if we wanted all columns
84+
except the "mm" columns above::
4985

50-
Last but not least, selectors can be passed to
51-
:ref:`skrub DataOps <user_guide_data_ops_index>` when applying an
52-
estimator with the :func:`skrub.DataOp.skb.apply` function:
86+
>>> SelectCols(s.all() - s.cols("height_mm", "width_mm")).fit_transform(df)
87+
kind ID
88+
0 A4 4
89+
1 A3 3
5390

54-
>>> import skrub
55-
>>> from sklearn.preprocessing import StandardScaler
56-
>>> skrub.X(df).skb.apply(StandardScaler(), cols=mm_cols)
57-
<Apply StandardScaler>
58-
Result:
59-
―――――――
60-
kind ID height_mm width_mm
61-
0 A4 4 -1.0 -1.0
62-
1 A3 3 1.0 1.0
91+
This module provides several kinds of selectors, which allow to select columns by
92+
name, data type, contents, or according to arbitrary user-provided rules::
6393

64-
Selectors can be used within the :class:`skrub.SelectCols` class, implementing `fit` and `transform`, as demoed below::
94+
>>> SelectCols(s.numeric()).fit_transform(df)
95+
height_mm width_mm ID
96+
0 297.0 210.0 4
97+
1 420.0 297.0 3
6598

66-
>>> from skrub import SelectCols
99+
>>> SelectCols(s.glob('*_mm')).fit_transform(df)
100+
height_mm width_mm
101+
0 297.0 210.0
102+
1 420.0 297.0
67103

68-
Type of selectors
69-
-----------------
104+
.. seealso::
70105

71-
:func:`~skrub.selectors.all` is another simple selector, especially useful for default
72-
arguments since it keeps all columns:
106+
* :ref:`selectors_details` explains more the various selectors
73107

74-
>>> SelectCols(cols=s.all()).fit_transform(df)
75-
height_mm width_mm kind ID
76-
0 297.0 210.0 A4 4
77-
1 420.0 297.0 A3 3
108+
* :ref:`selectors_ref` gives the exhaustive list of selectors.
78109

79-
Selectors can be combined with operators, for example if we wanted all columns
80-
except the "mm" columns above:
110+
* :ref:`user_guide_advanced_selectors`
81111

82-
>>> SelectCols(s.all() - s.cols("height_mm", "width_mm")).fit_transform(df)
83-
kind ID
84-
0 A4 4
85-
1 A3 3
112+
Combining selectors
113+
-------------------
86114

87-
This module provides several kinds of selectors, which allow to select columns by
88-
name, data type, contents, or according to arbitrary user-provided rules.
115+
The available operators are ``|``, ``&``, ``-``, ``^`` with the meaning of usual
116+
python sets, and ``~`` to invert a selection::
89117

90-
>>> SelectCols(s.numeric()).fit_transform(df)
91-
height_mm width_mm ID
92-
0 297.0 210.0 4
93-
1 420.0 297.0 3
118+
>>> SelectCols(s.glob('*_mm')).fit_transform(df)
119+
height_mm width_mm
120+
0 297.0 210.0
121+
1 420.0 297.0
94122

95-
>>> SelectCols(s.glob('*_mm')).fit_transform(df)
96-
height_mm width_mm
97-
0 297.0 210.0
98-
1 420.0 297.0
123+
>>> SelectCols(~s.glob('*_mm')).fit_transform(df)
124+
kind ID
125+
0 A4 4
126+
1 A3 3
99127

100-
See :ref:`selectors_ref` for an exhaustive list.
128+
>>> SelectCols(s.glob('*_mm') | s.cols('ID')).fit_transform(df)
129+
height_mm width_mm ID
130+
0 297.0 210.0 4
131+
1 420.0 297.0 3
101132

102-
The available operators are ``|``, ``&``, ``-``, ``^`` with the meaning of usual
103-
python sets, and ``~`` to invert a selection.
133+
>>> SelectCols(s.glob('*_mm') & s.glob('height_*')).fit_transform(df)
134+
height_mm
135+
0 297.0
136+
1 420.0
137+
138+
>>> SelectCols(s.glob('*_mm') ^ s.string()).fit_transform(df)
139+
height_mm width_mm kind
140+
0 297.0 210.0 A4
141+
1 420.0 297.0 A3
104142

105-
>>> SelectCols(s.glob('*_mm')).fit_transform(df)
106-
height_mm width_mm
107-
0 297.0 210.0
108-
1 420.0 297.0
143+
The operators respect the usual short-circuit rules. For example, the
144+
following selector won't compute the cardinality of non-categorical columns::
109145

110-
>>> SelectCols(~s.glob('*_mm')).fit_transform(df)
111-
kind ID
112-
0 A4 4
113-
1 A3 3
146+
>>> s.categorical() & s.cardinality_below(10)
147+
(categorical() & cardinality_below(10))
114148

115-
>>> SelectCols(s.glob('*_mm') | s.cols('ID')).fit_transform(df)
116-
height_mm width_mm ID
117-
0 297.0 210.0 4
118-
1 420.0 297.0 3
119149

120-
>>> SelectCols(s.glob('*_mm') & s.glob('height_*')).fit_transform(df)
121-
height_mm
122-
0 297.0
123-
1 420.0
150+
.. _selectors_and_transformer:
124151

125-
>>> SelectCols(s.glob('*_mm') ^ s.string()).fit_transform(df)
126-
height_mm width_mm kind
127-
0 297.0 210.0 A4
128-
1 420.0 297.0 A3
152+
Using selectors with other skrub transformers
153+
-------------------------------------------------
129154

130-
The operators respect the usual short-circuit rules. For example, the
131-
following selector won't compute the cardinality of non-categorical columns:
155+
Skrub transformers are designed to be used in conjunction with other transformers
156+
that operate on columns to improve their versatility.
157+
158+
For example, we can drop columns that have more unique values than a certain amount
159+
by combining :func:`cardinality_below` with :class:`skrub.DropCols`.
160+
We first select the columns that have more than 3 unique values, then we invert the
161+
selector and finally transform the dataframe.
132162

133-
>>> s.categorical() & s.cardinality_below(10)
134-
(categorical() & cardinality_below(10))
163+
>>> df = pd.DataFrame({
164+
... "not a lot": [1, 1, 1, 2, 2],
165+
... "too_many": [1,2,3,4,5]})
166+
167+
>>> from skrub import DropCols
168+
>>> DropCols(cols=~s.cardinality_below(3)).fit_transform(df)
169+
not a lot
170+
0 1
171+
1 1
172+
2 1
173+
3 2
174+
4 2
175+
176+
Selectors can be used in conjunction with |ApplyToCols| to transform columns
177+
based on specific requirements.
178+
179+
Consider the following example:
180+
181+
>>> import pandas as pd
182+
>>> data = {
183+
... "subject": ["Math", "English", "History", "Science", "Art"],
184+
... "grade": [5, 4, 3, 4, 3]
185+
... }
186+
>>> df = pd.DataFrame(data)
187+
>>> df
188+
subject grade
189+
0 Math 5
190+
1 English 4
191+
2 History 3
192+
3 Science 4
193+
4 Art 3
194+
195+
We might want to apply the |StandardScaler| only to the numeric column. We can
196+
do this like this:
197+
198+
>>> from skrub import ApplyToCols
199+
>>> from sklearn.preprocessing import StandardScaler
200+
>>> ApplyToCols(StandardScaler(), cols=s.numeric()).fit_transform(df)
201+
subject grade
202+
0 Math 1.603567
203+
1 English 0.267261
204+
2 History -1.069045
205+
3 Science 0.267261
206+
4 Art -1.069045

‎doc/modules/multi_column_operations/type_of_selectors.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
1+
.. _selectors_details:
2+
13
Selecting based on dtype or data properties
24
-------------------------------------------
5+
36
Selectors can filter columns based on different conditions.
47

58
:func:`~skrub.selectors.all` is a simple selector, especially useful for default

0 commit comments

Comments
 (0)