Skip to content

Commit dffb2d2

Browse files
author
Eloi Massoulié
committed
Merge remote-tracking branch 'upstream/main' into issue-1265-expose-cleaner-and-tablevectorizer-transformations
2 parents 054ddbf + c50109f commit dffb2d2

111 files changed

Lines changed: 35759 additions & 29554 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/changelog.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ jobs:
1818
run: |
1919
echo "PR_NUMBER=${{ github.event.pull_request.number }}" >> $GITHUB_ENV
2020
echo "TAGGED_MILESTONE=${{ github.event.pull_request.milestone.title }}" >> $GITHUB_ENV
21-
- uses: actions/checkout@v6
21+
- uses: actions/checkout@v7
2222
with:
2323
fetch-depth: '0'
2424
- name: Check the changelog entry

.github/workflows/check_stub_files_diff.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ jobs:
1414
runs-on: ubuntu-latest
1515

1616
steps:
17-
- uses: actions/checkout@v6
17+
- uses: actions/checkout@v7
1818
- uses: prefix-dev/setup-pixi@v0.9.6
1919
with:
20-
pixi-version: v0.59.0
20+
pixi-version: v0.68.0
2121
frozen: true
2222

2323
- name: Check stub file for `_data_ops.py` is up-to-date

.github/workflows/run-code-format-checks.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ jobs:
1414
runs-on: ubuntu-latest
1515

1616
steps:
17-
- uses: actions/checkout@v6
17+
- uses: actions/checkout@v7
1818
- uses: prefix-dev/setup-pixi@v0.9.6
1919
with:
20-
pixi-version: v0.59.0
20+
pixi-version: v0.68.0
2121
frozen: true
2222

2323
- name: Run tests

.github/workflows/test-javascript.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,10 @@ jobs:
1212
test:
1313
runs-on: ubuntu-latest
1414
steps:
15-
- uses: actions/checkout@v6
15+
- uses: actions/checkout@v7
1616
- uses: prefix-dev/setup-pixi@v0.9.6
1717
with:
18-
pixi-version: v0.59.0
18+
pixi-version: v0.68.0
1919
environments: ci-py314-latest-optional-deps
2020
# we can freeze the environment and manually bump the dependencies to the
2121
# latest version time to time.

.github/workflows/testing.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,10 @@ jobs:
2424
- ci-py314-polars-without-pyarrow
2525
runs-on: ${{ matrix.os }}
2626
steps:
27-
- uses: actions/checkout@v6
27+
- uses: actions/checkout@v7
2828
- uses: prefix-dev/setup-pixi@v0.9.6
2929
with:
30-
pixi-version: v0.59.0
30+
pixi-version: v0.68.0
3131
environments: ${{ matrix.environment }}
3232
# we can freeze the environment and manually bump the dependencies to the
3333
# latest version time to time.
@@ -47,7 +47,7 @@ jobs:
4747
outputs:
4848
run: ${{ github.event_name == 'schedule' || contains(steps.get_head_commit_message.outputs.COMMIT_MSG, '[deps nightly]') }}
4949
steps:
50-
- uses: actions/checkout@v6
50+
- uses: actions/checkout@v7
5151
with:
5252
ref: ${{ github.event.pull_request.head.sha }}
5353
- id: get_head_commit_message
@@ -60,10 +60,10 @@ jobs:
6060
runs-on: ubuntu-latest
6161
name: Test against nightly dependencies
6262
steps:
63-
- uses: actions/checkout@v6
63+
- uses: actions/checkout@v7
6464
- uses: prefix-dev/setup-pixi@v0.9.6
6565
with:
66-
pixi-version: v0.59.0
66+
pixi-version: v0.68.0
6767
environments: ci-nightly-deps
6868
# we can freeze the environment and manually bump the dependencies to the
6969
# latest version time to time.

.github/workflows/update_pixi_lock_files.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,10 @@ jobs:
2121
fail-fast: false
2222

2323
steps:
24-
- uses: actions/checkout@v6
24+
- uses: actions/checkout@v7
2525
- uses: prefix-dev/setup-pixi@v0.9.6
2626
with:
27-
pixi-version: v0.59.0
27+
pixi-version: v0.68.0
2828
run-install: false
2929

3030
- name: Remove the current lock file

CHANGES.rst

Lines changed: 57 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,15 @@ New Features
3131
:meth:`DataOp.skb.eval`, :meth:`SkrubLearner.predict`, etc., or in
3232
:meth:`DataOp.skb.find` or :meth:`SkrubLearner.truncated_after`. :pr:`2062` by
3333
:user:`Jérôme Dockès <jeromedockes>`.
34+
- The :class:`SessionEncoder` is now available. This encoder adds a `session_id`
35+
column, which groups together events that occur within the given session gap.
36+
Additionally, it is possible to provide a ``split_by`` column or list of columns
37+
(e.g., user ID or (user ID, user device)) to compute sessions for each grouping
38+
value.
39+
:pr:`1930` by :user:`Riccardo Cappuzzo <rcap107>`.
40+
- A new synthetic dataset generator for timestamped data and session-based
41+
operations has been added: :meth:`~skrub.datasets.make_retail_events`.
42+
:pr:`1930` by :user:`Riccardo Cappuzzo <rcap107>`.
3443
- The :class:`DropSimilar` transformer has been added, for removing columns in a
3544
dataframe that present high correlation with other columns. :pr:`2023` by
3645
:user:`Eloi Massoulié <emassoulie>`.
@@ -40,11 +49,20 @@ New Features
4049
Additionally, negative numbers indicated with parentheses can be converted to the
4150
regular numeric format (``(432)`` becomes ``-432``). :pr:`1772` by :user:`Gabriela
4251
Gómez Jiménez <gabrielapgomezji>`.
52+
- :meth:`TableReport.json` now includes histogram data for numeric and datetime
53+
columns (the bin count and edges, and numbers of low and high outliers). Now
54+
``json()`` contains all the information shown in the report html rendering,
55+
including the plots. :pr:`2164` by :user:`Jérôme Dockès <jeromedockes>`.
56+
- Added :func:`skrub.selectors.object` to select columns with the ``object``
57+
(pandas) or ``pl.Object`` (polars) dtype. :pr:`2171` by :user:`Omkar Kabde
58+
<omkar-334>`.
4359

4460
Changes
4561
-------
46-
- :meth:`choose_from` now transparently converts `outcomes` to a list when it is another type of sequence. :pr:`2100` by
47-
:user:`aidbar <aidbar>`.
62+
- Grouped Examples into subject-specific sections. :pr:`2102` by
63+
:user:`Maureen Githaiga <maureen-githaiga>`.
64+
- :meth:`choose_from` now transparently converts `outcomes` to a list when it is
65+
another type of sequence. :pr:`2100` by :user:`aidbar <aidbar>`.
4866
- An unnecessary warning that was raised when passing a numpy array to the
4967
TableVectorizer has been removed. :pr:`1908` by
5068
:user:`Sandrine Henry <sandrineh>`.
@@ -54,16 +72,42 @@ Changes
5472
:pr:`2096` by :user:`Ayesha Siddiqua <siddiqua-tamk>`.
5573
- The :class:`TableReport` can now be exported in markdown format with ``.markdown``.
5674
:pr:`2048` by :user:`Riccardo Cappuzzo <rcap107>`.
75+
- The minimum required version of matplotlib has been increased from 3.4.3 to 3.6.1.
76+
:pr:`2159` by :user:`Riccardo Cappuzzo <rcap107>`.
77+
- :meth:`SkrubLearner.score` has been enhanced when the DataOp used
78+
:meth:`DataOp.skb.with_scoring`. During scoring, predict(), predict_proba()
79+
etc. are cached to avoid recomputation when multiple scorers are used (or one
80+
scorer calls them several times). Moreover it is possible to pass
81+
``return_predictions=True`` to also retrieve any predictions that have been
82+
computed during scoring, in addition to the scores. Finally, in cases where we
83+
already have the predictions but want the result of score() without
84+
recomputing them, it is possible to provide them in the environment passed to
85+
``score({..., "_skrub_predictions": {"predict_proba": ...}})``.
86+
:pr:`2195` by :user:`Jérôme Dockès <jeromedockes>`.
87+
- :meth:`SkrubLearner.find_fitted_estimator` now supports searching for the
88+
apply node by ID or callable predicate as alternatives to the node name.
89+
:pr:`2194` by :user:`Jérôme Dockès <jeromedockes>`.
5790

5891
Bugfixes
5992
--------
93+
- :class:`MinHashEncoder` with the default ``hashing="fast"`` now uses every
94+
n-gram size in ``ngram_range`` (the upper bound is inclusive, as documented
95+
and as already done by ``hashing="murmur"``). Previously the largest size was
96+
dropped, so the default ``ngram_range=(2, 4)`` ignored 4-grams and a
97+
single-size range such as ``(3, 3)`` produced the same constant encoding for
98+
every string. :pr:`2168` by :user:`José Maia <glitch-ux>`.
6099
- A bug in how the :class:`TableVectorizer` and :class:`Cleaner` treated columns
61100
duration columns in pandas and polars has been fixed. Now, both classes convert
62101
durations to the total number of seconds (with fractional part). This is done
63102
by the new transformer :class:`DurationToFloat`. :pr:`2069` by
64103
:user:`Riccardo Cappuzzo <rcap107>`.
65-
66-
104+
- An error that could arise when running ``TableReport`` on dataframes containing
105+
double dollar (``$$``) signs has been fixed.
106+
:pr:`2154` by :user:`Katerina Michenina <Michenina-Lab>`,
107+
:user:`CecilyTS <CecilyTS>`, :user:`Eve Rabin <eve2705>`.
108+
- An error that happened when running ``TableReport`` or ``column_associations``
109+
on some dataframes with non-string column names has been fixed in :pr:`2179`
110+
by :user:`Jérôme Dockès <jeromedockes>`.
67111

68112
Deprecations
69113
------------
@@ -99,6 +143,9 @@ New Features
99143
- A new dataframe generator, :func:`datasets.toy_cities`, has been added for
100144
use cases on dataframes with variable sizes and variable correlation between
101145
columns. :pr:`2042` by :user:`Eloi Massoulié <emassoulie>`.
146+
- A new selector function, :func:`selectors.drop`, has been added to drop columns
147+
from a dataframe using a selector. It mirrors the behavior of :func:`selectors.select`.
148+
:pr:`2108` by :user:`Mary Njoroge <Maryahcee>`.
102149

103150
Changes
104151
-------
@@ -135,6 +182,9 @@ Changes
135182
<jeromedockes>`.
136183
- The ``exclude_cols`` of :meth:`DataOp.skb.apply` can now be a DataOp.
137184
:pr:`2050` by :user:`Jérôme Dockès <jeromedockes>`.
185+
- Skrub estimators now correctly show links to the documentation in the HTML
186+
representation that is generated for notebooks. :pr:`2036` by :user:`Riccardo
187+
Cappuzzo <rcap107>`.
138188

139189
Bugfixes
140190
--------
@@ -182,6 +232,9 @@ New Features
182232
- :func:`selectors.has_nulls` now takes a ``proportion`` parameter, which allows
183233
selecting columns that have a fraction of null values above the given threshold.
184234
:pr:`1881` by :user:`Gabriela Gómez Jiménez <gabrielapgomezji>`.
235+
- Added a new dataset, :func:`fetch_electricity_usage`, which contains electricity usage data
236+
for several French cities and corresponding weather data.
237+
:pr:`2013` by :user:`Lisa McBride<lisaleemcb>`.
185238

186239

187240
Changes

doc/_templates/index.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ <h5 class="feature-title">Inspect it, apply it to new data</h5>
152152
</div>
153153
<p class="feature-text-see-also">
154154
<!-- name of the userguide file on DataOps -->
155-
<a href="{{ pathto('auto_tutorials/1110_data_ops_intro') }}">Discover the skrub DataOps →</a>
155+
<a href="{{ pathto('data_ops') }}">Discover the skrub DataOps →</a>
156156
</p>
157157
</div>
158158
<div class="col-md-7">

doc/api_reference.py

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@
6767
"ApplyToCols",
6868
"SelectCols",
6969
"DropCols",
70+
"Drop",
7071
],
7172
}
7273
],
@@ -89,6 +90,7 @@
8990
"SimilarityEncoder",
9091
"ToCategorical",
9192
"DatetimeEncoder",
93+
"SessionEncoder",
9294
"ToDatetime",
9395
"ToFloat",
9496
],
@@ -128,24 +130,6 @@
128130
},
129131
],
130132
},
131-
"joining": {
132-
"title": "Joining dataframes",
133-
"short_summary": None,
134-
"description": None,
135-
"sections": [
136-
{
137-
"description": None,
138-
"autosummary": [
139-
"Joiner",
140-
"AggJoiner",
141-
"MultiAggJoiner",
142-
"AggTarget",
143-
"InterpolationJoiner",
144-
"fuzzy_join",
145-
],
146-
},
147-
],
148-
},
149133
"selectors": {
150134
"title": "Selectors",
151135
"short_summary": None,
@@ -175,6 +159,7 @@
175159
"selectors.inv",
176160
"selectors.make_selector",
177161
"selectors.numeric",
162+
"selectors.object",
178163
"selectors.regex",
179164
"selectors.select",
180165
"selectors.string",
@@ -313,6 +298,24 @@
313298
}
314299
],
315300
},
301+
"joining": {
302+
"title": "Joining dataframes",
303+
"short_summary": None,
304+
"description": None,
305+
"sections": [
306+
{
307+
"description": None,
308+
"autosummary": [
309+
"Joiner",
310+
"AggJoiner",
311+
"MultiAggJoiner",
312+
"AggTarget",
313+
"InterpolationJoiner",
314+
"fuzzy_join",
315+
],
316+
},
317+
],
318+
},
316319
"datasets": {
317320
"title": "Datasets",
318321
"short_summary": None,
@@ -326,6 +329,7 @@
326329
"datasets.fetch_country_happiness",
327330
"datasets.fetch_credit_fraud",
328331
"datasets.fetch_drug_directory",
332+
"datasets.fetch_electricity_forecasting",
329333
"datasets.fetch_employee_salaries",
330334
"datasets.fetch_flight_delays",
331335
"datasets.fetch_medical_charge",
@@ -338,6 +342,9 @@
338342
"datasets.get_data_dir",
339343
"datasets.make_deduplication_data",
340344
"datasets.toy_orders",
345+
"datasets.toy_products",
346+
"datasets.toy_cities",
347+
"datasets.make_retail_events",
341348
],
342349
}
343350
],

doc/conf.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@
7070
"sphinx.ext.linkcode",
7171
"sphinx.ext.autodoc.typehints",
7272
# contrib
73+
"sphinx_design",
7374
"numpydoc",
7475
"sphinx_issues",
7576
"sphinx_copybutton",

0 commit comments

Comments
 (0)