CHANGES.rst

.. _changes:

===============
Release history
===============

.. currentmodule:: skrub

Ongoing Development
===================

New features
------------

Changes
-------

Bugfixes
--------

Release 0.7.2
=============

Changes
-------
- The :class:`StringEncoder` now exposes the ``vocabulary`` parameter from the parent
  :class:`TfidfVectorizer`.
  :pr:`1819` by :user:`Eloi Massoulié <emassoulie>`
- :func:`compute_ngram_distance` has been renamed to :func:`_compute_ngram_distance` and is now a private function.
  :pr:`1838` by :user:`Siddharth Baleja <siddharthbaleja>`.
- The repository wheel has been made smaller by removing some material that was
  not necessary for using the library. Benchmarks are now available in a separate
  `repository <https://github.com/skrub-data/skrub-benchmarks>`__.
  :pr:`1893` by :user:`Riccardo Cappuzzo <rcap107>`.


Bugfixes
--------
- Fixed some issues related to the release of Pandas 3.0. :pr:`1855` by :user:`Riccardo Cappuzzo <rcap107>`.

Release 0.7.1
=============

New features
------------
- A new dataset, :func:`fetch_california_housing`, has been added to the
  :mod:`skrub.datasets` module. It allows to get a redundancy copy of the scikit-learn
  :func:`fetch_california_housing` function.
  :pr:`1830` by :user:`Guillaume Lemaitre <glemaitre>`.

Bugfixes
--------
- :class:`DropCols` and :class:`SelectCols:` attributes were renamed to end
  with an underscore, in order to follow a scikit-learn convention which is
  used to determine if an estimator is fitted. :pr:`1813` by :user:`Auguste
  Baum <auguste-probabl>`.

Release 0.7.0
=============

New features
------------
- It is now possible to tune the choices in a :class:`DataOp` with `Optuna
  <https://optuna.readthedocs.io/en/stable/>`_. See
  :ref:`example_optuna_choices` for an example.
  :pr:`1661` by :user:`Jérôme Dockès <jeromedockes>`.
- :meth:`DataOp.skb.apply` now allows passing extra named arguments to the
  estimator's methods through the parameters ``fit_kwargs``, ``predict_kwargs``
  etc. :pr:`1642` by :user:`Jérôme Dockès <jeromedockes>`.
- TableReport now displays the mean statistic for boolean columns.
  :pr:`1647` by :user:`Abdelhakim Benechehab <abenechehab>`.
- :meth:`DataOp.skb.get_vars` allows inspecting all the variables, or all the
  named dataops, in a :class:`DataOp`. This lets us easily know what keys should
  be present in the ``environment`` dictionary we pass to
  :meth:`DataOp.skb.eval` or to :meth:`SkrubLearner.fit`,
  :meth:`SkrubLearner.predict`, etc.
  :pr:`1646` by :user:`Jérôme Dockès <jeromedockes>`.
- :meth:`DataOp.skb.iter_cv_splits` iterates over the training and testing
  environments produced by a CV splitter -- similar to
  :meth:`DataOp.skb.train_test_split` but for multiple cross-validation splits.
  :pr:`1653` by :user:`Jérôme Dockès <jeromedockes>`.
- :class:`TableReport` now supports ``np.array``. :pr:`1676` by :user:`Nisma Amjad <Nismamjad1>`.
- :meth:`DataOp.skb.full_report` now accepts a new parameter, ``title``, that is displayed
  in the html report.
  :pr:`1654` by :user:`Marie Sacksick <MarieSacksick>`.
- :class:`TableReport` now includes the ``open_tab`` parameter, which lets the
  user select which tab should be opened when the ``TableReport`` is
  rendered. :pr:`1737` by :user:`Riccardo Cappuzzo<rcap107>`.

Changes
-------
- The minimum supported version of Python has been increased to 3.10. Additionally,
  the minimum supported versions of scikit-learn and requests are 1.4.2 and 2.27.1
  respectively. Support for python 3.14 has been added.
  :pr:`1572` by :user:`Riccardo Cappuzzo<rcap107>`.
- The :meth:`DataOp.skb.full_report` method now deletes reports created with
  ``output_dir=None`` after 7 days. :pr:`1657` by :user:`Simon Dierickx <simon.dierickx>`.
- The :func:`tabular_pipeline` uses a :class:`SquashingScaler` instead of a
  :class:`StandardScaler` for centering and scaling numerical features
  when linear models are used.
  :pr:`1644` by :user:`Simon Dierickx <dierickxsimon>`
- The transformer :class:`ToFloat`, previously called ``ToFloat32``, is now public.
  :pr:`1687` by :user:`Marie Sacksick <MarieSacksick>`.
- Improved the error message raised when a Polars lazyframe is passed to
  :class:`TableReport`, clarifying that ``.collect()`` must be called first.
  :pr:`1767` by :user:`Fatima Ben Kadour <fatiben2002>`.
- Computing the associations in :class:`TableReport` is now deterministic and can
  be controlled by the new parameter ``subsampling_seed`` of the global configuration.
  :pr:`1775` by :user:`Thomas S. <thomass-dev>`.
- Added ``cast_to_str`` parameter to :class:`Cleaner` to prevent unintended
  conversion of list/object-like columns to strings unless explicitly enabled.
  :pr:`1789` by :user:`PilliSiddharth`.

Bugfixes
--------
- The :meth:`skrub.cross_validate` function now raises a specific exception if the wrong variable
  type is passed.
  :pr:`1799` by :user:`Eloi Massoulié<emassoulie>`
- Fixed various issues with some transformers by adding ``get_feature_names_out``
  to all single column transformers.
  :pr:`1666` by :user:`Riccardo Cappuzzo<rcap107>`.
- Issues occurring when :meth:`DataOp.skb.apply` was passed a DataOp as the
  estimator have been fixed in :pr:`1671` by :user:`Jérôme Dockès
  <jeromedockes>`.
- :class:`TableReport` could raise an error while trying to check if Polars
  columns with some dtypes (lists, structs) are sorted. It would not indicate
  Polars columns sorted in descending order. Fixed in :pr:`1673` by
  :user:`Jérôme Dockès <jeromedockes>`.
- Fixed nightly checks and added support for upcoming library versions, including Pandas
  v3.0. :pr:`1664` by :user:`Auguste Baum <auguste-probabl>` and
  :user:`Riccardo Cappuzzo <rcap107>`.
- Fixed the use of :class:`TableReport` and :class:`Cleaner` with Polars dataframes
  containing a column with empty string as name.
  :pr:`1722` by :user:`Marie Sacksick <MarieSacksick>`.
- Fixed an issue where :class:`TableReport` would fail when computing associations
  for Polars dataframes if PyArrow was not installed.
  :pr:`1742` by :user:`Riccardo Cappuzzo <rcap107>`.
- Fixed an issue in the Data Ops report generation in cases where the DataOp
  contained escape characters or were spanning multiple lines.
  :pr:`1764` by :user:`Riccardo Cappuzzo <rcap107>`.
- Added :meth:`get_feature_names_out` to :class:`Cleaner` for consistency with the
  :class:`TableVectorizer` and other transformers. :pr:`1762` by
  :user:`Riccardo Cappuzzo <rcap107>`.
- Improve error message when :class:`TextEncoder` is used without the optional
  transformers dependencies. :pr:`1769` by :user:`Fangxuan Zhou <fxzhou22>`.
- Accessing ``.skb.applied_estimator`` on a :class:`DataOp` after calling
  ``.skb.set_name()``, ``.skb.set_description()``, ``.skb.mark_as_X()`` or
  ``.skb.mark_as_y()`` used to raise an error, this has been fixed in :pr:`1782`
  by :user:`Jérôme Dockès <jeromedockes>`.
- Fixed potential issues that could arise in :meth:`ParamSearch.plot_results`
  when NaN values were present in the cross-validation results.
  :pr:`1800` by :user:`Riccardo Cappuzzo <rcap107>`.

Release 0.6.2
=============

New features
------------
- The :meth:`DataOp.skb.full_report` now displays the time each node took to
  evaluate. :pr:`1596` by :user:`Jérôme Dockès <jeromedockes>`.

Changes
-------
- Ken embeddings are now deprecated, the functions :func:`datasets.get_ken_embeddings`,
  :func:`datasets.get_ken_table_aliases`, and :func:`datasets.get_ken_types` will be
  removed in the next release of skrub.
  :pr:`1546` by :user:`Vincent Maladiere <Vincent-Maladiere>`.
- Improved error messages when a DataOp is being sent to dispatched functions.
  :pr:`1607` by :user:`Riccardo Cappuzzo<rcap107>`.
- The accepted values for the parameter ``how`` of :meth:`DataOp.skb.apply` have
  changed. The new values are ``"auto"`` (unchanged), ``"cols"`` to wrap the
  transformer in :class:`ApplyToCols`, ``"frame"`` to wrap the transformer in
  :class:`ApplyToFrame`, or ``"no_wrap"`` for no wrapping. The old values are
  deprecated and will result in an error in a future release.
  :pr:`1628` by :user:`Jérôme Dockès <jeromedockes>`.
- The parameter ``splitter`` of :meth:`DataOp.skb.train_test_split` has been
  renamed ``split_func``. :pr:`1630` by :user:`Jérôme Dockès <jeromedockes>`.
- KEN embeddings and all the relevant functions have been removed from skrub.
  :pr:`1567` by :user:`Riccardo Cappuzzo<rcap107>`.
- The objects ``tabular_learner`` and ``DropIfTooManyNulls`` were removed. Use
  :func:`tabular_pipeline` and :class:`DropUninformative` instead.
  :pr:`1567` by :user:`Riccardo Cappuzzo<rcap107>`.
- The skrub global configuration now includes a parameter for setting the default
  verbosity of the :class:`TableReport`.
  :pr:`1567` by :user:`Riccardo Cappuzzo<rcap107>`.

Bugfixes
--------

- Fixed a compatibility bug with Polars 1.32.3 that may cause `ToFloat32` to fail
  when applied to categorical columns. :pr:`1570` by :user:`Riccardo Cappuzzo<rcap107>`.
- Fixed the display of DataOp objects in google colab cell outputs (no output
  was displayed). :pr:`1590` by :user:`Jérôme Dockès <jeromedockes>`.
- Fixed an error that occurred when using ``.skb.concat`` with a pandas dataframe
  with column names that aren't strings. :pr:`1594` by :user:`Riccardo Cappuzzo<rcap107>`.
- Fixed the range from which :func:`choose_float` and :func:`choose_int` sample
  values when ``log=False`` and ``n_steps`` is ``None``. It was between ``low``
  and ``low + high``, now it is between ``low`` and ``high``. :pr:`1603` by
  :user:`Jérôme Dockès <jeromedockes>`.
- DataOp hyperparameter search would raise an error when doing classification
  and using the ``scoring`` parameter, when the dataop contained no variables.
  Fixed in :pr:`1601` by :user:`Jérôme Dockès <jeromedockes>`.
- :class:`SkrubLearner` used to do a prediction on the train set during
  ``fit()``, this has been fixed.
  :pr:`1610` by :user:`Jérôme Dockès <jeromedockes>`.
- :class:`DataOp` would raise errors when containing subclasses of list, tuple
  or dict that cannot be initialized with an instance of the builtin type (such
  as classes created by ``collections.namedtuple``), this has been fixed.
  DataOps now only recurse into the builtin collections to evaluate their items
  (not into their subclasses). If you need the items evaluated (ie if they
  contain DataOps or Choices), store them in one of the builtin collections.
  :pr:`1612` by :user:`Jérôme Dockès <jeromedockes>`.
- :meth:`SkrubLearner.report` with ``mode="fit"`` used to display the dataops
  themselves, rather than their outputs, in the report. This has been fixed in
  :pr:`1623` by :user:`Jérôme Dockès <jeromedockes>`.
- Fixed a bug that happened when ``get_feature_names_out`` was called on instances
  of the :class:`DatetimeEncoder`. :pr:`1622` by :user:`Riccardo Cappuzzo<rcap107>`.

Release 0.6.1
===================

Bugfixes
--------

- ``get_feature_names_out`` now works correctly when used by :class:`GapEncoder`,
  :class:`DropCols`, :class:`SelectCols:` from within a scikit-learn ``Pipeline``. In
  addition, :class:`DropCols`'s ``get_feature_names_out`` method now returns the
  names of the columns that are not dropped, rather than the names of the columns
  that are dropped. :pr:`1543` by :user:`Riccardo Cappuzzo<rcap107>`.


Release 0.6.0
=============

Highlights
----------
- Major feature! Skrub DataOps are a powerful new way of
  combining dataframe transformations over multiple tables, and machine learning
  pipelines. DataOps can be combined to form compled data plans, that can be used
  to train and tune machine learning models. Then, the DataOps plans can be exported
  as ``Learners`` (:class:`skrub.SkrubLearner`), standalone objects that can be
  used on new data. More detail about the DataOps can be found in the
  :ref:`User guide <user_guide_data_ops_index>` and in the
  :ref:`examples <data_ops_examples_ref>`.

- The :class:`TableReport` has been improved with many new features. Series are
  now supported directly. It is now
  possible to skip computing column associations and generating plots when the
  number of columns in the dataframe exceeds a user-defined threshold. Columns with
  high cardinality and sorted columns are now highlighted in the report.

- :mod:`selectors`, :class:`ApplyToCols` and :class:`ApplyToFrame` are now available,
  providing utilities for selecting columns to which a transformer should be applied
  in a flexible way. For more details, see the :ref:`User guide <user_guide_selectors>`
  and the :ref:`example <sphx_glr_auto_examples_0090_apply_to_cols.py>`.

- The :class:`SquashingScaler` has been added: it robustly rescales and smoothly
  clips numeric columns, enabling more robust handling of numeric columns
  with neural networks. See the :ref:`example <sphx_glr_auto_examples_0100_squashing_scaler.py>`

New features
------------

- The Skrub DataOps are new mechanism for building machine-learning
  pipelines that handle multiple tables and easily describing their
  hyperparameter spaces. Main PR: :pr:`1233` by :user:`Jérôme Dockès <jeromedockes>`.
  Additional work from other contributors can be found
  `here <https://github.com/skrub-data/skrub/issues?q=merged%3A%3C2025-07-24%20label%3Adata_ops>`_:
  :user:`Vincent Maladiere <Vincent-Maladiere>` provided very important help by
  trying the DataOps on many use-cases and datasets, providing feedback and
  suggesting improvements, improving the examples (including creating all the
  figures in the examples) and adding jitter to the parallel coordinate plots,
  :user:`Riccardo Cappuzzo<rcap107>` experimented with the DataOps,
  suggested improvements and improved the examples, :user:`Gaël Varoquaux
  <gaelvaroquaux>` , :user:`Guillaume Lemaitre <glemaitre>`, :user:`Adrin Jalali
  <adrinjalali>`, :user:`Olivier Grisel <ogrisel>` and others participated
  through many discussions in defining the requirements and the public API.
  See :ref:`the examples <data_ops_examples_ref>` for
  an introduction.

- The :mod:`selectors` module provides utilities for selecting columns to which
  a transformer should be applied in a flexible way. The module was created in
  :pr:`895` by :user:`Jérôme Dockès <jeromedockes>` and added to the public API
  in :pr:`1341` by :user:`Jérôme Dockès <jeromedockes>`.

- The :class:`DropUninformative` transformer is now available. This transformer
  employs different heuristics to detect columns that are not likely to bring
  useful information for training a model.
  The current implementation includes detection of columns that contain only a
  single value (constant columns), only missing values, or all unique values (such
  as IDs). :pr:`1313` by :user:`Riccardo Cappuzzo<rcap107>`.

- :func:`get_config`, :func:`set_config` and :func:`config_context` are now available
  to configure settings for dataframes display and expressions. :func:`patch_display`
  and :func:`unpatch_display` are deprecated and will be removed in the next release
  of skrub. :pr:`1427` by :user:`Vincent Maladiere <Vincent-Maladiere>`.
  The global configuration includes the parameter ``cardinality_threshold`` that
  controls the threshold value used to warn user if they have high cardinality
  columns in their dataset. :pr:`1498` by :user:`rouk1 <rouk1>`.
  Additionally, the parameter ``float_precision``
  controls the number of significant digits displayed for floating-point values
  in reports. :pr:`1470` by :user:`George S <georgescutelnicu>`.

- Added the :class:`SquashingScaler`, a transformer that
  robustly rescales and smoothly clips numeric columns,
  enabling more robust handling of numeric columns
  with neural networks. :pr:`1310` by :user:`Vincent Maladiere <Vincent-Maladiere>` and
  :user:`David Holzmüller <dholzmueller>`.

- :func:`datasets.toy_order` is now available to create a toy dataframe and
  corresponding targets for examples.
  :pr:`1485` by :user:`Antoine Canaguier-Durand <canag>`.

- :class:`ApplyToCols` and :class:`ApplyToFrame` are now available to apply transformers
  on a set of columns independently and jointly respectively.
  :pr:`1478` by :user:`Vincent Maladiere<Vincent-Maladiere>`.


Changes
-------
.. warning::
  The default high cardinality encoder for both :class:`TableVectorizer` and
  :meth:`tabular_learner` (now :meth:`tabular_pipeline`) has been changed from
  :class:`GapEncoder` to :class:`StringEncoder`. :pr:`1354` by
  :user:`Riccardo Cappuzzo<rcap107>`.

- The ``tabular_learner`` function has been deprecated in favor of :func:`tabular_pipeline` to honor
  its scikit-learn pipeline cultural heritage, and remove the ambiguity with the data
  ops Learner. :pr:`1493` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

- :class:`StringEncoder` now exposes the ``stop_words`` argument, which is passed to the
  underlying vectorizer (:class:`~sklearn.feature_extraction.text.TfidfVectorizer`,
  or :class:`~sklearn.feature_extraction.text.HashingVectorizer`). :pr:`1415` by
  :user:`Vincent Maladiere <Vincent-Maladiere>`.

- A new parameter ``max_association_columns`` has been added to the
  :class:`TableReport` to skip association computation when the number of columns
  exceeds the specified value. :pr:`1304` by :user:`Victoria Shevchenko <victoris93>`.

- The `packaging` dependency was removed.
  :pr:`1307` by :user:`Jovan Stojanovic <jovan-stojanovic>`

- :class:`TextEncoder`, :class:`StringEncoder` and :class:`GapEncoder` now compute the
  total standard deviation norm during training, which is a global constant, and
  normalize the vector outputs by performing element-wise division on all entries.
  :pr:`1274` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

- The :class:`DropIfTooManyNulls` transformer has been replaced by the
  :class:`DropUninformative` transformer and will be removed in a future release.
  :pr:`1313` by :user:`Riccardo Cappuzzo<rcap107>`

- The :func:`concat_horizontal` function was replaced with :func:`concat`. Horizontal or vertical concatenation
  is now controlled by the `axis` parameter. :pr:`1334` by :user:`Parasa V Prajwal <pvprajwal>`.

- The :class:`TableVectorizer` and :class:`Cleaner` now accept a `datetime_format`
  parameter for specifying the format to use when parsing datetime columns.
  :pr:`1358` by :user:`Riccardo Cappuzzo<rcap107>`.

- The :class:`SimpleCleaner` has been removed. use :class:`Cleaner` instead. :pr:`1370` by :user:`Riccardo Cappuzzo<rcap107>`.

- The periodic encoding for the ``day_in_year`` has been removed from the :class:`DatetimeEncoder` as it was
  redundant. The feature itself is still added if the flag is set to ``True``. :pr:`1396` by :user:`Riccardo Cappuzzo<rcap107>`.

- The naming scheme used for the features generated by :class:`TextEncoder`, :class:`StringEncoder`, :class:`MinHashEncoder`,
  :class:`DatetimeEncoder` has been standardized. Now features generated by all encoders have indices in the range
  ``[0, n_components-1]``, rather than ``[1, n_components]``. Additionally, columns with empty name are assigned a default
  name that depends on the encoder used. :pr:`1405` by :user:`Riccardo Cappuzzo<rcap107>`.

- The optional dependencies 'dev', 'doc', 'lint' and 'test' have been coalesced into
  'dev'. :pr:`1404` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

- The :class:`TableReport` now supports Series in addition to Dataframes. :pr:`1420` by :user:`Vitor Pohlenz<vitorpohlenz>`.

- The :class:`Cleaner` now exposes a parameter to convert numeric values to float32. :pr:`1440` by
  :user:`Riccardo Cappuzzo<rcap107>`.

- The :class:`TableReport` now shows if columns are sorted. :pr:`1512` by :user:`Dea María Léon<DeaMariaLeon>`.


Bugfixes
--------
- Fixed a bug that caused the :class:`StringEncoder` and :class:`TextEncoder` to raise an exception if the
  input column was a Categorical datatype. :pr:`1401` by :user:`Riccardo Cappuzzo<rcap107>`.

Documentation
-------------
A large number of improvements to the examples, docstrings, and the documentation
website have been made. Contributors include :user:`Vincent Maladiere <Vincent-Maladiere>`,
:user:`Riccardo Cappuzzo<rcap107>`, :user:`Jérôme Dockès <jeromedockes>`,
:user:`Gael Varoquaux <gaelvaroquaux>`, :user:`Gabriela Gómez Jiménez <gabrielapgomezji>`,
:user:`Sylvain Combettes <sylvaincom>`, :user:`Frits Hermans <fritshermans>`,
:user:`Vitor Pohlenz <vitorpohlenz>`, :user:`Arturo Amor Quiroz <ArturoAmorQ>`,
:user:`Marie Sacksick <MarieSacksick>`, :user:`Emilien Battel <emilienbattel09>`,
:user:`George El Haber <gmhaber>`, :user:`Antoine Canaguier-Durand <canag>`, and
:user:`Lionel Kusch <lionelkusch>`.


Release 0.5.4
=============

Maintenance
-----------
* Make ``skrub`` compatible with scikit-learn 1.7.
  :pr:`1434` by :user:`Vincent Maladiere <Vincent-Maladiere>`.


Release 0.5.3
=============

Changes
-------

- The :class:`SimpleCleaner` has been renamed to :class:`Cleaner`. Use of the
  name :class:`SimpleCleaner` is deprecated and will result in an error in some
  future release of skrub. :pr:`1275` by :user:`Riccardo Cappuzzo<rcap107>`.

- A new parameter ``max_plot_columns`` has been added to the
  :class:`TableReport` and :func:`patch_display` to skip column plots when the
  number of columns exceeds the specified value. :pr:`1255` by :user:`Priscilla
  Baah<priscilla-b>`.


Release 0.5.2
=============

New features
------------

- The :class:`TableReport` now switches its visual theme between light and dark according to the user preferences.
  :pr:`1201` by :user:`rouk1 <rouk1>`.

- Adding a new way to control the location of the data directory, using envar ``SKRUB_DATA_DIRECTORY``.
  :pr:`1215` by :user:`Thomas S. <thomass-dev>`

- The :class:`DatetimeEncoder` now supports periodic encoding of datetime features
  with trigonometric functions and B-splines transformers.
  :pr:`1235` by :user:`Riccardo Cappuzzo<rcap107>`.

- The :class:`TableReport` now also compute Pearson's correlation for numeric values.
  :pr:`1203` by :user:`Reshama Shaikh <reshamas>` and
  :user:`Vincent Maladiere <Vincent-Maladiere>`.

- The :class:`SimpleCleaner` is now available (⚠️ it was renamed to
  :class:`Cleaner` in skrub ``0.5.3``.). This transformer is a lightweight
  pre-processor that applies some of the transformations applied by the
  :class:`TableVectorizer`, with a simpler interface. :pr:`1266` by
  :user:`Riccardo Cappuzzo<rcap107>` and :user:`Jerome Dockes <jeromedockes>` .

Changes
-------

- The estimator returned by :func:`tabular_learner` now uses spline encoding of
  datetime features when the supervised learner is not a model based on decision
  trees such as random forests or gradient boosting. :pr:`1264` by
  :user:`Guillaume Lemaitre <glemaitre>`.

- The "distribution" tab of the ``TableReport`` now stacks cards horizontally to avoid adding
  vertical space.
  :pr:`1259` by :user:`Gaël Varoquaux <gaelvaroquaux>`

- Progress messages when generating a ``TableReport`` are now written to stderr instead of stdout.
  :pr:`1236` by :user:`Priscilla Baah<priscilla-b>`

- Optimize the :class:`StringEncoder`: lower memory footprint and faster execution in some cases.
<<<<<<< HEAD
  :pr:`1248` by :user:`Gaël Varoquaux <gaelvaroquaux>`

Bug fixes
---------
- :class:`StringEncoder` now works correctly in presence of null values.
=======
>>>>>>> upstream/main
  :pr:`1248` by :user:`Gaël Varoquaux <gaelvaroquaux>`

Bug fixes
---------
- :class:`StringEncoder` now works correctly in presence of null values.
  :pr:`1224` by :user:`Jérôme Dockès <jeromedockes>`.

- The :meth:`TableVectorizer.get_feature_names_out` method now works when used in a
  scikit-learn pipeline by exposing the `input_features` parameter.
  :pr:`1258` by :user:`Guillaume Lemaitre <glemaitre>`.


Release 0.5.1
=============

New features
------------
* The :class:`StringEncoder` encodes strings using tf-idf and truncated SVD
  decomposition and provides a cheaper alternative to :class:`GapEncoder`.
  :pr:`1159` by :user:`Riccardo Cappuzzo<rcap107>`.

Changes
-------
* New dataset fetching methods have been added: :func:`fetch_videogame_sales`,
  :func:`fetch_bike_sharing`, :func:`fetch_flight_delays`,
  :func:`fetch_country_happiness`, and removed :func:`fetch_road_safety`.
  :pr:`1218` by :user:`Vincent Maladiere <Vincent-Maladiere>`

Bug fixes
---------

Maintenance
-----------

Release 0.4.1
=============

Changes
-------

* :class:`TableReport` has `write_html` method. :pr:`1190` by :user:`Mojdeh Rastgoo<mrastgoo>`.

* A new parameter ``verbose`` has been added to the :class:`TableReport` to toggle on or off the
  printing of progress information when a report is being generated.
  :pr:`1182` by :user:`Priscilla Baah<priscilla-b>`.

* A parameter ``verbose`` has been added to the :func:`patch_display` to toggle on or off the
  printing of progress information when a table report is being generated.
  :pr:`1188` by :user:`Priscilla Baah<priscilla-b>`.

* :func:`tabular_learner` accepts the alias ``"regression"`` for the option
  ``"regressor"`` and ``"classification"`` for ``"classifier"``.
  :pr:`1180` by :user:`Mojdeh Rastgoo <mrastgoo>`.

Bug fixes
---------
* Generating a ``TableReport`` could have an effect on the matplotib
  configuration which could cause plots not to display inline in jupyter
  notebooks any more. This has been fixed in skrub in :pr:`1172` by
  :user:`Jérôme Dockès <jeromedockes>` and the matplotlib issue can be tracked
  `here <https://github.com/matplotlib/matplotlib/issues/25041>`_.

* The labels on bar plots in the ``TableReport`` for columns of object dtypes
  that have a repr spanning multiple lines could be unreadable. This has been
  fixed in :pr:`1196` by :user:`Jérôme Dockès <jeromedockes>`.

* Improve the performance of :func:`deduplicate` by removing some unnecessary
  computations. :pr:`1193` by :user:`Jérôme Dockès <jeromedockes>`.

Maintenance
-----------
* Make ``skrub`` compatible with scikit-learn 1.6.
  :pr:`1169` by :user:`Guillaume Lemaitre <glemaitre>`.

Release 0.4.0
=============

Highlights
----------
* The :class:`TextEncoder` can extract embeddings from a string column with  a deep
  learning language model (possibly downloaded from the HuggingFace Hub).

* Several improvements to the :class:`TableReport` such as better support for
  other scripts than the latin alphabet in the bar plot labels, smaller report
  sizes, clipping the outliers to better see the details of distributions in
  histograms. See the full changelog for details.

* The :class:`TableVectorizer` can now drop columns that contain a fraction of
  null values above a user-chosen threshold.

New features
------------
* The :class:`TextEncoder` is now available to encode string columns with
  diverse entries.
  It allows the representation of table entries as embeddings computed by a deep
  learning language model. The weights of this model can be fetched locally
  or from the HuggingFace Hub.
  :pr:`1077` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

* The :func:`column_associations` function has been added. It computes a
  pairwise measure of statistical dependence between all columns in a dataframe
  (the same as shown in the :class:`TableReport`). :pr:`1109` by :user:`Jérôme
  Dockès <jeromedockes>`.

* The :func:`patch_display` function has been added. It changes the display of
  pandas and polars dataframes in jupyter notebooks to replace them with a
  :class:`TableReport`. This can be undone with :func:`unpatch_display`.
  :pr:`1108` by :user:`Jérôme Dockès <jeromedockes>`

Major changes
-------------
* :class:`AggJoiner`, :class:`AggTarget` and :class:`MultiAggJoiner` now require
  the `operations` argument. They do not split columns by type anymore, but
  apply `operations` on all selected cols. "median" is now supported, "hist" and
  "value_counts" are no longer supported. :pr:`1116` by :user:`Théo Jolivet <TheooJ>`.

* The :class:`AggTarget` no longer supports `y` inputs of type list. :pr:`1116`
  by :user:`Théo Jolivet <TheooJ>`.

Minor changes
-------------

* The column filter selection dropdown in the tablereport is smaller and its
  label has been removed to save space. :pr:`1107` by :user:`Jérôme Dockès
  <jeromedockes>`.

* The TableReport now uses the font size of its parent element when inserted
  into another page. This makes it smaller in pages that use a smaller font size
  than the browser default such as VSCode in some configurations. It also makes
  it easier to control its size when inserting it in a web page by setting the
  font size of its parent element. A few other small adjustments have also been
  made to make it a bit more compact. :pr:`1098` by :user:`Jérôme Dockès
  <jeromedockes>`.

* Display of labels in the plots of the TableReport, especially for other
  scripts than the latin alphabet, has improved.

  - before, some characters could be missing and replaced by empty boxes.
  - before, when the text is truncated, the ellipsis "..." could appear on the
    wrong side for right-to-left scripts.

  Moreover, when the text contains line breaks it now appears all on one line.
  Note this only affects the labels in the plots; the rest of the report did not
  have these problems.
  :pr:`1097` by :user:`Jérôme Dockès <jeromedockes>`
  and :pr:`1138` by :user:`Jérôme Dockès <jeromedockes>`.

* In the TableReport it is now possible, before clicking any of the cells, to
  reach the dataframe sample table and activate a cell with tab key navigation.
  :pr:`1101` by :user:`Jérôme Dockès <jeromedockes>`.

* The "Column name" column of the "summary statistics" table in the TableReport
  is now always visible when scrolling the table. :pr:`1102` by :user:`Jérôme
  Dockès <jeromedockes>`.

* Added parameter `drop_null_fraction` to `TableVectorizer` to drop columns based
  on whether they contain a fraction of nulls larger than the given threshold.
  :pr:`1115` and :pr:`1149` by :user:`Riccardo Cappuzzo <rcap107>`.

* The :class:`TableReport` now provides more helpful output for columns of dtype
  TimeDelta / Duration. :pr:`1152` by :user:`Jérôme Dockès <jeromedockes>`.

* The :class:`TableReport` now also reports the number of unique values for
  numeric columns. :pr:`1154` by :user:`Jérôme Dockès <jeromedockes>`.

* The :class:`TableReport`, when plotting histograms, now detects outliers and
  clips the range of data shown in the histogram. This allows seeing more detail
  in the shown distribution. :pr:`1157` by :user:`Jérôme Dockès <jeromedockes>`.

Bug fixes
---------

* The :class:`TableReport` could raise an exception when one of the columns
  contained datetimes with time zones and missing values; this has been fixed in
  :pr:`1114` by :user:`Jérôme Dockès <jeromedockes>`.

* In scikit-learn versions older than 1.4 the :class:`TableVectorizer` could
  fail on polars dataframes when used with the default parameters. This has been
  fixed in :pr:`1122` by :user:`Jérôme Dockès <jeromedockes>`.

* The :class:`TableReport` would raise an exception when the input (pandas)
  dataframe contained several columns with the same name. This has been fixed in
  :pr:`1125` by :user:`Jérôme Dockès <jeromedockes>`.

* The :class:`TableReport` would raise an exception when a column contained
  infinite values. This has been fixed in :pr:`1150` by :user:`Jérôme Dockès
  <jeromedockes>` and :pr:`1151` by Jérôme Dockès.

Release 0.3.1
=============

Minor changes
-------------

* For tree-based models, :func:`tabular_learner` now adds
  `handle_unknown='use_encoded_value'` to the `OrdinalEncoder`, to avoid
  errors with new categories in the test set. This is consistent with the
  setting of `OneHotEncoder` used by default in the
  :class:`TableVectorizer`. :pr:`1078` by :user:`Gaël Varoquaux <gaelvaroquaux>`

* The reports created by :class:`TableReport`, when inserted in an html page (or
  displayed in a notebook), now use the same font as the surrounding page.
  :pr:`1038` by :user:`Jérôme Dockès <jeromedockes>`.

* The content of the dataframe corresponding to the currently selected table
  cell in the TableReport can be copied without actually selecting the text (as
  in a spreadsheet).
  :pr:`1048` by :user:`Jérôme Dockès <jeromedockes>`.

* The selection of content displayed in the TableReport's copy-paste boxes has
  been removed. Now they always display the value of the selected item. When
  copied, the repr of the selected item is copied to the clipboard.
  :pr:`1058` by :user:`Jérôme Dockès <jeromedockes>`.

* A "stats" panel has been added to the TableReport, showing summary statistics
  for all columns (number of missing values, mean, etc. -- similar to
  ``pandas.info()`` ) in a table. It can be sorted by each column.
  :pr:`1056` and :pr:`1068` by :user:`Jérôme Dockès <jeromedockes>`.

* The credit fraud dataset is now available with the
  :func:`fetch_credit_fraud function`.
  :pr:`1053` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

* Added zero padding for column names in :class:`MinHashEncoder` to improve column ordering consistency.
  :pr:`1069` by :user:`Shreekant Nandiyawar <Shree7676>`.

* The selection in the TableReport's sample table can now be manipulated with
  the keyboard. :pr:`1065` by :user:`Jérôme Dockès <jeromedockes>`.

* The ``TableReport`` now displays the pandas (multi-)index, and has a better
  display & interaction of pandas columns when the columns are a MultiIndex.
  :pr:`1083` by :user:`Jérôme Dockès <jeromedockes>`.

* It is possible to control the number of rows displayed by the TableReport in
  the "sample" tab panel by specifying ``n_rows``.
  :pr:`1083` by :user:`Jérôme Dockès <jeromedockes>`.

* the `TableReport` used to raise an exception when the dataframe contained
  unhashable types such as python lists. This has been fixed in :pr:`1087` by
  :user:`Jérôme Dockès <jeromedockes>`.

* Display's columns name with the HTML representation of the fitted TableVectorizer.
  This has been fixed in :pr:`1093` by :user:`Shreekant Nandiyawar <Shree7676>`.

* AggTarget will now work even when y is a Series and not raise any error.
  This has been fixed in :pr:`1094` by :user:`Shreekant Nandiyawar <Shree7676>`.

Release 0.3.0
=============

Highlights
----------
* Polars dataframes are now supported across all ``skrub`` estimators.
* :class:`TableReport` generates an interactive report for a dataframe. This
  `page <https://skrub-data.org/skrub-reports/examples/>`_ regroups some
  precomputed examples.

Major changes
-------------
* The :class:`InterpolationJoiner` now supports polars dataframes. :pr:`1016`
  by :user:`Théo Jolivet <TheooJ>`.
* The :class:`TableReport` provides an interactive report on a dataframe's
  contents: an overview, summary statistics and plots, statistical associations
  between columns. It can be displayed in a jupyter notebook, a browser tab or
  saved as a static HTML page. :pr:`984` by :user:`Jérôme Dockès <jeromedockes>`.

Minor changes
-------------
* :class:`Joiner` and :func:`fuzzy_join` used to raise an error when columns
  with the same name appeared in the main and auxiliary table (after adding the
  suffix). This is now allowed and a random string is inserted in the duplicate
  column to ensure all names are unique.
  :pr:`1014` by :user:`Jérôme Dockès <jeromedockes>`.

* :class:`AggJoiner` and :class:`AggTarget` could produce outputs whose column
  names varied across calls to `transform` in some cases in the presence of
  duplicate column names, now the output names are always the same.
  :pr:`1013` by :user:`Jérôme Dockès <jeromedockes>`.

* In some cases :class:`AggJoiner` and :class:`AggTarget` inserted a column in
  the output named "index" containing the pandas index of the auxiliary table.
  This has been corrected.
  :pr:`1020` by :user:`Jérôme Dockès <jeromedockes>`.

Release 0.2.0
=============

Major changes
-------------
* The :class:`Joiner` has been adapted to support polars dataframes. :pr:`945` by :user:`Théo Jolivet <TheooJ>`.

* The :class:`TableVectorizer` now consistently applies the same transformation
  across different calls to `transform`. There also have been some breaking
  changes to its functionality: (i) all transformations are now applied
  independently to each column, i.e. it does not perform multivariate
  transformations (ii) in ``specific_transformers`` the same column may not be
  used twice (go through 2 different transformers).
  :pr:`902` by :user:`Jérôme Dockès <jeromedockes>`.

* Some parameters of :class:`TableVectorizer` have been renamed:
  `high_cardinality_transformer` → `high_cardinality`,
  `low_cardinality_transformer` → `low_cardinality`,
  `datetime_transformer` → `datetime`, `numeric_transformer` → `numeric`.
  :pr:`947` by :user:`Jérôme Dockès <jeromedockes>`.

* The :class:`GapEncoder` and :class:`MinHashEncoder` are now a single-column
  transformers: their ``fit``, ``fit_transform`` and ``transform`` methods
  accept a single column (a pandas or polars Series). Dataframes and numpy
  arrays are not accepted.
  :pr:`920` and :pr:`923` by :user:`Jérôme Dockès <jeromedockes>`.

* Added the :class:`MultiAggJoiner` that allows to augment a main table with
  multiple auxiliary tables. :pr:`876` by :user:`Théo Jolivet <TheooJ>`.

* :class:`AggJoiner` now only accepts a single table as an input, and some of its
  parameters were renamed to be consistent with the :class:`MultiAggJoiner`.
  It now has a ``key``` parameter that allows to join main and auxiliary tables that share
  the same column names. :pr:`876` by :user:`Théo Jolivet <TheooJ>`.

* :func:`tabular_learner` has been added to easily create a supervised
  learner that works well on tabular data. :pr:`926` by :user:`Jérôme Dockès
  <jeromedockes>`.

Minor changes
-------------

* :class:`GapEncoder` and :class:`MinHashEncoder` used to modify their input
  in-place, replacing missing values with a string. They no longer do so. Their
  parameter `handle_missing` has been removed; now missing values are always
  treated as the empty string.
  :pr:`930` by :user:`Jérôme Dockès <jeromedockes>`.

* The minimum supported python version is now 3.9
  :pr:`939` by :user:`Jérôme Dockès <jeromedockes>`.

* Skrub supports numpy 2. :pr:`946` by :user:`Jérôme Dockès <jeromedockes>`.

* :func:`~datasets.fetch_ken_embeddings` now add suffix even with the default
  value for the parameter `pca_components`.
  :pr:`956` by :user:`Guillaume Lemaitre <glemaitre>`.

* :class:`Joiner` now performs some preprocessing (the same as done by the
  :class:`TableVectorizer`, eg trying to parse dates, converting pandas object
  columns with mixed types to a single type) on the joining columns before
  vectorizing them. :pr:`972` by :user:`Jérôme Dockès <jeromedockes>`.

skrub release 0.1.1
===================

This is a bugfix release to adapt to the most recent versions of pandas (2.2) and
scikit-learn (1.5). There are no major changes to the functionality of skrub.


skrub release 0.1.0
===================


Major changes
-------------
* :class:`TargetEncoder` has been removed in favor of
  :class:`sklearn.preprocessing.TargetEncoder`, available since scikit-learn 1.3.

* :class:`Joiner` and :func:`fuzzy_join` support several ways of rescaling
  distances; ``match_score`` has been replaced by ``max_dist``; bugs which
  prevented the Joiner to consistently vectorize inputs and accept or reject
  matches across calls to transform have been fixed. :pr:`821` by :user:`Jérôme
  Dockès <jeromedockes>`.

* :class:`InterpolationJoiner` was added to join two tables by using
  machine-learning to infer the matching rows from the second table.
  :pr:`742` by :user:`Jérôme Dockès <jeromedockes>`.

* Pipelines including :class:`TableVectorizer` can now be grid-searched, since
  we can now call `set_params` on the default transformers of :class:`TableVectorizer`.
  :pr:`814` by :user:`Vincent Maladiere <Vincent-Maladiere>`

* :func:`to_datetime` is now available to support pandas.to_datetime
  over dataframes and 2d arrays.
  :pr:`784` by :user:`Vincent Maladiere <Vincent-Maladiere>`

* Some parameters of :class:`Joiner` have changed. The goal is to harmonize
  parameters across all estimator that perform join(-like) operations, as
  discussed in `#751 <https://github.com/skrub-data/skrub/discussions/751>`_.
  :pr:`757` by :user:`Jérôme Dockès <jeromedockes>`.

* :func:`dataframe.pd_join`, :func:`dataframe.pd_aggregate`,
  :func:`dataframe.pl_join` and :func:`dataframe.pl_aggregate`
  are now available in the dataframe submodule.
  :pr:`733` by :user:`Vincent Maladiere <Vincent-Maladiere>`

* :class:`FeatureAugmenter` is renamed to :class:`Joiner`.
  :pr:`674` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* :func:`fuzzy_join` and :class:`FeatureAugmenter` can now join on datetime columns.
  :pr:`552` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* :class:`Joiner` now supports joining on multiple column keys.
  :pr:`674` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* The signatures of all encoders and functions have been revised to enforce
  cleaner calls. This means that some arguments that could previously be passed
  positionally now have to be passed as keywords.
  :pr:`514` by :user:`Lilian Boulard <LilianBoulard>`.

* Parallelized the :class:`GapEncoder` column-wise. Parameters `n_jobs` and `verbose`
  added to the signature. :pr:`582` by :user:`Lilian Boulard <LilianBoulard>`

* Introducing :class:`AggJoiner`, a transformer performing
  aggregation on auxiliary tables followed by left-joining on a base table.
  :pr:`600` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

* Introducing :class:`AggTarget`, a transformer performing
  aggregation on the target y, followed by left-joining on a base table.
  :pr:`600` by :user:`Vincent Maladiere <Vincent-Maladiere>`.

* Added the :class:`SelectCols` and :class:`DropCols` transformers that allow
  selecting a subset of a dataframe's columns inside of a pipeline. :pr:`804` by
  :user:`Jérôme Dockès <jeromedockes>`.


Minor changes
-------------
* :class:`DatetimeEncoder` doesn't remove constant features anymore.
  It also supports an 'errors' argument to raise or coerce errors during
  transform, and a 'add_total_seconds' argument to include the number of
  seconds since Epoch.
  :pr:`784` by :user:`Vincent Maladiere <Vincent-Maladiere>`

* Scaling of ``matching_score`` in :func:`fuzzy_join` is now between 0 and 1; it used to be between 0.5 and 1. Moreover, the division by 0 error that occurred when all rows had a perfect match has been fixed. :pr:`802` by :user:`Jérôme Dockès <jeromedockes>`.

* :class:`TableVectorizer` is now able to apply parallelism at the column level rather than the transformer level. This is the default for univariate transformers, like :class:`MinHashEncoder`, and :class:`GapEncoder`.
  :pr:`592` by :user:`Leo Grinsztajn <LeoGrin>`

* ``inverse_transform`` in :class:`SimilarityEncoder` now works as expected; it used to raise an exception. :pr:`801` by :user:`Jérôme Dockès <jeromedockes>`.

* :class:`TableVectorizer` propagate the `n_jobs` parameter to the underlying
  transformers except if the underlying transformer already set explicitly `n_jobs`.
  :pr:`761` by :user:`Leo Grinsztajn <LeoGrin>`, :user:`Guillaume Lemaitre <glemaitre>`,
  and :user:`Jerome Dockes <jeromedockes>`.


* Parallelized the :func:`deduplicate` function. Parameter `n_jobs`
  added to the signature. :pr:`618` by :user:`Jovan Stojanovic <jovan-stojanovic>`
  and :user:`Lilian Boulard <LilianBoulard>`

* Functions :func:`datasets.fetch_ken_embeddings`, :func:`datasets.fetch_ken_table_aliases`
  and :func:`datasets.fetch_ken_types` have been renamed.
  :pr:`602` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* Make `pyarrow` an optional dependencies to facilitate the integration
  with `pyodide`.
  :pr:`639` by :user:`Guillaume Lemaitre <glemaitre>`.

* Bumped minimal required Python version to 3.10. :pr:`606` by
  :user:`Gael Varoquaux <GaelVaroquaux>`

* Bumped minimal required versions for the dependencies:
  - numpy >= 1.23.5
  - scipy >= 1.9.3
  - scikit-learn >= 1.2.1
  - pandas >= 1.5.3 :pr:`613` by :user:`Lilian Boulard <LilianBoulard>`

* You can now pass column-specific transformers to :class:`TableVectorizer`
  using the `specific_transformers` argument.
  :pr:`583` by :user:`Lilian Boulard <LilianBoulard>`.

* Do not support 1-D array (and pandas Series) in :class:`TableVectorizer`. Pass a
  2-D array (or a pandas DataFrame) with a single column instead. This change is for
  compliance with the scikit-learn API.
  :pr:`647` by :user:`Guillaume Lemaitre <glemaitre>`

* Fixes a bug in :class:`TableVectorizer` with `remainder`: it is now cloned if it's
  a transformer so that the same instance is not shared between different
  transformers.
  :pr:`678` by :user:`Guillaume Lemaitre <glemaitre>`

* :class:`GapEncoder` speedup :pr:`680` by :user:`Leo Grinsztajn <LeoGrin>`

  - Improved :class:`GapEncoder`'s early stopping logic. The parameters `tol` and `min_iter`
    have been removed. The parameter `max_no_improvement` can now be used to control the
    early stopping.
    :pr:`663` by :user:`Simona Maggio <simonamaggio>`
    :pr:`593` by  :user:`Lilian Boulard <LilianBoulard>`
    :pr:`681` by  :user:`Leo Grinsztajn <LeoGrin>`

  - Implementation improvement leading to a ~x5 speedup for each iteration.

  - Better default hyperparameters: `batch_size` now defaults to 1024, and `max_iter_e_steps`
    to 1.

* Removed the `most_frequent` and `k-means` strategies from the :class:`SimilarityEncoder`.
  These strategy were used for scalability reasons, but we recommend using the :class:`MinHashEncoder`
  or the :class:`GapEncoder` instead. :pr:`596` by :user:`Leo Grinsztajn <LeoGrin>`

* Removed the `similarity` argument from the :class:`SimilarityEncoder` constructor,
  as we only support the ngram similarity. :pr:`596` by :user:`Leo Grinsztajn <LeoGrin>`

* Added the `analyzer` parameter to the :class:`SimilarityEncoder` to allow word counts
  for similarity measures. :pr:`619` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* skrub now uses modern type hints introduced in PEP 585.
  :pr:`609` by :user:`Lilian Boulard <LilianBoulard>`

* Some bug fixes for :class:`TableVectorizer` ( :pr:`579`):

  - `check_is_fitted` now looks at `"transformers_"` rather than `"columns_"`
  - the default of the `remainder` parameter in the docstring is now `"passthrough"`
    instead of `"drop"` to match the implementation.
  - uint8 and int8 dtypes are now considered as numeric columns.

* Removed the leading "<" and trailing ">" symbols from KEN entities
  and types.
  :pr:`601` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* Add `get_feature_names_out` method to :class:`MinHashEncoder`.
  :pr:`616` by :user:`Leo Grinsztajn <LeoGrin>`

* Removed `requests` from the requirements. :pr:`613` by :user:`Lilian Boulard <LilianBoulard>`

* :class:`TableVectorizer` now handles mixed types columns without failing
  by converting them to string before type inference.
  :pr:`623`by :user:`Leo Grinsztajn <LeoGrin>`

* Moved the default storage location of data to the user's home folder.
  :pr:`652` by :user:`Felix Lefebvre <flefebv>` and
  :user:`Gael Varoquaux <GaelVaroquaux>`

* Fixed bug when using :class:`TableVectorizer`'s `transform` method on
  categorical columns with missing values.
  :pr:`644` by :user:`Leo Grinsztajn <LeoGrin>`

* :class:`TableVectorizer` never output a sparse matrix by default. This can be changed by
  increasing the `sparse_threshold` parameter. :pr:`646` by :user:`Leo Grinsztajn <LeoGrin>`

* :class:`TableVectorizer` doesn't fail anymore if an inferred type doesn't work during transform.
  The new entries not matching the type are replaced by missing values. :pr:`666` by :user:`Leo Grinsztajn <LeoGrin>`

- Dataset fetcher :func:`datasets.fetch_employee_salaries` now has a parameter
  `overload_job_titles` to allow overloading the job titles
  (`employee_position_title`) with the column `underfilled_job_title`,
  which provides some more information about the job title.
  :pr:`581` by :user:`Lilian Boulard <LilianBoulard>`

* Fix bugs which was triggered when `extract_until` was "year", "month", "microseconds"
  or "nanoseconds", and add the option to set it to `None` to only extract `total_time`,
  the time from epoch. :class:`DatetimeEncoder`. :pr:`743` by :user:`Leo Grinsztajn <LeoGrin>`

Before skrub: dirty_cat
========================

Skrub was born from the `dirty_cat <http://dirty-cat.github.io>`__
package.

Dirty-cat release 0.4.1
==========================

Major changes
-------------
* :func:`fuzzy_join` and :class:`FeatureAugmenter` can now join on numeric columns based on the euclidean distance.
  :pr:`530` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* :func:`fuzzy_join` and :class:`FeatureAugmenter` can perform many-to-many joins on lists of numeric or string key columns.
  :pr:`530` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* :func:`GapEncoder.transform` will not continue fitting of the instance anymore.
  It makes functions that depend on it (:func:`~GapEncoder.get_feature_names_out`,
  :func:`~GapEncoder.score`, etc.) deterministic once fitted.
  :pr:`548` by :user:`Lilian Boulard <LilianBoulard>`

* :func:`fuzzy_join` and :class:`FeatureAugmenter` now perform joins on missing values as in `pandas.merge`
  but raises a warning. :pr:`522` and :pr:`529` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* Added :func:`get_ken_table_aliases` and :func:`get_ken_types` for exploring
  KEN embeddings. :pr:`539` by :user:`Lilian Boulard <LilianBoulard>`.


Minor changes
-------------
* Improvement of date column detection and date format inference in :class:`TableVectorizer`. The
  format inference now tries to find a format which works for all non-missing values of the column, and only
  tries pandas default inference if it fails.
  :pr:`543` by :user:`Leo Grinsztajn <LeoGrin>`
  :pr:`587` by :user:`Leo Grinsztajn <LeoGrin>`



Dirty-cat Release 0.4.0
=========================

Major changes
-------------
* `SuperVectorizer` is renamed as :class:`TableVectorizer`, a warning is raised when using the old name.
  :pr:`484` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* New experimental feature: joining tables using :func:`fuzzy_join` by approximate key matching. Matches are based
  on string similarities and the nearest neighbors matches are found for each category.
  :pr:`291` by :user:`Jovan Stojanovic <jovan-stojanovic>` and :user:`Leo Grinsztajn <LeoGrin>`

* New experimental feature: :class:`FeatureAugmenter`, a transformer
  that augments with :func:`fuzzy_join` the number of features in a main table by using information from auxiliary tables.
  :pr:`409` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* Unnecessary API has been made private: everything (files, functions, classes)
  starting with an underscore shouldn't be imported in your code. :pr:`331` by :user:`Lilian Boulard <LilianBoulard>`

* The :class:`MinHashEncoder` now supports a `n_jobs` parameter to parallelize
  the hashes computation. :pr:`267` by :user:`Leo Grinsztajn <LeoGrin>` and :user:`Lilian Boulard <LilianBoulard>`.

* New experimental feature: deduplicating misspelled categories using :func:`deduplicate` by clustering string distances.
  This function works best when there are significantly more duplicates than underlying categories.
  :pr:`339` by :user:`Moritz Boos <mjboos>`.

Minor changes
-------------
* Add example `Wikipedia embeddings to enrich the data`. :pr:`487` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* **datasets.fetching**: contains a new function :func:`get_ken_embeddings` that can be used to download Wikipedia
  embeddings and filter them by type.

* **datasets.fetching**: contains a new function :func:`fetch_world_bank_indicator` that can be used to download indicators
  from the World Bank Open Data platform.
  :pr:`291` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* Removed example `Fitting scalable, non-linear models on data with dirty categories`. :pr:`386` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* :class:`MinHashEncoder`'s :func:`minhash` method is no longer public. :pr:`379` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* Fetching functions now have an additional argument ``directory``,
  which can be used to specify where to save and load from datasets.
  :pr:`432` by :user:`Lilian Boulard <LilianBoulard>`

* Fetching functions now have an additional argument ``directory``,
  which can be used to specify where to save and load from datasets.
  :pr:`432` and :pr:`453` by :user:`Lilian Boulard <LilianBoulard>`

* The :class:`TableVectorizer`'s default `OneHotEncoder` for low cardinality categorical variables now defaults
  to `handle_unknown="ignore"` instead of `handle_unknown="error"` (for sklearn >= 1.0.0).
  This means that categories seen only at test time will be encoded by a vector of zeroes instead of raising an error. :pr:`473` by :user:`Leo Grinsztajn <LeoGrin>`

Bug fixes
---------

* The :class:`MinHashEncoder` now considers `None` and empty strings as missing values, rather
  than raising an error. :pr:`378` by :user:`Gael Varoquaux <GaelVaroquaux>`

Dirty-cat Release 0.3.0
==========================

Major changes
-------------

* New encoder: :class:`DatetimeEncoder` can transform a datetime column into several numeric columns
  (year, month, day, hour, minute, second, ...). It is now the default transformer used
  in the :class:`TableVectorizer` for datetime columns. :pr:`239` by :user:`Leo Grinsztajn <LeoGrin>`

* The :class:`TableVectorizer` has seen some major improvements and bug fixes:

  - Fixes the automatic casting logic in ``transform``.
  - To avoid dimensionality explosion when a feature has two unique values, the default encoder (:class:`~sklearn.preprocessing.OneHotEncoder`) now drops one of the two vectors (see parameter `drop="if_binary"`).
  - ``fit_transform`` and ``transform`` can now return unencoded features, like the :class:`~sklearn.compose.ColumnTransformer`'s behavior. Previously, a ``RuntimeError`` was raised.

  :pr:`300` by :user:`Lilian Boulard <LilianBoulard>`

* **Backward-incompatible change in the TableVectorizer**:
  To apply ``remainder`` to features (with the ``*_transformer`` parameters),
  the value ``'remainder'`` must be passed, instead of ``None`` in previous versions.
  ``None`` now indicates that we want to use the default transformer. :pr:`303` by :user:`Lilian Boulard <LilianBoulard>`

* Support for Python 3.6 and 3.7 has been dropped. Python >= 3.8 is now required. :pr:`289` by :user:`Lilian Boulard <LilianBoulard>`

* Bumped minimum dependencies:

  - scikit-learn>=0.23
  - scipy>=1.4.0
  - numpy>=1.17.3
  - pandas>=1.2.0 :pr:`299` and :pr:`300` by :user:`Lilian Boulard <LilianBoulard>`

* Dropped support for Jaro, Jaro-Winkler and Levenshtein distances.

  - The :class:`SimilarityEncoder` now exclusively uses ``ngram`` for similarities,
    and the `similarity` parameter is deprecated. It will be removed in 0.5. :pr:`282` by :user:`Lilian Boulard <LilianBoulard>`

Notes
-----

* The ``transformers_`` attribute of the :class:`TableVectorizer` now contains column
  names instead of column indices for the "remainder" columns. :pr:`266` by :user:`Leo Grinsztajn <LeoGrin>`


Dirty-cat Release 0.2.2
=========================

Bug fixes
---------

* Fixed a bug in the :class:`TableVectorizer` causing a :class:`FutureWarning`
  when using the :func:`get_feature_names_out` method. :pr:`262` by :user:`Lilian Boulard <LilianBoulard>`


Dirty-cat Release 0.2.1
==========================

Major changes
-------------

* Improvements to the :class:`TableVectorizer`

    - Type detection works better: handles dates, numerics columns encoded as strings, or numeric columns containing strings for missing values.

  :pr:`238` by :user:`Leo Grinsztajn <LeoGrin>`

* :func:`get_feature_names` becomes :func:`get_feature_names_out`, following changes in the scikit-learn API.
  :func:`get_feature_names` is deprecated in scikit-learn > 1.0. :pr:`241` by :user:`Gael Varoquaux <GaelVaroquaux>`

* Improvements to the :class:`MinHashEncoder`
    - It is now possible to fit multiple columns simultaneously with the :class:`MinHashEncoder`.
      Very useful when using for instance the :func:`~sklearn.compose.make_column_transformer` function,
      on multiple columns.

  :pr:`243` by :user:`Jovan Stojanovic <jovan-stojanovic>`


Bug-fixes
---------

* Fixed a bug that resulted in the :class:`GapEncoder` ignoring the analyzer argument. :pr:`242` by :user:`Jovan Stojanovic <jovan-stojanovic>`

* :class:`GapEncoder`'s `get_feature_names_out` now accepts all iterators, not just lists. :pr:`255` by :user:`Lilian Boulard <LilianBoulard>`

* Fixed :class:`DeprecationWarning` raised by the usage of `distutils.version.LooseVersion`. :pr:`261` by :user:`Lilian Boulard <LilianBoulard>`

Notes
-----

* Remove trailing imports in the :class:`MinHashEncoder`.

* Fix typos and update links for website.

* Documentation of the :class:`TableVectorizer` and the :class:`SimilarityEncoder` improved.

Dirty-cat Release 0.2.0
=========================

Also see pre-release 0.2.0a1 below for additional changes.

Major changes
-------------

* Bump minimum dependencies:

  - scikit-learn (>=0.21.0) :pr:`202` by :user:`Lilian Boulard <LilianBoulard>`
  - pandas (>=1.1.5) **! NEW REQUIREMENT !** :pr:`155` by :user:`Lilian Boulard <LilianBoulard>`

* **datasets.fetching** - backward-incompatible changes to the example
  datasets fetchers:

  - The backend has changed: we now exclusively fetch the datasets from OpenML.
    End users should not see any difference regarding this.
  - The frontend, however, changed a little: the fetching functions stay the same
    but their return values were modified in favor of a more Pythonic interface.
    Refer to the docstrings of functions `dirty_cat.datasets.fetch_*`
    for more information.
  - The example notebooks were updated to reflect these changes. :pr:`155` by :user:`Lilian Boulard <LilianBoulard>`

* **Backward incompatible change to** :class:`MinHashEncoder`: The :class:`MinHashEncoder` now
  only supports two dimensional inputs of shape (N_samples, 1).
  :pr:`185` by :user:`Lilian Boulard <LilianBoulard>` and :user:`Alexis Cvetkov <alexis-cvetkov>`.

* Update `handle_missing` parameters:

  - :class:`GapEncoder`: the default value "zero_impute" becomes "empty_impute" (see doc).
  - :class:`MinHashEncoder`: the default value "" becomes "zero_impute" (see doc).

  :pr:`210` by :user:`Alexis Cvetkov <alexis-cvetkov>`.

* Add a method "get_feature_names_out" for the :class:`GapEncoder` and the :class:`TableVectorizer`,
  since `get_feature_names` will be depreciated in scikit-learn 1.2. :pr:`216` by :user:`Alexis Cvetkov <alexis-cvetkov>`

Notes
-----

* Removed hard-coded CSV file `dirty_cat/data/FiveThirtyEight_Midwest_Survey.csv`.


* Improvements to the :class:`TableVectorizer`

  - Missing values are not systematically imputed anymore
  - Type casting and per-column imputation are now learnt during fitting
  - Several bugfixes

  :pr:`201` by :user:`Lilian Boulard <LilianBoulard>`

Dirty-cat Release 0.2.0a1
============================

Version 0.2.0a1 is a pre-release.
To try it, you have to install it manually using::

    pip install --pre dirty_cat==0.2.0a1

or from the GitHub repository::

    pip install git+https://github.com/dirty-cat/dirty_cat.git

Major changes
-------------

* Bump minimum dependencies:

  - Python (>= 3.6)
  - NumPy (>= 1.16)
  - SciPy (>= 1.2)
  - scikit-learn (>= 0.20.0)

* :class:`TableVectorizer`: Added automatic transform through the
  :class:`TableVectorizer` class. It transforms
  columns automatically based on their type. It provides a replacement
  for scikit-learn's :class:`~sklearn.compose.ColumnTransformer` simpler to use on heterogeneous
  pandas DataFrame. :pr:`167` by :user:`Lilian Boulard <LilianBoulard>`

* **Backward incompatible change to** :class:`GapEncoder`: The :class:`GapEncoder` now only
  supports two-dimensional inputs of shape (n_samples, n_features).
  Internally, features are encoded by independent :class:`GapEncoder` models,
  and are then concatenated into a single matrix.
  :pr:`185` by :user:`Lilian Boulard <LilianBoulard>` and :user:`Alexis Cvetkov <alexis-cvetkov>`.


Bug-fixes
---------

* Fix `get_feature_names` for scikit-learn > 0.21. :pr:`216` by :user:`Alexis Cvetkov <alexis-cvetkov>`


Dirty-cat Release 0.1.1
========================

Major changes
-------------

Bug-fixes
---------

* RuntimeWarnings due to overflow in :class:`GapEncoder`. :pr:`161` by :user:`Alexis Cvetkov <alexis-cvetkov>`


Dirty-cat Release 0.1.0
=========================

Major changes
-------------

* :class:`GapEncoder`: Added online Gamma-Poisson factorization through the
  :class:`GapEncoder` class. This method discovers latent categories formed
  via combinations of substrings, and encodes string data as combinations of
  these categories. To be used if interpretability is important. :pr:`153` by :user:`Alexis Cvetkov <alexis-cvetkov>`

Bug-fixes
---------

* Multiprocessing exception in notebook. :pr:`154` by :user:`Lilian Boulard <LilianBoulard>`


Dirty-cat Release 0.0.7
========================

* **MinHashEncoder**: Added ``minhash_encoder.py`` and ``fast_hast.py`` files
  that implement minhash encoding through the :class:`MinHashEncoder` class.
  This method allows for fast and scalable encoding of string categorical
  variables.

* **datasets.fetch_employee_salaries**: change the origin of download for employee_salaries.

  - The function now return a bunch with a dataframe under the field "data",
    and not the path to the csv file.
  - The field "description" has been renamed to "DESCR".

* **SimilarityEncoder**: Fixed a bug when using the Jaro-Winkler distance as a
  similarity metric. Our implementation now accurately reproduces the behaviour
  of the ``python-Levenshtein`` implementation.

* **SimilarityEncoder**: Added a `handle_missing` attribute to allow encoding
  with missing values.

* **TargetEncoder**: Added a `handle_missing` attribute to allow encoding
  with missing values.

* **MinHashEncoder**: Added a `handle_missing` attribute to allow encoding
  with missing values.

Dirty-cat Release 0.0.6
=========================

* **SimilarityEncoder**: Accelerate ``SimilarityEncoder.transform``, by:

  - computing the vocabulary count vectors in ``fit`` instead of ``transform``
  - computing the similarities in parallel using ``joblib``. This option can be
    turned on/off via the ``n_jobs`` attribute of the :class:`SimilarityEncoder`.

* **SimilarityEncoder**: Fix a bug that was preventing a :class:`SimilarityEncoder`
  to be created when ``categories`` was a list.

* **SimilarityEncoder**: Set the dtype passed to the ngram similarity
  to float32, which reduces memory consumption during encoding.

Dirty-cat Release 0.0.5
========================

* **SimilarityEncoder**: Change the default ngram range to (2, 4) which
  performs better empirically.

* **SimilarityEncoder**: Added a `most_frequent` strategy to define
  prototype categories for large-scale learning.

* **SimilarityEncoder**: Added a `k-means` strategy to define prototype
  categories for large-scale learning.

* **SimilarityEncoder**: Added the possibility to use hashing ngrams for
  stateless fitting with the ngram similarity.

* **SimilarityEncoder**: Performance improvements in the ngram similarity.

* **SimilarityEncoder**: Expose a `get_feature_names` method.