DOC - More fixes and improvements to the user guide (skrub-data#1574)

Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Gael Varoquaux <gael.varoquaux@normalesup.org>

Author: 3 people

CHANGES.rst

@@ -75,7 +75,7 @@ Highlights
   to train and tune machine learning models. Then, the DataOps plans can be exported
   as ``Learners`` (:class:`skrub.SkrubLearner`), standalone objects that can be
   used on new data. More detail about the DataOps can be found in the
-  :ref:`User guide <userguide_data_ops>` and in the
+  :ref:`User guide <user_guide_data_ops_index>` and in the
   :ref:`examples <data_ops_examples_ref>`.
 
 - The :class:`TableReport` has been improved with many new features. Series are
@@ -86,12 +86,12 @@ Highlights
 
 - :mod:`selectors`, :class:`ApplyToCols` and :class:`ApplyToFrame` are now available,
   providing utilities for selecting columns to which a transformer should be applied
-  in a flexible way. For more details, see the :ref:`User guide <userguide_selectors>`
-  and the :ref:`example <sphx_glr_auto_examples_10_apply_on_cols.py>`.
+  in a flexible way. For more details, see the :ref:`User guide <user_guide_selectors>`
+  and the :ref:`example <sphx_glr_auto_examples_09_apply_to_cols.py>`.
 
 - The :class:`SquashingScaler` has been added: it robustly rescales and smoothly
-  clips numerical columns, enabling more robust handling of numerical columns
-  with neural networks. See the :ref:`example <sphx_glr_auto_examples_11_squashing_scaler.py>`
+  clips numeric columns, enabling more robust handling of numeric columns
+  with neural networks. See the :ref:`example <sphx_glr_auto_examples_10_squashing_scaler.py>`
 
 New features
 ------------
@@ -137,8 +137,8 @@ New features
   in reports. :pr:`1470` by :user:`George S <georgescutelnicu>`.
 
 - Added the :class:`SquashingScaler`, a transformer that
-  robustly rescales and smoothly clips numerical columns,
-  enabling more robust handling of numerical columns
+  robustly rescales and smoothly clips numeric columns,
+  enabling more robust handling of numeric columns
   with neural networks. :pr:`1310` by :user:`Vincent Maladiere <Vincent-Maladiere>` and
   :user:`David Holzmüller <dholzmueller>`.
 
@@ -206,7 +206,7 @@ Changes
 
 - The :class:`TableReport` now supports Series in addition to Dataframes. :pr:`1420` by :user:`Vitor Pohlenz<vitorpohlenz>`.
 
-- The :class:`Cleaner` now exposes a parameter to convert numerical values to float32. :pr:`1440` by
+- The :class:`Cleaner` now exposes a parameter to convert numeric values to float32. :pr:`1440` by
   :user:`Riccardo Cappuzzo<rcap107>`.
 
 - The :class:`TableReport` now shows if columns are sorted. :pr:`1512` by :user:`Dea María Léon<DeaMariaLeon>`.
@@ -337,8 +337,7 @@ Release 0.4.1
 Changes
 -------
 
-* :class: `TableReport` has `write_html` method
-  :pr:`1190` by :user:`Mojdeh Rastgoo<mrastgoo>`.
+* :class:`TableReport` has `write_html` method. :pr:`1190` by :user:`Mojdeh Rastgoo<mrastgoo>`.
 
 * A new parameter ``verbose`` has been added to the :class:`TableReport` to toggle on or off the
   printing of progress information when a report is being generated.
@@ -806,7 +805,7 @@ Minor changes
   - `check_is_fitted` now looks at `"transformers_"` rather than `"columns_"`
   - the default of the `remainder` parameter in the docstring is now `"passthrough"`
     instead of `"drop"` to match the implementation.
-  - uint8 and int8 dtypes are now considered as numerical columns.
+  - uint8 and int8 dtypes are now considered as numeric columns.
 
 * Removed the leading "<" and trailing ">" symbols from KEN entities
   and types.
@@ -856,10 +855,10 @@ Dirty-cat release 0.4.1
 
 Major changes
 -------------
-* :func:`fuzzy_join` and :class:`FeatureAugmenter` can now join on numerical columns based on the euclidean distance.
+* :func:`fuzzy_join` and :class:`FeatureAugmenter` can now join on numeric columns based on the euclidean distance.
   :pr:`530` by :user:`Jovan Stojanovic <jovan-stojanovic>`
 
-* :func:`fuzzy_join` and :class:`FeatureAugmenter` can perform many-to-many joins on lists of numerical or string key columns.
+* :func:`fuzzy_join` and :class:`FeatureAugmenter` can perform many-to-many joins on lists of numeric or string key columns.
   :pr:`530` by :user:`Jovan Stojanovic <jovan-stojanovic>`
 
 * :func:`GapEncoder.transform` will not continue fitting of the instance anymore.
@@ -949,7 +948,7 @@ Dirty-cat Release 0.3.0
 Major changes
 -------------
 
-* New encoder: :class:`DatetimeEncoder` can transform a datetime column into several numerical columns
+* New encoder: :class:`DatetimeEncoder` can transform a datetime column into several numeric columns
   (year, month, day, hour, minute, second, ...). It is now the default transformer used
   in the :class:`TableVectorizer` for datetime columns. :pr:`239` by :user:`Leo Grinsztajn <LeoGrin>`
 

doc/_templates/index.html

@@ -147,7 +147,7 @@ <h5 class="feature-title">Inspect it, apply it to new data</h5>
       </div>
       <p class="feature-text-see-also">
         <!-- name of the userguide file on DataOps -->
-        <a href="userguide_data_ops.html">Discover the skrub DataOps →</a>
+        <a href="{{ pathto('auto_examples/data_ops/11_data_ops_intro') }}">Discover the skrub DataOps →</a>
       </p>
     </div>
     <div class="col-md-7">

doc/api_reference.py

@@ -56,13 +56,13 @@
         "sections": [
             {
                 "description": (
-                    "See :ref:`End-to-End pipeline <userguide_tablevectorizer>` for "
-                    "further details. For more flexibility and control to build "
-                    "pipelines, see the :ref:`skrub DataOps <userguide_data_ops>`."
+                    "See :ref:`End-to-End pipeline"
+                    " <user_guide_building_pipeline_index>` for further details. For"
+                    " more flexibility and control to build pipelines, see the"
+                    " :ref:`skrub DataOps <user_guide_data_ops_index>`."
                 ),
                 "autosummary": [
                     "tabular_pipeline",
-                    "tabular_learner",
                     "TableVectorizer",
                     "SelectCols",
                     "DropCols",
@@ -79,7 +79,8 @@
         "sections": [
             {
                 "description": (
-                    "See :ref:`encoding <userguide_encoders>` for further details."
+                    "See :ref:`encoding <user_guide_encoders_index>` for further"
+                    " details."
                 ),
                 "autosummary": [
                     "StringEncoder",
@@ -153,7 +154,7 @@
             {
                 "description": (
                     "Contains method to select columns in a dataframe. See the"
-                    " :ref:`selectors <userguide_selectors>` section for further"
+                    " :ref:`selectors <user_guide_selectors>` section for further"
                     " details."
                 ),
                 "autosummary": [
@@ -186,8 +187,8 @@
         "sections": [
             {
                 "description": (
-                    "Generalizing the scikit-learn pipeline. "
-                    "See :ref:`skrub DataOps <userguide_data_ops>` for further details."
+                    "Generalizing the scikit-learn pipeline. See :ref:`skrub DataOps"
+                    " <user_guide_data_ops_index>` for further details."
                 ),
                 "autosummary": ["var", "X", "y", "as_data_op", "deferred"],
                 "template": "base.rst",

doc/column_level_featurizing.rst

@@ -0,0 +1,18 @@
+.. _user_guide_encoders_index:
+
+Column-level feature extraction
+===============================
+
+Skrub provides various transformers that help with feature engineering numeric,
+datetime and categorical data. The encoders covered in this section convert the
+raw features found in an input dataframe into numeric features that can be used
+directly by machine learning models.
+
+.. include:: includes/big_toc_css.rst
+
+.. toctree::
+   :maxdepth: 3
+
+   modules/column_level_featurizing/feature_engineering_categorical
+   modules/column_level_featurizing/feature_engineering_datetimes
+   modules/column_level_featurizing/robust_scaling

doc/conf.py

@@ -353,7 +353,7 @@
         "skrub Documentation",
         author,
         "skrub",
-        "Prepping tables for machine learning.",
+        "Machine learning with dataframes",
         "Data Science",
     ),
 ]
@@ -397,7 +397,7 @@ def notebook_modification_function(notebook_content, notebook_filename):
             "</div>",
         ]
     )
-
+    # TODO: remove this when we remove KEN embeddings
     if "06_ken_embeddings_example" in notebook_filename:
         message_class = "danger"
         message = (

doc/configuration_and_utils.rst

@@ -0,0 +1,13 @@
+.. _user_guide_configuration_index:
+
+Configuration and dataset utilities
+=========================================
+
+This section covers the skrub global configurations and dataset utilities.
+
+.. toctree::
+   :maxdepth: 2
+
+   modules/configuration_and_utils/customizing_configuration
+   modules/configuration_and_utils/deduplicate_categorical_data
+   modules/configuration_and_utils/fetching_datasets

doc/data_ops.rst

@@ -0,0 +1,71 @@
+.. _user_guide_data_ops_index:
+
+Complex multi-table pipelines with Data Ops
+===========================================
+
+Skrub provides an easy way to build complex, flexible machine learning pipelines.
+There are several needs that are not easily addressed with standard scikit-learn
+tools such as :class:`~sklearn.pipeline.Pipeline` and
+:class:`~sklearn.compose.ColumnTransformer`, and for which the Skrub DataOps offer
+a solution:
+
+- Multiple tables: We often have several tables of different shapes (for
+  example, "Customers", "Orders", and "Products" tables) that need to be
+  processed and assembled into a design matrix ``X``. The target ``y`` may also
+  be the result of some data processing. Standard scikit-learn estimators do not
+  support this, as they expect right away a single design matrix ``X`` and a
+  target array ``y``, with one row per observation.
+- DataFrame wrangling: Performing typical DataFrame operations such as
+  projections, joins, and aggregations should be possible and allow leveraging
+  the powerful and familiar APIs of `Pandas <https://pandas.pydata.org>`_ or
+  `Polars <https://docs.pola.rs/>`_.
+- Hyperparameter tuning: Choices of estimators, hyperparameters, and even
+  the pipeline architecture can be guided by validation scores. Specifying
+  ranges of possible values outside of the pipeline itself (as in
+  :class:`~sklearn.model_selection.GridSearchCV`) is difficult in complex
+  pipelines.
+- Iterative development: Building a pipeline step by step while inspecting
+  intermediate results allows for a short feedback loop and early discovery of
+  errors.
+
+In this section we cover all about the skrub Data Ops, from starting out with a
+simple example, to more advanced concepts like parameter tuning and and pipeline
+validation.
+
+Data Ops basic concepts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 3
+
+   modules/data_ops/basics/what_are_data_ops
+   modules/data_ops/basics/building_data_ops_plan
+   modules/data_ops/basics/using_previews
+   modules/data_ops/basics/direct_access_methods
+   modules/data_ops/basics/control_flow
+   modules/data_ops/basics/data_ops_vs_alternatives
+
+Building a complex pipeline with the skrub Data Ops
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 2
+
+   modules/data_ops/ml_pipeline/applying_ml_estimators
+   modules/data_ops/ml_pipeline/applying_different_transformers
+   modules/data_ops/ml_pipeline/documenting_data_ops_plan
+   modules/data_ops/ml_pipeline/evaluating_debugging_data_ops
+   modules/data_ops/ml_pipeline/using_part_of_data_ops_plan
+   modules/data_ops/ml_pipeline/subsampling_data
+
+Tuning and validating Skrub DataOps plans
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 2
+
+   modules/data_ops/validation/tuning_validating_data_ops
+   modules/data_ops/validation/hyperparameter_tuning
+   modules/data_ops/validation/nested_cross_validation
+   modules/data_ops/validation/nesting_choices_choosing_pipelines
+   modules/data_ops/validation/exporting_data_ops

doc/default_wrangling.rst

@@ -0,0 +1,16 @@
+.. _user_guide_building_pipeline_index:
+
+Wrangling data with good defaults
+=================================
+
+This section covers how to build a predictive pipeline starting from a dataframe.
+The skrub objects described in this section can be used as strong defaults for
+building baseline pipelines, and can be customized for specific use cases.
+
+
+.. toctree::
+   :maxdepth: 3
+
+   modules/default_wrangling/cleaning_dataframes
+   modules/default_wrangling/table_vectorizer
+   modules/default_wrangling/tabular_pipeline

doc/documentation.rst

@@ -1,38 +1,44 @@
+.. |TableReport| replace:: :class:`~skrub.TableReport`
+.. |Cleaner| replace:: :class:`~skrub.Cleaner`
+.. |TableVectorizer| replace:: :class:`~skrub.TableVectorizer`
+.. |tabular_pipeline| replace:: :func:`~skrub.tabular_pipeline`
+
 .. _user_guide:
 
 User Guide
 ==========
-Skrub is a library that eases machine learning with dataframes
-for machine learning.
+Skrub is a library that eases machine learning with dataframes, from exploring
+dataframes to validating a machine-learning pipeline.
+
+The |TableReport| is a powerful data exploration tool, which can be followed by
+data sanitization and feature engineering tools in the |Cleaner| and |TableVectorizer|.
+The |tabular_pipeline| combines the two to build a strong baseline for dataframes.
+
+The skrub :ref:`column-level encoders<user_guide_encoders_index>` can be tweaked by the user for more
+specific needs.
+Various :ref:`multi-column transformers <user_guide_building_pipeline_index>` and the :ref:`selectors API<user_guide_selectors>`
+provide a high degree of control over which columns should be modified.
 
-Starting from rich, complex data stored in one or several dataframes, it helps
-performing the data wrangling necessary to produce a numeric array that is fed
-to a machine-learning model. This wrangling comprises joining tables (possibly
-with inexact matches), parsing structured data such as datetimes from text,
-and extracting numeric features from non-numeric data.
+More complex, multi-table scenarios can make use of the skrub :ref:`Data Ops <user_guide_data_ops_index>`,
+which enable constructing and validating pipelines that involve
+multiple dataframes and hyperparameter tuning.
 
-For those tasks, skrub does not replace pandas or polars. Instead, it
+Skrub does not replace pandas or polars. Instead, it
 leverages the dataframe libraries to provide more high-level building blocks that
 perform the data preprocessing steps that are typically needed in a machine learning
 pipeline.
 
-This guide demonstrates how to resolve various issues using Skrub's features.
-See the examples section for full code snippets.
 
 
 .. include:: includes/big_toc_css.rst
 
 .. toctree::
-   :maxdepth: 2
-
-   userguide_tablereport
-   userguide_encoders
-   userguide_datetimes
-   userguide_tablevectorizer
-   userguide_data_cleaning
-   userguide_selectors
-   userguide_data_ops
-   userguide_data_ops_ml_pipeline
-   userguide_data_ops_validation
-   userguide_joining_tables
-   userguide_utils
+   :maxdepth: 3
+
+   exploring_a_dataframe
+   default_wrangling
+   column_level_featurizing
+   multi_column_operations
+   data_ops
+   configuration_and_utils
+   joining_dataframes

doc/exploring_a_dataframe.rst

@@ -0,0 +1,14 @@
+.. _user_guide_exploring_a_dataframe_index:
+
+Exploring a Dataframe
+=====================
+
+This section covers the :class:`~skrub.TableReport` and how it can be used for exploring
+and understanding your dataframes.
+
+
+.. toctree::
+   :maxdepth: 3
+
+   modules/tablereport/exploring_dataframes_interactively
+   modules/tablereport/finding_correlated_columns