-
Notifications
You must be signed in to change notification settings - Fork 261
DOC - Improving getting started example #1627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
3e44636
864fc1a
9af00c2
18618f3
1ed618e
d6a6b4e
a7637aa
c57299d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -2,69 +2,44 @@ | |||||||||||||
| Getting Started | ||||||||||||||
| =============== | ||||||||||||||
|
|
||||||||||||||
| This guide showcases some of the features of ``skrub``, an open-source package | ||||||||||||||
| that aims at bridging the gap between tabular data stored in Pandas or Polars | ||||||||||||||
| dataframes, and machine-learning models. | ||||||||||||||
|
|
||||||||||||||
| Much of ``skrub`` revolves around simplifying many of the tasks that are involved | ||||||||||||||
| This guide showcases some of the features of skrub. | ||||||||||||||
| Much of skrub revolves around simplifying many of the tasks that are involved | ||||||||||||||
| in pre-processing raw data into a format that shallow or classic machine-learning | ||||||||||||||
| models can understand, that is, numerical data. | ||||||||||||||
|
|
||||||||||||||
| ``skrub`` does this by vectorizing, assembling, and encoding tabular data through | ||||||||||||||
| a number of features that we present in this example and the following. | ||||||||||||||
| Skrub achieves this by vectorizing, assembling, and encoding tabular data through | ||||||||||||||
| the features we present in this example and the following ones. | ||||||||||||||
|
|
||||||||||||||
| .. |TableReport| replace:: :class:`~skrub.TableReport` | ||||||||||||||
| .. |Cleaner| replace:: :class:`~skrub.Cleaner` | ||||||||||||||
| .. |set_config| replace:: :func:`~skrub.set_config` | ||||||||||||||
| .. |tabular_pipeline| replace:: :func:`~skrub.tabular_pipeline` | ||||||||||||||
| .. |TableVectorizer| replace:: :class:`~skrub.TableVectorizer` | ||||||||||||||
| .. |Joiner| replace:: :class:`~skrub.Joiner` | ||||||||||||||
| .. |SquashingScaler| replace:: :class:`~skrub.SquashingScaler` | ||||||||||||||
| .. |DatetimeEncoder| replace:: :class:`~skrub.DatetimeEncoder` | ||||||||||||||
| .. |ApplyToCols| replace:: :class:`~skrub.ApplyToCols` | ||||||||||||||
| .. |StringEncoder| replace:: :class:`~skrub.StringEncoder` | ||||||||||||||
| .. |TextEncoder| replace:: :class:`~skrub.TextEncoder` | ||||||||||||||
| """ | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Downloading example datasets | ||||||||||||||
| # ---------------------------- | ||||||||||||||
| # | ||||||||||||||
| # The :obj:`~skrub.datasets` module allows us to download tabular datasets and | ||||||||||||||
| # demonstrate ``skrub``'s features. | ||||||||||||||
| # | ||||||||||||||
| # .. note:: | ||||||||||||||
| # | ||||||||||||||
| # You can control the directory where the datasets are stored by: | ||||||||||||||
| # | ||||||||||||||
| # - setting in your environment the ``SKRUB_DATA_DIRECTORY`` variable to an | ||||||||||||||
| # absolute directory path, | ||||||||||||||
| # - using the parameter ``data_directory`` in fetch functions, which takes | ||||||||||||||
| # precedence over the envar. | ||||||||||||||
| # | ||||||||||||||
| # By default, the datasets are stored in a folder named "skrub_data" in the | ||||||||||||||
| # user home folder. | ||||||||||||||
|
|
||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Preliminary exploration with the |TableReport| | ||||||||||||||
| # ---------------------------------------------- | ||||||||||||||
| from skrub.datasets import fetch_employee_salaries | ||||||||||||||
|
|
||||||||||||||
| dataset = fetch_employee_salaries() | ||||||||||||||
| employees_df, salaries = dataset.X, dataset.y | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Explore all the available datasets in :ref:`datasets_ref`. | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Preliminary exploration and parsing of data | ||||||||||||||
| # ------------------------------------------------- | ||||||||||||||
| # Typically, the first operations that are done on new data involve data exploration | ||||||||||||||
| # and parsing. | ||||||||||||||
| # To quickly get an overview of a dataframe's contents, use the | ||||||||||||||
| # :class:`~skrub.TableReport`. | ||||||||||||||
| # Here, we also use the :class:`~skrub.Cleaner`, a transformer that cleans the | ||||||||||||||
| # dataframe by parsing nulls and dates, and by dropping "uninformative" columns | ||||||||||||||
| # (e.g., that contain too many nulls, or that are constant). | ||||||||||||||
| # | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| from skrub import Cleaner, TableReport | ||||||||||||||
|
|
||||||||||||||
| TableReport(employees_df) | ||||||||||||||
| # Typically, the first step with new data is exploration and parsing. | ||||||||||||||
| # To quickly get an overview of a dataframe's contents, use the |TableReport|. | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # From the Report above, we can see that there are datetime columns, so we use the | ||||||||||||||
| # :class:`~skrub.Cleaner` to parse them. | ||||||||||||||
| from skrub import TableReport | ||||||||||||||
|
|
||||||||||||||
| employees_df = Cleaner().fit_transform(employees_df) | ||||||||||||||
| TableReport(employees_df) | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
|
|
@@ -80,11 +55,27 @@ | |||||||||||||
| # | ||||||||||||||
| # .. _example reports: https://skrub-data.org/skrub-reports/examples/ | ||||||||||||||
| # .. _demo: https://skrub-data.org/skrub-reports/ | ||||||||||||||
| # | ||||||||||||||
| # From the Report above, we can see that there are datetime columns, so we use the | ||||||||||||||
| # |Cleaner| to parse them. | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe, we could simplify this text here because it is redundant with the next section. I would propose a broader statement than focusing on the datetime. For instance, we could state that after reading a table, it is common that the dtype are not inferred properly or contains unnecessary columns and that we use the And in the next section, we point out specifically to the datetime column. |
||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Sanitizing data with the |Cleaner| | ||||||||||||||
|
rcap107 marked this conversation as resolved.
|
||||||||||||||
| # ---------------------------------- | ||||||||||||||
| # Here, we use the |Cleaner|, a transformer that cleans the | ||||||||||||||
| # dataframe by parsing nulls and dates, and by dropping "uninformative" columns | ||||||||||||||
| # (e.g., columns with too many nulls or that are constant). | ||||||||||||||
| # | ||||||||||||||
|
|
||||||||||||||
| from skrub import Cleaner | ||||||||||||||
|
|
||||||||||||||
| employees_df = Cleaner().fit_transform(employees_df) | ||||||||||||||
| TableReport(employees_df) | ||||||||||||||
|
rcap107 marked this conversation as resolved.
|
||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # It is also possible to tell ``skrub`` to replace the default pandas & polars | ||||||||||||||
| # displays with ``TableReport`` by modifying the global config with | ||||||||||||||
| # :func:`~skrub.set_config`. | ||||||||||||||
| # It is also possible to tell skrub to replace the default pandas & polars | ||||||||||||||
| # displays with |TableReport| by modifying the global config with | ||||||||||||||
| # |set_config|. | ||||||||||||||
|
|
||||||||||||||
| from skrub import set_config | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -103,8 +94,8 @@ | |||||||||||||
| # Easily building a strong baseline for tabular machine learning | ||||||||||||||
| # -------------------------------------------------------------- | ||||||||||||||
| # | ||||||||||||||
| # The goal of ``skrub`` is to ease tabular data preparation for machine learning. | ||||||||||||||
| # The :func:`~skrub.tabular_pipeline` function provides an easy way to build a simple | ||||||||||||||
| # The goal of skrub is to ease tabular data preparation for machine learning. | ||||||||||||||
| # The |tabular_pipeline| function provides an easy way to build a simple | ||||||||||||||
| # but reliable machine learning model that works well on most tabular data. | ||||||||||||||
|
|
||||||||||||||
|
|
||||||||||||||
|
|
@@ -119,8 +110,8 @@ | |||||||||||||
|
|
||||||||||||||
|
rcap107 marked this conversation as resolved.
|
||||||||||||||
| # %% | ||||||||||||||
| # To handle rich tabular data and feed it to a machine learning model, the | ||||||||||||||
| # pipeline returned by :func:`~skrub.tabular_pipeline` preprocesses and encodes | ||||||||||||||
| # strings, categories and dates using the :class:`~skrub.TableVectorizer`. | ||||||||||||||
| # pipeline returned by |tabular_pipeline| preprocesses and encodes | ||||||||||||||
| # strings, categories and dates using the |TableVectorizer|. | ||||||||||||||
| # See its documentation or :ref:`sphx_glr_auto_examples_01_encodings.py` for | ||||||||||||||
| # more details. An overview of the chosen defaults is available in | ||||||||||||||
| # :ref:`user_guide_tabular_pipeline`. | ||||||||||||||
|
|
@@ -130,11 +121,11 @@ | |||||||||||||
| # Assembling data | ||||||||||||||
| # --------------- | ||||||||||||||
| # | ||||||||||||||
| # ``skrub`` allows imperfect assembly of data, such as joining dataframes | ||||||||||||||
| # on columns that contain typos. ``skrub``'s joiners have ``fit`` and | ||||||||||||||
| # Skrub allows imperfect assembly of data, such as joining dataframes | ||||||||||||||
|
rcap107 marked this conversation as resolved.
Outdated
|
||||||||||||||
| # on columns that contain typos. Skrub's joiners have ``fit`` and | ||||||||||||||
| # ``transform`` methods, storing information about the data across calls. | ||||||||||||||
| # | ||||||||||||||
| # The :class:`~skrub.Joiner` allows fuzzy-joining multiple tables, each row of | ||||||||||||||
| # The |Joiner| allows fuzzy-joining multiple tables, where each row of | ||||||||||||||
| # a main table will be augmented with values from the best match in the auxiliary table. | ||||||||||||||
| # You can control how distant fuzzy-matches are allowed to be with the | ||||||||||||||
| # ``max_dist`` parameter. | ||||||||||||||
|
|
@@ -169,32 +160,32 @@ | |||||||||||||
| joiner.fit_transform(airports) | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Information about countries have been added, even if the rows aren't exactly matching. | ||||||||||||||
| # Information about countries has been added, even if the rows aren't exactly matching. | ||||||||||||||
| # | ||||||||||||||
| # ``skrub`` allows to aggregate multiple tables according to various strategies: you | ||||||||||||||
| # can see other ways to join multiple tables in :ref:`userguide_joining_tables`. | ||||||||||||||
| # Skrub allows you to aggregate multiple tables according to various strategies. You | ||||||||||||||
| # can see other ways to join multiple tables in :ref:`user_guide_joining_dataframes`. | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Encoding any data as numerical features | ||||||||||||||
| # --------------------------------------- | ||||||||||||||
| # | ||||||||||||||
| # Tabular data can contain a variety of datatypes, ranging from numerical, to | ||||||||||||||
| # datetimes, to categories, strings, and text. Encoding features in a meaningful | ||||||||||||||
| # way requires a lot of effort and is a major part of the feature engineering | ||||||||||||||
| # process that is required to properly train machine learning models. | ||||||||||||||
| # Tabular data can contain a variety of datatypes, from numerical to | ||||||||||||||
| # datetimes, categories, strings, and text. Encoding features in a meaningful | ||||||||||||||
| # way requires significant effort and is a major part of the feature engineering | ||||||||||||||
| # process required to properly train machine learning models. | ||||||||||||||
| # | ||||||||||||||
| # ``skrub`` helps with this by providing various transformers that automatically | ||||||||||||||
| # Skrub helps with this by providing various transformers that automatically | ||||||||||||||
| # encode different datatypes into ``float32`` features. | ||||||||||||||
| # | ||||||||||||||
| # For **numerical features**, the :class:`~skrub.SquashingScaler` applies a robust | ||||||||||||||
| # For **numerical features**, the |SquashingScaler| applies a robust | ||||||||||||||
| # scaling technique that is less sensitive to outliers. Check the | ||||||||||||||
| # :ref:`relative example <sphx_glr_auto_examples_11_squashing_scaler.py>` | ||||||||||||||
| # :ref:`relative example <sphx_glr_auto_examples_10_squashing_scaler.py>` | ||||||||||||||
| # for more information on the feature. | ||||||||||||||
| # | ||||||||||||||
| # For **datetime columns**, ``skrub`` provides the :class:`~skrub.DatetimeEncoder` | ||||||||||||||
| # For **datetime columns**, skrub provides the |DatetimeEncoder| | ||||||||||||||
| # which can extract useful features such as year, month, day, as well as additional | ||||||||||||||
| # features such as weekday or day of year. Periodic encoding with trigonometric | ||||||||||||||
| # or spline features is also available. Refer to the :class:`~skrub.DatetimeEncoder` | ||||||||||||||
| # or spline features is also available. Refer to the |DatetimeEncoder| | ||||||||||||||
| # documentation for more detail. | ||||||||||||||
| # | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -211,10 +202,10 @@ | |||||||||||||
| data = Cleaner().fit_transform(data) | ||||||||||||||
| TableReport(data) | ||||||||||||||
| # %% | ||||||||||||||
| # ``skrub`` transformers are applied column-by-column, but it is possible to use | ||||||||||||||
| # the :class:`~skrub.ApplyToCols` meta-transformer to apply a transformer to | ||||||||||||||
| # Skrub transformers are applied column-by-column, but it's possible to use | ||||||||||||||
| # the |ApplyToCols| meta-transformer to apply a transformer to | ||||||||||||||
| # multiple columns at once. Complex column selection is possible using | ||||||||||||||
| # :ref:`skrub's column selectors <userguide_selectors>`. | ||||||||||||||
| # :ref:`skrub's column selectors <user_guide_selectors>`. | ||||||||||||||
|
|
||||||||||||||
| from skrub import ApplyToCols, DatetimeEncoder | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -224,10 +215,10 @@ | |||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Finally, when a column contains **categorical or string data**, it can be | ||||||||||||||
| # encoded using various encoders provided by ``skrub``. The default encoder is | ||||||||||||||
| # the :class:`~skrub.StringEncoder`, which encodes categories using | ||||||||||||||
| # encoded using various encoders provided by skrub. The default encoder is | ||||||||||||||
| # the |StringEncoder|, which encodes categories using | ||||||||||||||
| # `Latent Semantic Analysis (LSA) <https://scikit-learn.org/stable/modules/decomposition.html#about-truncated-svd-and-latent-semantic-analysis-(lsa)>`_. | ||||||||||||||
| # It is a simple and efficient way to encode categories, and works well in | ||||||||||||||
| # It is a simple and efficient way to encode categories and works well in | ||||||||||||||
| # practice. | ||||||||||||||
|
|
||||||||||||||
| data = pd.DataFrame( | ||||||||||||||
|
|
@@ -243,22 +234,22 @@ | |||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # If your data includes a lot of text, you may want to use the | ||||||||||||||
| # :class:`~skrub.TextEncoder`, | ||||||||||||||
| # |TextEncoder|, | ||||||||||||||
| # which uses pre-trained language models retrieved from the HuggingFace hub to | ||||||||||||||
| # create meaningful text embeddings. | ||||||||||||||
| # See :ref:`userguide_encoders` for more details on all the categorical encoders | ||||||||||||||
| # provided by ``skrub``, and :ref:`sphx_glr_auto_examples_01_encodings.py` for a | ||||||||||||||
| # See :ref:`user_guide_encoders_index` for more details on all the categorical encoders | ||||||||||||||
| # provided by skrub, and :ref:`sphx_glr_auto_examples_01_encodings.py` for a | ||||||||||||||
| # comparison between the different methods. | ||||||||||||||
|
|
||||||||||||||
| # %% | ||||||||||||||
| # Advanced use cases | ||||||||||||||
| # ---------------------- | ||||||||||||||
| # If your use case involves more complex data preparation, hyperparameter tuning, | ||||||||||||||
| # or model selection, if you want to build a multi-table pipeline that requires | ||||||||||||||
| # assembling and preparing multiple tables, or if you want to make sure that the | ||||||||||||||
| # data preparation can be reproduced exactly, you can use the ``skrub`` Data Ops, | ||||||||||||||
| # a powerful framework which provides tools to build complex data processing pipelines. | ||||||||||||||
| # See the relative :ref:`user guide <userguide_data_ops>` and the | ||||||||||||||
| # assembling and preparing multiple tables, or if you want to ensure that the | ||||||||||||||
| # data preparation can be reproduced exactly, you can use the skrub Data Ops, | ||||||||||||||
| # a powerful framework that provides tools to build complex data processing pipelines. | ||||||||||||||
| # See the related :ref:`user guide <user_guide_data_ops_index>` and the | ||||||||||||||
| # :ref:`data_ops_examples_ref` | ||||||||||||||
| # examples for more details. | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -267,11 +258,11 @@ | |||||||||||||
| # ---------- | ||||||||||||||
| # | ||||||||||||||
| # We have briefly covered pipeline creation, vectorizing, assembling, and encoding | ||||||||||||||
| # data. We presented the main functionalities of ``skrub``, but there is much | ||||||||||||||
| # more to it! | ||||||||||||||
| # data. We presented the main functionalities of skrub, but there is much | ||||||||||||||
| # more to explore! | ||||||||||||||
| # | ||||||||||||||
| # Please refer to our :ref:`user_guide` for a more in-depth presentation of | ||||||||||||||
| # ``skrub``'s concepts, or visit our | ||||||||||||||
| # skrub's concepts, or visit our | ||||||||||||||
| # `examples <https://skrub-data.org/stable/auto_examples>`_ for more | ||||||||||||||
| # illustrations of the tools that we provide! | ||||||||||||||
| # | ||||||||||||||
Uh oh!
There was an error while loading. Please reload this page.