Skrub Ops vs Apache Hamilton

[jklaise]

Just discovered Skrub and Skrub Ops in particular - the first thing I thought of, this looks quite similar to Apache Hamilton which is a dataflow "micro-orchestrator" (they would call something like Airflow a "macro-orchestrator"). I'm wondering what are the key differences between these two - Hamilton is more general purpose (although marketed towards data practitioners), whereas Skrub Ops seems more specialised towards ML, and primarily tabular data pipelines? I still need to do a deep dive on the Skrub Ops piece.

[jeromedockes]

Hello @jklaise , thanks for the interesting question. As you say the skrub DataOps are specialized for machine-learning:

• They can contain scikit-learn estimators that are fitted to data. This is the crucial point: the pipeline can be fitted to training data, store all the relevant state like model weights etc., then be applied to new data. I'm not familiar with Hamilton at all but from skimming through the docs I only see stateless functions being mentioned -- I may have missed something though.

• They enable cross-validation, by splitting the data into training and testing datasets. Importantly this can be done on a node inside the pipeline, not necessarily at the very beginning.

• They enable easily constructing hyperparameter search spaces. There are often choices to make in our pipeline, like the regularization strength of an estimator, what feature engineering to use, etc. and we want to pick the best hyperparameters based on a prediction score. Skrub allows easily declaring the choices and running the optimization to pick the best option as shown in this example

Moreover skrub DataOps have some aspects that make developing a pipeline iteratively easier: they can compute eager previews of the results, subsample the data for debugging, etc.
They are also tightly integrated with scikit-learn, pandas and polars to ensure that defining simple or complex machine-learning pipelines is easy.