Change StandardScaler to the SquashScaler for the tabular_pipline

[dierickxsimon]

Changing the scaler used for linear models from StandardScaler to SquashingScaler.

[rcap107]

Thanks a lot @dierickxsimon!

In the original issue there was a discussion on the value of max_absolute_value, so I'm paging @GaelVaroquaux or @dholzmueller to ask what should be used here

[dholzmueller]

I would say 5 or 10 might be good for max_absolute_value? maybe go conservative and use 10?

[GaelVaroquaux]

I'd rather go conversative

[rcap107]

Thanks a lot @dierickxsimon! This will be useful.

I left a few comments to improve the docs, we can merge after they are addressed.

Nice job!

[rcap107]

Suggested change

	- A :obj:`~skrub.SquashingScaler` for centering and standard scaling
	- A :obj:`~skrub.SquashingScaler` for centering and scaling

[dholzmueller]

I could relatively easily run some benchmarks to check different max_absolute_value with LinearModel. I'd prefer to do it in ~ two weeks though.

[rcap107]

I could relatively easily run some benchmarks to check different max_absolute_value with LinearModel. I'd prefer to do it in ~ two weeks though.

I think we can wait until then without problem

[rcap107]

@dholzmueller do you think you could run the benchmark this week? otherwise we can always set the value to 10 and open a new pr later to set a different value once we have the results

[dholzmueller]

Yes, I plan to do it this week, sometime after my deadline (which is tomorrow).

[rcap107]

Yes, I plan to do it this week, sometime after my deadline (which is tomorrow).

That's great, thanks a lot

[dholzmueller]

I ran linear regression with tuning and cross-validation ensembling on TALENT-regression (only applying the scalers to numerical features).

• The SquashingScaler is better than standardization, achieving win-rates of >70% vs standardization but only small improvements <2% in terms of average scores.
• small clipping values (3, 5) are (significantly) better in terms of win-rates / ranks / normalized scores, but clipping at 10 is better on average scores because it's much better on few datasets.

I could also run classification experiments with logistic regression easily (but it might take a few hours).

[dholzmueller]

For binary classification on TALENT with cross-entropy loss, I also tested 3, 5, and 10. Smaller values are better in all aggregations (win-rate, normalized scores, ranks, average error, etc.). For all values, the SquashingScaler is better than standardization.

Based on this, I would say we should probably take a smaller value like 3 or 5 unless we want to be good on the few datasets where having a linear function is important.

[rcap107]

I think we can go with 5, the tabular_pipeline is not intended to be the best in all situations, and I think that a value that is better in most scenarios is what we should go for here.

As I've mentioned IRL, @dholzmueller it would be great if we could put out a short post on how we chose the default value to add to the skrub-materials website.

@dierickxsimon I think we can merge after setting the value to 5

[dholzmueller]

Here is a draft for the post, can you fill in title / docs references or whatever is needed?

To compare the SquashingScaler to the StandardScaler for linear/logistic regression, we ran them (with max_abs_value in {3, 5, 10} for the SquashingScaler) on a version of the TALENT benchmark using pytabkit. We noticed the following trends:

• Across aggregation strategies like average score, average normalized score, average rank, win-rate, the SquashingScaler performed better than the StandardScaler for all values of max_abs_value. Typically, variants of the SquashingScaler won on around 70%-75% of datasets vs the StandardScaler, though often only by a small margin on the order of 1% error reduction.
• Smaller values of max_abs_value were often better, with 3 being the best on classification. For regression, smaller values were better in terms of win-rates, average normalized scores, and average ranks, but there were a few datasets where max_abs_value=10 achieved a much lower error. Perhaps this is because the ground-truth function on the dataset was close to linear.
  Based on the observations above, we chose max_abs_value=5 as a compromise.

Details: We used RMSE for regression and cross-entropy for classification, inner 8-fold cross-validation with ensembling, tuning on the TabRepo/TabArena search space (includes L1 or L2 regularization), 60%-40% train-test split, 2 repeats per dataset.

[rcap107]

Perfect, thanks a lot @dierickxsimon for the PR, and @dholzmueller for running the benchmarks!

[rcap107]

Here is a draft for the post, can you fill in title / docs references or whatever is needed?

To compare the SquashingScaler to the StandardScaler for linear/logistic regression, we ran them (with max_abs_value in {3, 5, 10} for the SquashingScaler) on a version of the TALENT benchmark using pytabkit. We noticed the following trends:

• Across aggregation strategies like average score, average normalized score, average rank, win-rate, the SquashingScaler performed better than the StandardScaler for all values of max_abs_value. Typically, variants of the SquashingScaler won on around 70%-75% of datasets vs the StandardScaler, though often only by a small margin on the order of 1% error reduction.
• Smaller values of max_abs_value were often better, with 3 being the best on classification. For regression, smaller values were better in terms of win-rates, average normalized scores, and average ranks, but there were a few datasets where max_abs_value=10 achieved a much lower error. Perhaps this is because the ground-truth function on the dataset was close to linear.
  Based on the observations above, we chose max_abs_value=5 as a compromise.

Details: We used RMSE for regression and cross-entropy for classification, inner 8-fold cross-validation with ensembling, tuning on the TabRepo/TabArena search space (includes L1 or L2 regularization), 60%-40% train-test split, 2 repeats per dataset.

Thanks a lot, this will end on the skrub-materials website, so I'll move the draft there and ping you when it's ready