Skip to content

Improve error message and add FAQ entry for stratified CV with Dataset (#1122) - #1152

Open
Sweeyya wants to merge 1 commit into
skorch-dev:masterfrom
Sweeyya:fix-1122-dataset-stratify
Open

Improve error message and add FAQ entry for stratified CV with Dataset (#1122)#1152
Sweeyya wants to merge 1 commit into
skorch-dev:masterfrom
Sweeyya:fix-1122-dataset-stratify

Conversation

@Sweeyya

@Sweeyya Sweeyya commented Aug 8, 2026

Copy link
Copy Markdown

Why

skorch's docs say you can pass a PyTorch or skorch Dataset directly to
.fit(). Doing so currently fails with a confusing error
(ValueError: Stratified CV requires explicitly passing a suitable y),
because the default validation split logic (ValidSplit, stratified by
default on NeuralNetClassifier) can't pull class labels out of a Dataset
object the way it can from raw arrays. Users hitting this had no indication
of what to do next (see the StackOverflow thread linked in the issue).

Per @BenjaminBossan's guidance on the issue, this fix does not guess y
from the dataset or silently disable stratification. It only makes the
existing error message and docs more actionable.

What changed

  • ValidSplit.__call__ now detects when the failing dataset is a
    skorch.dataset.Dataset or torch.utils.data.TensorDataset and extends
    the error message to name the type and point to three workarounds:
    predefined_split(), train_split=ValidSplit(5, stratified=False), or
    train_split=None. Any other type still gets the original message,
    unchanged.
  • Added a FAQ entry to docs/user/FAQ.rst covering this failure and the
    same three workarounds.
  • Added a CHANGES.md entry under Unreleased.

Closes #1122

Acceptance criteria

  • Tests added (skorch/tests/test_dataset.py: Dataset, TensorDataset,
    and generic array cases)
  • All tests passing (python -m pytest: 1648 passed, 104 skipped,
    1 xfailed)
  • Follows existing style and conventions in dataset.py and the test
    files
  • No breaking changes. Behavior is unchanged except for the two newly
    detected dataset types, where the message is strictly more
    informative

Before and after

Before:

ValueError: Stratified CV requires explicitly passing a suitable y.

After:

ValueError: Stratified CV requires explicitly passing a suitable y. You
passed a skorch.dataset.Dataset as dataset, which skorch cannot introspect
to obtain labels for stratification. To resolve this, either wrap your
validation data with skorch.helper.predefined_split and pass it as
train_split, disable stratification with train_split=ValidSplit(5,
stratified=False), or disable the internal validation split entirely with
train_split=None.

Testing

  • Ran the full test suite locally: 1648 passed, 104 skipped, 1 xfailed
  • Updated the pre-existing test_net.py test that hard coded the old
    message text
  • Manually reran the reproduction script from the issue and confirmed the
    new message
  • Built the docs locally with sphinx-build to confirm the FAQ entry
    renders correctly

@BenjaminBossan, ready for review whenever you have a chance. Thank you for
scoping this out and for the mentorship on my first contribution here.

Fixes the unclear ValueError raised by ValidSplit when a skorch Dataset
or TensorDataset is passed to .fit() with y=None and stratified=True.
The error now names the dataset type and points to the documented
workarounds (predefined_split, stratified=False, or train_split=None)
instead of a generic message.

Refs skorch-dev#1122

@BenjaminBossan BenjaminBossan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR to improve the error message. There is one merge conflict, could you please resolve it? Moreover, there is one more option for users if they encounter this error. This would affect the docs and also the error message.

Comment thread docs/user/FAQ.rst
.. code:: python

net = NeuralNetClassifier(MyModule, train_split=None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yet another option is to explicitly pass y, if the user can somehow retrieve it. Something like this would work: net.fit(dataset, y=y)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NeuralNetClassifier doesn't fit to a skorch.dataset.Dataset with the default train_split

2 participants