Skip to content

[MNT] Improve coverage of test framework for ptf-v2 and fix TiDE categorical embedding bug - #2345

Open
harshsomankar123-tech wants to merge 5 commits into
sktime:mainfrom
harshsomankar123-tech:improve-test-coverage-v2
Open

[MNT] Improve coverage of test framework for ptf-v2 and fix TiDE categorical embedding bug#2345
harshsomankar123-tech wants to merge 5 commits into
sktime:mainfrom
harshsomankar123-tech:improve-test-coverage-v2

Conversation

@harshsomankar123-tech

Copy link
Copy Markdown
Contributor

Reference Issues/PRs

Fixes #2328.

What does this implement/fix? Explain your changes.

This PR increases the test coverage of the V2 testing framework by introducing new data scenarios, parameterizing integration tests, adding validation-check tests, and resolving a categorical embedding bug in the TiDE V2 model.

Specifically:

  1. Diverse V2 Data Scenarios:
    • Added make_datasets_v2_without_covariates to verify estimator behaviors on datasets containing only targets and group IDs without any covariates.
    • Added make_datasets_v2_with_categoricals to verify estimators handling categorical features alongside continuous features.
  2. Parameterized Integration Tests:
    • Parameterized V2 model integration tests in test_all_estimators_v2.py across three data scenarios: with_covariates, without_covariates, and with_categoricals.
  3. New Test Validation Cases:
    • Added test_uninitialized_predict_error to assert that calling .predict() on an uninitialized model correctly raises a RuntimeError.
    • Added test_predict_save_to_dir to assert that calling .predict() with an output_dir successfully saves the prediction dictionary to a predictions.pkl file.
  4. Fixed TiDE Categorical Embedding Bug:
    • Updated TslibDataModule and EncoderDecoderTimeSeriesDataModule metadata builders to compute and include "categorical_cardinalities" (derived from the dataset's unique values/max values).
    • Updated TIDE model constructor to automatically resolve categorical cardinalities from the datamodule metadata when embs is not explicitly provided. This prevents the IndexError on categorical feature embedding projection.

What should a reviewer concentrate their feedback on?

  • Review the integration of "categorical_cardinalities" metadata propagation inside TslibDataModule and EncoderDecoderTimeSeriesDataModule.
  • Inspect the fallback mechanism in TIDE.__init__ that dynamically maps these cardinalities into embedding dimensions when embs is not explicitly supplied in model configuration.

Did you add any tests for the change?

Yes:

  • Added parameterized integration tests for all V2 models across the three new and existing data scenarios (with_covariates, without_covariates, and with_categoricals).
  • Added test_uninitialized_predict_error and test_predict_save_to_dir check cases in test_all_estimators_v2.py.

Any other comments?

All V2 estimators (DLinear, TimeXer, TFT, TiDE) were verified to pass the integration and validation test suites successfully (201 total tests passed).

PR checklist

  • The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.
  • Added/modified tests
  • Used pre-commit hooks when committing to ensure that code is compliant with hooks. Install hooks with pre-commit install.
    To run hooks independent of commit, execute pre-commit run --all-files

@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@e7c7965). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2345   +/-   ##
=======================================
  Coverage        ?   87.60%           
=======================================
  Files           ?      171           
  Lines           ?    10157           
  Branches        ?        0           
=======================================
  Hits            ?     8898           
  Misses          ?     1259           
  Partials        ?        0           
Flag Coverage Δ
cpu 87.60% <100.00%> (?)
pytest 87.60% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@harshsomankar123-tech
harshsomankar123-tech marked this pull request as ready for review July 16, 2026 17:31
@harshsomankar123-tech

Copy link
Copy Markdown
Contributor Author

@phoeenniixx @fkiraly could you please take a look!
Thanks!

@phoeenniixx phoeenniixx left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
This is a difficult task! Actually i was thinking of adding scenario classes and updating the test framework in a similar way as in sktime.

(also the tests are failing)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ENH] Improve the coverage of test framework for ptf-v2

2 participants