sktime/sklearn integration?

[fkiraly]

@anniegbryant, @benfulcher, I would like to congratulate you to this nice package, I really like the concept and it is quite nicely designed! There are also a lot of useful methods collected! Nice.

Now imo the next "big" question is integrability with the wider modelling ecosystem, e.g., can I use the pairwise time series metrics as components in sktime or sklearn. Where with "I", of course, I mean the wider user ecosystem.

Currently, I think there are a few blockers, but would you be interested to resolve them together?

Two main points imo from the codebase review:

• sklearn interoperable interfaces expect a few things such as __init__ signature related, and availability of get_params, set_params. You can get this for free by inheriting from scikit-base base classes, of course that's not the only way to satisfy the interface requirements.
• sktime has related classes which you could adopt or adapt, e.g., the BasePairwiseTransformerPanel. Options could involve, writing an adapter in sktime, or using the class in pyspi, the latter would give you testing for free by using check_estimator. Or, writing your own base class template based on scikit-base that marries the current interface definition with sklearn and sktime expectations.

Side points but synergistic points:

• testing could - and should - be more systematic for reliable use, e.g., CI on operating system and python version combinations. Happy to help setting this up if we set aside some time. Of course, the "sktime interface" option would take care of this as part of sktime, although bugfixing could become more clunky as we would have to push bug reports upstream (like in pycatch22).
• a good object/estimator search utility might be nice for the user, there are a lot of implemented objects! We could lift some components from sktime or skbase here.

[benfulcher]

Thanks @fkiraly for the kind words and enthusiasm! The compliments are best directed at @olivercliff who did the software dev for this project.

I personally don't have the time or python expertise to contribute much to software expansion efforts, but @olivercliff may be able to weigh in on this point. It's possible @anniegbryant may be able to help somewhat but will leave to her…

Ultimately would be great to have a student or keen software dev join the team—e.g., could be a good Google Summer of Code project. Will keep you posted…

[olivercliff]

Hi @fkiraly, glad to hear you like it! In fact, I designed the code with future integration of the sktime/sklearn framework in mind, which is probably why certain parts of it feel familiar (and hopefully the integration would not be too much of a hassle).

Your two main points, imo, would not only allow integration with sklearn/sktime, but also significantly improve the readability and usability of the standalone package. My thoughts after having a quick look at the code you referenced:

• The sklearn-base classes might be the more difficult aspect to implement, as it looks like it requires pyspi to handle data differently - is that correct? Many methods store certain results directly in the data object in order to extract statistics from these results later on; otherwise the computation time blows out significantly. I imagine there is a simpler way to achieve this using the sklearn framework but I have not come across it yet.
• Adopting the BasePairwiseTransformerPanel sounds achievable in a shorter period of time. Moreover, the arguments cover all cases that the methods in pyspi require (e.g., multivariate or bivariate) and extend in useful directions (e.g., handles NaN or not).

I am unfortunately quite short on time these days and don't work directly on the codebase anymore, so I think the idea of a GSoC project, as @benfulcher suggests, is a great way forward.

[bruAristimunha]

Hey @fkiraly, @benfulcher, @olivercliff!

Has there been any progress on the Google Summer of code? I might be interested in doing the sklearn integration, but I didn't find the project in the sktime projects list.

[fkiraly]

@bruAristimunha, apologies, I did not see this post!

Yes, we have been selected for GSoC 2024, and this would have been an excellent topic!

Unfortunately, the application deadline was April 2.

We could still work on this though?
We have a great (unpaid) mentoring programme!
https://github.com/sktime/mentoring/tree/main

Or perhaps @benfulcher has an academic internship available?

[fkiraly]

@benfulcher, @olivercliff, apologies, I missed the more recent discusion in my inbox.

Let us know if further collaboration here is of interest, we are going to kick off our summer workstreams in May.

[bruAristimunha]

Hi @fkiraly,

Unfortunately, doing unpaid work this way is not very interesting for me, but I appreciate the answer. It would be a "hard" project, with a lot of code, and a lot of time commitment.

Maybe next year if sktimes is selected.

[fkiraly]

@bruAristimunha, we did get selected 2024, getting paid would have required an application by April 2. Sorry that I did not see this.

How about an alternative idea then, @benfulcher: you (or someone from your team) could present pyspi in one of the sktime meet-ups, these are Fridays 4pm UTC at the moment. There is one free slot on April 26, and most of June is also available.

The aim would be to present pyspi and a potential integration project, I'm sure many members of the community and adjacent listeners would find this interesting, someone might take that up.

[benfulcher]

Ok sounds good thanks for the invite—would be happy to present pyspi. @jmoo2880 has done a bunch of work on it recently, getting it into a nice format (e.g., now pip installable). Trouble is that 4pm UTC seems to be 2am Sydney time, so it's not going to work at that timing.

[Spinachboul]

@benfulcher
Hi! I am one of the contributors of Sktime. The project seems to be interesting and it aligns well with my ideology, I am a mentee under @fkiraly and I have worked on a couple of interfacing use-cases before and would love to take this one on as a long term project in the form of an internship.
So wanted to know if you have any openings this year (early autumn time would be preferable)
Thanks!!

[benfulcher]

Hi @Spinachboul thanks for the message. Can you email me ben.fulcher@sydney.edu.au? Not sure what you had in mind, but happy to discuss.
Ben

[Spinachboul]

@benfulcher
Thanks for the reply! I have just sent an email describing idea there in brief!
Hope to hear soon..
Thanks again!!

[Spinachboul]

@benfulcher @fkiraly @olivercliff @joshuabmoore
I have been in regular contact with @fkiraly and @benfulcher about the scope of this project and since Ben believes this is a valuable addition to the pyspi and sktime ecoystem, I would like to describe the synopsis for the project below.

This is the proposal for which I have officially submitted the application for ESOC via Sktime.

Project Explaination

This project integrates pyspi with sktime and scikit-learn to create a compehensive time series analysis framework.

Core Components
l. Preprocessing Implementation
1.1 Detrending Mechanism

• Implement a linear trend removal technique by subtracting the mean from the data to inear trends.
• Develop a modular approach similar to tsfresh transformers that isolates trend components.
• Support multiple detrending methods (linear, polynomial) based on dataset characteristics.

1.2 Differencing Implementation

• Create transformation components akin to the "Integrated" component in ARIMA models.
• Implement configurable differencing orders (first-order, second-order).
• Calculate differences using:
  - First-order differencing:
  - Second-order differencing:

1.3 Normalization Implementation

• Implement Z-score normalization by subtracting the mean and dividing by the standard deviation.
• Develop Min-Max scaling functionality.
• Design normalization to support both global and local (windowed) approaches.
• Include functionality inspired by flight data normalization techniques to enhance
  time series comparability.

2. scikit-learn Compatibility

• Structure preprocessing steps as transformers that implement fit(), transform(),
  and fit_transform() methods.
• Ensure integration with scikit-learn pipelines following best practices seen in
  tsfresh examples.
• Properly handle time series data structures with appropriate indexing.

3. Pairwise Metrics Calculation

• Implement distance metrics for time series, including Dynamic Time Warping (DTW), Euclidean distance, and correlation-based metrics.
• Create efficient similarity matrix generators for large-scale time series data.
  Support both symmetric and asymmetric metrics for comparing time series.

4. Metric Recommender
4.1 Recommendation System Implementation
4.1.1 Data Characteristics Analysis

• Implement functionality to detect seasonality, trends, and stationarity in time
  series.
• Use statistical tests to identify distribution properties.
• Design feature extraction inspired by tsfresh to identify key characteristics.

4.1.2 Task-Specific Recommendations

• Build a knowledge base mapping data characteristics to appropriate metrics.
• Develop decision rules for selecting metrics based on:
• Time series length and sampling frequency.
  • Presence of outliers or missing data.
  • Intended modeling task (classification, clustering, forecasting).
  • Utilize transformer architecture principles to understand temporal dependencies.

4.1.3 Adaptation Mechanism

• Implement feedback loops to evaluate metric performance.
• Create weighted metric ensembles when no single metric is clearly optimal.
• Design progressive refinement techniques based on interim results.

5. PyspiPipeline: Workflow Optimization
5.1 Pipeline Optimization Implementation
5.1.1 Complexity Analysis

• Develop heuristics to estimate computational requirements based on data volume and characteristics.
• Create preprocessing profilers to measure the impact of each transformation step.
• Draw inspiration from sktime’s pipeline architectures for time series tasks.

5.1.2 Pipeline Reordering

• Implement a dependency graph for transformation steps.
• Design logic to identify redundant calculations and optimize execution order.
• Ensure transformation steps maintain data integrity through pipeline validation mechanisms.

5.1.3 Computational Cost Management

• Monitor resource usage during execution.
• Implement adaptive sampling strategies for large datasets.
• Develop caching mechanisms to store intermediate results efficiently.

5.1.4 Adaptive Metric Selection

• Implement an online assessment of metric relevance during execution.
• Create dynamic weighting of multiple metrics based on intermediate results.
• Design confidence estimation techniques inspired by transformer-based prediction models.

6. Data Flow Implementation
6.1 Data Ingestion Layer

• Develop flexible input adapters for various time series formats.
• Implement validation checks for data integrity and format consistency.
• Utilize sktime’s unified interface for handling time series data.

6.2 Preprocessing Orchestration

• Implement sequential and conditional application of transformations.
• Develop preprocessing pipelines using scikit-learn’s Pipeline structure.
• Validate preprocessing steps to ensure time series properties are maintained.

6.3 Metric Calculation Layer

• Enable parallel computation of selected metrics for efficiency.
• Implement efficient storage solutions for metric results.
• Design visualization components for comparing metric outputs.

---

This is the high level idea for the project, and of course each section needs an extensive detailing which can be discussed over the course of the timeline with the mentors @fkiraly , @benfulcher and @olivercliff

Thus any kind of feedback, or suggestions on the feasability, architecture adaptibilty or maybe any questions/doubts about the diagram (or explaination) is highly appreciated...
Thanks a lot!!

[fkiraly]

@Spinachboul, thanks for your thoughts! A lot of ideas clearly went into this.

However, I would advise: "start simple". It feels very overengineered - you are trying to build an all-bells-and-whistles solution, which is (a) unrealistic to do from scratch, and (b) perhaps not needed, and in the worst case adversely impacting user experience.

(I suspect this mirrors your enthusiasm for the project :-) which is probably good thing)

Where I would start is a simple, public and unified API for the SPI. Currently there is only an internal API which is not very searchable or accessible to users, except through the all-in-one accessor.

I would base this on skbase BaseObject, and try to design a base class for SPI that is as simple as possible (while still useful).

Meta-learning, pre-processing, and all that could be added later, or might also be out of scope.

"keep it crisp and prototype fast" would be my more general advice.

[Spinachboul]

@fkiraly
Thanks a lot for your feedback! Yes the ideas I proposed might be due to my excitement for working on this project, but while keeping the enthusiasm alive, I will try start simple and crisp, and as you mentioned, basing this upon skbase is probably a good idea to begin with. I'll focus on the fundamentals first and later we can discuss about the additional features which can be implemented after extensive tech discussions..
Thanks again!!!