[ENH] Salute Tensorflow, move to Keras 3

[Ball-Man]

Describe the feature or idea you want to propose

The deep learning side of the toolkit has great ergonomics, bringing sklearn's API and state of the art models close together. However, the Tensorflow dependency is quite inconvenient, forcing an extra Python version constraint on the toolkit, and forcing users on an arbitrary framework.

Describe your proposed solution

The toolkit is actually mainly built on top of the high level Keras API, with minimal pure Tensorflow usage. The latest major release of Keras, Keras 3, leaves the high level API intact, but is backend agnostic (compatible backends include TF, Torch, Jax). Most of the existing DL algorithms in aeon could easily be adapted to this new API by simply replacing the tf.keras calls with pure keras ones. User could then decide which of the supported backends to use based on what is more convenient for their project.

As a proof of concept, I adapted one of the simple DL algorithms (TimeCNNClassifier) to work with Keras 3. Here is the branch: https://github.com/Ball-Man/aeon/tree/keras3_concept
With minimal to no changes, relevant unit tests (specific to this classifier) are working as intended. As a qualitative approach, from the same branch, here you can see the very first example in the deep learning classification notebook working as intended (just the first cell, where Time CNN is used): https://github.com/Ball-Man/aeon/blob/keras3_concept/examples/classification/deep_learning.ipynb

The example works with Python 3.13 and uses Torch as backend.

Describe alternatives you've considered, if relevant

No response

Additional context

Feasibility

As a feasibility study, I inspected all modules of the aeon toolkit, looking for potential obstacles. Here is what I found out from a shallow search:

• All network definitions (aeon.networks), aside from weight normalization, seem to be adaptable by immediate replacement of the tf.keras calls.
• All DL classification models (aeon.classification.deep_learning) and regression models (aeon.regression.deep_learning) seem to be adaptable by immediate replacement of the tf.keras calls.
• All DL clustering models (aeon.clustering.deep_learning). aside from ae fcn and ae resnet, seem to be adaptable by immediate replacement of the tf.keras calls. This is despite the fact that the network definitions of ae fcn and resnet can easily be adapted.
• Rocket GPU can not immediately be adapted by replacing the tf calls. This would require using the new numpy-like interface from Keras 3, keras.ops.
• Some tests and notebooks would require keras.ops or other minor changes (e.g. aeon.networks.tests.test_all_networks).
• All the other problematic modules (weight normalization, ae fcn clustering and ae resnet clustering) should be adatpable using keras.ops.

I believe the value of such an update would be immense. I can work on it if/when needed.

edit: title formatting.

[MatthewMiddlehurst]

Hi, thanks for the suggestion.

This would be a large structural change, but we will discuss it. No guarantees that we agree or that the decision is made quickly because I'm sure it will be split 🙂.

If you wish to continue working on it, these are some concerns that would possibly hold it back currently. I am not super familiar with networks, so it may be trivial:

• How would the backend be set, can it be parameterised and can we default it to Tensorflow to keep the models close to their current state
• Is alternative backends a useful/popular enough user request to justify the ongoing maintenance. Currently, we only have to maintain the one. If anyone has an opinion it would be good to make it known
• How would this work for more complex models which require custom layers or other functionality which can not be generalised
• Would changing the ops change output/performance and how can we verify that. If code which previously runs fine on GPU starts causing problems that's not great

The current system is a bit rigid, but changing that and introducing more complexity in exchange is probably not what we want.
Being able to easily run published algorithms is one of our main goals.

[TonyBagnall]

hi, thank you for the useful comments and suggestions, I dont know too much about it but we are discussing it and will look at your branch

[Ball-Man]

Hi Matthew, Hi Tony,
Thank you for you taking some time to review these ideas!

Yes, it is quite the big change under the hood. Of course, I don't expect quick decisions. If it can help, I'll try and give an answer to Matthew's points. I'm throwing in some facts and some of my personal opinions:

How would the backend be set?

The backend for Keras can be changed setting an environment variable (KERAS_BACKEND). There may be different ways, but it must be done before Keras is actually imported. The Keras docs themselves use os.environ to get this done. The default backend is always Tensorflow (but it's not a hard dependency of the package). Of course, aeon could build on top of it with with some dedicated functions to switch the backend, or it could leave it to the user to change it using the env variable, given that the default is always tf.

How would this work for more complex models?

The keras.ops exposes a clone of the numpy interface, but under the hood it uses the selected backend. Even for more complex models, I don't see operations that may not be implemented through the numpy interface. In general, I don't really think backend specific interfaces are more expressive than that.

Of course, I may be wrong. And of course, users may be used to develop logic using their favorite framework directly. About this, keep in mind that interoperability between the backend specific objects and the Keras objects is always possible. I think this is showcase by the Keras docs here and there, like this notebook, showcasing how to build a custom layer specifically for the Tensorflow backend. My point here is that aeon could use the high level abstractions and be backend agnostic for (hopefully) everything, while leaving open the possibility to develop framework specific implementations to its users.

Would changing the ops change output/performance?

Performance wise, there should not be changes, as long as the Tensorflow backend is used. ops and all the rest of the API (layers, etc) are a wrapper on top of the chosen backend. If we assume the default backend, Tensorflow, then you are not really changing much, just adding some extra function calls (which we know have an impact, but let's say negligible in this context). Changing backend clearly means changing the implementation of these ops, which potentially means varying performance. How can we verify? Benchmarking everything, I suppose.

That said, using keras.ops is definitely the most delicate of the parts of this update. While it's necessary in very limited cases (mostly Rocket GPU, it looks like), the keras ops are mostly a clone of the numpy interface, which is not always identical to the low level Tensorflow interface. This calls for careful checks on the input and ouput shapes of some of the ops, and so on. Of course, unit tests and benchmarking would help a lot.

Of course, in case of performance degradation, it may be best to drop this idea.

Just some considerations

The reason I'm proposing this is that I see a high potential at "relatively low" cost. Switching an entire underlying framework for a project is generally a big deal, potentially not worth the pain. Here I see the chance to simplify dependencies and generalize over multiple frameworks, at a very reasonable cost. Maybe there is simply too much wiring and this switch is not possible, that's it, but it's definitely worth the investigation.

About being able to easily support published models, I absolutely get you. However, that fight is a hard one. Deep learning literature is a big mix of all frameworks, depending on the experience and clemency of the authors. Adapting models for the toolkit in the future may require skill and time in any case, whether you stick with Tensorflow or not. For example, Torch was almost as popular as TF in 2020 already.

Thank you all for your time.

[hadifawaz1999]

Thanks for opening this and for the thoughtful proposal, it’s clear that a lot of care went into outlining the motivations and mapping out the affected modules.

I wanted to share my personal perspective on this, as a user, contributor and maintainer. While I understand the benefits of moving toward a backend agnostic setup via Keras ops, I have some concerns about the trade offs involved, especially regarding long term maintainability and complexity.

A few points that stand out to me:

Risk of unnecessary disruption: Many of the current modules using TensorFlow are functioning well, and shifting away from it would require changes that could introduce instability or/and performance hits especially where Tensorflow ops are providing optimized GPU exec.

Backend inconsistency: Adopting different backends across similar modules (meaning: some using TensorFlow, others Keras 3, others potentially Torch) could make the codebase harder to navigate, test, and contribute to. A consistent backend strategy within a module or category tends to reduce confusion for contributors and users alike.

Implementation complexity: Supporting backend agnostic code often adds an abstraction layer that may not be necessary for aeon’s goals. Unless a module genuinely benefits from switching or supporting multiple backends, the added complexity could outweigh the flexibility gained.

My general comment on this kind of change is to question the need for backend agnostic in an open source project like aeon, and whether the effort required, both now and for future contributions, is truly justified. The goal should be to simplify things not only for users, but also for contributors.

That said, I appreciate the motivation to streamline things where possible. I’m personally more comfortable with incremental, case by case backend decisions where the gains are clear (example: for new modules where a different backend significantly improves development or performance).

Again, thanks for initiating this discussion, it's important we carefully weigh the costs and benefits before making major architectural shifts. We can try testing some estimators, am more than happy to answer any questions regarding the framework of all networks and deep learning and how they're connected together. The removal of tensorflow completely is not something i currently can support as it is still needed, but incremental branches to see how the changes can be smooth is something we can work on step by step.