[BUG] Arsenal predict not working properly?!

[tim-bsm]

Describe the issue

I was using sktime for Arsenal but because of the split - the reasons stated by Tony in this issue - and because of wanting to test, whether aeon is faster than sktime, I wanted to try aeon. As aeon is a fork of sktime, the basics are the same so it was kind of easy to switch to aeon. Well, that's what I thought. Turns out that the predict of the Arsenal algorithm now uses another type. I'm still trying to use the predict and can't get it to work.

For Arsenal, I'm fitting it in a different file and save its outcome with the joblib.dump function. The generated file is then being loaded back in the main script in which I'm trying to predict the class of my measurement. This always gives me the following error:

TypeError: No matching definition for argument type(s) array(float64, 1d, C), array(float64, 1d, C), Tuple(array(int32, 1d, C), array(int32, 1d, C), array(float32, 1d, C)), Tuple(array(int32, 1d, C), array(int32, 1d, C), array(float32, 1d, C)), int64

I'm trying to predict it like that:

acf = joblib.load(arsenal.joblib)
data = np.array(['0.0', '0.0', '0.0', ..., '0.0'], dtype=np.float64)
acf.predict(data)

The list contains 200 strings all being 0.0

After that I went to the examples directory in the aeon git-repo and tried to use the example written there. While reading this example I noticed that the line in "In [2]:" should be motions_test, motions_test_labels = load_basic_motions(split="test") instead of motions_test, motions_test_labels = load_basic_motions(split="train"), if I'm not completely misunderstanding the example. But this just as a side node.
So I tried to use the example with which I didn't get the above mentioned error anymore but instead a new one:

Terminating: fork() called from a process already using Gnu OpenMP, this is unsafe.

I assume that this is a direct cause of me using Pythons multiprocessing module to run the prediction parallel with another module?!

Is this a bug or am I doing something completely wrong? As I said, the example works great in sktime.

Suggest a potential alternative/fix

No response

Additional context

No response

[TonyBagnall]

hi and welcome to aeon. This is related to the use of joblib, but not really sure whats going on. As a sanity check, the code below all works fine. Have you got a complete code example so we can give it a go? Most likely someone else here will have a much better idea. It could be numba related, typing often is

thanks for the typo spot, we would appreciate a PR to fix it.

from aeon.classification.convolution_based import Arsenal
import numpy as np

X = np.random.random((10,20))
X2 = np.random.random((10,20))
y = np.array([0,1,0,1,0,1,0,1,0,1])
X3 = np.random.random((10,1,20))
X4 = np.random.random((10,3,20))
afc = Arsenal()
afc.fit(X,y)
print(afc.predict(X2))
afc.fit(X3,y)
print(afc.predict(X2))
afc.fit(X4,y)
print(afc.predict(X4))

[tim-bsm]

Hey @TonyBagnall thanks for the quick response. Here's a quick example. This only shows my problem with the types, as this should be the first one to solve.

from aeon.classification.convolution_based import Arsenal
import numpy as np
import joblib

def save_arsenal():
  print("saving")
  X = np.random.random((10, 20))
  y = np.array([0, 1, 2, 3, 3, 1, 0, 0, 2, 1])
  afc = Arsenal(n_estimators=15, n_jobs=-1, num_kernels=5000, random_state=42, rocket_transform='multirocket')
  afc.fit(X, y)
  joblib.dump(afc, "test.joblib")

def get_arsenal():
  print("getting")
  afc: Arsenal = joblib.load("test.joblib")

  # working
  X2 = np.random.random((10, 20))
  print(afc.predict(X2))

  # create new data
  tmp_meas = ['0.0']
  for _ in range(199):
    tmp_meas.append('0.0')
  print(tmp_meas)
  data = np.array([tmp_meas], dtype=np.float64) # this must be wrong?!
  print(data)

  # not working but does with sktime
  print(afc.predict(data))

save_arsenal()
get_arsenal()

[MatthewMiddlehurst]

Running this now. will take a look and get back to you.

[MatthewMiddlehurst]

So, it is crashing on print(afc.predict(data)) for me. But this makes sense since it is trained on a dataset with 20 series length then data has 200 series length. Thats not the error being raised however, so another issue to work through 🙂.

I think you have found an edge case where the transform does not accept a singular series, as this works:

  for _ in range(19):
    tmp_meas.append('0.0')
  data = np.array([tmp_meas, tmp_meas], dtype=np.float64)

I think if we revert the changes made to the class in #838 it should work @TonyBagnall?

[tim-bsm]

To be fair, I just adopted the code of @TonyBagnall to create a code quickly that produces my error. But yeah, normally the dataset will be around 200 - although not always the complete same amount of data (+-20?). But since this difference of +-20 works with sktime, it should with aeon as well, as this is the same base, right?

So if I understood you correctly, this is a bug which can be fixed on your side, right? If so, can it be done fairly simple or does it take long until it will be released?

[MatthewMiddlehurst]

Yeah we can fix this on our side. Though may take a while to make a release, since we have a lot of other changes currently that we don't want to rush through.

The estimator does not accept unequal length data, so any series length change from the original should not be accepted. It's a bit strange it is getting to the transform where your code is crashing, and I will create an issue to throw and appropriate error when it sees unequal length where that is not allowed. It does not look like the version I wrote originally in the other package has changed to accept unequal length, so I assume its a bug there as well.

Regarding your second error, I can run the notebook without any issues on Windows. I don't know this is a conflict with something else, I would need an example really.

[MatthewMiddlehurst]

Though saying that, it runs the below without issues (I think). Maybe we can configure it to accept unequal length relatively easily. We really need to revamp this code 🙂.

  for _ in range(199):
    tmp_meas.append('0.0')
  data = np.array([tmp_meas, tmp_meas], dtype=np.float64)

Purely implementation though, I'm not sure if the rocket transformation will be outputting any useful features for comparing 20 series length vs 200

[tim-bsm]

For the second part with multiprocessing, the following is crashing (i adopted the temporary fix of @MatthewMiddlehurst):

from aeon.classification.convolution_based import Arsenal
import numpy as np
import joblib
import multiprocessing as mp

def save_arsenal():
  print("saving")
  X = np.random.random((10, 20))
  y = np.array([0, 1, 2, 3, 3, 1, 0, 0, 2, 1])
  afc = Arsenal(n_estimators=15, n_jobs=-1, num_kernels=5000, random_state=42, rocket_transform='multirocket')
  afc.fit(X, y)
  joblib.dump(afc, "test.joblib")

def get_arsenal():
  print("getting")
  afc: Arsenal = joblib.load("test.joblib")

  # working
  X2 = np.random.random((10, 20))
  print(afc.predict(X2))

  # create new data
  tmp_meas = ['0.0']
  for _ in range(199):
    tmp_meas.append('0.0')
  print(tmp_meas)
  data = np.array([tmp_meas, tmp_meas], dtype=np.float64) # this must be wrong?!
  print(data)

  # not working but does with sktime
  print(afc.predict(data))

if __name__ == '__main__':
  save_arsenal()
  
  # this works (even with range(199))
  # get_arsenal()

  # this does not
  mp.Process(target=get_arsenal, args=()).start()

[tim-bsm]

And another part of the second problem was a typo i think. @TonyBagnall asked me to open a PR but I'm fairly new to GitHub and haven't really opened any real PR so far. So I'm a bit lost on this one to be honest.

Regarding your last comment about the 199 @MatthewMiddlehurst, it does work (can be run with my latest code snippet). I agree that the output is much likely going to be unusable. But as i said, my real values are somewhere around 200 and normally just changing +-4, sometimes +- ~20. So, as far as I noticed, Arsenal ist stable enough and the outcome is good.

[MatthewMiddlehurst]

The above code snippet actually runs without an exception for me. Not very familiar with in-built Python multithreading though, TBH. I'm on Python 3.10, Windows 10 OS.

Don't worry about the typo.

[tim-bsm]

Mhm, strange. For me it gives the error Terminating: fork() called from a process already using Gnu OpenMP, this is unsafe. on Ubuntu with Python 3.12. Can anybody test this on Linux?

[TonyBagnall]

hi, no worries about making the contribution, but if you want to figure out how to do it we are more than happy to help. There is a guide here
https://www.aeon-toolkit.org/en/stable/contributing.html
starts with this
https://www.aeon-toolkit.org/en/stable/developer_guide/dev_installation.html
feel free to join our slack, there are lots of people there happy to help. We have all been in the position of getting started, it can be confusing at first but you will get used to it soon. I'll look at the rest later