Ragged tensor error on single animal network

[AkariGong]

Hi,

I am trying to use SLEAP to label a zebrafish in the videos. My videos were recorded by two cameras and their resolutions are different: one was 1014x1353, and the other one was 1080x1920. When I trained the network, it ran for a few epoch and then gave the error below. The screenshot of GUI and training_config.json are attached to the post.

When I looked up this error, I found #585 and realized that the scale of my animal also changes across the video. When the fish swim closed to the camera it becomes larger and when it swims away from the camera it becomes smaller. Is this the reason causing the ragged tensor problem? I was unable to find the final solution from that post, so I am looking for extra help here.

Thank you,
Ming

Error message from the terminal:
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) INVALID_ARGUMENT: Invalid row split size.
[[{{node RaggedToTensor/RaggedTensorToTensor}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_4]]
(1) INVALID_ARGUMENT: Invalid row split size.
[[{{node RaggedToTensor/RaggedTensorToTensor}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_7356]

Function call stack:
train_function -> train_function

My software versions are:
SLEAP: 1.4.1a2
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0

training_config.json

[talmo]

Hi @AkariGong,

Looks like we missed your post originally -- sorry about that!

Are you still stuck on this error?

Have you tried the stable version of SLEAP v1.4.1?

Does this happen during training (at the end perhaps?) or inference?

Let us know when you get a chance!

Cheers,

Talmo