Using train.pkg.slp, val.pkg.slp, test.pkg.slp for training in python, how to cofigure model and hyperparameter for training.

[Powerwoody]

Hey, i am having trouble getting my training going without my kernel crashing (to much memory ussage) am i doing something wrong here. I want to be able to just specify the models to be used in python and configure hyperparameters and then run training. I am really stuck.

Thanks in advance. :)

CODE:
labeled_frames = []
num_frames = annotations.shape[0]

for frame_idx in range(num_frames):
# Extracting the keypoints for the current frame
pts = annotations[frame_idx].copy() # shape: (num_keypoints, 2)

# Creating a single instance from these points
instance = sio.Instance.from_numpy(points=pts, skeleton=skeleton)

# Creating a LabeledFrame
lf = sio.LabeledFrame(
    video=video,
    frame_idx=frame_idx,
    instances=[instance]
)

labeled_frames.append(lf)

LABELS_SAVE_PATH = dir_path + "slp_files/" + file_name.replace(".h5", ".slp")

labels = sio.Labels(
videos=[video],
skeletons=[skeleton],
labeled_frames=labeled_frames
)

labels.save(LABELS_SAVE_PATH)

labels = sio.Labels(
videos=[video],
skeletons=[skeleton],
labeled_frames=labeled_frames
)

labels.save(LABELS_SAVE_PATH)

Load labels:

labels = sio.load_file(dir_path + "slp_files/" + file_name.replace(".h5", ".slp"))
save_dir = dir_path + "slp_files/splits/"

Make splits and export with embedded images:

labels.make_training_splits(n_train=0.8, n_val=0.1, n_test=0.1, save_dir=save_dir, seed=42)

Initialize the default training job configuration.

config = TrainingJobConfig()

TRAINING_SLP_FILE = dir_path + "slp_files/splits/train.pkg.slp"
VALIDATION_SLP_FILE = dir_path + "slp_files/splits/val.pkg.slp"
TEST_SLP_FILE = dir_path + "slp_files/splits/test.pkg.slp"

config.data.labels.training_labels = TRAINING_SLP_FILE
config.data.labels.validation_labels = VALIDATION_SLP_FILE
config.data.labels.test_labels = TEST_SLP_FILE

config.model.backbone.unet = UNetConfig()
config.model.heads.centered_instance = CenteredInstanceConfmapsHeadConfig()

config.optimization.augmentation_config.rotate = True

config.outputs.run_name = "baseline_model.topdown"

config.save_json(dir_path + "config_files/config.json")

trainer = sleap.nn.training.Trainer.from_config(config)

trainer.train()

[gitttt-1234]

Hi @Powerwoody ,

Could you please share the training_config.json?

Thanks,

Divya

[Powerwoody]

Sure here it is Thanks a lot. (I might need to do the centroid model since i want to do a top-down, not sure how to do that in python)
training_config.json

I am using a sligthly different setup know just for context:

class SleapWrapper:
"""
A wrapper for converting a DeepPoseKit-style .h5 file to a .slp file
compatible with SLEAP.

Attributes:
h5_file_path (str): Path to the input DeepPoseKit .h5 file.
skeleton_csv_path (str): Path to a CSV that defines the skeleton edges.
The CSV should have columns like [name, parent, ...].
output_dir (str): Directory where output files (.mp4, .slp) will be saved.
"""

def init(self, h5_file_path: str, skeleton_csv_path: str, output_dir: str):
self.h5_file_path = h5_file_path
self.skeleton_csv_path = skeleton_csv_path
self.output_dir = output_dir

# Basic checks for file paths
if not os.path.isfile(self.h5_file_path):
    raise FileNotFoundError(f"h5_file_path does not exist: {self.h5_file_path}")
if not os.path.isfile(self.skeleton_csv_path):
    raise FileNotFoundError(f"skeleton_csv_path does not exist: {self.skeleton_csv_path}")
if not os.path.isdir(self.output_dir):
    os.makedirs(self.output_dir, exist_ok=True)

@staticmethod
def replace_missing_points(annotations: np.ndarray) -> np.ndarray:
"""
Replace missing (-1, -1) pairs with NaNs in the annotations array.

Args:
    annotations (np.ndarray): Array of shape (n_frames, n_keypoints, 2) 
                              containing x, y coordinates.

Returns:
    np.ndarray: Modified annotations with missing points replaced by NaNs.
"""
mask = (annotations[:, :, 0] == -1) & (annotations[:, :, 1] == -1)
annotations[mask] = np.nan
return annotations

def convert_to_sleap(self) -> None:
"""
Convert a DeepPoseKit-style .h5 file into a SLEAP .slp file for training.

This method:
    1. Reads images and annotations from the .h5 file.
    2. Replaces missing (-1, -1) points with NaN values.
    3. Reads skeleton definitions from the CSV file.
    4. Creates a .mp4 video from the image frames.
    5. Builds LabeledFrame objects for each frame.
    6. Saves the final SLEAP labels (.pkg.slp) in the specified output directory.
"""
# Derive base filename and output paths
base_filename = os.path.splitext(os.path.basename(self.h5_file_path))[0]
video_file_path = os.path.join(self.output_dir, base_filename + ".mp4")
sleap_file_path = os.path.join(self.output_dir, base_filename + ".pkg.slp")

# Check if either output file already exists; if so, skip conversion.
if os.path.exists(video_file_path) or os.path.exists(sleap_file_path):
    print(f"Output file(s) already exist for '{base_filename}'. Skipping conversion.")
    return

# 1. Read the .h5 file
with h5py.File(self.h5_file_path, "r") as f:
    annotations = f["annotations"][:]  # shape: (n_frames, n_keypoints, 2)
    images = f["images"][:]            # shape: (n_frames, height, width, channels)

# 2. Preprocess annotations
annotations = self.replace_missing_points(annotations)

# 3. Read the skeleton CSV and build edges
skeleton_data = pd.read_csv(self.skeleton_csv_path, header=0)
edges = []
for name, parent, *rest in skeleton_data.itertuples(index=False, name=None):
    if pd.notna(parent):  # Only add edges if parent is not NaN
        edges.append((parent, name))

skeleton = sio.Skeleton(
    nodes=skeleton_data["name"].tolist(),
    edges=edges
)

# 4. Create a .mp4 video from image frames
try:
    with sio.VideoWriter(video_file_path) as writer:
        for frame in images:
            writer(frame)
except Exception as e:
    raise RuntimeError(f"Error while writing video: {e}")

video = sio.load_video(video_file_path)

# 5. Build LabeledFrame objects for each frame
labeled_frames = []
for frame_idx in range(annotations.shape[0]):
    pts = annotations[frame_idx]
    instance = sio.Instance.from_numpy(points=pts, skeleton=skeleton)
    labeled_frame = sio.LabeledFrame(
        video=video,
        frame_idx=frame_idx,
        instances=[instance],
    )
    labeled_frames.append(labeled_frame)

# 6. Build a Labels object and save to disk
labels = sio.Labels(videos=[video], skeletons=[skeleton], labeled_frames=labeled_frames)
labels.save(sleap_file_path)

print(f"Video saved to: {video_file_path}")
print(f"SLEAP labels saved to: {sleap_file_path}")

wrapper.convert_to_sleap()

RUN_NAME = "test_run"

Initialize the default training job configuration.

config = TrainingJobConfig()

Data config:

config.data.labels.training_labels = H5_PATH.replace(".h5", ".pkg.slp")
config.data.labels.validation_fraction = 0.1

Model config:

config.model.backbone.unet = UNetConfig()
config.model.heads.centered_instance = CenteredInstanceConfmapsHeadConfig()

Optimization config:

config.optimization.augmentation_config.rotate = True

Outputs config:

config.outputs.tensorboard.write_logs = True
config.outputs.run_name = RUN_NAME

Initializing trainer from config

trainer = sleap.nn.training.Trainer.from_config(config)

trainer.train()

[gitttt-1234]

Hi @Powerwoody !

1. I see the output_stride parameter in the model_config is set to 1 (which is pretty high). Output stride controls the resolution of the confidence maps generated. Try setting it to 2 or 4.
2. Your image size is 1024x1024 and your crop size is 976. Do you have only one animal instance in your frame? If not, there might be some issue with labelling. Did you manually set the crop_size?
3. We disable gpu preallocation here before initiating the training. Could you try doing this?
4. Do you want to train a centroid model? We do have an option to train centroid model with SLEAP framework.

Let me know if you have any questions!

Thanks,

Divya

[Powerwoody]

I am not sure of how to upload the .slp file. :)

[gitttt-1234]

Hi @Powerwoody!

You're right, the reason why the centroid model is not performing is because the centroid in the SLEAP training pipeline is computed as the center of a bounding box constructed with the keypoints. Since your keypoints are across both the bat and the worm, the centroid is computed to be somewhere between the worm and the bat.

1. you could provide an anchor index (mention the name of the bodypart in the bat to be considered as centroid) in the config file before training. (For instance, config.model.heads.centered_instance.anchor_part = "head")
2. Do you need to track the worm position along with the predicting the pose of the bat?

Thanks,

Divya

[Powerwoody]

I am not sure if the tracking of the worm is necessary but what happens if the anchor keypoint is not visible in a frame? :) Maybe if the worm is important i could make it into its own instance?

[gitttt-1234]

yes it's better to train separate models to track bat and worm! If you only have bat instance labeled in a frame, then you don't need to specify the anchor part in the config.

Thanks,

Divya