Identifying problematic frames in ground-truth user predictions

[eringiglio]

Hi! My project is big enough that many of my user predictions (N = 3781 on the project I'm looking at--one of 16 total!) are generated by undergrads doing their best to assign points on a mouse from above. I have some reason to believe some of these user predicted 'ground-truth' frames are of, shall we see, varying quality. I've been writing a script to identify individual frames that my best-working models do the worst at predicting, which can then be turned into labeling suggestions that I can flip quickly through and double-check.

It'd be great to have an out-of-the-box way to do this, but I'm totally capable of writing my own (and am--love the commitment to command-line and Pythonic tools in this package generally!). What occurs to me is that this cannot be a problem specific only to me and my ridiculous monster project.

It's been a while since I collaborated on a Github project belonging to anyone else; is there interest in writing my script up so that it can be folded into the main program, or is this something that is largely just worth it to me?

I know there is a tool already that should do a big piece of this in sleap.info.metrics, matching two ranges of labels and finding the distance between each point of each, but it seems to be missing a necessary function (will submit a related Issue about that).

[eringiglio]

The other issue with this sleap.info.metrics tool is in #1517, fwiw.

[roomrys]

Hi @eringiglio,

is there interest in writing my script up so that it can be folded into the main program, or is this something that is largely just worth it to me?

I think this is something worth including. In our own lab we have also recently run into this after a large round of labeling by multiple different annotators. If you have a script/workflow that you already use, I'd be happy to collaborate on something to be rolled into main SLEAP.

Thanks,
Liezl

[eringiglio]

Well, it's been nine months, but I have a script running that has been very effective for me. Here's what I've got at the moment:

"""
Utility to identify the worst-predicted frames in a project so that I can move in and do some quality control. Should work alongside all files. MUST update the dictionary before any runs. 
"""
import sys
sys.path.append('/scratch.global/gigli029/sleap/analysis/msi_analysis/model_eval/')
import sleap
import pandas as pd
import os
import numpy as np
from datetime import datetime as dt
import shutil

#edit this upon running: consists of mice to run this pipeline for
mouseList=["donut3", "egg1", "egg2", "egg3", "fitti1", "fitti2", "granola3", "hash2", "oat5"]

#dict last updated 3/1/24
mouseDict=dict(
    bacon1=dict(centered="221103_05-18-2023.centered_instance",centroid="230823_08-23-2023.centroid"),
    bacon2=dict(centered="230823_08-23-2023.centered_instance",centroid="230613_06-28-2023.centroid"),
    cereal1=dict(centered="230904_09-06-2023.centered_instance",centroid="230214_143042_03-23-2023.centroid"),
    cereal2=dict(centered="230912_09-12-2023.centered_instance",centroid="230509_05-12-2023.centroid"),
    donut1=dict(centered="230907_09-08-2023.centered_instance",centroid="230726_08-25-2023.centroid"),
    donut2=dict(centered="230823_08-24-2023.centered_instance",centroid="230908_09-11-2023.centroid"),
    donut3=dict(centered="07-24-23230724_07-24-2023.centered_instance",centroid="230825_08-25-2023.centroid"),
    egg1=dict(centered="230913_09-13-2023.centered_instance",centroid="230913_09-13-2023.centroid"),
    egg2=dict(centered="230216_06-08-2023.centered_instance",centroid="230216_06-26-2023.centroid"),
    egg3=dict(centered="230825_08-25-2023.centered_instance",centroid="240215_02-19-2024.centroid"),
    fitti1=dict(centered="230905_09-08-2023.centered_instance",centroid="230905_09-08-2023.centroid"),
    fitti2=dict(centered="230310_161254_04-20-2023.centered_instance",centroid="230310_06-09-2023.centroid"),
    granola1=dict(centered="230724_07-24-2023.centered_instance",centroid="230323_06-26-2023.centroid"),
    granola3=dict(centered="230323_160956_05-09-2023.centered_instance",centroid="230323_160956_05-09-2023.centroid"),
    hash1=dict(centered="230324_124058_04-14-2023.centered_instance",centroid="230509_06-08-2023.centroid"),
    hash2=dict(centered="230904_09-12-2023.centered_instance",centroid="230509_05-12-2023.centroid"),
    lox1=dict(centered="231201_12-01-2023.centered_instance",centroid="221214_214421_03-23-2023.centroid"),
    lox2=dict(centered="230904_09-06-2023.centered_instance",centroid="221214_210338.centroid"),
    muffin1=dict(centered="230918_09-18-2023.centered_instance",centroid="230823_08-23-2023.centroid"),
    muffin2=dict(centered="230904_09-06-2023.centered_instance",centroid="230224_06-16-2023.centroid"),
    muffin3=dict(centered="230216_05-19-2023.centered_instance",centroid="230223_05-11-2023.centroid"),
    muffin4=dict(centered="230915_09-15-2023.centered_instance",centroid="230223_163747_03-31-2023.centroid"),
    oat2=dict(centered="230724_07-24-2023.centered_instance",centroid="230216_153112.centroid"),
    oat3=dict(centered="230915_09-15-2023.centered_instance",centroid="230216_155903_04-17-2023.centroid"),
    oat4=dict(centered="06-27-2023.centered_instance",centroid="230223_06-01-2023.centroid"),
    oat5=dict(centered="230302_06-23-2023.centered_instance",centroid="230302_06-23-2023.centroid"),
    poptart1=dict(centered="230221_05-11-2023.centered_instance",centroid="230509_06-16-2023.centroid"),
    poptart2=dict(centered="230302_131301_04-13-2023.centered_instance",centroid="230302_131301_04-13-2023.centroid"),
    poptart3=dict(centered="230302_132042_04-11-2023.centered_instance",centroid="230302_132042_04-05-2023.centroid"),
    poptart4=dict(centered="230915_09-15-2023.centered_instance",centroid="230828_08-29-2023.centroid"),
    poptart5=dict(centered="230302_133029_05-09-2023.centered_instance",centroid="230302_07-05-2023.centroid"))

mDir = "/scratch.global/gigli029/breakfast/males/sleap/"
fDir = "/scratch.global/gigli029/breakfast/females/sleap/"
imgDir = "/scratch.global/gigli029/sleap/data/model_quality_scores/"

def check_this_frameSet(mouseID,mouseDict):
    mDir = "/scratch.global/gigli029/breakfast/males/sleap/"
    fDir = "/scratch.global/gigli029/breakfast/females/sleap/"
    imgDir = "/scratch.global/gigli029/sleap/data/model_quality_scores/"
    if mouseID[0] in ['b','c','d','e','f','g','h']: #if it's a male mouse
        filename = os.path.join(mDir,mouseID,mouseID+".slp")
        mouseDir = os.path.join(mDir,mouseID)
        slp = sleap.load_file(filename)
        for video in slp.videos:
            oldName = video.filename
            oldName = oldName.replace('\\','/')
            root = oldName[oldName.rindex('/')+1:oldName.rindex('.')]
            newFile = [x for x in os.listdir(os.path.join('/scratch.global/gigli029/breakfast/males/final_videos/',mouseID)) if root in x][0]
            newName = os.path.join('/scratch.global/gigli029/breakfast/males/final_videos/',mouseID,newFile)
            video.backend.filename = newName
            video.filename = newName
        gtVal = sleap.Labels(slp.user_labeled_frames)  # all I care about here are user-labeled frames 
    else:
        filename = os.path.join(fDir,mouseID,mouseID+".slp")
        mouseDir = os.path.join(fDir,mouseID)
        slp = sleap.load_file(filename)
        for video in slp.videos:
            oldName = video.filename
            oldName = oldName.replace('\\','/')
            root = oldName[oldName.rindex('/')+1:oldName.rindex('.')]
            newFile = [x for x in os.listdir(os.path.join('/scratch.global/gigli029/breakfast/females/final_videos/',mouseID)) if root in x][0]
            newName = os.path.join('/scratch.global/gigli029/breakfast/females/final_videos/',mouseID,newFile)
            video.backend.filename = newName
            video.filename = newName
        gtVal = sleap.Labels(slp.user_labeled_frames)  # all I care about here are user-labeled frames 
    modelDir = os.path.join(mouseDir,'models')
    thisIMGdir = os.path.join(imgDir,mouseID)
    if os.path.isdir(thisIMGdir) == False:
        os.makedirs(thisIMGdir)
    print(filename)
    slp.suggestions = []
    for model in [mouseDict[mouseID]['centroid'],mouseDict[mouseID]['centered']]:
        distDict=dict(mouseID=[], model=[], videoID=[], frameID=[], dist=[])
        nodeNames = ['nose', 'ear_left', 'ear_right', 'tail_base', 'tail_tip', 'tail_mid', 'hind_left', 'hind_right', 'fore_left', 'fore_right']
        for key in nodeNames:
            distDict[key+"_dist"] = []
        predictor = sleap.load_model(os.path.join(modelDir,model))
        prVal = predictor.predict(gtVal) #not all of these frames may wind up predicted by the model
        for i in range(len(gtVal.user_instances)):
            gt = gtVal.user_instances[i]
            prList = [x for x in prVal.predicted_instances if x.video.filename == gt.video.filename and x.frame_idx == gt.frame_idx]
            if len(prList) == 1:
                pr = prList[0]
            elif len(prList) > 1:
                print("multiple prediction matches found")
                pr = prList[0]
                for i in range(1,len(prList)):
                    if pr.score < prList[i].score:
                        pr = prList[i]
            else:
                print("no matching prediction found")
                continue
            thisDist = []
            for j in nodeNames:
                gtPoint = gt[j]
                prPoint = pr[j]
                if prPoint.visible == True:
                    xDist = gtPoint.x - prPoint.x
                    yDist = gtPoint.y - prPoint.y
                    thisDist.append(np.hypot(xDist,yDist))
                else:
                    thisDist.append('')
            distDict['mouseID'].append(mouseID)
            distDict['model'].append(model)
            if gt.video.filename != pr.video.filename:
                print(gt.video.filename)
                print(pr.video.filename)
                print("oh no! mismatch")
            else:
                distDict['videoID'].append(gt.video.filename)
            if gt.frame_idx != pr.frame_idx:
                print(gt.frame_idx)
                print(pr.frame_idx)
                print("oh no! mismatch")
            else:
                distDict['frameID'].append(gt.frame_idx)
            distDict['dist'].append(np.nanmean(thisDist))
            for k in range(len(nodeNames)):
                distDict[nodeNames[k]+"_dist"].append(thisDist[k])
        distDF = pd.DataFrame(distDict)
        distDF.to_csv(os.path.join(thisIMGdir,mouseID+'_'+model+'_groundToPredict.csv'))
        checks=distDF[distDF.dist> (distDF.dist.mean() + 3*distDF.dist.std())]
        checkList = list(checks.index)
        for key in distDF.keys()[3:len(distDF.keys())]:
            checks=distDF[distDF[key]>(distDF[key].mean() + 3*distDF[key].std())]
            checkList = checkList + list(checks.index)
        for i in checkList:
            thisCheck = distDF.loc[i]
            thisVid=[x for x in slp.videos if x.filename==thisCheck.videoID][0]
            slp.add_suggestion(video=thisVid,frame_idx=str(thisCheck.frameID))
        print(len(slp.suggestions))
    slp.save(os.path.join(mouseDir,mouseID+"_doublecheck.slp"))

Please let me know if you'd still like to work together to merge it into the main SLEAP routine when you get back from Cosyne. It's been very handy!