When the players are detected in the first frame, instead of running detections on all frames thereafter, we need to track the players. See the link for example https://drive.google.com/file/d/1FwGkApZwX03mCpvA1BD1aRT1nVa6QMnZ/view?usp=sharing