Input / output pipeline

[Tortoise17]

Earlier I was working with the action spotting. Now I am trying to understand that how to reach to the replay grounding from that task further. I am trying to understand the workflow and how the input output works. It would be your great help if you can hint me with guide and advise what is the input required for this repository to get the json as output . Or in easy words, how to get the json export of replay_grounding and what input is required for this.

I understood that to reach to this point, it is maybe dependent on field calibration which needs the bounding boxes and hemography. Please can you help me to understand how can I get the inference out and requirements of input?

[SilvioGiancola]

Hi @Tortoise17 , you do not need any calibration nor bounding boxes to perform grounding. All you need are the labels from the v2, in particular Labels-v2.json and Labels-replay.json. You can have full tutorial of this task from last year challenge: https://youtu.be/u3CkxY85wAo . It will give you an overview of the input you need, and how the baselines work.

[Tortoise17]

@SilvioGiancola Great help. I will go through it and will ask further question. You are really a good supervisor.

[Tortoise17]

@SilvioGiancola thank you again. One question. If we make / construct of inference of this repository, we get the Labels-replay.json from the input of Labels-v2.json and ResNet_PCT_512.npy ? or something different? because I want to understand how to get the Labels-replay.json for any single video.mp4

[SilvioGiancola]

Hi @Tortoise17 , the labels-replay.json is a by-product of the Labels-cameras.json, which embed all the information of the cameras during the broadcast (camera type, live/replay, transition type, ...). The grounding task takes as input the information of where a replay occurs, and it outputs the time in the broadcast that action occurred live. We do not provide any code for inference, but I foresee several way to proceed:

1. Provide a "complete game video" + "replay video": you can input those 2 videos to a pre-trained model and infer the time the "replay video" occurred live in the "complete game video".
2. Provide a "complete game video" + "replay annotation time": you can input the video and identify the replay clip, that you can then feed into the model as well.
   In both case, you will be using a Siamese network that will provide encodings of the replay video and the encoding in time of the full game video. The task consist in finding the encodings of the replay video in the encoding in time of the full game video.
   Makes sense? :)

[hwazni]

Hi @Tortoise17 , the labels-replay.json is a by-product of the Labels-cameras.json, which embed all the information of the cameras during the broadcast (camera type, live/replay, transition type, ...). The grounding task takes as input the information of where a replay occurs, and it outputs the time in the broadcast that action occurred live. We do not provide any code for inference, but I foresee several way to proceed:

1. Provide a "complete game video" + "replay video": you can input those 2 videos to a pre-trained model and infer the time the "replay video" occurred live in the "complete game video".
2. Provide a "complete game video" + "replay annotation time": you can input the video and identify the replay clip, that you can then feed into the model as well.
   In both case, you will be using a Siamese network that will provide encodings of the replay video and the encoding in time of the full game video. The task consist in finding the encodings of the replay video in the encoding in time of the full game video.
   Makes sense? :)

Hi @SilvioGiancola, can you provide more details concerning the two last points. Is there any new models that can do so or any new papers?