Inference guidance

[hienhayho]

Thank you for your efforts and dedication.

I would like to perform inference on a 30-second video using the provided source code. While I have reviewed the instructions, I find some aspects unclear. Could you kindly provide a detailed, step-by-step guide to help me carry out the video inference process?

[jyrao]

Sorry for making you confused. I recommend you send me with details of your problems, I will reply you once I see it.

Generally, what you only need to do is do modify some hyper-parameters in inference.py as:

`

parser.add_argument("--open_visual_encoder", type=bool, default=False)

parser.add_argument("--open_llm_decoder", type=bool, default=False)

parser.add_argument("--ckpt_path", type=str, default="FILE_PATH_TO_MODEL_CHECKPOINT")

parser.add_argument("--csv_out_path", type=str, default="inference/sample.csv")

parser.add_argument("--matchtime_json", type=str, default="train_data/json/MatchTime/classification_test.json")

parser.add_argument("--matchtime_video_base", type=str, default="FOLDER_OF_VIDEO_CLIPS_OF_MATCHTIME")

`

with chechpoints on huggingface page and well-defined csv files and videos as I show in github repo would be enough.

[hienhayho]

Hi, @jyrao
I noticed that the inference.py file contains a model for capturing commentary. Could you please guide me on how to run the classifier as well?
Thank you very much.

[jyrao]

That's a very good point, thanks for this question. Actually, all you need to do is to load the classification model ([MatchVision_classifier.py]https://github.com/jyrao/UniSoccer/blob/main/model/MatchVision_classifier.py) with pretrained checkpoints.

With same dataset and dataloader input the features to model, the output will be the order of the classes after the classifier. The type is in the order as default config file shows: ["var", "end of half game", "clearance", "second yellow card", "injury", "ball possession", "throw in", "show added time", "shot off target", "start of half game", "substitution", "saved by goal-keeper", "red card", "lead to corner", "ball out of play", "off side", "goal", "penalty", "yellow card", "foul lead to penalty", "corner", "free kick", "foul with no card"]. The order of You might find more clue on last issue post in #4.

As for multiple request for this classification, I would think of providing a more clear version of code/handbook on how to fulfill such classification😃

[hienhayho]

Hi, @jyrao
I successfully ran the classifier; however, I am encountering issues with the commentary model. Specifically, it produces similar outputs for different video shots.
Here are my code for commentary task:

def load_commentary_model(model_path: str, device_ids: list[int], ckpt_path: str):
    commentary_model_type = matchvoice_model_all_blocks
    devices = [torch.device(f"cuda:{device_id}") for device_id in device_ids]

    model = commentary_model_type(
        visual_encoder_checkpoint=model_path,
        num_features=768,
        need_temporal="yes",
        open_visual_encoder=True,
        open_llm_decoder=True,
    )

    model = model.to(devices[0])
    model = DataParallel(model, device_ids=device_ids)
    checkpoint = torch.load(ckpt_path, map_location="cpu")["state_dict"]
    model.load_state_dict(checkpoint, strict=False)
    model.eval()

    print("========== Commentary model loaded ==========")

    return model

commentary_model = load_commentary_model("weights/downstream_commentary_all_open.pth", [0], "weights/downstream_commentary_all_open.pth")

I passed downstream_commentary_all_open.pth for this model_path parameter.

Then, I run it with:

val_pbar = tqdm(
    test_dataloader, total=len(test_dataloader), desc="Inference caption ..."
)
with torch.no_grad():
    model = (
        commentary_model.module
        if isinstance(commentary_model, torch.nn.DataParallel)
        else commentary_model
    )
    model = model.to(self.devices[0])
    torch.cuda.empty_cache()
    for samples in val_pbar:
        samples["frames"] = samples["frames"].to(self.devices[0])
        temp_res_text, anonymized, video_path = model(samples, True)

Here is one example of my son file:

[
    {
        "video": "video.mp4",
        "caption": "",
        "comments_text": "",
        "comments_text_anonymized": ""
    }
]

I see that temp_res_text is similar as the same in different video shot. How can I fix this, and what should I pass to model_path and ckpt_path ?

Thanks a lot.

[jyrao]

I want to know how your model works now. Normally there will be two different cases:

1. The model is saying words with no sense, it will be like similar problems as Reproduction Issue #3 (comment). You should modify the hyperparameters to open visual encoder and llm decoder since you are using all_open.pth as checkpoint.

2. If the model kept saying soccer words, I didn't meet the situation of highly similar outputs. If this situation happens, could you please send your videos to me to check if the gap between my dataset defers too much?

[DAVID-Hown]

@jyrao
I also encountered the same problem and I successfully loaded the model. I see that csv_out_path.csv is similar as the same in different video shot. Is it for this reason that my test video clip is within 10 seconds？
According to the previous issues, I only modified the path of data loading (matchtime_json/matchtime_video_base), and all the other code remained consistent.

class matchvoice_model_all_blocks(nn.Module):
    def __init__(self,
                 # Visual Encoder
                 load_checkpoint = True,
                 visual_encoder_checkpoint = "pretrained_contrastive.pth",
                 # LLM part
                 llm_ckpt = "LLM-Research/Meta-Llama-3-8B-Instruct",
                 tokenizer_ckpt = "LLM-Research/Meta-Llama-3-8B-Instruct",
                 # Q-former part
                 max_frame_pos = 128,
                 window = 30,
                 num_query_tokens = 32,
                 num_video_query_token = 32,
                 num_features = 512,
                #  device = "cuda:0",
                 inference = False,
                 file_path = './words_world/merge.pkl',
                 need_temporal = True,
                 encoder_type = "spatial_and_temporal",
                 open_visual_encoder = False,
                 open_llm_decoder = False,
                 llm_lora_rank = 16,
                 llm_lora_dropout = 0.05,
                 **kwargs,
                 ):
parser.add_argument("--open_visual_encoder", type=bool, default=True)
parser.add_argument("--open_llm_decoder", type=bool, default=True)
parser.add_argument("--ckpt_path", type=str, default="downstream_commentary_all_open.pth")

[DAVID-Hown]

@hienhayho May I ask if you have solved this problem? Is it convenient for you to communicate how it was solved.

[hienhayho]

Hi @DAVID-Hown, I was able to run it with a 30-second video, but it's been a while since I last did it, so I don't remember how I fixed it.