Classifier inference

[Eurususu]

Hi!
It is very exciting work!

But there are same issues happen:

I want to test the performance of classification of events. So I use MatchVision_Classifier and then load the classifier checkpoint,modify the names of key. But in the end the output shape is not correct.

from model.MatchVision_classifier import MatchVision_Classifier
import torch

classifier = MatchVision_Classifier()
checkpoint = torch.load("weights/pretrained_classification.pth",map_location='cpu')
new_static_dict = {key.replace("module.", ""): value for key, value in checkpoint['state_dict'].items()}
classifier.load_state_dict(new_static_dict,strict=False)

run error as shown:

RuntimeError: Error(s) in loading state_dict for MatchVision_Classifier:
	size mismatch for classifier.weight: copying a param with shape torch.Size([23, 768]) from checkpoint, the shape in current model is torch.Size([24, 768]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([23]) from checkpoint, the shape in current model is torch.Size([24]).

how to infer the classifier model?

[jyrao]

Yeah, actually you should use 23 as number of classification, the exact order of 23 classes are the same as the config files shows. The reason I omit one class is because the examples are too few so I train the model for avoiding long-tail issues since this backbone is targeting on a more universal understanding.

Hope this information can help you. You might see how the dataset exactly looks like in near future since we are still considering the format to release our data due to some copyright issues. I believe you can have a more clear view of such task. Sincerely thanks for your message and great questions!

[Eurususu]

Yeah, actually you should use 23 as number of classification, the exact order of 23 classes are the same as the config files shows. The reason I omit one class is because the examples are too few so I train the model for avoiding long-tail issues since this backbone is targeting on a more universal understanding.

Hope this information can help you. You might see how the dataset exactly looks like in near future since we are still considering the format to release our data due to some copyright issues. I believe you can have a more clear view of such task. Sincerely thanks for your message and great questions!

ok,thanks!

[Eurususu]

Hi,@jyrao
So I tried to infer with some test videos, but the result was not very satisfactory.

First I create json file,like:

[
  {
    "video": "segment_1.mp4",
    "caption": ""
  },
  {
    "video": "segment_2.mp4",
    "caption": ""
  },
  {
    "video": "segment_3.mp4",
    "caption": ""
  },
  {
    "video": "segment_4.mp4",
    "caption": ""
  }
]

then I add test term in pretrain_classification.py like:

test = dict(
            json = ["./train_data/json/custom/classification_test.json"],
            keywords = ["var", "end of half game", "clearance", "second yellow card", "injury", "ball possession", "throw in", "show added time", "shot off target", "start of half game", "substitution", "saved by goal-keeper", "red card", "lead to corner", "ball out of play", "off side", "goal", "penalty", "yellow card", "foul lead to penalty", "corner", "free kick", "foul with no card"],
            video_base = ["/home/jia/dataset/videos/football_video/test_videos"],
            batch_size = 4,
            num_workers = 4,
            sample = 'middle',
            balanced_or_not = 'unbalanced'
        )

finally infer with code:

import sys, os
sys.path.append('PATH_TO_FOLDER_OF_THIS_PROJECT')
from dataset.video_dataset import VideoCaptionDataset, VideoCaptionDataset_Balanced
from model.MatchVision_classifier import MatchVision_Classifier
import torch
from torch.utils.data import DataLoader
from tqdm import tqdm
import argparse
import importlib.util

def load_config(path):
    spec = importlib.util.spec_from_file_location("config", path)
    config = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(config)
    return config.config

def main():
    ############## Configs ################
    parser = argparse.ArgumentParser(description="Load a Python config file.")
    parser.add_argument('--config_path', type=str, default="config/pretrain_classification.py", help='The path to the Python config file')
    parser.add_argument('--checkpoint_path', type=str, default="weights/pretrained_classification.pth", help='The path to the checkpoint file')
    args = parser.parse_args()

    config = load_config(args.config_path)
    checkpoint_path = args.checkpoint_path

    # Dataset configuration
    config_dataset = config["dataset"]
    config_test_dataset = config_dataset["test"]

    # Model configuration
    config_training_settings = config["training_settings"]
    device_ids = config_training_settings["device_ids"]
    classifier_transformer_type = config_training_settings["classifier_transformer_type"]
    encoder_type = config_training_settings["encoder_type"]
    use_transformer = config_training_settings["use_transformer"]

    # Set up the device
    devices = [torch.device(f'cuda:{i}') for i in device_ids]

    ############## Dataset ################
    test_dataset_type = None
    if config_test_dataset["balanced_or_not"] == "balanced":
        test_dataset_type = VideoCaptionDataset_Balanced
    else:
        test_dataset_type = VideoCaptionDataset

    test_dataset = test_dataset_type(
        json_file=config_test_dataset["json"],
        video_base_dir=config_test_dataset["video_base"],
        sample=config_test_dataset["sample"],
        keywords=config_test_dataset["keywords"],
    )

    test_data_loader = DataLoader(
        test_dataset,
        batch_size=config_test_dataset["batch_size"],
        num_workers=config_test_dataset["num_workers"],
        shuffle=False,
        pin_memory=True,
        persistent_workers=True
    )

    ############## Model ################
    classifier = MatchVision_Classifier(
        keywords=config_test_dataset["keywords"],
        classifier_transformer_type=classifier_transformer_type,
        vision_encoder_type=encoder_type,
        use_transformer=use_transformer
    ).eval()

    # Load checkpoint
    checkpoint = torch.load(checkpoint_path, map_location="cpu")
    new_state_dict = {key.replace('module.', ''): value for key, value in checkpoint['state_dict'].items()}
    classifier.load_state_dict(new_state_dict)

    # Move model to device and wrap with DataParallel
    classifier = classifier.to(devices[0])
    classifier = torch.nn.DataParallel(classifier, device_ids=device_ids)

    ############## Inference ################
    all_predictions = []
    test_progress_bar = tqdm(enumerate(test_data_loader), total=len(test_data_loader), desc="Inference")

    with torch.no_grad():
        for batch_idx, (frames, _) in test_progress_bar:  # Captions not needed for inference
            frames = frames.to(devices[0])

            # Forward pass
            logits = classifier.module.get_logits(frames)
            predictions = classifier.module.get_types(logits)
            all_predictions.append(predictions.cpu())

    # Combine all predictions into a single tensor
    all_predictions = torch.cat(all_predictions, dim=0)
    print(all_predictions)

    # # Save predictions to a file or process further
    # output_file = "predictions.pt"
    # torch.save(all_predictions, output_file)
    # print(f"Predictions saved to {output_file}")

if __name__ == "__main__":
    main()

>>>>
tensor([[10, 18,  2, 22, 20],
        [13, 20,  8, 10,  2],
        [ 8, 20, 13,  2, 22],
        [20,  2, 11, 13,  8]])

the result is not well, is there somthing wrong?
Since the fps of my videos are 60 fps, will this lead to poor results? I tried to change them to 30 fps, but it didn't improve much either.

Can you help me analyze the situation？

videos were updated to baidu disk:
link: test videos code: r7w7

[jyrao]

I've saw your raw videos. Let me answer these questions one by one:

1. We have bare restrict around FPS since the logic of our preprocessor is to sample 30 frames with equal step in the video. The only requirement for video to run the pipeline successfully is to have 30 frames.

2. To possibly explain such situations, followings are some of my analysis:

a) Our model concentrates on videos of key events, seg1 might be kinda normal to classify. Even "ball posession" means dense ball sessions and passes around mid-field.
b) Seg2 is showing good classification on "lead to corner" and Seg3 has top-2 accurately classifying on "corner"
c) Seg4 is somehow start from "corner", that's why it classify as this.

So I don't think those are failure examples, especially considering more factors of video color, camera quality. Such gap is not sensitive to us but for the matrix and features.

Also, to be noted, I saw your videos are like single-camera recording for non/half-professional games, that might have some gap since our model always learn patterns from those various camera shots from literally professional European leagues settings (e.g.: they would have close-up for player with corner post, yellow card by referee.......). That's somehow it works in our video-caption pairs for training sets.

btw, I'm curious your setting to use such project. It's kinda like an application on Chinese junior league. I also want to know more about your projects. Hope these can help you understand better with this model.

[Eurususu]

Hi, @jyrao
Thanks replying. I'll show you the contents of our project
The goal of our project is to bring AI into various sports events, making them more automated and intelligent. For instance, an automated directing algorithm can capture the key areas of a game, enabling live event broadcasts without requiring professional directors. Additionally, it includes features like automatic highlight editing, AI-powered commentary, player analysis, and more.

Most of the video sources we use are recorded on-site using regular cameras, so the video quality may not be great, especially under poor lighting conditions at night. The footage is not as clear as what professional cameras can capture in official sports venues. As for the events, most of them are amateur matches (without close-up for player with corner post, yellow card by referee.......)

However, achieving all these functionalities is not an easy task. For the directing algorithm, we are currently using YOLO for detection combined with some directing logic. But due to the video quality issues and YOLO’s occasional misdetections, it’s challenging to achieve perfect live broadcasts. There are times when the directing algorithm fails to capture the focal points of the game.

As for automatic highlight editing, it requires event classification capabilities, which is why I want to test the performance of classification models. Considering the cost constraints (since we are using consumer-grade equipment), we also need to optimize the inference efficiency of the models. Event classification would not only help with highlight editing but also support other features like match analysis, player insights, and AI commentary, serving as the foundation for all these functionalities.

Currently, we are improving the directing algorithm and exploring event classification. If we can develop a robust classification model, it will represent a significant breakthrough.

As for your earlier mention of a single camera, that's not actually the case. We are using a panoramic camera setup, which stitches together footage from multiple cameras to create a single panoramic view. The directing algorithm identifies key areas within this panoramic footage. However, due to the small size of targets in the panoramic view, motion blur, the overall video quality (where small objects may get pixelated during the encoding and decoding process) and the YOLO object detection algorithm finds it very difficult to detect targets when the background is complex. we’ve hit some bottlenecks. To make improvements, we may need to explore alternative approaches rather than relying on detecting balls and players through object detection. Do you have any suggestions for effective methods to achieve this?

For event classification, we use footage from the directed camera feed. This ensures the information is more focused, and the resolution is lower, which is beneficial for model inference efficiency.

You can refer to the panoramic video after stitching
link: panoramic video code:ij3g

[jyrao]

Thanks for sharing, I believe the college research would always have , but I'm literally excited to see such solid attempt of you. Actually highlight detection and streaming commentary of soccer games are exactly in my research plan in 2025, it will be great if you try them up by that time.

As for such detection, we tried yolo for tracking before and actually it shows pretty good result. I can recommend you several works around tracking, game state reconstruction:

• https://arxiv.org/abs/2404.08401
• https://github.com/SoccerNet/sn-gamestate
• https://github.com/facebookresearch/sam2

For me myself, I might have several suggestions:

1. You might seperate the task step by step like agent. For example, to detect sidelines first, then detect players, then recognize players .etc
2. After that, you can locate the bottlenecks via labeling ground truth for each part to control the variable.
3. For the part of bottleneck, I believe there should be somehow sota work, or you could try using those GPT-4, claude, videoChat .etc

Those are just some rough thoughts come to my mind according to the past experience. Hope everything goes well in your project. I'm always believing it's a insdustry with promising future. 一起加油！

[Eurususu]

Thanks for suggestions!
In fact, we use a manual approach for field calibration, where the four corner points of the field are manually marked to define the event area. Why do we do it this way? It's because the lines in real match scenarios can sometimes be very complex, and the model might struggle to distinguish them clearly, as seen in the panoramic video I shared with you earlier. On the path to full automation, I believe a certain level of manual intervention is still necessary.

I’ll try your suggestions step by step, but for now, I’ll primarily focus on the directing algorithm and event classification algorithm, as these two are the most fundamental and important.

If there are any new developments, I’ll be sure to share them with you.一起进步！
By the way, what do you think about using reinforcement learning for the directing algorithm? Have you worked in this area before?

[jyrao]

I see, I think such full view camera would be great for calibration since when we use the broadcast video, we would easily lose track due to the camera-shot change. And you can worry less about video quality since our backbone is using 224224 as frame input, if you got higher resolution they would also be resized to 224224.

As for reinforcement learning, I didn't try it on visual perception of soccer but I believe it would useful for other aspects of tactic, analysis, performance. That is one of my research plan in the future.

Appreciate if you would like to share the updates of your projects in the future. You can send me email or if you would you can leave your WeChat here. For those general or technical problems I still prefer you publish on github issue that can help more people understand this project. Thank you so much!

[Eurususu]

emm, here it is my WeChat ID:johnson_jia,I am looking forward more technical exchanges with you, if possible.