video-classification pipeline looks for image processors

[merveenoyan]

System Info

4.57.1

Who can help?

@zucchini-nlp I can take a stab at this sometime

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import pipeline, infer_device
import torch
device = infer_device()
checkpoint = "facebook/vjepa2-vitl-fpc64-256"
pipe = pipeline("video-classification", model=checkpoint, device=device)

Expected behavior

Full trace:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[/usr/local/lib/python3.12/dist-packages/transformers/image_processing_base.py](https://localhost:8080/#) in get_image_processor_dict(cls, pretrained_model_name_or_path, **kwargs)
    353                 ]
--> 354                 resolved_image_processor_file = resolved_image_processor_files[0]
    355             except OSError:

IndexError: list index out of range

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
5 frames
[/usr/local/lib/python3.12/dist-packages/transformers/image_processing_base.py](https://localhost:8080/#) in get_image_processor_dict(cls, pretrained_model_name_or_path, **kwargs)
    359             except Exception:
    360                 # For any other exception, we throw a generic error.
--> 361                 raise OSError(
    362                     f"Can't load image processor for '{pretrained_model_name_or_path}'. If you were trying to load"
    363                     " it from 'https://huggingface.co/models', make sure you don't have a local directory with the"

OSError: Can't load image processor for 'facebook/vjepa2-vitl-fpc64-256'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/vjepa2-vitl-fpc64-256' is the correct path to a directory containing a preprocessor_config.json file

[FWKMultiverse]

1. Wrong model type: The facebook/vjepa2-vitl-fpc64-256 model you selected is not a video classification model. It's a base model trained to "understand" video (called V-JEPA). It's good at feature extraction, but it's not trained to answer "What is this video?" (e.g., 'playing basketball', 'cooking').

2. The pipeline requires a processor: When you call pipeline("video-classification", ...), it tries to download two things:

The model

The video processor, which has a configuration file named preprocessor_config.json

3. What happened: The pipeline found the model, but couldn't find the processor (because it wasn't provided), so it crashed and returned an OSError error.

Update Libraries
This is a solution that works 90% of the time. Open the Terminal (where you run the code) and run this command to update the necessary libraries to the latest version:

pip install --upgrade transformers huggingface_hub

This is the modified code, switching to a model designed directly for video-classification (e.g. VideoMAE) and adding error catching (Try...Except).

# Import necessary libraries
import torch
from transformers import pipeline
import requests  # Used to download a sample video
import os        # Used to check if file exists

# --- FIX 1: Robust Device Selection ---
# 'infer_device()' is deprecated. This is the modern, robust way
# to select the best available hardware (GPU > Apple Silicon > CPU).
if torch.cuda.is_available():
    device = torch.device("cuda")
    print("Using GPU (NVIDIA CUDA)")
elif torch.backends.mps.is_available():
    device = torch.device("mps")
    print("Using GPU (Apple MPS)")
else:
    device = torch.device("cpu")
    print("Using CPU")

# --- FIX 2: Choose a Correct Model ---
# The model 'facebook/vjepa2-vitl-fpc64-256' is a base model (a feature extractor)
# and does not have a classification head or a preprocessor config file.
# We must use a model that is fine-tuned for 'video-classification'.
# 'MCG-NJU/videomae-base-finetuned-kinetics400' is a popular choice.
checkpoint = "MCG-NJU/videomae-base-finetuned-kinetics400"

# --- FIX 3: Error Handling (Try...Except Block) ---
# This block will try to run the code. If any error occurs
# (e.g., internet down, model name wrong), it will 'catch'
# the exception and print a friendly message instead of crashing.
try:
    print(f"Loading pipeline with model: {checkpoint}...")
    
    # Load the pipeline with the correct model and device
    pipe = pipeline(
        "video-classification", 
        model=checkpoint, 
        device=device
    )
    print("Pipeline loaded successfully.")

    # --- Example Usage: Classify a sample video ---
    
    # You can replace this with the path to your own video file
    video_path = "sample_cat_video.mp4"
    video_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cat.mp4"

    # Download the sample video if we don't have it
    if not os.path.exists(video_path):
        print(f"Downloading sample video to {video_path}...")
        response = requests.get(video_url)
        response.raise_for_status() # Raise an error if download fails
        with open(video_path, "wb") as f:
            f.write(response.content)
        print("Download complete.")

    # Run the classification!
    print(f"Classifying video: {video_path}...")
    results = pipe(video_path)
    
    print("\n--- Classification Results ---")
    for result in results:
        print(f"Label: {result['label']}, Score: {result['score']:.4f}")

except Exception as e:
    # This is the 'catch' block. It runs if anything in 'try' fails.
    print(f"\n--- AN ERROR OCCURRED ---")
    print(f"Error Type: {type(e).__name__}")
    print(f"Details: {e}")
    print("Failed to load pipeline or classify video. Please check your model name and internet connection.")

[merveenoyan]

@FWKMultiverse this is probably an AI comment, please do not write AI comments in this repository, because the comment might mislead other devs.

VideoMAE-base for instance is a headless backbone just like VJEPA2, but you can load it like this. pipeline handles it. the issue here is that old models have image processors while new ones have video, which is not supported by the pipeline. e.g.

from transformers import pipeline

pipe = pipeline("video-classification", "MCG-NJU/videomae-base-short-ssv2")

This works and you only get this warning which is shown for headless backbone models.

Some weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base-short-ssv2 and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
preprocessor_config.json: 100%

The pipeline abstraction automatically initializes a random head for you.