Audio from video in jinja by zucchini-nlp · Pull Request #49382 · huggingface/transformers

zucchini-nlp · 2026-10-07T08:36:13Z

Models that don't have explicit format for video+audio or aren't trained for it can use pass audio in conversation to indicate where to add audio tokens. This works in plug and play mode for any omni modal model

messages = [
            {
                "role": "user",
                "content": [
                    {"type": "video", "path": video_file_path},
                    {"type": "audio"}, # pass without any url/path if load_audio_from_video, it'll be auto-decoded from video URL
                    {"type": "text", "text": "Which of these animals is making the sound?"},
                ],
            },
        ]

inputs = processor.apply_chat_template(
            messages,
            add_generation_prompt=True,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
            # num_frames=2, # if you're sampling the video, provide necessary kwargs
            load_audio_backend="torchcodec", # video formats arent all supported by librosa, torchcodec is recommended
            load_audio_from_video=True, # to make sure that the audio track is decoder and passed to processor.__call__
        ).to("cuda")

print(inputs.keys()) # has one video and one audio

Omni models with strict formatting and interleaving of audio and video per frame/chunk can inherit from OmniModalProcessorMixin and indicate how to replace_video_audio_token. For those models the conversation holds only a video entry and the rest is handled by processors, since model has a strict format that it's trained with!

import torch
from transformers import EmbeddingGemma2Processor, EmbeddingGemma2VideoProcessor, AutoModel

model_id = "my_local_embeddinggemma2"
processor = EmbeddingGemma2Processor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="cuda", dtype="float16")


messages_audio = [
    {
        'role': 'user',
        'content': [
            {"type": "audio", "path": "/raid/raushan/audio.wav"},
            {'type': 'text', 'text': "What do you hear in this audio track?"}
        ]
    }
]

messages_silent_video = [
    {
        'role': 'user',
        'content': [
            {"type": "video", "video": "/raid/raushan/karate.mp4"},
            {'type': 'text', 'text': "What do you hear in this audio track?"}
        ]
    }
]


messages_video_with_audio = [
        {
            'role': 'user',
            'content': [
                {"type": "video", "path": "https://huggingface.co/datasets/hf-internal-testing/fixtures_videos/resolve/main/Cooking_cake.mp4"},
                {'type': 'text', 'text': "Describe what the man in the video is saying."}
            ]
        }
    ]

# Any order of video and audio within a batch works!
inputs = processor.apply_chat_template(
        [messages_audio, messages_video_with_audio, messages_silent_video],
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        overflow_strategy="uniform",
        max_frames=11,
        do_sample_frames=True,
        load_audio_from_video=True,
        add_generation_prompt=True,
        enable_thinking=False,
        add_timestamps=True, # audio+video can't work with no timstamps for EG2
        truncation=False, # try enabling it to truncate audio to 16k
        max_length=16_000, # is used by text/audio both unless we pass nested dict
    ).to(model.device)

with torch.no_grad():
    out = model(**inputs)
    print(f"Logits: {out.last_hidden_state.shape}")

HuggingFaceDocBuilderDev · 2026-10-07T08:45:52Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

vasqu

Wondering whether we should wait for Guarin's PR first but overall looks nice. Its a bit messy on the git diff tho because of just moving things 😢 not on you

vasqu · 2026-10-08T15:29:58Z

+        sscp_num_layers = 2
+        sscp_kernel, sscp_stride, sscp_padding = 3, 2, 1


lets add a note that we should change this for future additions not on you but good to raise awareness cc @eustlb

Meaning that we do not hardcode values

vasqu · 2026-10-08T15:43:26Z

+            **kwargs,
+        )
+
+        # TODO @guarin: move it to processor level argument. Used by EG2 and qwen-omni series


how much does that PR need? Maybe let's be patient and merge that first -> simpler moving here

Hmm can we not already declare load_audio_from_video as a processor kwarg instead of video? Afaik it doesn't need defaults or anything so should already work.

Also apply_chat_template(..., load_audio_from_video, processor_kwargs,) is now weird because load_audio_from_video should be in processor_kwargs and not on its own.

that would need us to break qwen models, where it is in video_kwargs atm. Imo it's better to keep TypedDict deprecation separate from this PR

Also apply_chat_template(..., load_audio_from_video, processor_kwargs,)

hmm yeah, but i think some kwargs are okay to be an arg of its own as long as it is used by chat_template, same as return_tensors which is consumed via jinja directly and also by processors

Hmm but then we add it to video kwargs now just to deprecate it again. And then we have to support bc for the deprecated version.

yeah 😢 we already have a "bad" pattern in old models so BC cycle is unevitable, no?

schechterh · 2026-10-13T08:01:41Z

Thank you for all your work to enable this awesome feature :)

Besides the OmniModalProcessorMixin inheritance issue (#49382 (comment)) and the question of where to put special tokens for EG2 (#49382 (comment)), here's a few issues I found:

Empty audio segments & token count mismatch on the last segment in replace_video_with_audio_token
In the per-second audio loop:

if (processed_audio_tokens + num_audio_tokens_per_sec) > num_audio_tokens_total:
    num_audio_tokens_per_sec = num_audio_tokens_total - processed_audio_tokens
segments.append(f"{audio_ts} {self.audio_token * num_audio_tokens_per_sec}")
processed_audio_tokens += num_audio_tokens_per_sec

Two issues come up here:

When audio is shorter than the video: Once processed_audio_tokens reaches num_audio_tokens_total, num_audio_tokens_per_sec becomes 0, and subsequent iterations still append bare timestamps without any audio tokens. When there are no audio tokens remaining, no timestamp or special token delimiters should be emitted for that second.
When num_audio_tokens_total is slightly larger than total_seconds * 25: On a standard 10.0s video with AAC audio, we actually produce 251 soft tokens (just over 25 * 10). Capping every 1s segment at 25 tokens only places 250 <|audio|> tokens in input_ids, causing ValueError: Audio features and audio tokens do not match, tokens: 250, features: .... The last audio segment should absorb any remaining tokens (remaining = num_audio_tokens_total - processed_audio_tokens) so the placeholder count in input_ids always matches compute_sscp(audio_mask).

Fractional framerates (e.g. 29.97, the standard NTSC framerate) shift timestamps and double audio chunks:
In EmbeddingGemma2VideoProcessor.sample_frames, sampling a 29.97 fps video at fps=1 picks frame indices 0, 29, 59, 89, ..., which have timestamps 0.0, 0.9676, 1.9686, 2.9696, ...:

Using int(seconds // 60) / int(seconds % 60) floors 0.0 to 00:00, 0.9676 to 00:00, 1.9686 to 00:01, etc., so frames 0 and 1 both get 00:00 and all subsequent timestamps are shifted off by 1s (this actually also affects the video-only timestamp formatting in replace_video_token).
math.ceil(metadata.timestamps[i] - metadata.timestamps[i - 1]) evaluates math.ceil(59/29.97 - 29/29.97) = math.ceil(1.001001) = 2, which inserts two 1-second audio blocks after every single 1fps frame, instead of just one.

To solve these, we should round frame timestamps to the nearest integer second (frame_seconds = [round(s) for s in metadata.timestamps]), and determine the audio seconds after frame i via range(frame_seconds[i], frame_seconds[i + 1]) for i < len(frame_seconds) - 1 and range(frame_seconds[-1], max(1, int(metadata.duration))) for the last frame (matching max(1, int(metadata.duration * fps)) in sample_frames). Note this does have some issues when fps > 1 but IIRC we had agreed to support fps<=1 as the primary use case here.

Audio tokens are dropped when add_timestamps=False even though load_audio_from_video=True:
In EmbeddingGemma2Processor.replace_video_token, if not add_timestamps: returns early before checking if kwargs.get("load_audio_from_video") and video_to_audio_indices[video_idx] is not None:, which means calling load_audio_from_video=True without explicitly passing add_timestamps=True extracts input_features for the audio track but inserts 0 <|audio|> tokens into input_ids. When load_audio_from_video=True, we should check load_audio_from_video before the if not add_timestamps: early return. As a sidenote, we should note that we recommend adding timestamps when using load_audio_from_video for EG2.
Silent video with load_audio_from_video=True raises IndexError:
In OmniModalProcessorMixin.__call__ (src/transformers/processing_utils.py), when all input videos are silent and no standalone audio is passed, self._resolve_audio_from_video returns audio = []. Then if audio is not None and self._audio_processor is not None: evaluates to True for [] and calls self._process_audio([]), which raises IndexError: list index out of range. Checking if audio and self._audio_processor is not None: (or returning None from _resolve_audio_from_video when audio_resolved is empty) should fix this.

zucchini-nlp · 2026-10-13T14:05:42Z

@schechterh thanks for review and testing it. I addressed the edge cases you mentioned in my last PR and added BOA/EOA tokens to wrap each audio frame. For add_timestamps=False, load_audio_from_video=True - i decided to raise an error since that is the recommended way for EG2's training setting

zucchini-nlp · 2026-10-13T15:50:09Z

@vasqu @guarin i think it is now cleaned up and I fixed edge cases for EG2. A few tests need fixing and I will be off tmrw, feel free to take over if you need it for release or i will fix CI on thursday

vasqu · 2026-10-13T15:52:52Z

No rush here, @guarin can you make a pass here first?

guarin

Super nice update, modular inheritance with OmniModelProcessorMixin works now well 👍🏼 Left some small comments, will have a closer look tomorrow

guarin · 2026-10-13T16:39:39Z

+
+    def replace_video_with_audio_token(
+        self, video_inputs: dict, video_idx: int, audio_inputs: dict, audio_idx: int, **kwargs: VideosKwargs
+    ) -> str:
+        raise NotImplementedError()
+


If I understand correctly this method is not called by default from anywhere. The idea is that users override it and then call it from within replace_video_token?

ideally we could call one of two depending on load-audio-from-video but then it forces EG2 to override all. It has additional args to check - dep on that it chooses how to add special tokens. 😿
I think since feat is yet not used widely we can keep it optional, maybe move into base class later on

guarin · 2026-10-13T16:46:07Z

+            "load_audio_from_video": load_audio_from_video,
+        }


Imo no need to set load_audio_from_video here as it is already known by the caller of the method

oh it is to pass the arg down to processor, since it is not in kwargs by default 🫠
after you pr we can try to put it in kwargs by default and deprecate the standalone arg - currently it is coming from main as standalone

…hat?

github-actions · 2026-10-15T20:55:14Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: embedding_gemma2, gemma4, gemma4_unified, qwen2_5_omni, qwen3_omni_moe

github-actions · 2026-10-15T21:06:26Z

CI recap

Dashboard: View test results in Grafana
Latest run: 38805616629:1
Result: failure | Jobs: 15 | Tests: 202,415 | Failures: 1 | Duration: 13h 53m

zucchini-nlp · 2026-10-16T00:25:40Z

run-slow: embedding_gemma2

zucchini-nlp · 2026-10-16T00:26:12Z

@vasqu should be ready - added the slow test as well. Lets see if expectations match with runners

github-actions · 2026-10-16T00:27:01Z

CI (Nvidia + AMD)

Nvidia CI

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	9bc7dd55	workflow commit (merge commit)
PR	f53dee46	branch commit (from PR)
main	b0cdc61e	base commit (on `main`)

Model CI Report

❌ 1 new failed tests from this PR 😭

embedding_gemma2:
tests/models/embedding_gemma2/test_modeling_embedding_gemma2.py::EmbeddingGemma2IntegrationTest::test_model_video_and_audio (✅ ⟹ ❌)

AMD CI

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	9bc7dd55	workflow commit (merge commit)
PR	f53dee46	branch commit (from PR)
main	b0cdc61e	base commit (on `main`)

Model CI Report

❌ 1 new failed tests from this PR 😭

embedding_gemma2:
tests/models/embedding_gemma2/test_modeling_embedding_gemma2.py::EmbeddingGemma2IntegrationTest::test_model_video_and_audio (✅ ⟹ ❌)

molbap · 2026-10-16T08:59:16Z

+        frame_seconds = [round(s) for s in metadata.timestamps]
+        num_frames = len(frame_seconds)
+
+        # Mirrors `max(1, int(metadata.duration * fps))` in `sample_frames` for fps=1
+        total_seconds = max(1, int(metadata.duration))
+
+        num_audio_tokens_per_sec = self.audio_frame_boundary(seconds=1)
+
+        mapped_audio_idx = video_to_audio_indices[video_idx]
+        audio_mask = processed_audio["input_features_mask"][mapped_audio_idx]
+        num_audio_tokens_total = self.compute_sscp(audio_mask)
+
+        # Each 1s chunk of video/audio is wrapped in special tokens:
+        # "00:00 <BOI> [frame image tokens] <EOI> 00:00 <BOA> [audio tokens] <EOA> ..."
+        segments = []
+        processed_audio_tokens = 0
+        for i, frame_sec in enumerate(frame_seconds):
+            # vision frame block, labeled with its own (rounded) timestamp
+            segments.append(
+                f"{self.timestamp_fmt(frame_sec)} {self.boi_token}{self.video_token * num_soft_tokens}{self.eoi_token}"
+            )
+
+            # 1s audio chunks that follow this frame before the next one
+            if i < num_frames - 1:
+                audio_seconds = range(frame_sec, frame_seconds[i + 1])
+            else:
+                audio_seconds = range(frame_sec, total_seconds)
+
+            for sec in audio_seconds:
+                n_tokens = num_audio_tokens_per_sec
+
+                # Clamp to account for accumulated rounding/truncation error
+                n_tokens = min(n_tokens, num_audio_tokens_total - processed_audio_tokens)
+
+                # The very last audio chunk absorbs any leftover tokens from rounding/sscp
+                if i == num_frames - 1 and sec == audio_seconds[-1]:
+                    n_tokens += max(0, num_audio_tokens_total - processed_audio_tokens - n_tokens)
+
+                if n_tokens > 0:
+                    segments.append(
+                        f"{self.timestamp_fmt(sec)} {self.boa_token}{self.audio_token * n_tokens}{self.eoa_token}"
+                    )
+                    processed_audio_tokens += n_tokens
+


I explored a bit this part and there might be an issue, LMK if I'm wrong. since we round, say at fps=2 on a 10s video the last range will be (9.5, 10) rounded so (10, 10) which is empty. in this case at the end there will be some leftover audio tokens since their range is none.

so they would need to be added before the segments are joined, like

leftover_tokens = num_audio_tokens_total - processed_audio_tokens if leftover_tokens > 0: # ...compute last second timestamp and add as many audio tokens as needed in segments

deorecate and ley users decicde where to add

c2b4ee5

guarin reviewed Oct 7, 2026

View reviewed changes

Comment thread src/transformers/processing_utils.py Outdated

zucchini-nlp added 10 commits October 7, 2026 11:01

let both be passed but order is wrong yet

eb1d7ce

move it all over here

03b33cc

nit

3640528

split it into overridable fn

3ce18d8

fix the order

91d5e88

CI repo consistency

a3dff7b

modular fix and revert bad rebase

3032525

whta?

6f61b65

eg2 modular and tests

7f29f93

actually fixing it might breaks other models...

082ae5c

vasqu reviewed Oct 8, 2026

View reviewed changes

guarin reviewed Oct 12, 2026

View reviewed changes

Comment thread src/transformers/processing_utils.py Outdated

Comment thread src/transformers/processing_utils.py Outdated

Comment thread src/transformers/models/embedding_gemma2/modular_embedding_gemma2.py Outdated

Comment thread src/transformers/processing_utils.py

guarin reviewed Oct 12, 2026

View reviewed changes

Comment thread src/transformers/processing_utils.py

zucchini-nlp added 3 commits October 13, 2026 15:03

some comments and BOA/EOA

a9052fa

nits

ff6bf21

henrique's big comment

b3950cc

zucchini-nlp added 3 commits October 13, 2026 16:06

Merge remote-tracking branch 'upstream/main' into audio-from-video

b55aa03

fix repo

20e0a3d

fix some tests, maybe need @require_torchcodec_greater now :(

ca68623

guarin reviewed Oct 13, 2026

View reviewed changes

zucchini-nlp added 3 commits October 15, 2026 21:37

fix CI i hope

94923ec

dont allow old qwen - we need to completely override processor one day

e430154

skip this one

1b8574d

zucchini-nlp added 2 commits October 15, 2026 22:28

add bucket

fe7cb0c

very very weird, but runners dont have latest torchcodec yet, maybe t…

f53dee4

…hat?

molbap reviewed Oct 16, 2026

View reviewed changes

		sscp_num_layers = 2
		sscp_kernel, sscp_stride, sscp_padding = 3, 2, 1

Conversation

zucchini-nlp commented Oct 7, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Oct 7, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

schechterh commented Oct 13, 2026

Uh oh!

zucchini-nlp commented Oct 13, 2026

Uh oh!

zucchini-nlp commented Oct 13, 2026

Uh oh!

vasqu commented Oct 13, 2026

Uh oh!

guarin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026

CI recap

Uh oh!

zucchini-nlp commented Oct 16, 2026

Uh oh!

zucchini-nlp commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI (Nvidia + AMD)

Nvidia CI

Commit Info

Model CI Report

AMD CI

Commit Info

Model CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Oct 7, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading