Grounded Multilingual Audio Description
Modern multimodal captioning systems (e.g., GPT-4V, Qwen-VL, LLaVA) inherit systematic biases from English-centric vision encoders such as CLIP, leading to hallucinations and weak grounding. These limitations make them unreliable for applications that require precise and culturally adaptive narration.We propose GMAD, a modular alternative to end-to-end captioning. GMAD replaces monolithic vision–language models with a language-neutral Temporal Scene Graph, constructed from independent detectors, trackers, and attribute models. Narration is generated only after this structured representation is formed, using text-only LLMs directly in the target language. GMAD demonstrates that accurate multilingual grounded narration does not require multimodal end-to-end learning. Instead, a clean separation between perception and language generation yields more reliable, bias-resistant descriptions across languages and cultures.
GMAD is a modular pipeline for generating grounded multilingual audio descriptions from raw video. Visual perception is executed entirely through specialized vision models detection tracking and attribute extraction. Narration is produced by text only LLMs ensuring low hallucination and full language flexibility.
GMAD proceeds through a transparent sequence of stages.
Splits the video into scenes and extracts representative frames.
Uses Florence2 YOLO or RTDETR to identify objects in each frame.
Associates detections across time using a sparse ByteTrackLite tracker.
Chooses a canonical frame per object and crops entities for analysis.
Extracts colors materials textures and coarse attributes from each crop.
Builds a symbolic representation of all objects their attributes their relations and all temporal intervals.
Uses QwenVL to detect high level actions and scene events.
A text only LLM merges the TSG with Qwen cues generating grounded narration directly in the target language without translation.
Narration audio is synthesized and aligned with the original video.
GMAD/ scene_segmentation/ detection_tracking/ clip_enrichment/ tsg_graph_construction/ qwen/ fusion_layer/ Audio_Layering/ tts/ prompts.yaml config.yml