MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models

Yifan Xu; Chao Zhang; Ruifei Ma; Fei Gao; Zhifei Yang; Jiaxing Qi; Zhipeng Chen

High-level overview of MotionEnhancer and observations on VDM attention.

MotionEnhancer distills motion priors from a video diffusion model and uses them to guide the text-to-vision attention of a vision-language model during supervised fine-tuning.

TL;DR. We leverage motion-aware attention from a pretrained Video Diffusion Model to improve fine-grained motion understanding in Vision-Language Models, without modifying the VLM architecture or introducing additional inference-time tools.

Abstract

Vision-Language Models (VLMs) have achieved impressive performance in video understanding, but they are often better at high-level event and story understanding than at capturing fine-grained motion details. In contrast, Video Diffusion Models (VDMs) must explicitly model temporal dynamics in order to generate coherent videos.

We introduce MotionEnhancer, a simple framework that distills motion priors from a powerful VDM and transfers them to a VLM through text-to-vision attention alignment. To extract motion-focused supervision from the VDM, MotionEnhancer introduces two parameter-free modules: Motion-sensitive Head Selection (MHS) and Motion-salient Text Token Identification (MTTI). These modules identify motion-relevant attention heads and text tokens in a computation-only manner.

Experiments on MotionBench and FAVOR-Bench show consistent improvements across different VLM backbones and model scales, demonstrating that motion priors learned by video generative models can effectively enhance fine-grained video motion understanding.

Motivation

Current video VLMs commonly process videos as sequences of visual frames and are optimized for semantic understanding and language generation. Such objectives can often be satisfied using appearance cues and high-level event information, which may leave subtle temporal changes under-modeled.

VLM

Strong semantics, limited motion sensitivity

VLMs are highly effective at recognizing objects, scenes, and macro-events, but can overlook fine-grained changes in action order, motion direction, repetition, and object dynamics.

VDM

Motion knowledge emerges from video generation

Video diffusion models must reconstruct and generate temporally coherent content. Their internal attention therefore contains useful signals about where motion-related concepts are grounded across space and time.

Can video generation teach video understanding how things move?

Method

MotionEnhancer uses a frozen Video Diffusion Model as a source of motion priors. Attention maps are extracted from the VDM through DDIM inversion/sampling, refined to retain motion-sensitive information, and then used as auxiliary supervision for VLM fine-tuning.

Overview of MotionEnhancer. VDM attention is refined by MHS and MTTI and then aligned with the VLM's text-to-vision attention during supervised fine-tuning.

1

Motion Prior Extraction

We obtain attention maps from a frozen VDM during DDIM sampling after inversion. The extraction process is performed offline, allowing the resulting motion priors to be reused across different VLMs and ablation settings.

2

Motion-sensitive Head Selection (MHS)

MHS identifies motion-relevant VDM attention heads using three complementary measurements: Diagonal Focus Coefficient (DFC), Temporal Continuity Score (TCS), and Diagonal Saliency Ratio (DSR). The top-scoring heads are aggregated to form cleaner motion-aware attention.

3

Motion-salient Text Token Identification (MTTI)

MTTI evaluates each text token using both its average attention and its inter-frame attention variation. Tokens with stronger saliency and temporal dynamics are selected for motion-focused alignment.

Attention Alignment

After refinement, the VDM provides a motion-aware text-to-vision attention target. The corresponding VLM attention is aligned to this target during supervised fine-tuning, jointly optimized with the standard autoregressive objective.

Parameter-free MHS Parameter-free MTTI Offline VDM prior extraction No inference-time tool calling Applicable to different VLM backbones

Results

MotionEnhancer consistently improves fine-grained motion understanding across both Qwen2.5-VL and InternVL3. Notably, the 3B Qwen2.5-VL model enhanced by MotionEnhancer surpasses the vanilla 7B counterpart on both evaluated benchmarks.

MotionBench — Overall Accuracy

Backbone	Base	+ MotionEnhancer
Qwen2.5-VL-3B	53.56	56.60 (+3.04)
Qwen2.5-VL-7B	52.81	57.04 (+4.23)
InternVL3-2B	53.96	55.50 (+1.54)
InternVL3-8B	54.88	57.69 (+2.81)

FAVOR-Bench — Overall Accuracy

Backbone	Base	+ MotionEnhancer
Qwen2.5-VL-3B	37.43	44.53 (+7.10)
Qwen2.5-VL-7B	42.61	46.88 (+4.27)
InternVL3-2B	39.27	43.71 (+4.44)
InternVL3-8B	45.82	48.94 (+3.12)

Numbers above report Overall accuracy. Improvements are measured against the corresponding reproduced vanilla backbone.

Citation

If you find MotionEnhancer useful for your research, please consider citing our CVPR 2026 paper.

@InProceedings{Xu_2026_CVPR,
  author    = {Xu, Yifan and Zhang, Chao and Ma, Ruifei and Gao, Fei and Yang, Zhifei and Qi, Jiaxing and Chen, Zhipeng},
  title     = {MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2026},
  pages     = {2778--2787}
}