This repository provides a curated collection of papers, benchmarks, and resources from our survey:
"AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation" (IEEE TCSVT'26).
📝 Authors: Dahyeon Kye, Changhyun Roh, Sukhun Ko, Chanho Eom, Jihyong Oh†
🎓 Institution: CMLab, Chung-Ang University
Video Frame Interpolation (VFI) is a fundamental Low-Level Vision (LLV) task that synthesizes intermediate frames between existing ones while maintaining spatial and temporal coherence. VFI techniques have evolved from classical motion compensation-based approach to deep learning-based approach, including kernel-, flow-, hybrid-, phase-, GAN-, Transformer-, Mamba-, and more recently diffusion model-based approach. We introduce AceVFI, the most comprehensive survey on VFI to date, covering over 250+ papers across these approaches. We systematically organize and describe VFI methodologies, detailing the core principles, design assumptions, and technical characteristics of each approach. We categorize the learning paradigm of VFI methods namely, Center-Time Frame Interpolation (CTFI) and Arbitrary-Time Frame Interpolation (ATFI). We analyze key challenges of VFI such as large motion, occlusion, lighting variation, and non-linear motion. In addition, we review standard datasets, loss functions, evaluation metrics. We examine applications of VFI including event-based, cartoon, medical image VFI and joint VFI with other LLV tasks. We conclude by outlining promising future research directions to support continued progress in the field. This survey aims to serve as a unified reference for both newcomers and experts seeking a deep understanding of modern VFI landscapes.
- 🆕 2026-06: Updated AAAI’26, ICLR’26, ICML’26, and CVPR’26 papers.
- 📌 2026-02: Accepted for publication in IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT); the final version will be released soon.
- 📌 2025-06: Paper released to ArXiv.
- 🚀 2025-05: Repository initialized.
If you find this survey helpful, please consider citing us:
@ARTICLE{11427010,
author={Kye, Dahyeon and Roh, Changhyun and Ko, Sukhun and Eom, Chanho and Oh, Jihyong},
journal={IEEE Transactions on Circuits and Systems for Video Technology},
title={AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation},
year={2026},
volume={},
number={},
pages={1-1},
keywords={Videos;Interpolation;Kernel;Measurement;Surveys;Pipelines;Feature extraction;Training;Costs;Circuits and systems;Video Frame Interpolation;Generative Inbetweening;Video Generation;Low-Level Vision},
doi={10.1109/TCSVT.2026.3672288}}
We welcome contributions from the VFI research community!
If you have a new method, dataset, or related resource that fits within the scope of this VFI repository, please feel free to submit a pull request (PR) with the following:
A brief description of your method/resource.
Relevant links (e.g., arXiv, project page, code).
Suggested placement (e.g., under “2.3 Diffusion Model-based", "6.4 Joint Task”).
Our maintainers will review your submission and merge it if appropriate. We hope this page will grow into a collaborative hub for Video Frame Interpolation (VFI) research.
You can find the preprint of our survey here:
📄 arXiv:2506.01061
The overview of our survey paper:
We categorize recent VFI papers by methodology (up to Jul. 4, 2026).
| Title | Publication | Date |
|---|---|---|
| PhaseNet for Video Frame Interpolation | CVPR | 2018 |
| Phase-based Frame Interpolation for Video | CVPR | 2015 |
| Title | Publication | Date |
|---|---|---|
| ST-MFNet: A Spatio-temporal Multi-flow Network for Frame Interpolation | CVPR | 2022 |
| Video Frame Interpolation via Down--up Scale Generative Adversarial Networks | Computer Vision and Image Understanding | 2022 |
| Multi-scale Attention Generative Adversarial Networks for Video Frame Interpolation | IEEE Access | 2020 |
| Efficient Video Frame Interpolation Using Generative Adversarial Networks | Applied Sciences | 2020 |
| Frame-GAN: Increasing the Frame Rate of Gait Videos with Generative Adversarial Networks | Neurocomputing | 2020 |
| AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation | CVPR | 2020 |
| Zoom-in-to-check: Boosting Video Interpolation Via Instance-level Discrimination | CVPR | 2019 |
| Generating Realistic Videos From Keyframes with Concatenated Gans | IEEE Transactions on Circuits and Systems for Video Technology | 2018 |
| Frame Interpolation with Multi-scale Deep Loss Functions and Generative Adversarial Networks | arXiv | 2017 |
| Frame Interpolation Using Generative Adversarial Networks | arXiv | 2017 |
| Title | Publication | Date |
|---|---|---|
| LC-Mamba: Local and Continuous Mamba with Shifted Windows for Frame Interpolation | CVPR | 2025 |
| VFIMamba: Video Frame Interpolation with State Space Models | NeurIPS | 2024 |
| Title | Publication | Date |
|---|---|---|
| Boost Video Frame Interpolation via Motion Adaptation | BMVC | 2023 |
| Scene-Adaptive Video Frame Interpolation via Meta-Learning | CVPR | 2020 |
| Title | Publication | Date |
|---|---|---|
| Surface-Aware Feed-Forward Quadratic Gaussian for Frame Interpolation with Large Motion | NeurIPS | 2025 |
| Title | Publication | Date |
|---|---|---|
| DiffPCI: Large Motion Point Cloud frame Interpolation with Diffusion Model | ICCV | 2025 |
We include commonly used datasets for evaluating VFI performance.
Datasets are categorized into Triplet and Multi-frame types depending on the supervision format.
Early learning-based VFI approaches primarily rely on triplet datasets, where two input frames are used to predict the temporally centered GT frame.
| Dataset | Venue | Type | Resolution | Split | #Videos / #Triplets | URL |
|---|---|---|---|---|---|---|
| Middlebury | IJCV'11 | 🔹 T | ≤ 640×480 (VGA) | test | 12 | 🔗 |
| UCF101 | CRCV'12 | 🔹 T | 256×256 | test | 379 | 🔗 |
| Vimeo90K | IJCV'19 | 🔹 T | 448×256 | train / test | 51,312 / 3,782 | 🔗 |
| SNU-FILM | AAAI'20 | 🔹 T | ≤ 1280×720 (HD) | test | 1,240 | 🔗 |
| ATD-12K | CVPR'21 | 🔹 T | 1280×720, 1920×1080 (FHD) | train / test | 10,000 / 2,000 | 🔗 |
Multi-frame datasets enable dense temporal supervision and are commonly used in both CTFI and ATFI settings. They support flexible frame sampling and evaluation under diverse temporal intervals.
| Dataset | Venue | Type | Resolution | Split | #Videos / #Triplets | URL |
|---|---|---|---|---|---|---|
| Xiph | - | 🔸 M | 2048x1080 (2K), 3840×2160 (4K) | test | 8 | 🔗 |
| KITTI | CVPR'12 | 🔸 M | 1240×376 | train / test | 194 / 195 | 🔗 |
| DAVIS | CVPR'16 | 🔸 M | 1920×1080 | train / test | 30 / 20 | 🔗 |
| HD | TPAMI'19 | 🔸 M | 960×544, 1280×720, 1920×1080 | test | 11 | 🔗 |
| Sintel | ECCV'12 | 🔸 M | 1024×436 | train / test | 23 / 12 | 🔗 |
| Adobe240 | CVPR'17 | 🔸 M | 1280×720 | train / test | 61 / 10 | 🔗 |
| GOPRO | CVPR'17 | 🔸 M | 1280×720 | train / test | 22 / 11 | 🔗 |
| X4K1000FPS | ICCV'21 | 🔸 M | 4096×2160 | train / test | 4,408 / 15 | 🔗 |
| WebVid-10M | ICCV'21 | 🔸 M | varied | train | 10M | 🔗 |
| LAVIB | NeurIPS'24 | 🔸 M | 4096×2160 | train / test | 188,644 / 53,494 | 🔗 |
| OpenVid | ICLR'25 | 🔸 M | ≥ 512×512, 1920×1080 | train | 1M | 🔗 |
🔹 T (Triplet dataset): Two input frames predict the center frame
🔸 M (Multi-frame dataset): Multiple frames allow dense temporal supervision
This section summarizes commonly used metrics for evaluating the quality of video frame interpolation (VFI) results.
These metrics compare each interpolated frame to its ground truth (GT) reference on a pixel level.
-
PSNR (Peak Signal-to-Noise Ratio)
Measures reconstruction fidelity via Mean Squared Error (MSE).
📌 Higher is better, but it often doesn't align with human perception, especially in high-frequency regions. -
SSIM (Structural Similarity Index)
Compares luminance, contrast, and texture to evaluate structural similarity.
📌 More perceptually aligned than PSNR. Higher SSIM indicates stronger similarity. -
IE (Interpolation Error)
Root-mean-square error between the interpolated and GT frame.
📌 Simple and intuitive but limited in perceptual relevance.
These metrics better reflect human perception by analyzing textures, semantics, and style.
-
NIQE (Natural Image Quality Evaluator)
A no-reference metric using statistical deviations from natural images.
📌 Lower NIQE implies higher natural image quality. -
FID (Fréchet Inception Distance)
Measures distributional difference in features between generated and GT frames.
📌 Lower FID indicates better semantic alignment. -
LPIPS (Learned Perceptual Image Patch Similarity)
Uses deep features to assess perceptual similarity.
📌 Lower LPIPS = better perceptual similarity. -
FloLPIPS
Motion-aware LPIPS variant that uses optical flow for weighting. -
STLPIPS
Shift-tolerant version of LPIPS, robust to slight misalignments. -
DISTS (Deep Image Structure and Texture Similarity)
Separately evaluates structure and texture using deep features.
📌 Balances local detail and global coherence.
These metrics evaluate spatiotemporal coherence across video sequences, important for smooth motion and consistency.
-
VSFA (Video Spatial-Feature Aggregation)
No-reference model estimating perceptual quality from human-labeled videos using deep recurrent features. -
tOF (temporal Optical Flow consistency)
Measures how consistent optical flow is across frames.
📌 Lower tOF = smoother motion continuity. -
FVD (Fréchet Video Distance)
Uses I3D features to compare real vs generated video distributions.
📌 Lower FVD = better realism and temporal quality. -
FVMD (Fréchet Video Motion Distance)
Enhances FVD by disentangling motion from appearance for better motion consistency evaluation. -
VBench
Large-scale, no-reference benchmark for evaluating fidelity, coherence, and realism using semantic video representations.
📌 Ideal for reference-free evaluation.