Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces

Abstract

Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4× without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method ProMAG includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to direct extensions of existing video tokenizers. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a reduced token budget.

ProMAG Reconstruction

We show a comparison of reconstruction results for our method ProMAG at different temporal compression ratios 4×, 8× and 16× temporal compression with our progressive growing approach. Note: For all cases in this work we use 8× spatial compression. ProMAG can achieve similar reconstruction quality without much degradation, even though we increase the temporal comression from 4× to 16× keeping the same number of channels in latent space.

16 channel Latent (z_dim=16)

Citation

Acknowledgements

We thank Kuldeep Kulkarni, Cusuh Ham, and Gunjan Aggarwal for proofreading the paper. We are also grateful to Richard Zhang, Zhifei Zhang, Jianming Zhang, Gunjan Aggarwal, Kevin Duarte, Yang Zhou, Zhan Xu, and Lakshya for several fruitful discussions. The website template is taken from Custom Diffusion (which was built on DreamFusion's project page).
* This work was done while Yitian Zhang was an intern at Adobe Research.