Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces

Aniruddha Mahapatra¹

Long Mai¹

Yitian Zhang^1,2

David Bourgin¹

Feng Liu¹

¹ Adobe Research ² Northeastern University

ArXiv Project Gallery

First row: videos showing reconstruction results under different settings.
Second row: videos showing text-to-video geenrating results with different latent spaces.
Third row: videos corresponding to figures in the main paper.

Reconstruction Comparison
4X (Baselines) Reconstruction Comparison
8X Reconstruction Comparison
16X Reconstruction Comparison
Overlapping Chunks
Text-to-Video Generation
16X Latent Text-to-Video Generation
4X v/s 16X Latent Text-to-Video Generation
16X Latent Long Video Text-to-Video Generation
Overlapping Chunks
Figure 2
Main Paper Figure 3
Main Paper Figure 5
Main Paper Figure 9
Main Paper

Reconstruction Comparison (16× Temporal Compression)

We show a comparison of reconstruction results of directly extending MagViTv2 to 16× temporal compression against our method ProMAG at 16× with our progressive growing approach. ProMAG at 16× temporal compression has much more accurate reconstructions and not not contain artifacts observed in the reconstruction results of directly extending MagViTv2 to 16× temporal compression.

8 channel Latent (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

Ground-Truth

MagViTv2-16× (z_dim=8)

ProMAG-16× (z_dim=8)

16 channel Latent (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)

Ground-Truth

MagViTv2-16× (z_dim=16)

ProMAG-16× (z_dim=16)