TokenFlow AI (Text to Video) Tool : What is it & How to Use?2 min read

July 25, 2023Written by PY

Powering Text-Driven Video Editing

TokenFlow’s method revolves around a crucial insight: maintaining temporal consistency in video editing is closely tied to the temporal consistency of its feature representation. The video’s natural features exhibit a shared, temporally consistent representation, which tends to break when edited frame by frame. TokenFlow’s brilliance lies in ensuring the same level of feature consistency as seen in the original video features.

Diffusion Features

The key discovery driving TokenFlow’s success is that a temporally-consistent edit can be achieved by enforcing consistency on the internal diffusion features across frames during the editing process. The process unfolds in two main steps:

Joint Editing with Extended Attention: The journey begins by sampling keyframes from the noisy video and jointly editing them using an extended-attention block. This results in a set of edited tokens, forming the foundation for the subsequent steps.
Propagating Edited Tokens: Next, the edited tokens are skillfully propagated across the video, guided by the pre-computed correspondences of the original video features. This delicate process ensures that the edited video maintains its temporal coherence while aligning with the text prompt.

TokenFlow Editing Results

TokenFlow’s capabilities are nothing short of awe-inspiring, as it produces consistent and high-quality video edits that bring text prompts to life. Hovering over the videos reveals the original video and the text prompts that influenced the remarkable transformations. From an ice sculpture to a robotic wolf, from a Van Gogh painting to a mesmerizing origami stork, the possibilities are boundless.

Setting New Standards through Comparison

TokenFlow’s achievements shine through when compared to other state-of-the-art video editing models:

Text-to-video [1]
Tune-a-video [2]
Gen-1 [3]
Per frame PnP [4]

Each comparison highlights TokenFlow’s unique strengths and sets it apart as a true game-changer in text-driven video editing.

TokenFlow: Diffusion

The creators of TokenFlow, Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel, have propelled the field of generative AI forward. Their research, presented in the article “TokenFlow: Consistent Diffusion Features for Consistent Video Editing,” published on arXiv, opens up new possibilities for creative expression and storytelling through text-driven video editing.