MoRoute1T2V
TArmed man walks through a desolate cornfield.
↓
1Sun Yat-sen University 2Orange Team, Moku Lab, HUJING Digital Media & Entertainment Group 3Huazhong University of Science and Technology
*Corresponding author
One model handles text-, image-, and video-conditioned tasks. Hover a card to play. T text · I image · V video.
V
V
V
V
Multimodal video generation aims to generate and edit videos conditioned on arbitrary combinations of text, images, and videos within a single model, allowing diverse tasks to share complementary data and generative priors. Unifying these tasks requires multimodal understanding of diverse conditions, which is typically provided by a pretrained vision-language model (VLM). A key challenge is how to connect the VLM's hierarchical multimodal representations with a pretrained video diffusion transformer (DiT). Existing methods either inject features from only the final or a few manually selected VLM layers, or jointly train architecture-matched understanding and generation streams, making it difficult to reuse heterogeneous pretrained backbones.
We introduce MoRoute, a unified multimodal video generation framework that formulates a frozen VLM and a pretrained video DiT with different architectures as heterogeneous experts connected through dynamic layer routing. For each input, a lightweight block-wise router enables every DiT block to select the VLM layer most relevant to its generation stage, thereby learning an adaptive correspondence between multimodal understanding and video synthesis. MoRoute further incorporates reference images and source videos directly into the DiT token sequence through unified in-context conditioning, preserving fine-grained visual details across diverse generation and editing tasks. Experiments on IntelligentVBench, OpenVE-Bench, and RefVIE-Bench show that MoRoute consistently surpasses the best competing method on each benchmark, improving the average score by 0.15, 0.18, and 0.34 on a 1–5 scale, respectively.
A heterogeneous Mixture-of-Transformers that links a frozen VLM and a pretrained video DiT — of different architectures — through learned per-block layer routing. Our instantiation freezes Qwen3.5-9B (32 layers, d=4096) as the condition encoder and equips Wan2.1-T2V-14B (40 blocks, d=5120) with the router.
One framework covers text-, image-, and video-conditioned generation and editing. Joint multi-task training lets diverse tasks share complementary data and generative priors — a data-scarce task such as character replacement borrows real-world priors from reference-to-video generation without ever seeing a real paired sample.
The frozen VLM and the pretrained DiT are cast as heterogeneous experts in a Mixture-of-Transformers. Every VLM layer is summarized into a routing key, and a learnable per-block query selects the most relevant layer via straight-through argmax — reusing both pretrained backbones without architectural constraints.
Reference images and source videos are concatenated directly into the DiT input sequence with slotted temporal RoPE, sparse attention, and dual timestep modulation — ensuring fine-grained visual fidelity while supporting diverse generation and editing tasks in a single model.
Reference-guided generation and source-conditioned editing. Hover a case to play all clips; click an input to enlarge.
1Subject talks while a woman stands still behind him.
2Woman speaks to the camera over a new background.
3Man walks from porch to courtyard, stopping to talk (first & last frame).
1Replace the background with a dynamic ancient Roman forum.
2Turn the entire bread loaf into a miniature lush forest.
3Overlay a tracked dark-green backpack onto the walking person.
1Add the reference scarf around the person's neck.
2Replace the girl's orange cloth with a Chinese Mamianqun skirt.
3Replace the background with the scene in the reference image.
All scores are 1–5 ratings produced by Gemini-2.5-Pro under the official benchmark prompts; every baseline was re-run from its official codebase under the same protocol.
IntelligentVBench
3.98AVG · +0.15 over best baselineOpenVE-Bench
3.83AVG · +0.18 over best baselineRefVIE-Bench
4.11AVG · +0.34 over best baseline| Method | IntelligentVBench | OpenVE-Bench | RefVIE-Bench | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| IF↑ | CP↑ | VQ↑ | AVG↑ | IC↑ | CD↑ | VQ↑ | AVG↑ | Subj.↑ | BG↑ | AVG↑ | |
| VACE | 3.16 | 3.07 | 3.23 | 3.15 | 1.52 | 1.52 | 1.52 | 1.52 | 1.00 | 1.40 | 1.20 |
| Bernini | 3.79 | 3.75 | 3.46 | 3.67 | 3.93 | 3.49 | 3.54 | 3.65 | 3.94 | 3.60 | 3.77 |
| Kiwi-Edit† | 3.89 | 3.78 | 3.67 | 3.78 | 3.28 | 2.82 | 2.99 | 3.03 | 3.00 | 2.53 | 2.77 |
| Omni-Video 2† | 3.00 | 2.33 | 3.22 | 2.85 | 3.28 | 3.02 | 3.12 | 3.14 | 3.17 | 2.27 | 2.72 |
| OmniWeaving | 3.92 | 3.81 | 3.76 | 3.83 | 3.16 | 2.67 | 2.65 | 2.82 | 3.11 | 3.33 | 3.22 |
| MoRoute (Ours) | 4.08 | 4.02 | 3.84 | 3.98 | 4.07 | 3.66 | 3.76 | 3.83 | 4.43 | 3.78 | 4.11 |
Best results are highlighted; second best are underlined. †Kiwi-Edit and Omni-Video 2 do not support I2V-style generation, so their IntelligentVBench columns cover the TIV2V subset only (shaded) and are excluded from the IntelligentVBench ranking. IF: Instruction Following · CP: Condition Preserving · IC: Instruction Compliance · CD: Consistency and Detail Fidelity · VQ: Visual Quality · Subj./BG: subject- and background-editing category averages.
Per-task-family gains over the best competing method
| Task Family | Benchmark | Best Baseline AVG | MoRoute AVG | Gain |
|---|---|---|---|---|
| Compositional MI2V | IntelligentVBench | 3.91 | 3.93 | +0.02 |
| Implicit I2V | IntelligentVBench | 3.81 | 3.85 | +0.04 |
| Interpolative DI2V | IntelligentVBench | 3.70 | 3.86 | +0.16 |
| TIV2V generation/editing | IntelligentVBench | 4.04 | 4.26 | +0.22 |
| TV2V editing | OpenVE-Bench | 3.65 | 3.83 | +0.18 |
| Reference subject editing | RefVIE-Bench | 3.94 | 4.43 | +0.49 |
| Reference background editing | RefVIE-Bench | 3.60 | 3.78 | +0.18 |
Per-subtask and per-editing-type breakdowns are reported in the appendix of the paper.
@article{gao2026moroute,
title = {MoRoute: Dynamic Routing for In-Context Multimodal Video Generation},
author = {Gao, Chong and Ma, Jie and Peng, Zhan and Wang, Chongxiao and Wu, Haoxue and Liang, Jun and Li, Guanbin and Li, Jing},
journal = {arXiv preprint arXiv:2607.29545},
year = {2026}
}