Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
ControlVideo: Training-free Controllable Text-to-video Generation
Authors: Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, XIAOPENG ZHANG, Wangmeng Zuo, Qi Tian
ICLR 2024 | Venue PDF | LLM Run Details | Input Tokens: 20,209 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,998 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments demonstrate that our Control Video outperforms the state-of-the-arts both quantitatively and qualitatively. It is worth noting that, thanks to the efficient designs, Control Video could generate both short and long videos within several minutes using one NVIDIA 2080Ti. |
| Researcher Affiliation | Collaboration | 1Harbin Institute of Technology 2Huawei Cloud |
| Pseudocode | Yes | Algorithm 1 Interleaved-frame smoother |
| Open Source Code | Yes | Code and videos are available at this link. |
| Open Datasets | Yes | To evaluate our Control Video, we collect 25 object-centric videos from DAVIS dataset (Pont-Tuset et al., 2017) and manually annotate their source descriptions. |
| Dataset Splits | No | The paper mentions using 125 motion-prompt pairs as an evaluation dataset but does not specify explicit training, validation, or test dataset splits needed for reproduction. |
| Hardware Specification | Yes | Control Video could generate both short and long videos within several minutes using one NVIDIA 2080Ti. |
| Software Dependencies | No | The paper mentions Control Net, RIFE, and xFormers but does not provide specific version numbers for these software dependencies. |
| Experiment Setup | Yes | The synthesized short videos are of length 15, while the long videos usually contain about 100 frames. Unless otherwise noted, their resolution is both 512 512. During sampling, we adopt DDIM sampling (Song et al., 2020a) with 50 timesteps, and interleaved-frame smoother is performed on predicted RGB frames at timesteps {30, 31} by default. |