CogVideoX

What it is

CogVideoX is an open-source family of text-to-video models developed by Zhipu AI together with Tsinghua University. The first member, CogVideoX-2B, was released on 6 August 2024, and a larger CogVideoX-5B followed on 27 August 2024^[1]. The project describes itself as the open-source release of the video generation model behind Zhipu's Qingying (Ying) product^[1]. Ying, Zhipu's text-to-video product, had launched in July 2024 and generated six-second clips in about 30 seconds^[2]. Putting model weights in public hands is the main thing that distinguishes CogVideoX from a closed video product: outside researchers and developers can download the model and run it themselves.

Where it came from

The model line has a longer history than its 2024 releases suggest. It descends from CogVideo, which was open-sourced on 19 May 2022 and is described as the first open-source large Transformer-based text-to-video model^[1]. Zhipu had announced in March 2024 that it was developing Sora-like video technology as a path toward artificial general intelligence^[2]. The technical report, titled "CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer," was first submitted to arXiv on 12 August 2024 by 18 authors, the last of them Jie Tang^[3].

How it works

The model is a diffusion model, meaning it learns to turn random noise into video step by step, in the way that a sculptor removes material rather than adding it. What video adds to image generation is time: frames have to stay consistent with each other. CogVideoX handles this with a 3D variational autoencoder (VAE) that compresses video along both spatial and temporal dimensions^[3], so the diffusion process works on a compact representation instead of raw pixels. According to the paper, the model can generate 10-second videos aligned with text prompts at 16 frames per second and a resolution of 768x1360^[3].

Releases and licensing

The initial CogVideoX-2B and CogVideoX-5B models generate 720x480 video^[1]. An image-to-video variant, CogVideoX-5B-I2V, followed on 19 September 2024, letting users start from a still image rather than only text^[1]. On 8 November 2024 the CogVideoX1.5-5B and 5B-I2V models raised output to 1360x768 at 16 frames per second, for clips of 5 or 10 seconds^[1].

The licensing is not uniform across the family. CogVideoX-2B is released under Apache 2.0, while the 5B models use a separate CogVideoX license^[1]. That split matters for anyone planning commercial use, since the terms differ between the small and large variants.

What was made public

The authors state that the 3D causal VAE, the video caption model and CogVideoX itself are all publicly available^[3]. Releasing the caption model alongside the generator is notable because text-video pairs with good descriptions are a bottleneck for training video models; a captioner that turns clips into usable text is part of the pipeline, not just an afterthought.

The authors also report that the model achieves state-of-the-art performance across multiple machine metrics and human evaluations^[3]. Such comparisons are the authors' own claims in their technical report, and they depend on which benchmarks and baselines are chosen.

Its place in the line

CogVideoX sits at the point where a company's commercial video product and an open research release share the same underlying technology. Zhipu framed video generation as a route toward general-purpose AI^[2], and the open releases made that work inspectable and reusable outside the company. The trajectory from the 2022 CogVideo model to the 2024 CogVideoX family, and then to the higher-resolution 1.5 models within the same year, shows a fairly rapid cadence of iteration on resolution, clip length and input modalities, including the move from text-only generation to image-to-video^[1].