Yi-VL

What it is

Yi-VL-34B and Yi-VL-6B are a pair of open vision-language models released by 01.AI on 23 January 2024 ^[1]. A vision-language model, sometimes called a multimodal model, can take an image together with a text question and answer in text. Ask it what is on a table, what a chart shows or what a sign says, and it will look at the picture and reply. The 34B and 6B in the names refer to the size of the language model at the core of each version, as explained in the Yi-34B entry.

The models handle English and Chinese, which means the questions and answers can be in either language ^[2]. They take images at a resolution of 448x448 pixels.

How it works

The design follows the LLaVA architecture, a common recipe for building vision-language models. It has three parts. The first is a vision encoder, a vision transformer initialised from CLIP ViT-H/14. CLIP is an image model trained to match pictures with text descriptions, and the ViT-H/14 version is a large version of it ^[2]. The second is a projection module, a two-layer MLP that translates the visual features into a form the language model can read. The third is the language model itself, which is initialised from Yi-34B-Chat, the chat version of the Yi-34B model ^[2].

A simple way to picture this is that the vision encoder turns an image into a list of numerical features, the projection module converts those features into the same kind of input the language model handles, and the language model then writes an answer as if it were reading a description of the picture.

Training

Training happened in three stages. The first stage used about 100 million image-text pairs, mainly from the LAION-400M dataset, at a resolution of 224x224. The second stage used about 25 million pairs at 448x448. The third stage used about 1 million pairs to train the full model ^[2]. Going from low to high resolution in stages is a common way to keep the cost of training manageable while still reaching good detail at the end.

Training Yi-VL-34B took 128 NVIDIA A800 (80GB) GPUs for about 10 days ^[2]. The A800 is a data-centre GPU, a processor designed for large-scale computing, and the 80GB refers to its memory.

Results

Yi-VL-34B ranked first among open-source models on the MMMU and CMMMU multimodal benchmarks, based on data available up to January 2024 ^[1][2]. MMMU is a test of college-level questions that combine images and text, and CMMMU is its Chinese counterpart. The company reported an MMMU accuracy of 41.6% for Yi-VL-34B, compared with 55.7% for GPT-4V, a closed model from OpenAI ^[3]. The figures show that the open model was behind the best closed model on this test at the time, while leading other open models.

Limits

The model accepts only one image per input, and it is limited to visual question answering, meaning it answers questions about a single picture. It can also hallucinate, that is, describe content that is not present in the image ^[2]. Readers should treat its answers as a helpful first reading rather than a guaranteed description.

Licence and later versions

The weights are distributed under the Apache 2.0 licence, which allows use, modification and redistribution, including commercial use, as long as the licence notices are kept ^[2]. The weights were published on Hugging Face. Later, 01.AI offered a hosted image understanding model called Yi-Vision through its API platform ^[3].

Its place in the Yi family

Yi-VL extends the Yi language models from text into images. It was built on Yi-34B, which means its design depends on the language model family described elsewhere in this archive. For readers, the main lesson is that a vision-language model is a language model with an added way of seeing, and its quality depends on both parts.