01.AI open-sources the Yi-VL vision-language models

Background

By early 2024 01.AI had already released its open Yi base models and was building a family of models around them. A language model that handles only text cannot see pictures, so a separate line of work was needed to connect images and text. Vision-language models, also called multimodal models, are the kind of system that connects the two. 01.AI's Yi-34B had been trained on text, and the company's next step was to extend that model to images.

To understand the release, it helps to know what kinds of tests are used. Benchmarks are standard exams that let different models be compared on the same questions. MMMU is a benchmark of college-level questions that combine images and text, and CMMMU is its Chinese-language counterpart. Results on these exams show how well a model combines what it sees with what it knows.

What was released

On 23 January 2024 01.AI released two models, Yi-VL-34B and Yi-VL-6B ^[1]. The two sizes differ in the size of the language model at their core. Both were published as open weights on Hugging Face, the platform where developers share models, datasets and model cards ^[2].

The models combine a CLIP ViT-H/14-based vision encoder with Yi chat language models through a projection module. The design follows the LLaVA style, in which a vision encoder is connected to a language model with a small adapter that translates image features into language-model inputs ^[2]. CLIP is an image model trained to match pictures with text, so its encoder already understands a wide range of visual content before the Yi part begins to work.

The models handle English and Chinese and take images at 448x448 resolution ^[2]. Using a fixed resolution means that every image is resized to the same input size before the model reads it.

Key details

Yi-VL-34B ranked first among open-source models on the MMMU and CMMMU benchmarks as of January 2024 ^[1]. This ranking applied only to open models, so it does not mean the model beat the closed systems that were then considered the strongest.

Training Yi-VL-34B used 128 NVIDIA A800 GPUs for about 10 days ^[2]. A GPU is the processor type that does most of the work in training large models, and the A800 is a model designed for data centres.

The weights were released under the Apache 2.0 licence, which permits commercial use as long as the licence notices are kept ^[2]. The model card on Hugging Face lists this licence, and it is the document a reader should check for the current terms.

Impact

The release gave developers in both English- and Chinese-speaking communities an open model that could read an image and answer questions about it. Because the weights were available, people could run the model themselves, test it on their own pictures and adapt it for their own applications. The release also showed that a company which had built its reputation on text models could move into vision with a model that performed well on bilingual benchmarks.

The limits of the release are as important as its strengths. The models accept one image per input and are limited to visual question answering, and they can produce content that is not in the picture. Anyone building on them needs to test these behaviours for their own use case.

For a general reader, the event is a good example of how open-source AI works in practice: a company publishes weights, a community tests them against shared benchmarks, and the results are judged by people outside the company.