NVLM 1.0

What it is

NVLM is a family of multimodal large language models built by NVIDIA. A multimodal model is one that handles more than one kind of input — in this case images alongside text — rather than text alone. The family was described in a paper posted to arXiv on 17 September 2024 by Wenliang Dai and nine co-authors, among them Bryan Catanzaro and Wei Ping.^[1]

The design question behind it

At the time the paper appeared, work on vision-language models had settled into two broad architectural camps. One is the decoder-only style, of which LLaVA is a representative: image information is fed into the language model's own input stream, and a single transformer decodes over the combined sequence. The other is the cross-attention style, of which Flamingo is a representative: dedicated attention layers are inserted into the language model so that it can look at image features without those features becoming ordinary tokens. The NVLM paper compares the two designs and proposes a hybrid architecture that combines them.^[1]

The paper also introduces one-dimensional tile tagging, a scheme for handling high-resolution images. Instead of treating a large image as a single input, the image is divided into tiles, and the tiles are labelled along a single dimension so the model can tell them apart and keep their arrangement.^[1]

Results and claims

The authors reported state-of-the-art vision-language results, claiming performance that rivalled GPT-4o as well as open models such as Llama 3-V 405B and InternVL 2.^[1] These comparisons come from the authors themselves.

One observation the paper highlights concerns text ability. After multimodal training, the model improved over its own language-model backbone on text-only tasks, with gains in mathematics and coding.^[1] That is notable because adding image training to a language model does not automatically leave its text skills intact.

The authors also argued that dataset quality and task diversity matter more than scale.^[1] In other words, the recipe they credit is not simply more data or more parameters, but better-chosen data covering a wider range of tasks.

The open release

NVIDIA released the weights of its flagship model, NVLM-D-72B, on Hugging Face; Wikipedia dates this open release to October 2024.^[1][2] The weights carry a Creative Commons Attribution-NonCommercial 4.0 licence, with the Qwen2-72B-Instruct and InternViT licences also applying, and NVIDIA stated that the model was ready for non-commercial use.^[3]

NVLM-D-72B is a decoder-only model. According to NVIDIA, it handles optical character recognition, multimodal reasoning, localisation, common-sense reasoning, world knowledge and coding.^[3]

Alongside the weights, training code was published in NVIDIA's Megatron-LM repository. The model was trained with the legacy Megatron-LM codebase before being adapted to Hugging Face, and NVIDIA noted that numerical differences exist between the Megatron and Hugging Face implementations.^[3] That caveat matters for anyone trying to reproduce or fine-tune the model: the same weights run through two codebases will not produce bit-identical numbers.

Where it sits

NVLM is NVIDIA's contribution to the group of openly downloadable multimodal models, and the paper places it directly against both proprietary systems and other open models.^[1] Its release is unusually complete for a research model — architecture description, weights and training code — but the non-commercial licence and the stacked licence terms of its components constrain what can be done with it.^[3] The paper's architectural comparison and its argument that data quality and task diversity outweigh scale are the parts most likely to be reused by later work.^[1]