Vision-Language-Action model (VLA)

What it is

A vision-language-action model — usually shortened to VLA — is a single neural network that reads camera images and a written instruction and directly produces the commands that drive a robot. The authors of OpenVLA describe this class of model as large policies pretrained on a combination of internet-scale vision-language data and diverse robot demonstrations ^[1]. The word “policy” is the key: the output is not a description or a caption of the scene but an action, the thing the robot does next.

How it works

The usual recipe starts from a vision-language model (VLM) — a network already trained to connect images with text — and adds a way to emit actions. Different teams handle that last step differently.

Physical Intelligence describes π0 as a vision-language-action flow model that augments a pretrained vision-language model with continuous action outputs through flow matching ^[2]. Rather than picking one entry from a fixed list of commands, the model generates a smooth continuous trajectory.

AgiBot's ViLLA architecture takes another route: it extends vision-language-action modelling by predicting latent action tokens, which serve as a bridge between image-text inputs and robot actions ^[4]. Actions are represented as tokens, in roughly the same spirit as words in a sentence.

Some systems split the job in two. NVIDIA says the System 2 module of its GR00T N1 humanoid model is powered by a vision-language model that reasons about the environment and plans actions ^[5], leaving fast motor-level execution to another part of the system.

A small example makes the shape of the problem clear. Suppose a camera looks at a table and a person says “pick up the red block”. The model has to locate the block in the image, connect the phrase “red block” to that region, and then output a sequence of joint or end-effector commands that move the arm toward it. No part of that chain is hand-written for this particular block; the same weights are meant to handle the next instruction and the next scene.

Where it came from

Vision-language-action models did not appear out of nowhere — they build on vision-language models that already pair images with language. But the OpenVLA authors noted that existing vision-language-action models were largely closed and inaccessible to the public before OpenVLA ^[1], which is the situation that paper was written against. The framing it pushed — a large policy pretrained jointly on internet-scale vision-language data and diverse robot demonstrations ^[1] — became a common way to describe the category.

Where it is used

Figure describes Helix as a generalist vision-language-action model that unifies perception, language understanding and learned control ^[3], and says it was the first vision-language-action model to output high-rate continuous control of the entire humanoid upper body ^[3]. Physical Intelligence positions π0 as a vision-language-action flow model built on a pretrained vision-language backbone ^[2]. On the humanoid side, NVIDIA's GR00T N1 gives the reasoning and planning role to a vision-language model inside its System 2 module ^[5]. AgiBot's ViLLA applies the same idea through latent action tokens ^[4]. Across these systems the common thread is that one model connects what the camera sees and what the instruction says to what the motors do.

Limitations

The systems above do not agree on how an action should be represented: tokens in one design, continuous flow trajectories in another, a separate planning module in a third ^[2][4][5]. That divergence suggests the field has no settled recipe yet. Robot demonstrations are also far harder to collect than image-text pairs, which is why pretraining leans so heavily on vision-language data ^[1]. And a model that controls a whole humanoid upper body at high rate ^[3] or plans actions from a reasoned view of the environment ^[5] still has to be trusted with physical behaviour, where mistakes are not merely wrong captions.