OpenVLA

What OpenVLA is

OpenVLA is a vision-language-action model: a single neural network that takes an image from a robot's camera together with a natural-language instruction and outputs the actions the robot should take ^[1]. Models of this kind collapse what used to be a pipeline — a perception module, a planner, and a low-level controller — into one network trained end to end on demonstrations. OpenVLA has 7 billion parameters, and it is open source: the weights, not only the paper, were published ^[1]. The paper appeared on arXiv on 13 June 2024 as arXiv:2406.09246 ^[1].

How it was built

The model was trained on 970,000 real-world robot demonstrations ^[1]. Architecturally it builds on a Llama 2 language model, to which a visual encoder is attached; that encoder fuses features from two pretrained vision models, DINOv2 and SigLIP ^[1]. Both of those are encoders trained beforehand on large collections of images, so the policy starts from visual representations that were not learned from robot data alone. Training the final model took a cluster of 64 A100 GPUs running for 15 days ^[2].

Results

The authors compared OpenVLA with RT-2-X, a much larger closed model with 55 billion parameters. Across 29 tasks and multiple robot embodiments, OpenVLA came out ahead by 16.5% in absolute task success rate while having 7 times fewer parameters ^[1]. When fine-tuned to a particular task or robot, OpenVLA also beat Diffusion Policy, a widely used imitation-learning method, by 20.4% ^[1]. The evaluations were run on two hardware setups: the WidowX setup from Bridge V2 and the Google Robot from the RT-series papers ^[2].

What was released

Alongside the paper, the authors released model checkpoints, fine-tuning notebooks and a PyTorch codebase that supports training on Open X-Embodiment datasets ^[1]. They reported that the model can be fine-tuned on consumer GPUs using low-rank adaptation (LoRA), and served efficiently with quantization without losing downstream success rate ^[1]. According to the project page, the released model can control multiple robots out of the box, and new robot setups can be accommodated through parameter-efficient fine-tuning ^[2].

Where it sits

Generalist robot policies that map language and images directly to actions had been demonstrated mainly through large closed models, of which RT-2-X is the comparison point here ^[1]. OpenVLA's contribution is to put a model of this class, together with the code and recipes needed to adapt it, into open release: a group can start from the published checkpoint and fine-tune rather than train a model of this size from scratch ^[1]. That matters because a new robot arm or a new camera arrangement is rarely handled well by a policy trained elsewhere; the practical route is fine-tuning on a modest amount of new data, which the released notebooks and the low-rank adaptation results are meant to make feasible on ordinary hardware ^[1]. The model is therefore less a finished product than a starting point — a generalist policy that ships with the tooling to specialise it ^{[1] [2]}.