Vision-language-action models (VLAs), explained

A vision-language-action model, or VLA, is a neural network that takes camera images and a natural-language instruction — “put the cup in the sink” — and outputs the actions a robot should take. It is the dominant design for general-purpose robot policies, and the main reason physical AI is moving quickly.

How a VLA works

A VLA has three jobs, and usually three corresponding parts.

See and read. A vision encoder turns each camera frame into tokens. The instruction is tokenized as text. Many VLAs also take the robot’s own state — joint angles, gripper position — as input.

Reason. A transformer backbone processes those tokens together. Crucially, this backbone is almost always a pretrained vision-language model, already trained on enormous amounts of image and text data. That is where the model’s knowledge of what a “cup” or a “sink” is comes from.

Act. An output stage turns the backbone’s result into robot actions: typically a target pose for the end effector or target joint positions, plus a gripper command.

The whole thing is then trained on robot demonstrations — recordings of what the cameras saw and what the robot did, mostly collected by people teleoperating robots.

Two ways to output actions

The clearest dividing line between VLA architectures is the last step.

Actions as tokens

The first VLAs treated actions like words. Each dimension of an action is split into a fixed number of bins, each bin is a token, and the model predicts action tokens the same way a language model predicts the next word. RT-2 introduced this approach and OpenVLA made it available as open source.

It is simple and reuses the language model unchanged. Its weaknesses are that binning loses precision, and generating tokens one at a time is slow — a real limit when a task needs fast, smooth motion.

Actions as continuous trajectories

The alternative attaches a separate action head that generates continuous motion using diffusion or flow matching — the same family of techniques used to generate images. Instead of one action, the head produces a short sequence of future actions at once, known as an action chunk. π0 from Physical Intelligence is the best-known example.

This gives smoother, higher-frequency control and suits dexterous tasks. The price is a more complicated model and training process.

Dual-system designs

A third pattern splits the model in two: a large, slow vision-language model that interprets the scene and the instruction a few times per second, and a small, fast action model that turns its output into motion at a much higher rate. NVIDIA’s GR00T N1 describes itself this way, borrowing the “System 2 and System 1” language of slow and fast thinking.

Models worth knowing

Model	From	Year	Why it matters
RT-2	Google DeepMind	2023	Showed that a vision-language model can output actions as text tokens and carry web knowledge into robot control
RT-X / Open X-Embodiment	Google DeepMind and partner labs	2023	Pooled more than a million robot trajectories from 22 robot types; showed that training across embodiments helps
OpenVLA	Stanford, UC Berkeley and collaborators	2024	An open-source 7-billion-parameter VLA, trained on 970,000 robot episodes, that anyone can fine-tune
π0	Physical Intelligence	2024	Flow-matching action head on a pretrained VLM, producing actions at up to 50 Hz; demonstrated long, dexterous tasks such as folding laundry
GR00T N1	NVIDIA	2025	Open foundation model aimed at humanoid robots: a vision-language model paired with a diffusion-based action module
Gemini Robotics	Google DeepMind	2025	A VLA built on Gemini 2.0, announced alongside an embodied-reasoning model for spatial understanding

This table will date quickly. Treat it as a map of ideas, not a leaderboard.

Where VLAs still fail

Generalization is narrower than demos suggest. A model that handles new objects may still fail when the camera moves, the lighting changes or the table is a different height. Reliable performance on a specific task almost always needs fine-tuning on data from that task and that robot.

Evaluation is hard. There is no equivalent of a language benchmark that can be run in an afternoon. Measuring a robot policy means running a physical robot many times, and results rarely transfer between labs.

They are slow relative to control. Large backbones run at a few hertz. Action chunking and dual-system designs exist largely to hide that latency.

They offer no guarantees. A VLA will occasionally do something wrong with complete confidence. In a deployed system that is handled the same way as any other unreliable component: by an independent safety layer and a risk assessment that assume the model can output anything.