Vision-Language-Action Models: How They Work and Why Their Data Decides Performance

A vision-language-action (VLA) model lets a robot follow plain-language instructions. Tell a farm robot to “put the ripe tomatoes in the left crate” and the model has to spot which tomatoes are ripe, find the left crate and move each one without crushing it.

NVIDIA’s GR00T, Google DeepMind’s Gemini Robotics and open models such as OpenVLA all work this way. Many of these models are now free to download, so building the model is rarely the hardest part. The difficult part is teaching it with good examples of the task.

Getting those examples right decides whether a robot works outside the lab. For ML engineers and robotics leads, that makes data collection and annotation the central engineering problem. For executives funding physical AI, it decides which investments pay off.

How Do Vision-Language-Action Models Work?

A VLA model works in three stages that are trained together. One part reads the camera images, another reads the instructions and a third turns both into movement.

1) The Vision Encoder

The vision encoder is the robot’s eyesight. It turns each camera image into information the model can use, such as which tomatoes are red and where the crate sits.

Robots usually have more than one camera. A camera on the wrist shows the tomato up close as the gripper closes, while an overhead camera shows the whole crate. Sensors in the arm add what no camera can see, such as the exact angle of each joint.

2) The Language Backbone

The language backbone understands the instruction. Rather than training this part from scratch, developers borrow a model that already understands language and images.

That borrowed knowledge is why a robot can recognise a tomato it has never handled. Knowing what a tomato looks like is still different from knowing how to pick one up. That skill only comes from recorded demonstrations of the real task.

3) The Action Decoder

The action decoder turns the model’s understanding into movement. This is where VLA models differ most.

For instance, OpenVLA predicts movement in small steps, much as a language model predicts one word after another. Physical Intelligence’s pi0 uses a technique called flow matching to plan a short run of smooth movements at once, which suits delicate two-handed tasks such as folding laundry. Google DeepMind’s Gemini Robotics 1.5 splits the job between a model that plans the task and another that carries it out.

Whichever design a team chooses, the model only learns what its examples show it. If every training example has the tomatoes in the same spot, the robot will struggle the first time they are somewhere else.

Why Is Robot Demonstration Data So Hard to Scale?

Language models learned from text that already existed on the internet. Robot examples have to be made one at a time, usually by a person steering the robot through the task while its cameras and sensors record what happens.

Even the largest public collection is small by comparison. The Open X-Embodiment project pooled just over one million robot examples from 34 research labs, covering 22 types of robot. According to the OXE-AugE team’s ICML 2026 paper, four of those robot types account for more than 85% of the real data, so a model trained on it sees a handful of robots far more often than the rest.

The research so far points to two lessons: pooling data from many robots helps and the quality of each example matters even more than the total count.

What Pooling Data Across Robots Achieves

The Open X-Embodiment team trained one model on data pooled from many labs and tested it on tasks where each lab had only a small dataset of its own. The pooled model performed 50% better on average than the models each lab had built alone. Seeing many robots, rooms and objects taught it things no single lab could.

Pooled data still has limits. Physical Intelligence trained its openpi base models on more than 10,000 hours of robot data, yet it warns that pi0 was built for its own robots and may not work as well on others. A team with different hardware still needs examples recorded on that hardware.

Why Data Quality Outweighs Volume

Hugging Face’s SmolVLA shows how far careful data can go. It learned from fewer than 30,000 examples, yet its developers report results comparable to models ten times its size.

The team cleaned the data first. Many task labels in the community datasets were noisy, so they used an AI model to write short, clear task descriptions. They also gave every camera a standard name, such as top, wrist or side.

For teams recording their own data, Hugging Face’s SmolVLA fine-tuning guidance suggests about 50 examples as a starting point, spread across the variations the robot will meet. Its own example recorded 10 demonstrations at each of five different cube positions.

What Does High-Quality VLA Training Data Look Like?

A training example only helps if the video, the arm’s recorded movements and the instruction all describe the same moment of the same task. Four checks catch the problems that matter most.

1) Synchronised Streams

The video and the recorded movements must line up to the split second. If the movement data runs half a second behind, the frame showing the gripper reaching for a tomato gets paired with the moment the gripper closes. The robot then learns to close its grip too early. Recording more examples will not fix a timing error.

2) Instructions That Match What the Robot Did

One word in an instruction can change what a robot does. In the Open X-Embodiment tests, “move apple on cloth” and “move apple near cloth” produced different movements. If an operator meant to place a tomato in the crate but set it down beside it, the instruction has to describe what actually happened.

3) Variation and Recovery

A robot that has only seen tomatoes in one position will hesitate when they are somewhere else. Training examples should cover the different positions, lighting and crate layouts the robot will meet in real use.

Examples of mistakes matter too. The researchers behind FLARE, presented at CVPR 2026, point out that earlier VLAs learned only from flawless demonstrations, which left them with no example of recovering from a missed grasp. A useful recovery example records the tomato slipping, where it landed and how the robot picked it back up.

4) Consistent Action Conventions

Robots don’t all describe movement the same way. On one robot, a recorded value might mean “move the gripper 5 cm to the right”, while on another the same value means a different direction. Mixing those records without converting them teaches the model the wrong movement.

Where Does VLA Data Annotation Go Wrong?

Annotation turns raw recordings into training examples by adding labels, such as what the task was, where each step begins and whether the attempt worked. In robotics data annotation, mistakes tend to cluster in four places.

1) Segmenting Long Tasks into Steps

Picking a tomato involves several steps, from reaching and gripping to lifting and placing. Newer models such as pi0.5 use those steps directly, naming the next step in words before acting.

Annotators need shared rules for where each step starts and ends. If one annotator marks the grip as its own step and another folds it into reaching, the model receives mixed signals. Checking inter-annotator agreement on a sample shows quickly where the rules need tightening.

2) Labelling Outcomes Honestly

Every example should be marked as a success or a failure, along with the moment things went wrong. Deleting failed attempts makes a dataset look cleaner but throws away the recovery examples robots need most.

3) Writing Instructions at Scale

One annotator writes “mug” and another writes “cup” for the same object. A shared vocabulary keeps labels consistent, while a few checked rewordings of each instruction help the model understand people who phrase things differently.

4) Reviewing Each Episode as One Record

The video can look right and the instruction can read well while the movement data is still out of step. Reviewers should check all three together, one example at a time, before it goes into the training set.

How Can Teams Source Reliable VLA Training Data?

Open models such as OpenVLA and SmolVLA mean teams no longer need to build a VLA from scratch. What they still need is a steady supply of examples showing what the robot was asked to do, what it did and how it recovered when something went wrong.

Aya Data collects and annotates egocentric robotic data, including video of people handling objects with their hands, which can support robot training alongside demonstrations recorded on robots. Our specialist partners across the UK, USA, India, Ghana and Kenya work with in-house quality assurance teams alongside data scientists and solutions architects to check every dataset before delivery.

Across every engagement, you receive one accountable partner for the experts, the data and its quality. If you’re planning a VLA dataset or struggling with demonstration quality, book a 15-minute discovery call to scope the data your robots need.

Frequently Asked Questions