Vision Language Action Models for Robot Control

Vision-Language-Action models replace the layered pipeline that has run robots for decades with one neural network that takes in what a camera sees, what a person asks for in plain language, and puts out motor commands directly. That shift matters because the old way of building robot control, chaining together separate modules for perception, planning, and actuation, breaks down at the seams between those modules. A perception system hands its output to a planner, the planner hands its output to a controller, and each handoff is a place where error creeps in and compounds rather than cancels out, because each module was tuned on its own, blind to how its mistakes would ripple through the next stage. Geometric and rule-based planners also demand that engineers hand-build a representation of every object, every environment, and every task the robot might face, which puts a hard ceiling on how far the system can generalize beyond what its designers anticipated. Add a new task, or swap in a different robot arm, and the fix usually means re-engineering several of these modules by hand, not retraining one system end to end.

A VLA model collapses that chain into a single foundation model. One review covering 183 contributions to the field describes a VLA in exactly these terms: a single foundation model that maps camera images and language instructions to robot actions with, in the review's words, "no assumptions about the specific robot." The perception-to-action mapping is learned from data rather than designed by hand, and because there's no longer a chain of separately optimized modules passing outputs to one another, the compounding-error problem that plagued classical pipelines is resolved by the architecture itself rather than by patchwork fixes bolted onto the joints.

What makes this kind of end-to-end learning possible at all is a transfer mechanism borrowed from a field well outside robotics. VLAs start life as vision-language models, pre-trained on internet-scale corpora of images paired with text, long before they ever see a robot demonstration. That pre-training imports scene understanding, object recognition, and a working grasp of multi-step instructions, the same kind of commonsense knowledge that classical robotics systems had to engineer by hand, one rule at a time. Only after that foundation is in place does fine-tuning on robot demonstration data specialize the model, turning general-purpose visual and linguistic understanding into the specific skill of producing physical action.

How the three-part unification works inside a VLA

The word "unification" in VLA is not a marketing flourish. Vision, language, and action are not processed as three sequential stages passed along a chain. They are learned jointly, inside one model, as shared representations, and grasping that joint structure is the key to understanding both what these systems can do and where they still fall short.

The architecture typically starts with a vision-language model backbone that encodes camera images and language instructions into a shared representation space, a format where visual and linguistic information sit side by side. From there, an action head, sometimes called an action token decoder, takes that shared representation and converts it into motor commands. How that action head is built turns out to be one of the biggest design decisions in the whole field. Some systems generate actions as discrete tokens, predicted one at a time the way a language model predicts the next word. Others use diffusion, generating a full sequence of movement by gradually refining noise into a plausible trajectory. The review in Drones 2026 points to a shift toward continuous, chunked action generation, with flow-matching and hybrid approaches gaining ground, because they sidestep two separate costs: the quantization bottleneck that comes from forcing continuous motion into discrete tokens, and the latency that multi-step diffusion adds at the moment a robot actually needs to move.

A basic fact about how these models work drives this architecture: they take the current camera view and the current instruction and produce an action, moment by moment. That reactive loop is the foundation on which everything else in a VLA is built, and it's also where the limits start. A system that only knows what's in front of it right now has no way to anticipate where a moving object will be in two seconds, no way to reason about what's hidden behind an obstacle, and no way to picture what the scene will look like once its own action has played out.

One design choice inside this architecture carries more weight than it first appears to. How a model represents the action it's about to take, whether as a special token, a numerical vector, or plain text, shapes what the model can still do once it's been turned into a robot controller. Research out of Princeton on a system called VLM2VLA found that representing actions as natural language text, rather than inventing new tokens or numerical formats, keeps the model inside the same statistical distribution it was pre-trained on. That single choice has consequences for how much of the model's original knowledge survives fine-tuning, and for how well it generalizes afterward, which is the subject the next section takes up directly.

What VLAs gain and risk from pre-trained world knowledge

The entire case for building a VLA on top of a vision-language model rests on inheritance: the robot gets to start from a system that already understands scenes, objects, and instructions, instead of learning all of that from scratch using robot data alone. But standard fine-tuning methods, the ones most VLA research treats as a default rather than a design choice, tend to produce a model that can move a robot's arm while losing much of what made the underlying vision-language model valuable to begin with. Most published VLA work treats that loss as background noise. It is closer to a structural problem built into how these models are usually adapted.

The name for this loss is catastrophic forgetting, and Princeton's 2025 ICLR paper, "Actions as Language," documents it directly: performance on visual reasoning, multilingual understanding, and open-world queries degrades once a vision-language model has been fine-tuned into a VLA using standard methods. The cause traces back to a mismatch in how actions get represented. Standard VLA training bridges the gap between a model pre-trained purely on language and a model that now has to output motor commands by bolting on special tokens or a separate action module. Both of those fixes pull the model away from the statistical distribution it learned during pre-training, and that drift produces the forgetting observed later. The consequence is not just a worse score on an academic benchmark. It undercuts the generalization that was the entire reason to build on a foundation model.

VLM2VLA, the Princeton system referenced above, resolves this tension by keeping actions inside language space. The model first generates a natural-language spatial plan, in plain text, before translating that plan into low-level motor commands, and it uses LoRA, a parameter-efficient fine-tuning method that updates small low-rank weight matrices instead of rewriting the whole network, so the base model's weights stay largely intact. The results back up the approach: VLM2VLA retains more than 85% of the base vision-language model's performance across twelve multimodal understanding benchmarks, a sharp contrast with conventional VLAs, which show substantial drops on the same measures. The payoff is not abstract. The system follows instructions given in Spanish, Mandarin, and Hindi without ever being trained on robot tasks in those languages, and it succeeds at open-world reasoning tasks, like picking up the item positioned above Ash Ketchum, where baseline systems built with conventional action representations perform close to zero.

What this shows is that the interface between language and action is a decision with real stakes. It determines how much of a vision-language model's accumulated knowledge survives the trip into a deployed robot policy. That reframes a question researchers have mostly asked at the level of architecture, which action head should we use, into a deeper one about where the boundary between language and action ought to sit. VLM2VLA offers one answer and proves forgetting can be addressed in principle. Whether that solution holds up as models and tasks scale further depends heavily on the training data available to push these systems forward. That dependence is where the field runs into a different kind of wall.

The data bottleneck's structural difference from LLMs

Large language models solved their data problem by scraping the internet: text is everywhere, and more of it could always be found. VLA models cannot solve their equivalent problem the same way, because the data they need does not exist on the internet waiting to be collected. Training the action-generating half of a VLA requires action-labeled trajectories: timestep-by-timestep records of what a robot actually did, paired with what it observed and what instruction it was following at the time. That kind of data comes only from physical collection, almost always through human teleoperation of a real robot, or through motion-capture systems that record human movement and then retarget it onto a robot's joints. Both methods are slow and expensive. Both are bound to the specific hardware used to collect them.

This creates a trade-off that has no clean workaround: datasets collected on real robots give accurate, physically grounded trajectories, but they're costly to produce and hard to scale across the range of tasks and robot bodies the field wants to cover. High fidelity costs scale. High scale costs fidelity. That tension drives nearly every data-collection decision VLA researchers make.

Two recent projects attack the bottleneck from different directions, and together they show where the field is heading. LingBot-VLA 2.0 treats egocentric human video as a proxy for robot data, curating roughly 60,000 hours of pre-training material that combines robot trajectories gathered across many different robot configurations with video recorded from a human's own point of view while performing tasks. A video representation model supplies semantic priors from that footage, and a depth estimation model extracts geometric cues, so the human video contributes structure the robot data alone would not provide. Xiaomi-Robotics-1 takes a different route, pre-training on a large volume of real-world manipulation trajectories gathered with UMI devices across a wide range of environments and tasks. During pre-training, scene state transitions are auto-labeled with language descriptions; during post-training, those labels are replaced with manually annotated instruction prompts designed to align the model with specific robot embodiments. Neither project treats human video or auto-labeled data as a replacement for real robot trajectories. Both treat it as a scalable bridge toward collecting less of the expensive kind.

Xiaomi-Robotics-1 also offers early evidence that robotics data might follow something like a scaling law, the pattern where more data and larger models reliably produce better performance, the dynamic that powered the last decade of progress in language models. The system shows consistent improvement with increased data and model size during pre-training, and that improvement carries over to post-training performance in real-robot environments it had never encountered before. That evidence covers manipulation tasks specifically. It says nothing yet about locomotion, aerial robotics, or dexterous hand control, so the scaling claim is real but bounded to the slice of the embodiment stack it was measured on. Whether VLAs scale the way LLMs did across the full range of robot bodies remains an open question the field has not yet answered.

What adding temporal foresight to a VLA buys

A model that can anticipate how a scene will change produces better actions than one that only reacts to what it sees right now, but the form that anticipation takes changes how useful it is. Predicting every pixel of a future frame burns representational capacity on parts of the scene that never move, the background, the fixed furniture, the static walls, while a sparser approach that tracks only the points that actually change focuses the model's attention where it matters.

FoMoVLA, developed by LiAuto and Tsinghua, builds this into the architecture by jointly learning two things at once: future feature prediction, where foresight tokens anticipate what the goal state of the scene will look like, and sparse 2D point tracking, which follows the continuous motion path that gets the scene from where it is now to that predicted goal. These two predictions aren't made separately and then stitched together after the fact. A future-conditioned cross-attention module, referred to in the research as FCCA, couples them directly, so the model's local estimate of how a specific point will move stays anchored to its global prediction of where the whole scene is heading. At inference time, once the model is actually deployed on a robot, all of these auxiliary prediction branches get discarded. The foresight training adds zero parameter overhead and negligible latency to the model that ends up running.

LingBot-VLA 2.0 pursues a related goal through a different route, introducing predictive dynamics modeling as a proxy training objective rather than a capability the deployed model exercises directly. The same video representation model and depth estimation model used to curate its training data also sharpen the system's temporal reasoning in dynamic environments. Because the approach extends the robot's action space beyond the standard dual-arm setup to include the head, waist, mobile base, and dexterous hand degrees of freedom, temporal foresight here applies to whole-body coordination, not arm movement alone. Training the model to anticipate future states as a proxy objective means it does not need explicit future-state labels once it's actually deployed and working.

That two independent research groups, LiAuto and Tsinghua on one side, and the team behind LingBot-VLA 2.0 on the other, converged on foresight as the answer to the reactive baseline's blind spot says something about where the field agrees the architecture needs to grow. Neither group is proposing to abandon the end-to-end VLA framework in favor of something else. Both are extending it, adding a layer of prediction on top of a mapping from perception to action that remains, beneath the additions, the same basic structure described earlier in this piece.

How cross-embodiment generalization exposes VLAs' unresolved structural challenge

The most ambitious claim behind VLA research is that a single model family could eventually control manipulator arms, mobile robots, and drones alike. That claim runs directly into a structural obstacle that no amount of pre-training or foresight modeling has yet resolved: a robot's action space, the specific set of joints, degrees of freedom, and control interfaces it exposes, differs so much from one robot to another that a model trained on one often fails outright on a different one. A quadruped moves nothing like a robot arm. An autonomous vehicle's controls share almost nothing in common with a dexterous hand. Training a model on a fixed set of robot types tends to produce a policy that cannot transfer to a robot whose action space looks fundamentally different, no matter how much data went into the original training run. The Drones 2026 review lists fourteen open research directions for the field, and cross-embodiment generalization threads through nearly every one of them.

The RoCo Challenge, held at AAAI 2026, offers a concrete look at what's achievable inside these constraints right now, showing how the field is navigating them in practice. Teams that performed best in the competition relied primarily on end-to-end VLA architectures. The overall winner, a team called IIGroup, fielded a system named ARC-VLA, a 3.6-billion-parameter model built on the π₀.₅ backbone. What set IIGroup apart from other competitors was not a larger model or a novel architecture, but a stability-first design philosophy: consistent observation protocols paired with explicit mechanisms for recovering from failure. That outcome points toward engineering discipline, careful handling of how a robot observes its environment and what it does when something goes wrong, as the lever currently moving the needle on precision-sensitive manipulation tasks, ahead of simply scaling model size further.

Bimanual coordination stands as the hardest test case inside this broader challenge, and it shows why cross-embodiment generalization is harder than it looks from a distance. Two robot arms, each with seven or more degrees of freedom, have to move in concert while the robot can only partially observe its own environment. That combination demands tightly coordinated, high-frequency control, and it strains a VLA model on two fronts simultaneously: the latency of generating and executing commands fast enough to keep both arms synchronized, and the generalization gap that opens up the moment the task or environment shifts even slightly from what the model saw during training. Bimanual manipulation is where the unresolved structural challenge of cross-embodiment generalization is most visible, and it marks the frontier where the next phase of VLA research will have to prove itself.