Foundation Models as Task Planners for Humanoid Robots

Foundation models have moved from the edges of humanoid robotics to its control center. The planning intelligence that decides what a robot should do next, and how to translate that decision into movement, now lives inside Vision-Language-Action models, and that shift is the subject of this piece.

Why humanoid robots needed a cognitive layer

A humanoid robot built on explicit programming or narrow task-specific learning can perform well inside the exact conditions its engineers anticipated. Ask it to do something slightly different, in a slightly different kitchen, with an object placed slightly out of its expected spot, and the system has no way to adapt. Each new task required its own programming effort or its own training run, and none of that effort transferred to the next task. The robot's body kept improving: better actuators, better joints, better hands. Its capacity to decide what to do with that body did not keep pace.

This produced a mismatch that defined the field for years. Humanoid platforms had the physical capability to perform dexterous, human-like work, but they lacked the nuanced decision-making that lets a person walk into an unfamiliar room and figure out what to do. A robot could follow a fixed script. It could not take a plain instruction like "clear the table and reload the dishwasher" and work out, on its own, what series of physical actions that implies in a room it had never seen configured that way before.

That gap, between open-ended instruction and executable motor command, is what Vision-Language-Action architectures were built to close. A robot needed something that could interpret a high-level goal stated in ordinary language, reason about what that goal requires given the scene in front of it, and decompose that reasoning into a sequence of actions its body could actually carry out, without an engineer rewriting its code for every new task. That requirement is the starting point for everything that follows.

How the Vision-Language-Action architecture connects language, perception, and motion

A Vision-Language-Action model gives a robot one framework that takes in a plain-language instruction, reads the scene through its cameras, and produces motor output, all without routing through a pre-programmed task routine for that specific job. The instruction "pick up the blue cup" and the camera feed showing a cluttered counter both go into the same system, and what comes out the other end is a sequence of joint commands.

The architecture that makes this work splits the job into two parts that run at two different speeds; NVIDIA's GR00T N1 paper describes this as a dual-system design modeled loosely on how human cognition separates deliberate thought from automatic action. One part of the system, the vision-language module, behaves like deliberate reasoning: it looks at the scene, reads the instruction, and figures out what needs to happen. In GR00T N1, this module runs at 10Hz on an NVIDIA L40 GPU, pacing itself to the speed that reasoning actually requires. A Diffusion Transformer, trained using a technique called action flow-matching, handles reflex: it takes what the reasoning module has worked out and turns it into continuous, moment-to-moment motor commands at 120Hz, fast enough to keep a robot's limbs moving smoothly and to correct for small errors as they happen.

The two modules are trained jointly, end to end, so the reasoning side and the acting side learn to coordinate with each other instead of being patched together through a brittle handoff.

Flow matching, the technique behind the fast-acting module, has become the default method for action generation across the field's leading model families, including the π0 family, GR00T, and SmolVLA. And the dual-system split itself, fast control paired with slower reasoning, appears across GR00T N1, Figure AI's Helix, and so-called Fast-in-Slow variants because it solves a timing conflict that any single-speed system runs into: language reasoning takes time to do well, but motor control can't wait for it. Separating the two speeds, while keeping them coupled, is what lets a robot think carefully and still move smoothly at the same time.

SayCan and the grounding problem in early LLM planners

The first generation of language-model-driven robot planners failed for a specific reason: they reasoned without any connection to physical reality. A large language model asked to plan a kitchen task could produce a plan that read as completely sensible, step by step, in plain language, while being impossible for the robot in front of it to execute, because the model had no way of knowing what the robot could reach, what objects actually existed in the room, or what the robot's current state even was. This is open-loop planning: language goes in, a plan comes out, and nothing in between checks that plan against the world.

Google's SayCan architecture addressed this by splitting the planning problem into two functions that check each other. The "Say" function is the language model's job: given a high-level instruction, propose the useful sub-goals that would accomplish it. The "Can" function is a learned affordance function, and it does the opposite kind of work: it scores each candidate action by how likely the robot is to actually be able to execute it, given its current state and surroundings. A plan only proceeds through steps that clear both bars, sensible in language and feasible in the world. An action that would require reaching for an object out of range, or behind an obstacle the robot can't move, gets filtered out before it ever becomes a command.

That second function is what makes the architecture work. Without it, a language model will happily reference objects that aren't there, propose sequences that contradict each other, or generate control instructions that are unsafe to run. The affordance function turns physical feasibility into a hard constraint on the plan.

SayCan was tested on a mobile robot that carried out 101 real-world instructions inside an office kitchen, and the instructions varied in length, language complexity, and the robot's starting state. Inside the training environment, the system achieved strong planning and execution success rates. Moved into a new, previously unseen kitchen, execution success dropped. The grounding function worked as intended inside the conditions it had learned, and performance declined once the environment changed. Grounding solves hallucination, but execution success dropped when the environment changed, showing that grounding alone does not solve generalization across new environments.

A structural limit matters here: the affordance function can only select among actions the robot already knows how to perform. It composes existing skills in new combinations. It cannot invent a new skill the robot was never equipped with, no matter how well the language model reasons about the task.

VLAs on real hardware: the GR00T N1 and Helix deployments

Moving from architecture to deployment is where the promise of VLA models meets the complexity of physical work, and the record so far shows real capability alongside a persistent gap between how these systems perform on benchmarks and how they perform in continuous operation.

NVIDIA introduced GR00T N1 in March 2025 (arXiv:2503.14734) as an open foundation model for generalist humanoid robots, and deployed it on the Fourier GR-1 humanoid for language-conditioned bimanual manipulation tasks. The model was trained on a mixture organized as a data pyramid: large-scale web data and human video forming the base, synthetic data generated through physics simulation in the middle, and real robot trajectories at the top, a structure built specifically to address the fact that any single humanoid platform's own collected data is far too small on its own to support large-scale pretraining. On standard simulation benchmarks across multiple robot embodiments, GR00T N1 outperformed state-of-the-art imitation learning baselines. NVIDIA has also reported, through its GR00T-Dreams simulation pipeline, that when it added synthetic motion data to training, task success rates improved substantially compared to training on real-world data alone.

Figure AI's Helix, also introduced in 2025, became the first VLA to output high-rate continuous control across a humanoid's entire upper body, a 35-degree-of-freedom action space running at 200Hz that covers individual finger movements, end-effector trajectories, head gaze, and torso posture together. Coordinating head and torso movement in the same control loop as the arms mattered because those movements have historically been a source of feedback instability in robotic control; Helix was built to handle gaze, posture, and arm motion as one coordinated problem. After three months of deployment in logistics work, barcode scanning success improved markedly as vision and control both got better. A later version, Helix 02, extended the approach to full-body behavior: a robot walked to a dishwasher, unloaded it, placed items around the room, reloaded the machine, and started it, completing 61 separate locomotion and manipulation actions in sequence without a reset or a human operator stepping in.

Figure AI's hardware has also run inside BMW's Spartanburg plant. Over an 11-month deployment, Figure 02 units contributed to producing tens of thousands of BMW X3 vehicles, loaded a large volume of sheet metal components, and ran shifts lasting 10 hours on weekdays, according to figures reported in KraneShares' May 2026 analysis of the deployment. Those figures come from the companies involved, so they should be read as company-stated results. Following that pilot, BMW extended its humanoid program into Europe: Hexagon Robotics, the robotics division of Hexagon AB, announced in February 2026 that its AEON humanoid would be deployed at BMW's Plant Leipzig, with a full-scale pilot targeting summer 2026 and covering high-voltage EV battery assembly and component manufacturing.

Taken together, these deployments show a consistent shape. Systems get better with more deployment time and more data, and performance on familiar, in-distribution conditions is reliably stronger than performance the moment conditions shift outside what the system has seen before. That's the same split SayCan exposed years earlier between its training kitchen and the new one.

Where VLA-driven planning reliably breaks down

Analysis of LLM-based robotic planning frameworks has found that task complexity and dynamic disturbances are the primary drivers of failure in VLA-driven planning, while model scale and environmental scale carry only marginal impact on performance. That finding points away from "bigger model" as the general fix and toward the specific conditions under which these systems break.

Several of those conditions recur across deployments. Perceptual occlusion, where an object the robot needs is partially hidden from its cameras, breaks the grounding assumptions the whole plan depends on. Unpredictable behavior from other agents in the workspace, a person walking through, an object getting moved by someone else, disrupts plans built on a snapshot of the world that no longer holds. Hallucination remains a live risk too: language models can still reference objects that don't exist, produce sequences that contradict each other, or generate control code that isn't safe to execute, and this counts as a safety risk.

There's also a ceiling built into the SayCan-style approach itself. An affordance-filtered planner can only recombine the skills it already has. When a task calls for a behavior the robot was never given, the plan fails no matter how good the language reasoning behind it is. Better reasoning doesn't expand the robot's repertoire of physical skills.

The most structurally important limitation is the embodiment gap. A VLA model trained on a huge, shared corpus of language and vision does not automatically become correct motion on a specific physical robot. Language and vision models can draw on internet-scale data because text and images are abundant and cheaply collected. Robotics can't collect observations paired with matching action commands at anything close to that scale, because each pairing needs a physical robot actually moving in a physical space. One robotics researcher has argued this point directly, and it remains an open question which parts of this gap scaling alone will eventually close and which parts will always require engineering specific to each robot body.

Compute availability compounds the problem. Strong LLM-based planning currently depends on large proprietary models, and smaller open-source alternatives frequently struggle to produce even valid planning syntax (PDDL) in tested scenarios, though results vary by model and by domain. That dependency raises the operational cost of deploying these systems and limits who can access the strongest versions of them.

Coordination introduces a separate failure mode once more than one robot is involved. Centralized LLM coordination, where one model plans for every robot, cannot self-correct mid-execution and degrades sharply as the number of robots grows. Fully decentralized coordination avoids that bottleneck but incurs heavy communication overhead between robots instead. Neither pure approach scales cleanly. Most serious multi-robot work treats this as an open design problem.

Each of these is a specific, named engineering problem rather than a reason to doubt the VLA approach generally. They describe what the next generation of research is trying to fix.

Why closed-loop feedback became a design requirement

If a system generates a full plan from language and executes it without checking back against reality, that open-loop planning consistently underperforms once it's deployed outside carefully controlled conditions. That weakness follows directly from the previous section: dynamic disturbances and perceptual failures are the primary sources of real-world failure, and an open-loop system has no mechanism for responding to either one once execution has started.

Closed-loop feedback is the response. Instead of trusting a plan made at one moment in time, a closed-loop system keeps checking that plan against what the environment actually looks like as execution proceeds, rather than against what it predicted the environment would look like.

Three mechanisms built on that principle have become standard practice. Iterative re-prompting handles constraint violations directly: when the robot encounters something that doesn't match its plan, the system sends the updated state back to the language model and asks for a revised plan on the spot. Self-debugging gives the system a way to catch its own mistakes: the model monitors its own execution for errors and generates a corrective sub-plan when something goes wrong. Retrieval-augmented generation draws on past experience: the planner pulls relevant execution traces from prior tasks to inform the current one, which reduces the rate of hallucinated claims about where objects are or what actions are actually feasible.

Together, these mechanisms mark the shift from treating feedback as a nice-to-have addition to treating it as a required part of the architecture. A planning system built on SayCan's grounding insight and a VLA's dual-speed architecture still needs a way to notice, mid-task, that the world has changed. Closed-loop feedback is how that noticing happens, and it's become the baseline expectation for any VLA system meant to operate outside a controlled lab.