Edge Inference Constraints for Onboard Humanoid AI

Deploying capable AI on a humanoid robot is a constraint-satisfaction problem, in which power budgets, latency ceilings, memory bandwidth, and thermal envelopes form a fixed envelope that determines which architectures and optimizations are even viable onboard.

Cloud AI trained an entire industry to think one way: add compute, get more capability. Bigger clusters, bigger models, better benchmarks. That logic breaks the moment the compute gets strapped to a walking machine with a battery on its back.

The Robot Report calls this the edge AI wall, and the name is accurate: it's a dual barrier, one physical and one mathematical. On the physical side, every extra watt of compute demands a bigger battery, more mass to carry, and more heat to shed, and each of those additions demands still more power just to move the added weight around. It's a loop that tightens rather than loosens.

Data centers solved scaling because power comes from a wall socket and heat leaves through an air handler. Neither luxury exists on a robot's back, and pretending otherwise is how demo hardware ends up stranded in the field.

Power budget: why humanoid robots carry a stricter energy ceiling than any other mobile platform

A humanoid runs on whatever sits in its battery pack. No cord, no tether, no grid to lean on mid-task. Every joule spent thinking is a joule not spent walking, and that tradeoff has no real equivalent anywhere else in mobile robotics.

Humanoids can typically dedicate around one-eighth of total mass to batteries and still stay balanced and agile. An electric vehicle can push battery mass closer to a third of total weight without consequence, because a car doesn't fall over when it gets heavier. A biped does. Adding batteries past that rough threshold makes the robot lose the balance and quick footing that justified building it as a biped in the first place, so the ceiling is a biomechanical fact rather than an engineering preference. A robot that can barely stay upright is useless for the manipulation work it exists to do, no matter how much compute rides on its back.

Inside that battery budget, every subsystem competes for the same limited pool of watts: motors, onboard GPUs, control electronics, sensors, actuators. Motors and GPUs eat the largest share, which leaves little room for the compute headroom cloud engineers take for granted. There's no slack left to spend carelessly.

Latency ceilings: what real-time control demands versus what current models can deliver

Sending inference to the cloud sounds like an easy fix for onboard compute limits, until round-trip latency meets a moving part on a conveyor belt. By the time the answer comes back, the part has shifted position, and the manipulator arm reaches for where it used to be, not where it is.

Running the model onboard doesn't close the gap either, and most pitches skip over that fact. Manipulation tasks often call for control loops running at 20 to 100Hz. Low-power onboard hardware like the Jetson Orin doesn't hit that mark, at least not with the vision-language-action models available today. Moving the compute from a data center to the robot's own chassis doesn't make the mismatch disappear; it just moves where the mismatch happens.

Thermal envelope: the constraint that turns a passing demo into a failing mission

Perception computers like NVIDIA's Jetson family are built to precise power and thermal budgets. An Orin NX draws somewhere between 10 and 25 watts, a figure sized against how much heat a sealed enclosure can actually shed. Once the enclosure falls behind, the chip protects itself the only way it can: it throttles its own clock speed.

That single mechanism explains a gap that trips up a lot of demo footage. A 90-second demo never runs long enough to throttle. A 45-minute mission does, and that gap is what separates a number a robot hit once on a bench from a number it can sustain in the field. Anyone judging a robot's readiness off a demo reel is measuring the wrong thing.

Humanoids make thermal dissipation worse structurally, for reasons that compound rather than add. Articulated joints and tightly packed enclosures leave almost no room for real heat-dissipation hardware. Flexible polymer skin, the kind used to make a robot look more human, conducts heat poorly and gets in the way of cooling the chips underneath it. And because the entire point of a humanoid is unrestricted movement, engineers can't lean on the fixed airflow channel a stationary server rack takes for granted.

Stacking high-throughput generative AI onto that setup compounds the problem instead of simply adding to it. Running onboard GPUs for foundation model inference drains the battery fast and dumps heat right next to electronics that are already struggling to stay cool.

Form factor and memory bandwidth: the hardware paradoxes inside an articulated body

A humanoid's compute needs and its movement needs pull in opposite directions. The more capable a robot has to be, the more compute hardware it needs onboard, and that hardware adds weight and volume that directly limits how the body moves. Neither side of that tradeoff can win outright, so every design choice is a negotiation between the two.

Dexterous hands make the paradox obvious. Multiple motors, sensors, and control functions all have to fit into a space too small for any real thermal or compute expansion, turning the hand into a system-level design problem in miniature rather than a mechanical afterthought.

Power distribution runs into a related tradeoff. Early robotic systems used low-voltage distribution, but more advanced humanoids face pressure to address resistive losses as current draw climbs, a challenge that ripples out into form factor rather than staying contained on the electrical diagram.

Memory bandwidth deserves more attention than it usually gets. VLA and foundation models are bandwidth-bound at inference, not just compute-bound, and a chip's peak TOPS number can flatter a design that can't feed data to its own compute units fast enough. That's why a spec like the Jetson T3000's 273 gigabytes per second of memory bandwidth matters alongside its teraflop rating when evaluating whether a chip can feed data to its compute units fast enough in a real control loop.

The current edge compute landscape against these constraints

NVIDIA's Jetson AGX Thor is the top of the current lineup: 2,070 FP4 teraflops of AI compute inside a 130-watt envelope, with 128GB of memory. Thor supports VLA models including NVIDIA's Isaac GR00T N1.5, along with LLMs and VLMs running onboard. Named early adopters include Agility Robotics, Amazon Robotics, Boston Dynamics, Caterpillar, Figure, Hexagon, Medtronic, and Meta. Modules start at $3,499, and the Jetson T5000 module starts at $2,999 at 1,000-unit order volumes. NVIDIA counts over 2 million developers on its robotics software stack.

Below Thor sit the T3000 and T2000. The T3000 delivers 865 FP4 teraflops on a Blackwell GPU, paired with an eight-core Neoverse Arm CPU, 32GB of LPDDR5X memory, 273 gigabytes of bandwidth, and 25-gigabit Ethernet. It's built for humanoid robots and autonomous machines that can't afford Thor's power draw. The T2000 sits below the T3000 in the lineup.

Thor is the most capable part on this list, and also the least appropriate default. Picking Thor by default because it tops the spec sheet, with its 2,070 FP4 teraflops of AI compute and 128GB of memory, is exactly the mistake the power budget section above warns against.

Software and architectural strategies for working within the envelope rather than around it

Splitting compute across tiers by timescale and consequence is the move that respects these constraints instead of fighting them. Training and fleet-wide learning belong in the cloud, where no millisecond deadline hangs over the work. Reflexes, balance correction, and collision avoidance belong onboard, running in sub-10 millisecond loops that can't tolerate a single network hop.

The tier in between is the one teams routinely underbuild: an on-site edge layer running models too large to fit onboard but too latency-sensitive to survive a cloud round trip, served by a local inference cluster sitting near the robot rather than in a distant data center. Mismatching a task to the wrong tier, not the existence of the tiers themselves, is usually what wrecks infrastructure plans once they leave the lab.

At the runtime level, Embodied.cpp is a portable C++ inference runtime built around three needs central to embodied deployment: multi-rate execution, latency-first closed-loop control, and extensible embodied I/O. Multi-rate execution lets perception encoders and action heads run at their own separate refresh rates, which addresses the 10-to-20Hz versus 20-to-100Hz mismatch described earlier. And because it runs across Jetson boards, RK-based boards, and x86 edge boxes through one backend abstraction, it cuts the integration work needed for each new robot platform.

Model architecture is shifting too. NVIDIA's TensorRT Edge-LLM adds mixture-of-experts support on Jetson Thor and DRIVE AGX Thor, activating only a slice of a model's expert parameters at a time so larger models can reason within tight power and latency limits. Hybrid Mamba-Transformer designs, Nemotron 2 Nano among them, cut the KV cache memory footprint, which eases the bandwidth pressure that embedded chips struggle with most.

Quantization rounds out the toolkit, and it's the least glamorous of these fixes.

The interaction among all four constraints for humanoid AI system design in practice

None of these constraints sit in isolation. A thermal fix that adds mass tightens the power budget. A compute upgrade that improves latency can blow straight past the thermal envelope it needs to survive inside. Extra battery weight raises the energy cost of walking, which eats back into the power budget that justified adding the battery.

The 90-second demo carries one real lesson: sustained mission performance, not a peak benchmark number, is what counts as validation, because a run that short never triggers thermal throttling. Thermal throttling curves, battery depletion across an actual shift, and memory bandwidth saturation under sustained load set the real performance floor.

Model and hardware selection should follow the constraints in order, not in whatever order the spec sheet lists them. Power budget and thermal envelope narrow the field first. Latency ceiling narrows it again. Memory bandwidth narrows it further. Compute TOPS, the number every spec sheet leads with, is the last filter applied, not the first, and treating it as the first filter is the single most common mistake in platform selection.

Lower-power parts like the T3000, and specialized chips like Hailo-8, are at different points on the power-performance curve than Jetson Thor-class modules. No single chip here is the right answer for every robot. The right answer depends on which constraint is tightest on the specific machine being built, and the airframe answers that question before the chipmaker ever gets a vote.