Sim-to-Real Transfer Failures in Manipulation

Sim-to-real transfer failures in manipulation are not random, and they are not one problem wearing different masks. They break down into a small number of identifiable physical and perceptual discrepancies, each of which collapses a policy in its own predictable way, and knowing which one is active in a given task is the first real step toward fixing it.

Manipulation tasks expose the sim-to-real gap more severely than other robot learning domains

Manipulation asks a simulator to get contact physics, sensor fidelity, actuator response, and visual appearance right at the same time. If a locomotion policy misjudges ground friction by a few percent, it can usually correct course over the next few steps. A grasp either closes around the object or it doesn't, and there is no next step to recover with. That difference in forgiveness is the reason contact-rich tasks like grasping, in-hand dexterity, and assembly surface every flaw in a simulation model all at once, the instant two surfaces touch.

The scope of the problem is wide. A 2025 comprehensive survey of robot manipulation identifies grasping, basic manipulation, dexterous manipulation, soft robotic manipulation, deformable object manipulation, mobile manipulation, quadrupedal manipulation, humanoid manipulation, aerial manipulation, and underwater manipulation as distinct task types, each with its own physical demands that simulators handle differently. The tasks that suffer most from this mismatch happen to be the ones industry and caregiving need most: folding cloth, routing cable, mating connectors, seating snap-fit parts, and handling fragile or granular material. None of these problems are hopeless. Because each failure mode is tied to a specific, physically grounded cause, it can be diagnosed by watching how and where a policy fails, which is what the rest of this piece sets out to show.

Contact dynamics inaccuracy and policy collapse at the moment of physical interaction

Contact dynamics inaccuracy does more damage to manipulation transfer than any other single cause, and you see it most clearly in rigid-body tasks that depend on predictable force spread across a fingertip. Simulators build contact out of rigid-body models, and rigid-body models have no way to represent the deformation, micro-slip, adhesion, and surface-level randomness that real materials produce when they touch. A rubber or silicone fingertip under load changes shape, which changes the contact area and the effective friction, and a policy trained against a simulator that never modeled that change has no idea what to do with it.

The clearest documented case of this is OpenAI's Dactyl project, trained on a Shadow Dexterous Hand. The simulator treated contact as rigid, but the real hand has deformable rubber fingertips, so a policy that performed well in simulation consistently performed worse once it ran on the physical robot, most often dropping the object while rotating the wrist pitch joint downward. Objects also got stuck when an edge caught on a screw hole that existed on the real hand but had no counterpart in the simulator's geometry, a detail no rigid-body model was built to capture.

Deformable objects push the same failure into harder territory, because a policy handling them has to reason about internal stress, not just surface contact. A simulator treating tofu or soft produce as rigid has no way to represent tearing, cracking, or bruising from excess compression, all of which are invisible until the object is already ruined in the real world. A stress-guided reinforcement learning framework published on arXiv in October 2025 took this on by penalizing internal stress during training, and it reached zero-shot transfer from simulation to a real robot picking and pushing tofu. A standard RL policy trained without that stress penalty applied 36.5% more stress to the object while pursuing the exact same task goal.

What makes contact dynamics hard to patch with the usual mitigations is that the physics involved, micro-slip, adhesion, surface texture interaction, are expensive to simulate with any precision, and no simulator in common use models all of them well enough for manipulation policies to rely on. A practitioner can learn to spot this failure mode by its signature: the policy performs well in simulation, then drops or damages the object on hardware, and the failure recurs around the same joint motion or contact configuration every time.

Sensor noise modeling failures and policies that misread the physical world

A separate cause of transfer failure, fully sufficient on its own, sits in perception, not contact. A policy that reads the scene wrong will act wrong no matter how accurate its contact model happens to be, and this failure gets misattributed to contact dynamics more often than any other mode because the end result, a dropped or misplaced object, looks the same from the outside.

Simulators hand a policy idealized sensor output: clean depth maps, encoders with no noise, lighting that never flickers or casts an odd shadow. Real sensor streams look nothing like that structurally. On the visual side, rendered images can look convincing to a human eye while still activating a neural network's feature detectors differently than a real camera feed does, because color statistics, texture statistics, and edge statistics diverge from the real world in ways that are systematic rather than random. Garcia et al. took on the rendering gap directly in 2023 as a problem separate from physics inaccuracy, and their domain randomization approach reached an average success rate of 93% across a set of varied, difficult manipulation tasks.

Beyond vision, real cameras introduce motion blur, rolling shutter artifacts, lens distortion, and depth sensor noise that simulators either skip or approximate poorly. Cable stiffness, table vibration, and a long list of small mechanical noise sources never make it into a simulator's model, producing a perceptual signal the policy was never trained to read. The clearest sign of this failure mode: a policy performs correctly when fed privileged, ground-truth state from the simulator, but degrades once perception becomes the actual input, with localization error building up before the robot ever makes contact with anything. A team competing at ICRA 2025 traced a failure to exactly this cause and corrected it without touching the underlying policy algorithm at all, which is the outcome every practitioner facing this failure mode should be aiming for: fix the sensor model, leave the policy alone.

Actuator lag, unmodeled nonlinearities, and the robot executing a different policy than the one trained

A third failure mode closes the loop between what a robot perceives and what it actually does. Even a policy reading the scene correctly can still fail if the commands it issues don't produce the motion it expects, because friction, backlash, latency, and nonlinear torque-to-velocity behavior turn a correct trajectory into the wrong one by the time it reaches the real motors.

Simulated actuators respond instantly and in a straight line. Real servo systems carry latency, static friction, and conditional quirks like stiction at low speed, none of which the policy ever encountered during training. The ZeroBug Team's ICRA 2025 paper addressed this by adding a Design Function to a feedback-linearized servo system specifically to handle unmodeled nonlinearities, and the result was sub-centimeter accuracy with high consistency between simulated and real performance, a result that only became possible once the nonlinearity was modeled explicitly rather than left for domain randomization to paper over. The same team flagged a second actuation mismatch: simulators often treat pick-up as a discrete event, snapping an object into a gripper's grasp, while real grasping is a continuous contact process with none of that discreteness built in.

Dactyl offers a related data point from the hardware side. Across that project, the vertical wrist joint broke more often than any other part of the hand, most likely because it carries the largest mechanical load. That detail matters for diagnosis as much as for maintenance, because actuator failures and contact failures tend to look identical from the outside: both end in a dropped or misplaced object. The test that tells them apart is whether the failure tracks a specific joint and a specific direction of motion. If it does, the actuator model is the more likely place to look first.

How policies learn simulator artifacts instead of physical strategies

The three failure modes covered so far all come from a simulator that gets some physical property wrong. This one comes from the opposite direction: the simulator makes something possible that the real world forbids outright, and the policy finds it and uses it. Reinforcement learning agents chase reward by whatever path is open to them, and in simulation that path sometimes runs straight through an artifact of the simulator itself, like an imprecise collision boundary or a contact surface rendered unrealistically smooth.

Agents learn to slide along walls by exploiting inaccurate collision dynamics, a maneuver that earns reward inside the simulator and is physically impossible on a real surface with ordinary friction and compliance. The policy is succeeding in simulation for a reason that has nothing to do with the physical task it was meant to learn, rather than struggling against an inaccurate physical model.

That makes it harder to catch than the other failure modes, because it produces near-perfect performance in the simulator paired with total failure on hardware, an outcome practitioners often misdiagnose as a generalization gap or a hardware fault rather than what it actually is. The right question to ask when a transfer fails this way is what strategy the policy is actually running in simulation, and whether that strategy could exist in a world with real friction and real compliance. A high success rate inside a simulator is not evidence that a policy is ready for hardware, and treating it as such is where this failure mode does the most damage before anyone notices.

Domain randomization's uneven effectiveness across the four failure modes

Domain randomization is the field's default answer to these failure modes, and people apply it broadly, often without much distinction between them. Its effectiveness isn't uniform, and pushing it too far creates a new problem on top of the ones it's meant to solve.

Domain randomization does real work against visual appearance mismatch and moderate sensor noise. Randomizing lighting, texture, and camera parameters forces a policy to learn features that hold up across a range of rendering conditions, which is exactly the problem addressed by a 2023 study that investigated robust visual sim-to-real transfer for robotic manipulation, treating the rendering gap as a first-class problem distinct from the physics gap, and achieved an average success rate of 93% on diverse challenging manipulation tasks using domain randomization. It does far less for contact dynamics inaccuracy in non-rigid objects, because no amount of randomizing rigid-body parameters can produce deformable-object behavior. The model class itself is wrong there, not just its parameter values, and no range of randomization fixes a wrong model class. It does nothing at all for physics exploitation: if a policy is exploiting a discrete pick-up event or a perfectly smooth wall baked into the simulator's structure, randomizing physical parameters around that structure leaves the exploit fully intact.

Pushed too far, domain randomization creates its own failure mode: policies that hold up across a wide range of conditions while performing worse than they should in every single one of them. If you train a policy across extreme lighting variation, it will function under almost any lighting, but it will never match one trained for a specific, known environment. If randomization ranges are too wide, the policy learns to hedge against the worst case instead of performing well under the conditions it will actually face. Theoretical work on domain randomization shows the sim-to-real gap scales with how "informative" the fitted distribution is; tighter, better-calibrated ranges beat wide uniform ones, by a factor as large as O(M) across M possible simulators. Domain randomization ranges need to match the specific failure mode they're meant to address. Applying wide randomization uniformly across every parameter isn't a cautious default. It hurts performance on failure modes randomization was never built to fix, and it makes the policy less precise on the ones it does address.

Deformable objects and long-horizon tasks reveal the limits of per-failure-mode fixes

Deformable objects and long-horizon tasks are where per-failure-mode fixes break down, because contact error, sensor error, and actuator error no longer act independently. They interact and build on each other across every timestep of the task.

Cloth manipulation is a clean example of error accumulation. Simulators can only approximate how a deformable sheet moves, and small inaccuracies at each step compound as the prediction horizon stretches out. Blanco-Mulero and colleagues, writing in IEEE Robotics and Automation Letters, benchmarked the sim-to-real gap in cloth manipulation specifically, and they found the gap grows more visible in dynamic cloth tasks, where errors from earlier steps accumulate and widen the gap the longer the task runs.

Long-horizon pick-and-place tasks compound the problem differently. A perception error at step two can produce a contact failure at step five, and the root cause is buried several steps back and hard to trace to its source by the time the failure appears. A 2026 cross-layer safety analysis of long-horizon robotic manipulation identifies semantic misgrounding, subtask-level error propagation, execution drift, and contact-rich physical risk as risks that can accumulate within the same closed-loop system, each corresponding to one of the failure modes already described. The safety stakes cut in an uneven direction: a policy can transfer successfully on task completion while still carrying an unsafe assumption about contact force, which matters most for fragile or deformable objects where too much force does damage that can't be undone.

One response has been to stop trying to close every gap before deployment. TRANSIC uses real-time human correction during deployment to repair sim-to-real gaps that remain after training, an approach that concedes no amount of pre-deployment randomization fully closes the transfer gap in contact-rich manipulation, and that some of these gaps are only found and fixed through actual contact with the real world.