Understanding sim-to-real gap with failure clustering
TL;DR: When a robot fails in the real world, the root cause is rarely the single event you see. Group failures into clusters, characterize each cluster by sim-versus-real signal differences, then retrain or update the simulation domain so the next deployment closes the gap.
Why the gap feels vague
Most robotics teams start with the same symptom: the policy worked in simulation, but on the floor it misses picks, collides with fixtures, or hesitates in lighting conditions that never appeared during training. The natural response is to add more training data or randomize the simulator. That sometimes helps, but it is a blunt instrument.
The sharper approach is to ask: what kind of failure is this? Failures of the same kind share causes and share fixes. Clustering is the discipline of answering that question with data instead of intuition.
A practical cluster taxonomy
We sort production failures into four clusters before deciding what to change:
- Perception drift. The camera or sensor inputs look different in production than in simulation: lighting, occlusion, background clutter, or calibration.
- Dynamics mismatch. The physics of contact, friction, inertia, or actuator latency differ enough that the policy's action distribution is wrong.
- Task-scope overflow. The robot encounters a situation that was never represented in training: a new object type, an unseen layout, or an off-nominal procedure.
- Deployment-system noise. The failure is caused by network latency, stale configurations, or operator handoff gaps rather than the policy itself.
Each cluster points to a different fix. Perception drift usually means better domain randomization or real-to-sim calibration. Dynamics mismatch means tuning physics parameters or switching to a higher-fidelity contact model. Task-scope overflow means expanding the training distribution with curated edge cases. Deployment-system noise means fixing infrastructure before retraining anything.
How Sim2Real captures clusters
Sim2Real collects telemetry from each deployment and compares production trajectories against the nearest simulation rollouts. We compute a drift score per failure event, then group events whose drift signatures look similar. The result is a failure-cluster report that tells you:
- Which cluster is growing fastest.
- Which simulation parameters are most different for that cluster.
- Which retraining or sim-update action is most likely to reduce that cluster.
Close one cluster at a time
The goal is not to eliminate all sim-to-real gap in one pass. It is to turn an ambiguous pile of failures into a prioritized list of concrete fixes, ship one, measure whether the cluster shrinks, and then move to the next. That loop — detect, cluster, fix, validate — is how high-performing robotics teams scale from pilot to production.