Understanding sim-to-real gap with failure clustering
TL;DR: When a robot fails in the real world, the root cause is rarely the single event you see. Group failures into clusters, characterize each cluster by sim-versus-real signal differences, then retrain or update the simulation domain so the next deployment closes the gap.
Why the gap feels vague
Most robotics teams start with the same symptom: the policy worked in simulation, but on the floor it misses picks, collides with fixtures, or hesitates in lighting conditions that never appeared during training. The natural response is to add more training data or randomize the simulator. That sometimes helps, but it is a blunt instrument.
The sharper approach is to ask: what kind of failure is this? Failures of the same kind share causes and share fixes. Clustering is the discipline of answering that question with data instead of intuition.
A practical cluster taxonomy
Sim2Real stores each failure under one of six recognized tags. Any other value is stored as unknown. There is no task-mismatch or deployment-noise tag.
- Lighting variance. Glare or low light reported on the event. Three or more can produce a starter lighting override.
- Object tilt. Tray or object angle reported as affecting placement. Three or more can produce a starter contact and friction override.
- Clutter occlusion. Partially hidden grasp targets. Three or more can produce a starter clutter-density override.
- Physics mismatch. Friction or contact the event tags as diverging from simulation. The workspace lists the group. The starter file does not add a separate patch for this tag.
- Sensor mismatch. Sensor readings the event tags as diverging from simulation. The starter export does not rewrite a sensor noise model.
- Unknown. No recognized tag, or a value outside this list.
A lighting, contact, or clutter override is a starting file for your team to review. It is not a measured fix, and the product does not decide which change will shrink a group.
How Sim2Real captures clusters
Sim2Real groups failure events by the pattern tag your telemetry reports and shows counts and success rate per group; drift scores are stored when you send them. The workspace does not watch a robot live. During a pilot, the useful questions are:
- Which recognized tag has the most posted events.
- Whether that tag is one the exporter can turn into a starter override (lighting, contact, or clutter) or only a listed group (physics mismatch, sensor mismatch, unknown).
- What your team will compare on the next run. A later change in the reported success rate is not by itself proof of cause.
Close one cluster at a time
The goal is not to close every sim-to-real gap in one pass. Review the posted tags, change one thing your team owns, and compare the next run. Sim2Real does not measure whether a group shrank because of that change.