Five signs your simulation is drifting from reality
TL;DR: Sim-to-real drift is easiest to fix when you catch it early. Watch for success-rate divergence, collapsing model confidence, operator overrides, domain-randomization gaps, and stale simulator assets. Each is a leading indicator with a specific fix.
1. Success-rate divergence
The most direct signal is also the easiest to ignore. Track real-world task success rate alongside simulated success rate for the same task. When the two curves separate by more than a few percentage points, you have a drift problem. The gap usually grows slowly, so you need a dashboard that compares the two numbers side by side, not a one-off evaluation.
Fix: Do not wait for a large gap. Re-train on the most recent production episodes as soon as divergence exceeds your threshold, and update simulator parameters to match the observed success distribution.
2. Confidence collapse
Even when success rate looks acceptable, your policy may be operating near the edge of its competence. A drop in average model confidence on production inputs is an early warning that the input distribution has shifted away from training. Confidence is often the first metric to move, before failures become common.
Fix: Log confidence per cluster. If confidence drops for a specific object, lighting condition, or site, generate new training episodes that replay that condition in simulation.
3. Operator overrides
Operators are human drift detectors. When they repeatedly pause, correct, or bypass the robot for a particular situation, that is a signal. Structured override logs are more valuable than anecdotal complaints because they tell you exactly where and when the policy failed to generalize.
Fix: Treat operator overrides as labeled failures. Import them into your failure cluster, characterize the sim-real gap for that cluster, and add counterexamples to your training set.
4. Domain-randomization gaps
Domain randomization works only when the randomization space covers the real-world variation you will see. If production introduces a new lighting angle, object texture, or background pattern that was never randomized, the policy will struggle. The fix is not always more randomization; it is the right randomization.
Fix: Compare the parameter ranges in your simulator against the ranges observed in production telemetry. Expand ranges where production data exceeds your training envelope, and tighten them where you are wasting samples.
5. Stale asset versions
Simulators, URDFs, meshes, and textures drift from the real hardware and environment over time. A cabinet moved, a camera was replaced, or a gripper was retightened. Small physical changes compound into policy failures that look mysterious unless you version your simulation assets alongside the real world.
Fix: Version simulator assets and map production failures back to the asset version used for training. When a cluster correlates with an asset change, update the simulator and retrain.
Build the habit before the drift compounds
None of these signs are emergencies on their own. Together they form an early-warning system. The teams that scale sim-to-real successfully are the ones that instrument all five, review them weekly, and close the smallest gaps before they become large ones.