What Sim2Real actually returns after processing deployment telemetry.

1. Robot and task context

2. Observed symptoms

• Real-world grasp success: 64.3% (vs. 91.1% in the most recent simulation regression run)
• Failure mode: Gripper closes on specular vacuum-sealed packaging; item slips during transfer 81% of the time
• Time-of-day pattern: Failure rate climbs to 78% between 16:00 and 18:30 when the skylight raises scene luminance and the gripper shadow self-occludes the package face
• Operator overrides: Spiked 6.2× in the same window (mostly manual repositioning)
• Telemetry signal: Wrist-camera histograms show a bimodal luminance distribution with a clipped specular highlight that the simulation never produced

3. Clustered failure pattern

These failures are reported by the robot under the perception mismatch pattern — lighting variance and reflective material handling — and Sim2Real groups them accordingly. The cluster is distinct from the physics-mismatch and task-mismatch clusters the same robot is producing, which is why the recommendation can target the right layer of the stack. Failures your telemetry does not tag are grouped as unknown.

Within the perception-mismatch family, the cluster is further tagged specular_highlight_self_occlusion and low_angle_skylight. Both tags co-occur in 71% of the cluster’s failure episodes, which is why we treat them as a single recommended action rather than two independent ones.

4. Simulator assumptions that turned out to be wrong

• Lighting randomization range was set to a uniform distribution matching the simulation regression fixture, not the production scene. The skylight angle at end-of-aisle never appeared in any training episode.
• Material randomization included diffuse plastic and matte cardboard but excluded the vacuum-sealed reflective SKU that the pilot now ships 3× per day.
• Pose randomization kept the package face within ±15° of the gripper camera. Production logging showed the actual distribution spans ±42° because totes are tipped at varied fill levels.
• Sensor noise model assumed Gaussian noise on a calibrated sensor. Production cameras in this aisle show a clipped-highlight region that the sensor model does not produce.

5. Parameter patch (what Sim2Real ships back)

The cluster returns a JSON-mergeable starter patch. Below is the rendered shape — the live product exposes .json patch exports via the /api/export/isaacsim and /api/export/mujoco endpoints, whose exact schema differs from this illustration. The exported values are baseline starting points for the tagged pattern, not per-scene calibrated numbers; your engineering contact tunes them against your setup during the pilot.

{
  "cluster_id": "C-0042",
  "site": "Pilot Site B / Aisle 7B",
  "robot": "arm-6dof-rgbd-gripper-v3",
  "tags": ["perception_mismatch", "specular_highlight", "low_angle_skylight"],
  "patch": {
    "domain_randomization": {
      "lighting": {
        "skylight_intensity_range": [0.6, 1.4],
        "key_light_angle_range_deg": [10, 75],
        "add_secondary_skylight": true
      },
      "materials": {
        "add_specular_variants": ["vacuum_seal_clear", "vacuum_seal_metalized"],
        "specular_roughness_range": [0.02, 0.35]
      },
      "pose": {
        "package_face_yaw_range_deg": [-42, 42],
        "package_face_pitch_range_deg": [-25, 25]
      }
    },
    "training": {
      "increase_specular_weight_x": 2.5,
      "new_failure_mix": "specular_seal:0.35,matte:0.45,transparent:0.20",
      "recommended_min_episodes": 1200
    },
    "validation": {
      "holdout_set": "aisle_7b_last_48h",
      "regression_set": "isaac_sim_v2024.1_pilot_b"
    }
  },
  "evidence_refs": ["telem:7b-1742", "telem:7b-1881", "telem:7b-2104"]
}

6. Before / after on the held-out validation set Synthetic demo

Synthetic demo. These numbers are fabricated — they are not a customer result. They show the shape of the report your team would see; the live product reports actual measured deltas against your own held-out set.

7. Next retraining action

1. Apply the patch above to your Isaac Sim domain-randomization config, adjusting the ranges to your scene.
2. Author a specular_seal scenario for your next training run — scoped with your engineering contact during the pilot — alongside the existing scenario set.
3. Validate against the held-out aisle_7b_last_48h set before promoting the policy to production.
4. Telemetry from the updated policy is surfaced for the next cluster review, so the loop stays closed.