Skip to content

Does sim red-teaming predict real-robot behaviour?

The first objection to any simulation-only safety tool is "but it's only sim." This page is the honest answer, with citations — not a promise.

The short version

Provael is a pre-deployment scanner: it measures how often an attack drives a policy out of its safe envelope in simulation, before that policy ever touches hardware. The relevant research finding is that, for learned visuomotor policies, sim (and even edited-image) evaluation predicts real-robot failure well enough to be useful — without breaking real robots to find out. The strongest evidence below is about diffusion policies rather than VLAs; the VLA evidence is real but weaker, and that difference is marked per citation rather than averaged away.

  • Predictive Red Teaming (Majumdar et al., 2025) degrades a policy's inputs (lighting, textures, distractors, camera pose) in sim / on edited images and shows the predicted per-factor success tracks real-robot success closely — in the paper's own words, "less than 0.19 average difference between predicted and real success rates", across 500+ hardware trials in twelve off-nominal conditions. It identifies brittle factors without real-world rollouts. This result is on visuomotor diffusion policies, not VLAs. It is the strongest sim-predicts-real evidence on this page and it is not about the model class Provael tests, so it is a reason to think the approach is sound rather than a measurement about VLAs. arXiv:2502.06575 · https://arxiv.org/abs/2502.06575
  • Robustness benchmarks (LIBERO-Plus, LIBERO-PRO, 2025–26) independently find VLAs collapse under camera/position shifts and largely ignore language perturbations — the same failure modes Provael's instruction/visual families exercise. This one is about VLAs, but it measures robustness rather than sim-to-real agreement — it corroborates the failure modes, not the prediction claim. arXiv:2510.13626
  • Real-to-sim evaluation (SimplerEnv) is the methodology VLA papers use precisely because sim rankings track real hardware closely enough to compare policies. https://simpler-env.github.io/

What this does and does not claim

  • Does: give a reproducible, controlled, pre-deployment signal (calibrated ASR + 95% CI + benign-FPR control) that flags brittle policies early and cheaply, and that the literature says correlates with real-robot brittleness.
  • Does not: certify real-world safety, predict a specific robot's behaviour on a specific day, or replace hardware testing and functional-safety processes. Treat the ASR as a floor on susceptibility, not a guarantee. See compliance (evidence, not certification) and the README's "Scope and honest limitations."

How to keep your own results honest

  • Always run the none benign baseline and read every attack rate against that control.
  • Calibrate the predicate to a benign-FPR target (provael calibrate) so "unsafe" means "diverted past the policy's own benign envelope," not an arbitrary threshold.
  • Report CIs and n, not just point estimates — n = 10 per attack is a screen, not a proof.