Skip to content

Embodied AI Security Top 10 ↔ SafeVLA-Bench crosswalk

Defensive, sim-only. This is a taxonomy-comparability artifact. It runs no SafeVLA-Bench harness, publishes no comparative scores against their numbers, and drives no physical robot. See SAFETY.md.

mapping_status: the mapping ships; the numeric comparison does not

provael crosswalk --target safevla now exists and emits results/crosswalk/crosswalk.safevla.json, carrying the structural mapping below plus a machine-readable blocker field.

What it deliberately does not emit is a number. No provael SBU is placed beside a SafeVLA-Bench SBU, and --in has no effect on this target. The blocker is unchanged from when this page said nothing was implemented: our succ_but_unsafe shares their metric's name and has not been shown to share its units, on a predicate that is still uncalibrated.

Comparing what the two projects measure does not require their numbers to be commensurable, and stating the incommensurability in a committed artifact is more useful than withholding the mapping. Do not cite this page as coverage of SafeVLA-Bench's scenarios.

What this is (and is not)

Source (pinned). SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models, Fan, Xu, Sokolsky, Lee, Kong (University of Notre Dame; University of Pennsylvania), arXiv:2606.00773 (submitted 30 May 2026) · safevla.org. Quotations below are verbatim from the abstract.

Not SafeVLA (2025), which is an alignment defense with a near-identical name. Two different works. See PRIOR_ART.md.

Why this crosswalk is worth building

Both instantiate on LIBERO, and provael's one measured real-policy result is SmolVLA × libero_object. A shared substrate is what separates a tractable crosswalk from a hypothetical one: the same policy, on the same tasks, can in principle carry both an SBU and an ASR.

They "formalize task-aware safety requirements as Signal Temporal Logic (STL) specifications and report native success with two unsafe-success metrics: Succ-But-Unsafe (SBU) … and Violation Severity Index (VSI)".

The two measure different failures

SafeVLA-Bench provael
When it acts post-hoc — scores rollouts already produced pre-hoc — perturbs the input first
Who causes the failure nobody; the policy's own behaviour under ordinary instructions an adversary, by construction
Safety definition STL specifications over the trajectory a keep-out predicate, currently uncalibrated
Denominator rollouts of the native benchmark task matched (task, seed) pairs against a benign twin
Headline SBU, VSI ASR with a 95% interval and a benign control

Neither is a substitute for the other. A policy can score well on SBU and still have a high ASR; a policy with a low ASR can be routinely unsafe on its own. A safety case citing only one is answering half the question.

The blocker, stated plainly

provael already has a field called succ_but_unsafe (scoring/asr.py) whose docstring names SafeVLA-Bench, and it computes the same per-episode quadrant: task-success and unsafe. That shared name is exactly what makes a premature crosswalk dangerous.

It does not share their units. Theirs is an STL-violation judgement over a trajectory; ours is a boolean from an uncalibrated keep-out predicate — on the ten-task LIBERO run the benign control itself fired on 2/50 episodes, so our "unsafe" carries a false-positive floor their STL judgement does not. Placing the two figures in one column would repeat precisely the error the ForesightSafety-VLA crosswalk exists to avoid: borrowing a benchmark's vocabulary never implies borrowing its units.

Calibrating the predicate is the prerequisite, and it is not done — provael calibrate has never been run on LIBERO. Until it has, an SBU emitted by provael and an SBU reported by SafeVLA-Bench are two different quantities wearing one name, and publishing them adjacently would mislead in the direction that flatters us.

What implementing it would take

  1. Calibrate the keep-out predicate on LIBERO so "unsafe" has a defensible threshold. Not done — provael calibrate has never been run on LIBERO.
  2. Express at least one of their STL clauses as a provael danger predicate, and say which. Not done.
  3. ~~Emit crosswalk.safevla.json behind provael crosswalk --target safevla~~ — done, with the structural mapping and the blocker, and without the per-row numeric comparability the original plan assumed. That column needs steps 1 and 2 first; shipping it before them is the exact error this page was written to prevent.

Only step 3 was code, and it is the step that could be honestly separated from the other two. Steps 1 and 2 are measurements, and until they exist an SBU emitted by provael and an SBU reported by SafeVLA-Bench remain two quantities wearing one name.

Their finding a reader of our numbers should see

Independent of anything we measured: "high-SR tabletop baselines still leave 13 to 15 percent unsafe-episode rates, and 36 to 56 percent of successful RoboCasa-365 rollouts violate at least one active safety clause."

A policy can be unattacked and still unsafe at those rates. provael's ASR is blind to that floor by construction — it measures lift over a benign control, so a policy already unsafe 15% of the time with no adversary present does not show up in our headline at all. That is a real limitation of our metric, not a criticism of theirs.