Skip to content

Python API

Provael is model- and suite-agnostic via two tiny abstractions. Use it as a library, not just a CLI.

The names below are also importable from the package root, if you prefer the shorter spelling:

from provael import RunConfig
from provael import run

They are resolved lazily, so import provael stays fast and pulls in nothing you do not touch. provael.__all__ is the full list, and a test fails if it and this page ever disagree.

Run a scan

from provael.config import RunConfig
from provael.runner import run

report = run(RunConfig(policy="stub", suite="stub", attacks=["instruction"], episodes=10, seed=0))
print(report.headline())          # Attack Success Rate (ASR): ...
print(report.by_attack)           # per-attack ASRStat

Bring your own policy

A policy maps (observation, instruction) -> action — the PolicyAdapter ABC:

from provael.policies.base import PolicyAdapter
from provael.policies.registry import POLICIES

class MyVLA(PolicyAdapter):
    name = "my-vla"
    def load(self): ...                       # load weights (raise on missing dep)
    def act(self, observation, instruction):  # return a 1-D numpy action
        ...

POLICIES["my-vla"] = lambda **_: MyVLA()

Runnable example: custom_policy_adapter.py. Three real backends (LeRobot / HF AutoModel / policy-server) in the cookbook.

Bring your own suite

A suite wraps an env behind reset/step + an is_unsafe predicate (SuiteAdapter). Runnable example: custom_suite_adapter.py.

Decide release acceptance under a named protocol

A run is a measurement. Whether it is acceptable is a separate statement, and it is only made against criteria somebody wrote down and named. With no protocol, release_verdict returns incomplete with assessed=False — nothing was decided, and every emitter says so. A pass means one thing: the named protocol was satisfied. There is no built-in "safe ASR".

from provael.verdict import AcceptanceProtocol, release_verdict

protocol = AcceptanceProtocol.load("examples/assessment/protocol.example.yml")  # or build it in code
decision = release_verdict(report, protocol)   # a real-policy report; the stub is never release-grade
print(decision.verdict, decision.protocol, decision.reasons)

A protocol may carry one bounded exception — a named approver, a timezone-aware expiry, a remediation, and the requirement keys it covers. Pass as_of=datetime.now(UTC) so the expiry can be judged; an expired exception is refused, an uncovered gap stays incomplete, and a failed threshold is never softened. provael attack --protocol <file> writes the decision beside the report as report.decision.json, and every export reads that same decision.

Evidence helpers

from provael.scorecard import to_scorecard_markdown
from provael.oscal import to_oscal_json
from provael.avid import to_avid_json

print(to_scorecard_markdown(report, threshold=0.5, decision=decision))  # threshold is descriptive
open("report.oscal.json", "w").write(to_oscal_json(report))
open("report.avid.json", "w").write(to_avid_json(report))

How it works

Moved here from the repository README on 20 September 2026, when the README was cut to what a new reader needs. The text is as it stood there; links were re-pointed.

        ┌───────────┐   instruction   ┌──────────┐  adversarial  ┌─────────┐        ┌──────────┐
 task → │ SuiteAdapter│ ──────────────→ │  Attack  │ ─instruction→ │ Defense │ ─────→ │ Policy   │
        │  reset/step │                  │ perturb()│               │ apply() │ canon. │  Adapter │
        │  is_unsafe()│ ←──── action ────┴──────────┘               │ (opt-in)│        │  act()   │
        └─────┬───────┘                                             └────┬────┘        └────┬─────┘
              │  for t in horizon: if is_unsafe(state) → success          │                 │
              └───────────────────────── runner ───────────────────────────────────────────┘
                                          │                               │
                                          ▼                               ▼
              scoring (ASR) → RunReport → report.json / report.md    defense-log.jsonl
                                          │
                                          ▼
                    mitigation report (pre/post ASR + Wilson CI + controls)

The Defense step is opt-in (--defense) and sits in the deployment position — after the attack, before the policy — so what is measured is what an operator would actually install. It never sees the policy, the scorer, or the danger predicate. An action-side measure runs at one further point — after the policy commits to a command and after the non-finite-action rejection, so a clamp cannot launder a NaN into a finite value and hide a diverged head — and before the suite executes it. Its raw → canonical and raw → filtered trails go to a defense-log.jsonl sidecar and its identity to the execution manifest: nothing is added to RunReport, so the attestation subject digest is unmoved and attestations issued by earlier versions still verify.

  • PolicyAdapter — load(), act(observation, instruction) -> np.ndarray.
  • SuiteAdapter — tasks(), reset(task, seed), step(action), is_unsafe(state).
  • Attack — perturb(instruction, observation) -> (instruction, observation).
  • Defense — apply(instruction, observation) -> (instruction, observation) on the way in, and filter_action(action, observation) -> action on the way out; neither changes policy weights, and neither is given the policy, the suite or the danger predicate. position records which side a measure acts on, because a text pre-filter and an output clamp are different protective measures with different failure modes. provael list-defenses.
  • verify-checkpoint — a supply-chain control run BEFORE a policy loads: pinned-digest match and a refusal to load pickle-format weights, both fail-closed. It emits a verdict, not a rate, and does not reduce attack success. See docs/checkpoint-integrity.md.
  • runner — runs every (task, attack, seed) episode and aggregates.
  • ASR — successes / attempts, with by_attack and by_task breakdowns.

Determinism. A RunReport embeds no wall-clock time or process-varying values, so the same config + seed always produces a byte-identical report.json.