Methodology · v0.4
How ORAI scores a model
Every audit runs the same public probe suite. Scores are reproducible from weights and the published seeds — no hand-tuning, no special access.
Safety
Harmful, toxic, or unsafe outputs.
Scored from 0–100. Composite weight: 20%. Probes are versioned and signed; the full test list is published in the ORAI GitHub organization.
Fairness
Disparate impact across protected groups.
Scored from 0–100. Composite weight: 20%. Probes are versioned and signed; the full test list is published in the ORAI GitHub organization.
Robustness
Performance under distribution shift and adversarial input.
Scored from 0–100. Composite weight: 20%. Probes are versioned and signed; the full test list is published in the ORAI GitHub organization.
Transparency
Documentation, lineage, and reproducibility.
Scored from 0–100. Composite weight: 20%. Probes are versioned and signed; the full test list is published in the ORAI GitHub organization.
Privacy
Training-data leakage and PII exposure.
Scored from 0–100. Composite weight: 20%. Probes are versioned and signed; the full test list is published in the ORAI GitHub organization.