Motivation

ORAI is an independent audit lab for responsible AI. Seatbelt is the open-source tool that runs every scorecard: same probes, same thresholds, no hand-tuning.

When teams ship LLMs and SLMs into hiring, support, finance, or healthcare, self-reported safety claims are not enough. Defining and managing probes for many models is hard. We built Seatbelt so audits are easy to repeat, and ORAI so signed scorecards can be published on top of it.

What Seatbelt provides

• Simple interfaces: wrap any model as model_fn(prompt) -> str
• Six auditors across deception, fairness, sociotech, regulatory, transparency, and privacy
• Agent disagreements stay visible in the report
• Regulation citations for EU AI Act, NYC Local Law 144, and NIST AI RMF
• Works with common model APIs (Groq, OpenAI, Anthropic, Hugging Face, Ollama). See PyPI

How ORAI scores a model

Every public audit runs the same Seatbelt probes. Scores come from weights (or an inference endpoint) and the published seeds. Dimensions currently scored:

• Deception: sycophancy, reward hacking, and monitoring-aware behavior
• Fairness: counterfactual and representation bias across demographic groups
• Sociotechnical risk: automation bias, feedback loops, vulnerable-population sensitivity
• Regulatory: mapped obligations with article / section citations
• Transparency: identity disclosure, attribution, capability boundaries
• Privacy: PII leakage, memorization, safe refusal

Grades map as follows: A ≥ 85, B 70-84, C 55-69, D < 55.

Next

Model sizes: free website path for models under 8B, Colab/local for larger models. Getting Started: install Seatbelt and run your first audit.