ORAI
GitHubSubmit your model
MethodologyGetting Started

Methodology

Getting Started

Seatbelt is an open-source tool for responsible AI audits of LLMs and SLMs.

Seatbelt runs your model through six auditors and returns a pass / warn / fail scorecard you can use in CI. The library itself needs no API key. You only need one for the model endpoint you audit.

Installation

pip install seatbelt

Package details: seatbelt on PyPI. Source: o-rai/seatbelt.

Quickstart

Wrap your model as model_fn(prompt: str) -> str, then call audit():

import os
from groq import Groq
from seatbelt import audit, AuditConfig

client = Groq(api_key=os.environ["GROQ_API_KEY"])

def model_fn(prompt: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=512,
        temperature=0.0,
        seed=42,
    )
    return response.choices[0].message.content

config = AuditConfig(
    context="general purpose open-source assistant",
    probe_budget=15,
    verbose=True,
)

report = audit(model_fn=model_fn, config=config)
print(report.summary())
report.save("audit.json")
report.save("audit.md")

if report.has_failures():
    raise SystemExit("Audit failed. Do not deploy.")

Configuration

Common options on AuditConfig:

from seatbelt import AuditConfig

config = AuditConfig(
    context="HR candidate screening tool",
    pass_threshold=0.90,
    warn_threshold=0.63,
    regulations=["eu_ai_act", "nyc_ll144", "nist_rmf"],
    probe_budget=15,  # lower = faster; default 50
    verbose=True,
)

Running through ORAI

Want the hosted path? Submit a model on ORAI. We check the Hugging Face repo, open a live report page, and show the Seatbelt snippet to run locally.

  • Free tier: up to 3 model audits every 30 calendar days (models below 8B params)
  • Larger models: run Seatbelt in Colab or locally. See Model sizes
  • Quick demo: open the Colab notebook

Next

Tutorials: run the Colab demo and audit a Hugging Face model.