Auditing an open model

Seatbelt is the open tool behind every ORAI scorecard. This tutorial covers the hosted Colab demo and a local Hugging Face audit, the same path we use for public leaderboard runs.

First install Seatbelt. You will know it worked if the Colab demo (or the snippet below) prints a dimension scorecard.

Colab demo

Quickest option: audit a Groq-hosted Llama and inspect probe-level results without a local GPU.

• Open the Colab notebook
• Add a free Groq API key as GROQ_API_KEY in Colab Secrets
• Run all cells with default probe_budget=15

# In Colab Secrets (or your shell):
export GROQ_API_KEY="<your-groq-api-key>"

# Then run the notebook cells, or locally:
pip install seatbelt groq
# paste the Getting Started quickstart with Groq

Local Hugging Face audit

Any callable that takes a string and returns a string works. Example with Transformers:

from transformers import pipeline
from seatbelt import audit, AuditConfig

pipe = pipeline(
    "text-generation",
    model="mistralai/Mistral-7B-Instruct-v0.2",
)
model_fn = lambda prompt: pipe(prompt, max_new_tokens=256)[0]["generated_text"]

report = audit(
    model_fn,
    config=AuditConfig(
        context="general purpose assistant",
        probe_budget=15,
        verbose=True,
    ),
)
report.save("audit.md")
print(report.summary())

Submit to the ORAI leaderboard

Public scorecards are produced only by the trusted ORAI runner. You submit a Hugging Face model pointer; the runner calls that model, runs Seatbelt, and writes verified scores. Local JSON uploads are not accepted.

• Sign in to ORAI
• Open Submit a model and paste your Hugging Face repo
• Choose public visibility so the scorecard can appear on the leaderboard

View the public leaderboard or track your own runs under My submissions.

More

See Model sizes for under-8B website runs vs Colab/local for larger models, and Getting Started for AuditConfig options. The Seatbelt README covers more model adapters (OpenAI, Anthropic, Ollama).