Model sizes

Run Seatbelt yourself outside the free website runner when your model is too large for that path. The free web path is for models under 8B params (up to 3 audits every 30 calendar days).

Under 8B: Hugging Face on the website

For smaller open models, submit a public Hugging Face repo on ORAI. We queue the trusted Seatbelt runner against that pointer, write verified scores to your live report, and list public runs on the leaderboard after verification.

• Submit a model with a Hugging Face pointer and public visibility
• Free tier: models under 8B params, up to 3 audits every 30 calendar days
• Verified public scorecards can appear on the leaderboard

8B and larger: Colab, local GPU, or API

Larger weights usually will not finish on the free website runner. Install Seatbelt and wrap your model as model_fn(prompt) -> str, then call audit().

API-hosted models (easiest). Point model_fn at Groq, OpenAI, Anthropic, Hugging Face Inference, or a similar endpoint. The Colab demo does this against Llama 70B via Groq with probe_budget=15.

pip install seatbelt
# Then wrap your API client as model_fn(prompt) -> str
# See Getting Started for a Groq example

Local / your own GPU. Same Seatbelt call, but point model_fn at a Transformers, vLLM, or Ollama pipeline that can load the larger weights.

from transformers import pipeline
from seatbelt import audit, AuditConfig

pipe = pipeline("text-generation", model="your-org/your-large-model")
model_fn = lambda prompt: pipe(prompt, max_new_tokens=256)[0]["generated_text"]

report = audit(
    model_fn,
    config=AuditConfig(context="general purpose assistant", probe_budget=15),
)
report.save("audit.md")
print(report.summary())

Package: seatbelt on PyPI. Source and adapters: o-rai/seatbelt.

ORAI Enterprise for large models

For a verified ORAI report or scorecard on a model greater than 7B, use Enterprise (from $1,500~ per report). ORAI runs Seatbelt on the trusted runner so the result can be signed and, when public, eligible for the leaderboard.

If you want to add your own model (greater than 7B) to the leaderboard but do not need an Enterprise plan, reach out to us. We can talk through options.

See Pricing for Free and Enterprise details.

Can large models appear on the leaderboard?

Not from a self-run Colab or local audit. Those runs use the same probes and give you a local report (audit.md / JSON), but they do not auto-post to the public leaderboard. Uploaded JSON is never accepted, to avoid contamination.

Leaderboard entries come only from the trusted ORAI runner: the free website path for Hugging Face models under 8B, Enterprise for larger models, or by arrangement if you contact us about listing a model greater than 7B without an Enterprise plan.

Next

Getting Started: install Seatbelt. Tutorials: Colab demo and a local Hugging Face example.