Benchmark · November 12, 2026

SafetyD against
open guard models

We ran Vectoral's content-safety classifier, SafetyD, next to Llama Guard 3, ShieldGemma and Granite Guardian on five public prompt-safety datasets. For the story behind it, read SafetyD: The Why, the How and the What.

267 µsSafetyD median latency on its stage-1 path, 70× faster than the fastest guard model on a GPU
73.1%macro F1 for SafetyD's research setting, vs 75.7% for the best guard model, ShieldGemma 9B

01 · Latency

Median time to a verdict

Same 4,105 prompts for every system, drawn from the five datasets, timed per call. Most are short. On the ones over 2 KB, SafetyD's median rises to 2.5 ms; the guard models' rise to between 53 ms and 524 ms.

Linear scale, so SafetyD is the sliver at the left; hover a bar for p95 and p99.

Lower is better

Vectoral SafetyD267 µsp95 768 µs · p99 2.4 ms · stage 2: p50 129 ms, p99 327 ms
Llama Guard 3 1B19 msp95 24 ms · p99 35 ms
Granite Guardian 3 2B44 msp95 56 ms · p99 112 ms
ShieldGemma 2B69 msp95 78 ms · p99 128 ms
Llama Guard 3 8B73 msp95 111 ms · p99 201 ms
Granite Guardian 3 8B94 msp95 136 ms · p99 313 ms
ShieldGemma 9B229 msp95 275 ms · p99 425 ms
Median (p50) latency per classification call, linear scale. SafetyD's stage-1 path on CPU; guard models on GPU via Lemonade. Prompts SafetyD escalates to stage 2 are not in its bar; see above.

02 · False positives

How often a benign prompt gets flagged

A low false positive rate is important for auto-blocking. XSTest's safe prompts are ones a person could reasonably send that only sound dangerous.

XSTest · 250 safe prompts

Lower is better

Llama Guard 3 8B8.0B · Q4_K_M5.6%
Llama Guard 3 1B1.5B · Q8_06.4%
Vectoral SafetyDCPU · stage 2 on GPU9.2%
ShieldGemma 2B2.6B · Q4_K_M12.8%
ShieldGemma 9B9.2B · Q4_K_M18.0%
Granite Guardian 3 8B8.2B · Q5_K_M30.0%
Granite Guardian 3 2B2.5B · Q8_041.2%
False-positive rate on XSTest's safe prompts.

03 · Accuracy

The leaderboard view

F1One score for catching unsafe prompts without flagging safe ones. It combines precision (how many flagged prompts were really unsafe) and recall (how many unsafe prompts got flagged). at each system's own operating point, the way public guardrail leaderboards rank models. SafetyD's row is its research tier with the opt-in stage-2 veto, the setting that can be compared like-for-like with the guards; the shipped default is far more conservative and does not compete on F1.

Higher is better

SystemMacro F1F1 XSTestF1 ToxicChatF1 Aegis 2.0Recall OR-Bench toxicAUROCRecall @ 1% FPR
ShieldGemma 9BGoogle · 9.2B · Q4_K_M75.7%81.9%71.0%74.3%57.6%0.92139.6%
Granite Guardian 3 8BIBM · 8.2B · Q5_K_M73.4%83.5%55.7%80.9%98.6%0.93249.1%
Vectoral SafetyDVectoral · CPU · stage 2 on GPU73.1%80.4%60.5%78.4%82.6%0.89419.9%
Llama Guard 3 8BMeta · 8.0B · Q4_K_M70.4%86.5%48.0%76.8%82.8%0.88849.8%
Llama Guard 3 1BMeta · 1.5B · Q8_062.1%82.3%32.3%71.7%83.2%0.83343.4%
Granite Guardian 3 2BIBM · 2.5B · Q8_060.0%79.3%25.5%75.3%99.4%0.88719.1%
ShieldGemma 2BGoogle · 2.6B · Q4_K_M51.1%70.4%26.7%56.3%27.2%0.89426.7%

Macro F1 averages XSTest, ToxicChat and Aegis 2.0. AUROC and recall at 1% FPR pool those three sets and sweep each system's own score.

Datasets

XSTest450 prompts · 200 unsafe
Safe prompts that sound unsafe (“how do I kill a Python process”), mixed with unsafe ones.
ToxicChat1,000 prompts · 72 unsafe
Real user prompts to a public chatbot, about 7% toxic. The closest thing to production traffic.
Aegis 2.01,000 prompts · 533 unsafe
Human-labelled prompts across 23 harm categories.
OR-Bench 80k1,000 prompts · 0 unsafe
Benign prompts written to brush against ten harm categories, drawn from OR-Bench's general 80k set rather than its hard-1k over-refusal subset. Used here for latency timing.
OR-Bench toxic655 prompts · 655 unsafe
Unsafe prompts only. Measures recall.

Fairness notes

Latency is wall-clock on loopback after warm-up. SafetyD's first stage ran on the CPU; every guard model, including the ones its stage 2 calls, had the machine's GPU (Metal, through Lemonade). Without local GPUs, the guard models will be slower in production; SafetyD's stage 1 needs none.
ShieldGemma is asked about all four of its policies in one prompt, as Ollama ships it. Its model card suggests one policy per call, which costs more and may score better.
SafetyD's AUROC and matched-FPR recall are computed from its distilled head's probability (the additive model_score field), not from the block verdict, so they describe the first-stage model rather than the default posture; stage 2 does not change them.
Each mixed set is a hash-ordered sample of up to 1,000 prompts; XSTest and OR-Bench toxic are used whole. Guard models run on November 12, 2026; SafetyD on November 13, 2026 at commit 27789d2b, Lemonade 2026.45.3 (llama.cpp b11846), Apple M5 Max.
Every benchmark prompt here is excluded from SafetyD's training data by exact hash and near-duplicate match. The row shown is SafetyD's research tier with the stage-2 veto. The research tier alone would block 3.8% of ordinary traffic; with stage 2 that falls to 2.4%, and 2.8% of ordinary traffic escalates to the guard models. Raising stage 1's threshold to 0.40 instead gets 2.2% at 63.0% macro F1. The shipped default keeps the 0.1% false-positive design instead: it blocks nothing on these sets (0.0% F1, 0.0% false positives) and its tag-only row scores 21.3%.
Stage 2 is opt-in and off by default. It needs the two guard models hosted next to SafetyD, and an escalated prompt's text is sent to them.