Benchmark · November 12, 2026
We ran Vectoral's content-safety classifier, SafetyD, next to Llama Guard 3, ShieldGemma and Granite Guardian on five public prompt-safety datasets. For the story behind it, read SafetyD: The Why, the How and the What.
01 · Latency
Same 4,105 prompts for every system, drawn from the five datasets, timed per call. Most are short. On the ones over 2 KB, SafetyD's median rises to 2.5 ms; the guard models' rise to between 53 ms and 524 ms.
Linear scale, so SafetyD is the sliver at the left; hover a bar for p95 and p99.
Lower is better
02 · False positives
A low false positive rate is important for auto-blocking. XSTest's safe prompts are ones a person could reasonably send that only sound dangerous.
Lower is better
03 · Accuracy
F1One score for catching unsafe prompts without flagging safe ones. It combines precision (how many flagged prompts were really unsafe) and recall (how many unsafe prompts got flagged). at each system's own operating point, the way public guardrail leaderboards rank models. SafetyD's row is its research tier with the opt-in stage-2 veto, the setting that can be compared like-for-like with the guards; the shipped default is far more conservative and does not compete on F1.
Higher is better
| System | Macro F1 | F1 XSTest | F1 ToxicChat | F1 Aegis 2.0 | Recall OR-Bench toxic | AUROC | Recall @ 1% FPR |
|---|---|---|---|---|---|---|---|
| ShieldGemma 9BGoogle · 9.2B · Q4_K_M | 75.7% | 81.9% | 71.0% | 74.3% | 57.6% | 0.921 | 39.6% |
| Granite Guardian 3 8BIBM · 8.2B · Q5_K_M | 73.4% | 83.5% | 55.7% | 80.9% | 98.6% | 0.932 | 49.1% |
| Vectoral SafetyDVectoral · CPU · stage 2 on GPU | 73.1% | 80.4% | 60.5% | 78.4% | 82.6% | 0.894 | 19.9% |
| Llama Guard 3 8BMeta · 8.0B · Q4_K_M | 70.4% | 86.5% | 48.0% | 76.8% | 82.8% | 0.888 | 49.8% |
| Llama Guard 3 1BMeta · 1.5B · Q8_0 | 62.1% | 82.3% | 32.3% | 71.7% | 83.2% | 0.833 | 43.4% |
| Granite Guardian 3 2BIBM · 2.5B · Q8_0 | 60.0% | 79.3% | 25.5% | 75.3% | 99.4% | 0.887 | 19.1% |
| ShieldGemma 2BGoogle · 2.6B · Q4_K_M | 51.1% | 70.4% | 26.7% | 56.3% | 27.2% | 0.894 | 26.7% |
Macro F1 averages XSTest, ToxicChat and Aegis 2.0. AUROC and recall at 1% FPR pool those three sets and sweep each system's own score.
Datasets
Fairness notes