GuardBench Evaluation Report
Run: {{ run_id }} · Samples: {{ total_samples }} · Dataset SHA: {{ dataset_sha[:12] }}
· Git: {{ git_commit[:8] }} · Generated: {{ generated_at }}
Baseline: {{ baseline_name }} · Candidate: {{ candidate_name }}
{% if mcnemar_p is not none %} · McNemar p: {{ mcnemar_p }}{% endif %}
Baseline — Strict
Recall: {{ strict_base.recall }} ({{ strict_base.recall_lo }}–{{ strict_base.recall_hi }})
Precision: {{ strict_base.precision }}
F1: {{ strict_base.f1 }}
FPR: {{ strict_base.fpr }} ({{ strict_base.fpr_lo }}–{{ strict_base.fpr_hi }})
FNR: {{ strict_base.fnr }}
TP:{{ strict_base.tp }} FP:{{ strict_base.fp }} TN:{{ strict_base.tn }} FN:{{ strict_base.fn }}
Latency p50/p90/p99: {{ strict_base.latency_p50 }}/{{ strict_base.latency_p90 }}/{{ strict_base.latency_p99 }} ms
Candidate — Strict
Recall: {{ strict_cand.recall }} ({{ strict_cand.recall_lo }}–{{ strict_cand.recall_hi }})
Precision: {{ strict_cand.precision }}
F1: {{ strict_cand.f1 }}
FPR: {{ strict_cand.fpr }} ({{ strict_cand.fpr_lo }}–{{ strict_cand.fpr_hi }})
FNR: {{ strict_cand.fnr }}
TP:{{ strict_cand.tp }} FP:{{ strict_cand.fp }} TN:{{ strict_cand.tn }} FN:{{ strict_cand.fn }}
Latency p50/p90/p99: {{ strict_cand.latency_p50 }}/{{ strict_cand.latency_p90 }}/{{ strict_cand.latency_p99 }} ms
Slice Breakdown — Strict (Category × Language)
Baseline
| Category | Lang | N | Recall | FPR | FNR |
|---|
{% for s in strict_base_slices %}
| {{ s.category }} | {{ s.language }} | {{ s.n }} |
{{ s.recall }} |
{{ s.fpr }} |
{{ s.fnr }} |
{% endfor %}
Candidate
| Category | Lang | N | Recall | FPR | FNR |
|---|
{% for s in strict_cand_slices %}
| {{ s.category }} | {{ s.language }} | {{ s.n }} |
{{ s.recall }} |
{{ s.fpr }} |
{{ s.fnr }} |
{% endfor %}
Baseline — Lenient
Recall: {{ lenient_base.recall }}
Precision: {{ lenient_base.precision }}
F1: {{ lenient_base.f1 }}
FPR: {{ lenient_base.fpr }}
FNR: {{ lenient_base.fnr }}
TP:{{ lenient_base.tp }} FP:{{ lenient_base.fp }} TN:{{ lenient_base.tn }} FN:{{ lenient_base.fn }}
Candidate — Lenient
Recall: {{ lenient_cand.recall }}
Precision: {{ lenient_cand.precision }}
F1: {{ lenient_cand.f1 }}
FPR: {{ lenient_cand.fpr }}
FNR: {{ lenient_cand.fnr }}
TP:{{ lenient_cand.tp }} FP:{{ lenient_cand.fp }} TN:{{ lenient_cand.tn }} FN:{{ lenient_cand.fn }}
Latency Distribution (Baseline)
Latency Distribution (Candidate)
{% if sweep_data %}
Threshold Sweep (Candidate)
{% endif %}
{% if sample_results %}
Sample Results
| Text | Label | Category | Lang | Baseline | Candidate | {% if has_judge %}Judge | {% endif %}
|---|
{% for s in sample_results %}
| {{ s.text|e }} |
{{ s.label }} |
{{ s.category }} |
{{ s.language }} |
{{ s.baseline_pred }} |
{{ s.candidate_pred }} |
{% if has_judge %}{{ s.judge_verdict or '—' }} | {% endif %}
{% endfor %}
{% endif %}
Compliance Reference
| Framework | Requirement | How GuardBench Addresses It |
|---|
| EU AI Act Art. 9 |
Risk management system for high-risk AI |
Automated evaluation pipeline quantifies guard performance risk per slice, with CI gate enforcement |
| EU AI Act Art. 15 |
Accuracy, robustness and cybersecurity |
Per-slice recall/FPR metrics with Wilson confidence intervals; adversarial augmentation testing |
| NIST AI RMF Measure 2.5 |
AI risk monitoring and measurement |
Run history stored with dataset SHA + git commit for full reproducibility; regression detection via comparison thresholds |
| ISO 42001 §8.4 |
AI system performance assessment |
Structured EvalResults with McNemar significance test; HTML report exportable for audit evidence |