Release decision: {{ r.verdict }}
{{ r.reason }} {{ blurb }}
Suite {{ r.suite_name }} (hash {{ r.suite_hash }}) —
baseline {{ r.baseline_run_id[:12] }} vs candidate {{ r.candidate_run_id[:12] }}.
{% macro casetable(cases) -%}
{% if cases %}
| Case | Cat/Lang | Before | After | Failing assertion |
|---|
{% for c in cases %}
{{ c.id }}{% if c.critical %} critical{% endif %} |
{{ c.category }}/{{ c.language }} |
{{ c.before or '—' }} |
{{ c.after or '—' }} |
{{ '; '.join(c.failing) if c.failing else '—' }} |
{% endfor %}
{% else %}None.
{% endif %}
{%- endmacro %}
Critical regressions
{{ casetable(r.critical_regressions) }}
Improvements
{{ casetable(r.improvements) }}
Other regressions
{{ casetable(r.other_regressions) }}
Added cases
{{ casetable(r.added) }}
Removed cases
{{ casetable(r.removed) }}
Disputes, errors & coverage
| Baseline | Candidate |
|---|
| Judge disputes | {{ r.disputes.baseline }} | {{ r.disputes.candidate }} |
| Errors | {{ r.errors.baseline }} | {{ r.errors.candidate }} |
| Category | Evaluable / total |
|---|
{% for cov in r.coverage %}
| {{ cov.category }} | {{ cov.evaluable }} / {{ cov.total }} |
{% endfor %}
Latency (ms)
| Baseline | Candidate |
|---|
| p50 | {{ '%.0f'|format(r.latency.baseline_p50) }} | {{ '%.0f'|format(r.latency.candidate_p50) }} |
| p95 | {{ '%.0f'|format(r.latency.baseline_p95) }} | {{ '%.0f'|format(r.latency.candidate_p95) }} |
Cost per successful task
{{ r.cost_per_successful_task }}
Acceptance policy applied
{% for p in r.policy_plain %}- {{ p }}
{% endfor %}
Reproduce
- Suite hash:
{{ r.suite_hash }}
- Baseline run:
{{ r.baseline_run_id }} — target {{ target_json(r.baseline_target) }}
- Candidate run:
{{ r.candidate_run_id }} — target {{ target_json(r.candidate_target) }}
{% if r.rerun_command %}{{ r.rerun_command }}{% endif %}