Did it get worse? No
Nothing got worse compared with the reference. 2 checks are on the line: the bands they measured cover their thresholds. Every case could be judged. No case is suspended. The suite is unchanged from the reference. The files under test are the same as the reference. The system under test answered under the same configuration as the reference.
The files under test are the same as the reference.
Configured as in the reference.
| Parameter | This run | Reference |
|---|---|---|
max_tokens | 200 | 200 |
model | claude-haiku-4-5 | claude-haiku-4-5 |
provider | anthropic | anthropic |
resolved_model | claude-haiku-4-5-20251001 | claude-haiku-4-5-20251001 |
Configured as in the reference.
| Parameter | This run | Reference |
|---|---|---|
judge | anthropic/claude-haiku-4-5 | anthropic/claude-haiku-4-5 |
max_tokens | 400 | 400 |
model | claude-haiku-4-5 | claude-haiku-4-5 |
provider | anthropic | anthropic |
resolved_model | claude-haiku-4-5-20251001 | claude-haiku-4-5-20251001 |
Nothing in this section.
Nothing in this section.
Nothing in this section.
Nothing in this section.
Nothing in this section.
| Case | Check | What happened | Why |
|---|---|---|---|
gift-wrap | levenshtein | Score unchanged at 0.191404. | mean of 5 samples (0.174107, 0.190722, 0.210227, 0.198795, 0.183168) |
gift-wrap | llm_rubric | Score unchanged at 0.540000. | mean of 5 samples (0.616667, 0.666667, 0.433333, 0.450000, 0.533333) |
opening-hours | levenshtein | Score unchanged at 0.196253. | mean of 5 samples (0.212389, 0.205479, 0.200893, 0.182927, 0.179577) |
opening-hours | llm_rubric | Score unchanged at 0.343333. | mean of 5 samples (0.350000, 0.333333, 0.300000, 0.433333, 0.300000) |
order-a-book | levenshtein | Score unchanged at 0.271600. | mean of 5 samples (0.228137, 0.270270, 0.299517, 0.277778, 0.282297) |
order-a-book | llm_rubric | Score unchanged at 0.540000. | mean of 5 samples (0.683333, 0.816667, 0.533333, 0.366667, 0.300000) |
price-check | levenshtein | Score unchanged at 0.194523. | mean of 5 samples (0.189516, 0.216590, 0.179211, 0.182573, 0.204724) |
price-check | llm_rubric | Score unchanged at 0.310000. | mean of 5 samples (0.300000, 0.333333, 0.250000, 0.333333, 0.333333) |
signed-copies | levenshtein | Score unchanged at 0.197552. | mean of 5 samples (0.206349, 0.206226, 0.190299, 0.193916, 0.190972) |
signed-copies | llm_rubric | Score unchanged at 0.443333. | mean of 5 samples (0.300000, 0.716667, 0.466667, 0.300000, 0.433333) |
| Case | Check | Result | Declared band | Why |
|---|---|---|---|---|
calibration-half-right | llm_rubric | 0.743333 inside | 0.100000–0.900000 | mean of 5 samples (0.733333, 0.733333, 0.766667, 0.716667, 0.766667) |