Evaluation report — expense-triage

Customer

northwind

Suite

expense-triage

This run

staging — 2026-10-30T15:45:28.301076+00:00

Reference

staging — 2026-10-27T17:13:28.796999+00:00

Code version

391c780dcb88fafd15f72f8c7dc6c4a9e528b84c

Reference approved

2026-10-27T17:13:37.266387+00:00

Reference key

2026-10-27T17-13-28-796999-00-00-5af8d8dd1e55f5c5

Did it get worse? No

Nothing got worse compared with the reference. 1 check moved within noise. 2 checks are on the line: the bands they measured cover their thresholds. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

regressed 0
errored 0
new 0
missing 0
improved 1
unchanged 31

Overall

Measure	Result	What happened	Why
`precision`	0.800000 / 0.600000	All 20 cases counted.	precision 0.800000 = 8/10 (20 counted, 0 suspended, 0 could not be judged)
`precision[group=hotel]`	1.000000 / 0.600000	All 3 cases counted.	precision 1.000000 = 2/2 (3 counted, 0 suspended, 0 could not be judged)
`precision[group=meal]`	1.000000 / 0.600000	All 7 cases counted.	precision 1.000000 = 4/4 (7 counted, 0 suspended, 0 could not be judged)
`precision[group=taxi]`	1.000000 / 0.600000	All 4 cases counted.	precision 1.000000 = 1/1 (4 counted, 0 suspended, 0 could not be judged)
`precision[group=tools]`	0.500000 / 0.600000	All 3 cases counted.	precision 0.500000 = 1/2 (3 counted, 0 suspended, 0 could not be judged)
`precision[group=travel]`	0.000000 / 0.600000	All 3 cases counted.	precision 0.000000 = 0/1 (3 counted, 0 suspended, 0 could not be judged)
`accuracy`	0.900000 / 0.700000	All 20 cases counted.	accuracy 0.900000 = 18/20 (20 counted, 0 suspended, 0 could not be judged)
`accuracy[group=hotel]`	1.000000 / 0.700000	All 3 cases counted.	accuracy 1.000000 = 3/3 (3 counted, 0 suspended, 0 could not be judged)
`accuracy[group=meal]`	1.000000 / 0.700000	All 7 cases counted.	accuracy 1.000000 = 7/7 (7 counted, 0 suspended, 0 could not be judged)
`accuracy[group=taxi]`	1.000000 / 0.700000	All 4 cases counted.	accuracy 1.000000 = 4/4 (4 counted, 0 suspended, 0 could not be judged)
`accuracy[group=tools]`	0.666667 / 0.700000	All 3 cases counted.	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)
`accuracy[group=travel]`	0.666667 / 0.700000	All 3 cases counted.	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)

Some measures are below their threshold and are still not reported as having got worse: they were below it in the reference too. The threshold says the system does not meet the bar; the comparison says it has not moved. Both are true, and only movement decides the answer above.

What got worse (0)

Nothing in this section.

What could not be judged (0)

Nothing in this section.

What is set aside (0)

Nothing in this section.

What was added or removed (0)

Nothing in this section.

What got better (1)

Case	Check	What happened	Why
`dinner_client`	`agrees_with_mark`	Went from failing to passing (0.400000 → 0.600000).	mean of 5 samples (1.000000, 0.000000, 1.000000, 0.000000, 1.000000)

What stayed the same (31)

Case	Check	What happened	Why
`amount_over_cap`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`book_technical`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`coffee_two`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`courier_urgent`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`dinner_solo`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`duplicate_claim`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`hotel_capped`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`hotel_over`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`late_taxi`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`lunch_team`	`agrees_with_mark`	Score unchanged at 0.800000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 0.000000)
`meal_no_note`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`monitor_home`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`no_receipt`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`parking_airport`	`agrees_with_mark`	Score unchanged at 0.000000.	mean of 5 samples (0.000000, 0.000000, 0.000000, 0.000000, 0.000000)
`software_seat`	`agrees_with_mark`	Score unchanged at 0.000000.	mean of 5 samples (0.000000, 0.000000, 0.000000, 0.000000, 0.000000)
`taxi_long`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`taxi_short`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`train_standard`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
`weekend_bar`	`agrees_with_mark`	Score unchanged at 1.000000.	mean of 5 samples (1.000000, 1.000000, 1.000000, 1.000000, 1.000000)
	`accuracy[group=tools]`	Score unchanged at 0.666667.	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)
	`precision[group=travel]`	Score unchanged at 0.000000.	precision 0.000000 = 0/1 (3 counted, 0 suspended, 0 could not be judged)
	`precision[group=tools]`	Score unchanged at 0.500000.	precision 0.500000 = 1/2 (3 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=hotel]`	Score unchanged at 1.000000.	accuracy 1.000000 = 3/3 (3 counted, 0 suspended, 0 could not be judged)
	`precision`	Score unchanged at 0.800000.	precision 0.800000 = 8/10 (20 counted, 0 suspended, 0 could not be judged)
	`precision[group=meal]`	Score moved from 0.800000 to 1.000000 — inside the range of the reference's individual samples (0.666667–1.000000 across 5 samples); not counted as a regression.	precision 1.000000 = 4/4 (7 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=meal]`	Score unchanged at 1.000000.	accuracy 1.000000 = 7/7 (7 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=taxi]`	Score unchanged at 1.000000.	accuracy 1.000000 = 4/4 (4 counted, 0 suspended, 0 could not be judged)
	`precision[group=hotel]`	Score unchanged at 1.000000.	precision 1.000000 = 2/2 (3 counted, 0 suspended, 0 could not be judged)
	`precision[group=taxi]`	Score unchanged at 1.000000.	precision 1.000000 = 1/1 (4 counted, 0 suspended, 0 could not be judged)
	`accuracy[group=travel]`	Score unchanged at 0.666667.	accuracy 0.666667 = 2/3 (3 counted, 0 suspended, 0 could not be judged)
	`accuracy`	Score unchanged at 0.900000.	accuracy 0.900000 = 18/20 (20 counted, 0 suspended, 0 could not be judged)