Customer support emails get a helpful, accurate, policy-compliant reply
| fixture | addresses-the-ask | golden-reply | has-greeting | appropriate-tone | no-policy-invention | no-unauthorized-commitments |
|---|---|---|---|---|---|---|
| all | 78% [45–94%] n=9 | 100% [44–100%] n=3 | 100% [70–100%] n=9 | 100% [70–100%] n=9 | 89% [56–98%] n=9 | 56% [27–81%] n=9 |
| billing-dispute | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 0/3 |
| product-question | 2/3 | — | 3/3 | 3/3 | 2/3 | 2/3 |
| upgrade-request | 2/3 | — | 3/3 | 3/3 | 3/3 | 3/3 |
| eval | labelled runs | agreement | errors |
|---|---|---|---|
| addresses-the-ask | 3 | 100.0% | 0 false pass, 0 false fail |
| no-unauthorized-commitments | 3 | 100.0% | 0 false pass, 0 false fail |
A false pass is an output you failed and the judge passed. On a safe eval that is the error that matters.