
== general/confirm: 2500 of 2500 items answered by every external model (1932 units); errors {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.481 [0.461,0.501]  ECE 0.229 [0.210,0.249]  Brier 0.702 [0.674,0.728]  over-context 0  latency median 0.564s
  tev1:4b                    acc 0.479 [0.456,0.501]  ECE 0.142 [0.121,0.165]  Brier 0.681 [0.658,0.704]  over-context 0  latency median 0.298s
  tev1:0.8b                  acc 0.378 [0.356,0.398]  ECE 0.154 [0.136,0.177]  Brier 0.701 [0.685,0.716]  over-context 0  latency median 0.09s
  judgly-gemma-raw           acc 0.509 [0.489,0.531]  ECE 0.373 [0.351,0.393]  Brier 0.834 [0.799,0.868]
  judgly-gemma-h2            acc 0.509 [0.488,0.530]  ECE 0.051 [0.041,0.072]  Brier 0.562 [0.547,0.578]
  judgly-gemma-temperature   acc 0.509 [0.489,0.531]  ECE 0.087 [0.069,0.106]  Brier 0.579 [0.563,0.595]
  judgly-gemma-default       acc 0.509 [0.488,0.530]  ECE 0.051 [0.041,0.072]  Brier 0.562 [0.547,0.578]
  judgly-qwen-raw            acc 0.484 [0.466,0.506]  ECE 0.406 [0.382,0.428]  Brier 0.888 [0.846,0.928]
  judgly-qwen-h2             acc 0.463 [0.442,0.483]  ECE 0.076 [0.062,0.101]  Brier 0.624 [0.609,0.639]
  judgly-qwen-temperature    acc 0.484 [0.466,0.506]  ECE 0.047 [0.032,0.068]  Brier 0.597 [0.584,0.611]
  judgly-qwen-default        acc 0.484 [0.466,0.506]  ECE 0.047 [0.032,0.068]  Brier 0.597 [0.584,0.611]

== stance/confirm: 1780 of 1780 items answered by every external model (70 units); errors {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.595 [0.553,0.640]  ECE 0.264 [0.211,0.296]  Brier 0.657 [0.588,0.721]  over-context 0  latency median 0.924s
  tev1:4b                    acc 0.630 [0.603,0.680]  ECE 0.185 [0.152,0.215]  Brier 0.542 [0.477,0.583]  over-context 0  latency median 0.561s
  tev1:0.8b                  acc 0.451 [0.380,0.481]  ECE 0.304 [0.279,0.364]  Brier 0.794 [0.752,0.870]  over-context 0  latency median 0.122s
  judgly-gemma-raw           acc 0.655 [0.617,0.698]  ECE 0.300 [0.255,0.333]  Brier 0.630 [0.541,0.688]
  judgly-gemma-h2            acc 0.629 [0.590,0.670]  ECE 0.078 [0.046,0.104]  Brier 0.490 [0.446,0.530]
  judgly-gemma-temperature   acc 0.655 [0.617,0.698]  ECE 0.052 [0.028,0.079]  Brier 0.476 [0.416,0.511]
  judgly-gemma-default       acc 0.655 [0.617,0.698]  ECE 0.052 [0.028,0.079]  Brier 0.476 [0.416,0.511]
  judgly-qwen-raw            acc 0.568 [0.526,0.607]  ECE 0.383 [0.346,0.423]  Brier 0.809 [0.735,0.883]
  judgly-qwen-h2             acc 0.526 [0.460,0.556]  ECE 0.183 [0.152,0.226]  Brier 0.612 [0.579,0.683]
  judgly-qwen-temperature    acc 0.568 [0.526,0.607]  ECE 0.106 [0.078,0.151]  Brier 0.581 [0.544,0.630]
  judgly-qwen-default        acc 0.568 [0.526,0.607]  ECE 0.106 [0.078,0.151]  Brier 0.581 [0.544,0.630]

== general/final: 7879 of 7879 items answered by every external model (6803 units); errors {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.674 [0.663,0.684]  ECE 0.129 [0.120,0.139]  Brier 0.450 [0.437,0.463]  over-context 0  latency median 0.485s
  tev1:4b                    acc 0.657 [0.647,0.668]  ECE 0.052 [0.044,0.061]  Brier 0.431 [0.420,0.441]  over-context 0  latency median 0.224s
  tev1:0.8b                  acc 0.504 [0.493,0.516]  ECE 0.086 [0.075,0.096]  Brier 0.588 [0.578,0.597]  over-context 0  latency median 0.084s
  judgly-gemma-raw           acc 0.760 [0.751,0.770]  ECE 0.194 [0.185,0.204]  Brier 0.431 [0.415,0.448]
  judgly-gemma-h2            acc 0.737 [0.727,0.746]  ECE 0.020 [0.015,0.030]  Brier 0.348 [0.338,0.358]
  judgly-gemma-temperature   acc 0.760 [0.751,0.770]  ECE 0.017 [0.015,0.027]  Brier 0.325 [0.315,0.334]
  judgly-gemma-default       acc 0.737 [0.727,0.746]  ECE 0.020 [0.015,0.030]  Brier 0.348 [0.338,0.358]
  judgly-qwen-raw            acc 0.656 [0.645,0.666]  ECE 0.268 [0.259,0.279]  Brier 0.602 [0.584,0.621]
  judgly-qwen-h2             acc 0.639 [0.628,0.648]  ECE 0.029 [0.025,0.041]  Brier 0.450 [0.440,0.460]
  judgly-qwen-temperature    acc 0.656 [0.645,0.666]  ECE 0.034 [0.025,0.043]  Brier 0.440 [0.430,0.450]
  judgly-qwen-default        acc 0.656 [0.645,0.666]  ECE 0.034 [0.025,0.043]  Brier 0.440 [0.430,0.450]

== stance/final: 1343 of 1343 items answered by every external model (315 units); errors {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.817 [0.796,0.837]  ECE 0.113 [0.096,0.136]  Brier 0.312 [0.279,0.348]  over-context 0  latency median 0.449s
  tev1:4b                    acc 0.826 [0.804,0.845]  ECE 0.041 [0.032,0.066]  Brier 0.264 [0.238,0.294]  over-context 0  latency median 0.307s
  tev1:0.8b                  acc 0.577 [0.557,0.598]  ECE 0.111 [0.090,0.135]  Brier 0.556 [0.534,0.578]  over-context 0  latency median 0.085s
  judgly-gemma-raw           acc 0.827 [0.807,0.848]  ECE 0.142 [0.123,0.164]  Brier 0.310 [0.273,0.350]
  judgly-gemma-h2            acc 0.812 [0.792,0.834]  ECE 0.048 [0.034,0.070]  Brier 0.287 [0.264,0.310]
  judgly-gemma-temperature   acc 0.827 [0.807,0.848]  ECE 0.087 [0.068,0.107]  Brier 0.280 [0.258,0.304]
  judgly-gemma-default       acc 0.827 [0.807,0.848]  ECE 0.087 [0.068,0.107]  Brier 0.280 [0.258,0.304]
  judgly-qwen-raw            acc 0.778 [0.755,0.799]  ECE 0.187 [0.168,0.211]  Brier 0.410 [0.370,0.453]
  judgly-qwen-h2             acc 0.773 [0.751,0.794]  ECE 0.052 [0.035,0.077]  Brier 0.337 [0.316,0.358]
  judgly-qwen-temperature    acc 0.778 [0.755,0.799]  ECE 0.093 [0.070,0.115]  Brier 0.357 [0.335,0.380]
  judgly-qwen-default        acc 0.778 [0.755,0.799]  ECE 0.093 [0.070,0.115]  Brier 0.357 [0.335,0.380]

== general/bench: 2195 of 2231 items answered by every external model (559 units); errors {'nimble:9b': 0, 'tev1:4b': 36, 'tev1:0.8b': 36}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.711 [0.688,0.733]  ECE 0.056 [0.046,0.079]  Brier 0.401 [0.375,0.427]  over-context 0  latency median 0.807s
  tev1:4b                    acc 0.629 [0.607,0.654]  ECE 0.042 [0.029,0.063]  Brier 0.470 [0.446,0.491]  over-context 0  latency median 0.464s
  tev1:0.8b                  acc 0.456 [0.433,0.479]  ECE 0.178 [0.156,0.198]  Brier 0.655 [0.626,0.680]  over-context 0  latency median 0.107s
  judgly-gemma-raw           acc 0.716 [0.698,0.737]  ECE 0.238 [0.219,0.257]  Brier 0.508 [0.469,0.543]
  judgly-gemma-h2            acc 0.716 [0.696,0.735]  ECE 0.029 [0.020,0.048]  Brier 0.382 [0.364,0.401]
  judgly-gemma-temperature   acc 0.716 [0.698,0.737]  ECE 0.026 [0.018,0.046]  Brier 0.390 [0.367,0.411]
  judgly-gemma-default       acc 0.716 [0.696,0.735]  ECE 0.029 [0.020,0.048]  Brier 0.382 [0.364,0.401]
  judgly-qwen-raw            acc 0.592 [0.570,0.616]  ECE 0.342 [0.320,0.364]  Brier 0.730 [0.687,0.769]
  judgly-qwen-h2             acc 0.606 [0.586,0.628]  ECE 0.116 [0.102,0.136]  Brier 0.547 [0.524,0.568]
  judgly-qwen-temperature    acc 0.592 [0.570,0.616]  ECE 0.130 [0.116,0.151]  Brier 0.557 [0.535,0.576]
  judgly-qwen-default        acc 0.592 [0.570,0.616]  ECE 0.130 [0.116,0.151]  Brier 0.557 [0.535,0.576]

== general/final-flagged: 1000 of 1000 items answered by every external model (1000 units); errors {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.390 [0.359,0.420]  ECE 0.327 [0.300,0.356]  Brier 0.812 [0.774,0.852]  over-context 0  latency median 0.332s
  tev1:4b                    acc 0.421 [0.391,0.454]  ECE 0.183 [0.152,0.212]  Brier 0.676 [0.648,0.705]  over-context 0  latency median 0.216s
  tev1:0.8b                  acc 0.339 [0.312,0.370]  ECE 0.105 [0.078,0.133]  Brier 0.680 [0.669,0.690]  over-context 0  latency median 0.083s
  judgly-gemma-raw           acc 0.519 [0.489,0.549]  ECE 0.423 [0.391,0.454]  Brier 0.889 [0.833,0.943]
  judgly-gemma-h2            acc 0.512 [0.482,0.541]  ECE 0.106 [0.081,0.137]  Brier 0.621 [0.596,0.647]
  judgly-gemma-temperature   acc 0.519 [0.489,0.549]  ECE 0.104 [0.078,0.136]  Brier 0.603 [0.578,0.628]
  judgly-gemma-default       acc 0.512 [0.482,0.541]  ECE 0.106 [0.081,0.137]  Brier 0.621 [0.596,0.647]
  judgly-qwen-raw            acc 0.492 [0.464,0.522]  ECE 0.442 [0.412,0.472]  Brier 0.935 [0.882,0.990]
  judgly-qwen-h2             acc 0.487 [0.457,0.517]  ECE 0.046 [0.033,0.079]  Brier 0.620 [0.602,0.640]
  judgly-qwen-temperature    acc 0.492 [0.464,0.522]  ECE 0.037 [0.025,0.068]  Brier 0.617 [0.605,0.631]
  judgly-qwen-default        acc 0.492 [0.464,0.522]  ECE 0.037 [0.025,0.068]  Brier 0.617 [0.605,0.631]

== stance/final-flagged: 749 of 749 items answered by every external model (749 units); errors {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}; missing {'nimble:9b': 0, 'tev1:4b': 0, 'tev1:0.8b': 0}
  nimble:9b                  acc 0.474 [0.439,0.510]  ECE 0.404 [0.370,0.438]  Brier 0.879 [0.818,0.935]  over-context 0  latency median 0.395s
  tev1:4b                    acc 0.689 [0.657,0.722]  ECE 0.138 [0.108,0.170]  Brier 0.436 [0.393,0.478]  over-context 0  latency median 0.332s
  tev1:0.8b                  acc 0.617 [0.582,0.652]  ECE 0.076 [0.048,0.110]  Brier 0.491 [0.458,0.523]  over-context 0  latency median 0.09s
  judgly-gemma-raw           acc 0.750 [0.720,0.782]  ECE 0.201 [0.172,0.231]  Brier 0.441 [0.387,0.493]
  judgly-gemma-h2            acc 0.634 [0.599,0.669]  ECE 0.123 [0.093,0.158]  Brier 0.500 [0.467,0.530]
  judgly-gemma-temperature   acc 0.750 [0.720,0.782]  ECE 0.069 [0.060,0.104]  Brier 0.369 [0.340,0.399]
  judgly-gemma-default       acc 0.750 [0.720,0.782]  ECE 0.069 [0.060,0.104]  Brier 0.369 [0.340,0.399]
  judgly-qwen-raw            acc 0.718 [0.686,0.752]  ECE 0.227 [0.195,0.258]  Brier 0.510 [0.451,0.567]
  judgly-qwen-h2             acc 0.541 [0.509,0.577]  ECE 0.150 [0.125,0.193]  Brier 0.591 [0.554,0.622]
  judgly-qwen-temperature    acc 0.718 [0.686,0.752]  ECE 0.052 [0.030,0.085]  Brier 0.410 [0.379,0.440]
  judgly-qwen-default        acc 0.718 [0.686,0.752]  ECE 0.052 [0.030,0.085]  Brier 0.410 [0.379,0.440]
