model Xenova/ms-marco-MiniLM-L-6-v2 · load 3.4 s · store 318 records, 313 with a vector · window 36 per arm

task                         before after  gov score best other  pairs     ms
idor_endpoint/ticket              3     2      -3.27      -3.11     53   4193
idor_endpoint/paraphrase         25     1      -6.89      -7.19     57   4503
idor_endpoint/bug-report         36     1       -5.8      -7.33     66   4138
price_tamper/ticket            None  None       None      -4.86     64   4255
price_tamper/paraphrase           2     1      -6.84       -7.2     64   4294
price_tamper/bug-report           9     2     -10.81     -10.51     49   3641
react_fetch/ticket                2     2      -3.62      -3.43     58   3952
react_fetch/paraphrase            2     6      -9.16      -6.71     61   4090
react_fetch/bug-report           43    31     -11.09       -6.5     58   3876
judge_summary/ticket              2     1       0.03      -0.52     57   3972
judge_summary/paraphrase          6    21     -10.37      -8.17     61   5469
judge_summary/bug-report          1     1       0.39       -5.2     57   4329
exit_code_trust/ticket            2     1      -5.09      -5.47     57   4454
exit_code_trust/paraphrase       18     7       -7.5       -5.9     63   5067
exit_code_trust/bug-report        1     1       3.46      -2.63     60   3519
spa_tokens/ticket                 2     1       2.95      -1.13     61   4804
spa_tokens/paraphrase             3    16      -10.4      -7.72     59   4662
spa_tokens/bug-report             4    23      -9.28      -7.23     57   4921
rate_limiter/ticket               1     1       2.67      -1.99     58   4630
rate_limiter/paraphrase          21    37     -10.72      -4.34     58   4458
rate_limiter/bug-report           1     1      -1.42      -3.55     58   4780
ci_linter/ticket                  8     2       0.54       1.62     56   4442
ci_linter/paraphrase           None  None       None      -4.28     59   5095
ci_linter/bug-report             12     4      -5.03      -0.28     58   3924

held-out: governing lesson in the top 12 after reranking: {'ticket': '7/8', 'paraphrase': '4/8', 'bug-report': '6/8'} · median 4385 ms per task, max 5469 ms

authored tasks: governing rank after reranking (must stay in the top 12):
  idor_endpoint    rank 2  score -3.97
  price_tamper     rank 6  score -5.27
  react_fetch      rank 1  score -1.24
  judge_summary    rank 2  score -0.44
  exit_code_trust  rank 12  score -6.43
  spa_tokens       rank 1  score 1.18
  rate_limiter     rank 1  score 2.29
  ci_linter        rank 3  score -0.74

precision prompts: reranker scores of the fused window (top 5, and how many at or above -2 / 0 / 2):
  top=[-10.2, -10.7, -11.0, -11.1, -11.1]  n>=-2:0 n>=0:0 n>=2:0  'this seems like a major point to scrutinize'
  top=[-3.6, -5.9, -6.7, -6.9, -6.9]  n>=-2:0 n>=0:0 n>=2:0  'ok do the updates and update my site and the wri'
  top=[-9.0, -9.7, -10.4, -10.7, -10.7]  n>=-2:0 n>=0:0 n>=2:0  'what about the org knowwledge ideae is that a ni'
  top=[-4.9, -6.6, -6.8, -7.0, -7.9]  n>=-2:0 n>=0:0 n>=2:0  'give me a doc for that too'
  top=[-5.3, -6.1, -9.3, -9.4, -9.5]  n>=-2:0 n>=0:0 n>=2:0  'how do we come up with a good plan for governing'
  top=[-4.3, -7.2, -7.9, -8.2, -8.7]  n>=-2:0 n>=0:0 n>=2:0  'what about the search ?  and how this thing find'
