real prompts · split heldout (seed 11) · store ee5406f19cf9 · code e17cfe763aab
1500 pooled pairs, labels missing 0

task prompts: 120 person-typed · 44 with an applying lesson · 115 applying lessons in the pool
  arm                precision [95%]  applying found  prompts hit  briefed any  lessons/prompt  off-topic/prompt  repeat
  floor           9.8% [8.0%, 11.8%]      90 of 115      41 of 44     112 of 120             7.69              6.94      2%
  calibrated    16.5% [11.9%, 22.5%]      31 of 115      17 of 44      52 of 120             1.57              1.31      5%
  fallback      16.0% [11.6%, 21.7%]      32 of 115      19 of 44      58 of 120             1.67              1.40      4%
  backfill      17.8% [13.3%, 23.5%]      38 of 115      20 of 44      59 of 120             1.77              1.46      4%
  floor-3        12.1% [8.9%, 16.2%]      38 of 115      26 of 44     112 of 120             2.62              2.30      3%
  tiered          9.1% [7.5%, 10.9%]     102 of 115      39 of 44     112 of 120             9.35              8.50      2%
  tiered-full   17.8% [13.3%, 23.5%]      38 of 115      20 of 44      59 of 120             1.77              1.46      4%

null prompts: 60 person-typed · 12 with an applying lesson · 20 applying lessons in the pool
  arm                precision [95%]  applying found  prompts hit  briefed any  lessons/prompt  off-topic/prompt  repeat
  floor           8.8% [5.8%, 13.3%]      20 of 20       12 of 12      43 of 60              4.17              3.43      0%
  calibrated      0.0% [0.0%, 49.0%]       0 of 20        0 of 12       2 of 60              0.07              0.07      0%
  fallback       16.7% [3.0%, 56.4%]       1 of 20        1 of 12       4 of 60              0.13              0.08      0%
  backfill      40.0% [11.8%, 76.9%]       2 of 20        2 of 12       4 of 60              0.12              0.05      0%
  floor-3        12.4% [7.4%, 20.0%]      13 of 20        9 of 12      43 of 60              1.95              1.53      1%
  tiered          8.3% [5.4%, 12.6%]      19 of 20       12 of 12      43 of 60              4.25              3.50      0%
  tiered-full   40.0% [11.8%, 76.9%]       2 of 20        2 of 12       4 of 60              0.12              0.05      0%

F1 on task prompts (judged precision, pooled recall): floor 0.173 · calibrated 0.205 · fallback 0.203 · backfill 0.232 · floor-3 0.177 · tiered 0.165 · tiered-full 0.232

fix rule: backfill (most applying lessons found on task prompts among the fix arms at least as precise as calibrated)

tier rule: tiered (it reaches 102 applying lessons on task prompts, the floor 90; the rule asks at least as many)

sweep rule: backfill (highest F1 on task prompts among arms briefing at most five lessons a prompt)
