verify steps in window: old 70, main 69
per step: busy old 29.490 ms, main 29.470 ms; span old 29.965, main 29.989

| kernel | launches/step old | us/step old | launches/step main | us/step main | delta us/step |
|---|---:|---:|---:|---:|---:|
| `fused_compress_store_kernel(__nv_bfloat16 const*, __nv_bfloat16 const*, unsigned char*, lo` | 0.0 | 0.0 | 32.0 | 200.0 | +200.0 |
| `void cutlass::Kernel2<cutlass_80_wmma_tensorop_bf16_s161616gemm_bf16_16x16_64x2_nn_align8>` | 0.0 | 0.0 | 32.0 | 167.8 | +167.8 |
| `void tq_oscar_rotate_rows_kernel<256>(TqOscarProblems, int)` | 0.0 | 0.0 | 16.0 | 74.7 | +74.7 |
| `(anonymous namespace)::ni8_prefill_kernel((anonymous namespace)::Ni8Params)` | 16.0 | 535.6 | 16.0 | 593.3 | +57.7 |
| `void exl3_gemm_kernel<5, false, 2, 16, 32, 128, 4, 3>(__half const*, unsigned short const*` | 400.0 | 18964.8 | 400.0 | 19001.8 | +37.0 |
| `triton_red_fused__to_copy_add_addcmul_constant_pad_nd_exl3_linear_gemm_mean_mul_pow_rsqrt_` | 0.0 | 0.0 | 8.0 | 28.2 | +28.2 |
| `memcpy32_post` | 128.0 | 111.2 | 160.0 | 138.9 | +27.7 |
| `void at::native::index_elementwise_kernel<128, 4, at::native::index_copy_kernel_impl<at::n` | 48.0 | 87.2 | 58.0 | 104.4 | +17.2 |
| `triton_poi_fused__to_copy__unsafe_view_clone_split_view_0` | 0.0 | 0.0 | 16.0 | 16.6 | +16.6 |
| `triton_poi_fused__to_copy_cat_split_with_sizes_view_2` | 48.0 | 53.2 | 48.0 | 67.4 | +14.2 |
| `triton_per_fused__to_copy_add_arange_cat_clamp_copy_cos_dflash_varlen_attend_mean_mul_neg_` | 0.0 | 0.0 | 5.0 | 7.5 | +7.5 |
| `triton_per_fused__to_copy_add_arange_as_strided_cat_cos_dflash_place_block_write_expand_me` | 0.0 | 0.0 | 5.0 | 6.9 | +6.9 |
| `triton_poi_fused__to_copy_exl3_linear_gemm_mul_silu_13` | 0.0 | 0.0 | 5.0 | 6.2 | +6.2 |
| `void exl3_gemm_kernel<6, false, 2, 16, 32, 128, 4, 3>(__half const*, unsigned short const*` | 2.0 | 1615.8 | 2.0 | 1621.8 | +6.0 |
| `triton_poi_fused__to_copy_add_addcmul_constant_pad_nd_exl3_linear_gemm_mean_mul_pow_rsqrt_` | 0.0 | 0.0 | 5.0 | 5.4 | +5.4 |
| ... | | | | | |
| `fused_sigmoid_gating_delta_rule_update_kernel` | 96.0 | 1444.8 | 96.0 | 1298.4 | -146.5 |
| `void cutlass::Kernel2<cutlass_80_wmma_tensorop_bf16_s161616gemm_bf16_16x16_128x1_nn_align8` | 32.0 | 94.0 | 0.0 | 0.0 | -94.0 |
| `void exl3_gemm_kernel<6, false, 1, 16, 32, 128, 4, 3>(__half const*, unsigned short const*` | 56.0 | 2074.0 | 56.0 | 1993.1 | -80.9 |
| `fused_compress_store_kernel(__nv_bfloat16 const*, __nv_bfloat16 const*, unsigned char*, lo` | 16.0 | 62.7 | 0.0 | 0.0 | -62.7 |
| `triton_poi_fused__to_copy_add_arange_cat_cos_expand_index_copy_mean_mul_pow_remainder_rsqr` | 10.0 | 60.3 | 0.0 | 0.0 | -60.3 |
| `triton_red_fused__to_copy_add_addcmul_constant_pad_nd_exl3_linear_gemm_mean_mul_pow_rsqrt_` | 8.0 | 28.1 | 0.0 | 0.0 | -28.1 |
| `triton_poi_fused_copy__view_16` | 10.0 | 26.4 | 0.0 | 0.0 | -26.4 |
| `triton_poi_fused__to_copy_add_cat_mean_mul_neg_pow_rsqrt_slice_split_view_5` | 16.0 | 26.3 | 0.0 | 0.0 | -26.3 |
| `triton_poi_fused__to_copy_add_copy_index_mean_mul_pow_rsqrt_slice_split_unsqueeze_view_6` | 16.0 | 23.9 | 0.0 | 0.0 | -23.9 |
| `triton_poi_fused__to_copy_add_cat_mean_mul_neg_pow_rsqrt_slice_view_2` | 16.0 | 21.8 | 0.0 | 0.0 | -21.8 |
| `triton_poi_fused__to_copy_add_copy_index_mean_mul_pow_rsqrt_slice_unsqueeze_view_3` | 16.0 | 18.2 | 0.0 | 0.0 | -18.2 |
| `kernel_cutlass_kernel_split_d_tkvkernelscudaprefillturbo_attn_cute_bypassTurboAttnCuteBypa` | 5.0 | 110.2 | 5.0 | 93.1 | -17.2 |
