SimpleTuner CuTe attention

Kernel implementation synchronized with KohakuFA-CuTe revision
1afb72698c74b3a1b5958d3b80948458c9bb6e84.
https://github.com/SimpleTuner-io/KohakuFA-CuTe

The kernels use warp MMA on SM86 Ampere, Ada and RTX Blackwell,
WGMMA/TMA on Hopper, and native tcgen05/TMEM producer/consumer pipelines
on B200 and B300.
Architecture-specific configurations preserve the centered FP32 backward
calculation, separate unscaled row maxima and logarithmic normalization,
and late gradient scaling. B200 and B300 use fused probability/dS consumers
for 128-column backward tiles; B300 retains its hardware TMEM reductions.
Interior probability tiles omit per-element bounds checks; tails and causal
boundaries retain visibility checks with the same centered arithmetic.

Blackwell uses packed FP32 arithmetic, overlapped operand/statistic transfers,
and adjacent D512 gradient slices. BF16 D512 MHA on B300 at 512 tokens
and above uses a register-resident FP32 auxiliary with 256-wide gradient
accumulators and vectorized gradient staging.

SM120 backward shares score products across query column warps for
D256/D512, combines row corrections before publishing them, and stages KV
statistics for the selected dense D256 and causal D64 configurations.
Aligned D512 forward and backward use full-width tiles with shared score
producers and separate output consumers. Consumer GPU backward pairs QK/dP
producers and packs DS/P stores. SM86/SM89 use wider D512 operand copies;
Ada uses tuned D256 tiles and cache policy, while SM120 also tunes causal
D128 query sharing.

The producer/consumer architecture in blackwell_pipeline.py adapts KohakuFA's
sm100/fwd.py and sm100/bwd.py at revision
0c5ede6859307bedc0c6089523c16b1ca51b7da9:
https://github.com/KohakuBlueleaf/KohakuFA
KohakuFA is distributed under Apache-2.0, reproduced in LICENSE.Apache-2.0.
The CuTe port retains centered backward and single-owner gradient tiles.

Causal BF16 D256 GQA uses overlapping forward stages on Ada and independent
K/V staging with ordered softmax warpgroups on Hopper. Short D64 cross-attention
queries split across the key sequence on Ada/Hopper. Native build caches include
the C++ source hash so copied checkouts cannot retain stale dispatch code.
Batch/head products above 65,535 flatten onto grid-x for attention, mask
packing and backward output-delta preparation; smaller grids are unchanged.

Boolean masks are classified on every invocation, including CUDA graph replay.
Native eager autograd and compiler-safe custom operators share allocation and
TVM FFI dispatch. Compiled host entries are cached by device, tensor metadata
and sequence-length configuration, bounded to 128 entries and released before
Python shutdown.

row_view and operand_a adapt accumulator-layout transforms from NVIDIA CUTLASS:
examples/python/CuTeDSL/cute/ampere/kernel/attention/flash_attention_v2.py
WGMMA and TMA API usage follows the BSD-licensed Hopper examples:
examples/python/CuTeDSL/cute/hopper/kernel/dense_gemm/dense_gemm.py
hopper_operand adapts the register-operand layout conversion from:
examples/python/CuTeDSL/cute/hopper/kernel/attention/fmha.py
https://github.com/NVIDIA/cutlass
Reference revision: 0b55a2f691d69981583568fd9eb69687b1f0de8a
Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES.
BSD-3-Clause; the full license is included in LICENSE.

Packed Float32-to-BFloat16/Float16 conversion adapts the cvt_f16x2_f32 and
cvt_f16 helpers from FlashAttention:
flash_attn/cute/utils.py
https://github.com/Dao-AILab/flash-attention
Reference revision: 85d6dfe3573708b5632915950cd8b069b924ba52
Copyright (c) 2025, Tri Dao.
BSD-3-Clause; the full license and contributor list are included in LICENSE.
