TransformerBlock · 1.77M params · 3.86GFLOP · 85 ops → 34 fused · CPU/XLA
input f32(8, 128, 384)

memory
total         █████████████████████████  21.8 MB
params        ███████▉·················  6.76 MB
input         ········█▌···············  1.5 MB
intermediate  ··········█████████████▍·  12 MB
output        ························█  1.5 MB

layer                             FLOPs              share  ops  output
TransformerBlock                  3.86G  ██████████ 100.0%   85  f32(8, 128, 384)
  attention MultiHeadAttention    1.42G  ███▊······  36.8%   30  f32(8, 128, 384)
  attention_norm LayerNorm        2.37M  ··········   0.1%   16  f32(8, 128, 384)
  mlp_norm LayerNorm              2.37M  ··········   0.1%   16  f32(8, 128, 384)
  mlp_in Linear                   1.21G  ····██▉···  31.3%    1  f32(8, 128, 1536)
  mlp_out Linear                  1.21G  ·······███  31.3%    1  f32(8, 128, 384)