liteinfer benchmarks
Each config is scored against the one it improves on (vs base), against the start of its lineage (vs first), and against vLLM at the same batch width (vs vLLM). Above 1.00x is better. Each row names the config it improves on; vLLM rows are references, not competitors. Bars show standing on the headline metric of each table.
model Llama-3.2-1B-InstructISL 128OSL 256prompts 50 / 200decoding greedy, forced lengthlast run 2026-09-12T09:45:14+00:00
latency · Llama-3.2-1B-Instruct · ISL=128 OSL=256
These runs did not all use the same prompts.
| config | ITL p50 | ITL p95 | TTFT p50 | TTFT p95 | E2E p50 | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|---|---|
| liteinfer-nocacheNo KV cache: every step re-feeds the whole sequence · reference point · removed from the codebase | 16.5 | 16.5 | 14.8 | 16.1 | 4,215.6 | — | — | 0.32x~ |
| liteinfer-eagerKV cache via transformers DynamicCache · improves on liteinfer-nocache · removed from the codebase · which is no longer runnable, so this delta spans two engines | 13.7 | 13.9 | 15.7 | 16.3 | 3,521.2 | 1.20x | 1.20x | 0.38x~ |
| liteinfer-native-eagerKV cache as plain tensors, no DynamicCache · improves on liteinfer-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 13.7 | 13.9 | 14.8 | 16.1 | 3,518.5 | 1.00x | 1.20x | 0.38x~ |
| liteinfer-pagedPaged KV cache: fixed-size blocks from a pool · improves on liteinfer-native-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 15.3 | 15.3 | 19.3 | 21.1 | 3,910.7 | 0.90x | 1.08x | 0.34x~ |
| liteinfer-eager-b4Static batching, B=4, eager cache · improves on liteinfer-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 14.0 | 14.1 | 15.8 | 17.7 | 3,596.6 | 0.98x | 1.17x | 0.37x~ |
| liteinfer-native-eager-b4Static batching, B=4, native eager cache · improves on liteinfer-native-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 13.9 | 13.9 | 14.9 | 16.7 | 3,561.1 | 0.99x | 1.18x | 0.38x~ |
| liteinfer-paged-b4Static batching, B=4, paged cache · improves on liteinfer-paged · removed from the codebase · which is no longer runnable, so this delta spans two engines | 15.0 | 15.0 | 19.0 | 20.9 | 3,832.2 | 1.02x | 1.10x | 0.35x~ |
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 · which is no longer runnable, so this delta spans two engines | 15.4 | 15.5 | 16.1 | 17.9 | 3,946.9 | 0.97x~ | 1.07x~ | 0.34x |
| liteinfer-sdpaContinuous batching, attention through PyTorch SDPA · improves on liteinfer-continuous | 14.4 | 14.4 | 14.4 | 16.3 | 3,675.9 | 1.07x | 1.15x~ | 0.36x |
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 13.3 | 13.4 | 14.1 | 16.0 | 3,414.5 | 1.08x | 1.24x~ | 0.39x |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 6.5 | 7.1 | 15.2 | 17.6 | 1,685.6 | 2.04x~ | 2.52x~ | 0.80x |
| liteinfer-splitkCaptured decode, narrow batches splitting their key loop across programs · improves on liteinfer-graphs | 6.5 | 6.5 | 14.9 | 16.8 | 1,677.8 | 1.00x | 2.53x~ | 0.80x |
| vllmvLLM, one sequence at a time · reference point | 5.2 | 5.3 | 22.5 | 30.4 | 1,358.1 | — | — | — |
| vllm-b4vLLM, up to 4 concurrent sequences · reference point | 5.2 | 5.3 | 22.2 | 28.8 | 1,357.0 | — | — | — |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 5.2 | 5.2 | 22.4 | 28.7 | 1,356.5 | — | — | — |
latency · Llama-3.2-1B-Instruct · ISL=3584 OSL=128
| config | ITL p50 | ITL p95 | TTFT p50 | TTFT p95 | E2E p50 | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|---|---|
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 13.4 | 13.4 | 147.3 | 165.8 | 1,847.0 | — | — | — |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 7.8 | 7.8 | 148.2 | 167.2 | 1,139.5 | 1.71x~ | — | — |
| liteinfer-splitkCaptured decode, narrow batches splitting their key loop across programs · improves on liteinfer-graphs | 6.7 | 6.8 | 147.9 | 166.2 | 1,004.2 | 1.16x | — | — |
latency · Llama-3.2-1B-Instruct · ISL=15360 OSL=128
| config | ITL p50 | ITL p95 | TTFT p50 | TTFT p95 | E2E p50 | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|---|---|
| liteinfer-graphs-16kCaptured decode with a 16k context budget, one program per KV head · reference point | 12.0 | 12.1 | 1,963.1 | 2,032.8 | 3,493.4 | — | — | — |
| liteinfer-splitk-16kSplit decode where attention is most of the step: long context, narrow batch · improves on liteinfer-graphs-16k | 7.5 | 7.5 | 1,962.5 | 2,033.2 | 2,912.9 | 1.61x | 1.61x | — |
throughput · Llama-3.2-1B-Instruct · ISL=128 OSL=256
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-nocacheNo KV cache: every step re-feeds the whole sequence · reference point · removed from the codebase | 60.5 | 0.2 | 846.7 | — | — | 0.32x |
| liteinfer-eagerKV cache via transformers DynamicCache · improves on liteinfer-nocache · removed from the codebase · which is no longer runnable, so this delta spans two engines | 73.4 | 0.3 | 697.9 | 1.21x | 1.21x | 0.39x |
| liteinfer-native-eagerKV cache as plain tensors, no DynamicCache · improves on liteinfer-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 72.7 | 0.3 | 704.6 | 0.99x | 1.20x | 0.39x |
| liteinfer-pagedPaged KV cache: fixed-size blocks from a pool · improves on liteinfer-native-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 66.8 | 0.3 | 766.7 | 0.92x | 1.10x | 0.35x |
| liteinfer-eager-b4Static batching, B=4, eager cache · improves on liteinfer-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 281.6 | 1.1 | 181.8 | 3.84x | 4.66x | 0.39x |
| liteinfer-native-eager-b4Static batching, B=4, native eager cache · improves on liteinfer-native-eager · removed from the codebase · which is no longer runnable, so this delta spans two engines | 277.6 | 1.1 | 184.4 | 3.82x | 4.59x | 0.38x |
| liteinfer-paged-b4Static batching, B=4, paged cache · improves on liteinfer-paged · removed from the codebase · which is no longer runnable, so this delta spans two engines | 252.9 | 1.0 | 202.4 | 3.79x | 4.18x | 0.35x |
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 · which is no longer runnable, so this delta spans two engines | 1,688.7 | 6.6 | 30.3 | 6.68x~ | 27.93x~ | 0.38x |
| liteinfer-sdpaContinuous batching, attention through PyTorch SDPA · improves on liteinfer-continuous | 1,804.6 | 7.0 | 28.4 | 1.07x | 29.84x~ | 0.41x |
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 1,997.1 | 7.8 | 25.6 | 1.11x | 33.03x~ | 0.45x |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 3,374.3 | 13.2 | 15.2 | 1.69x | 55.80x~ | 0.77x |
| liteinfer-graphs-b128Captured decode at 128 concurrent sequences · reference point | 8,060.4 | 31.5 | 6.4 | — | — | 0.74x |
| liteinfer-packedPrefill as one flat token run; measure it on a mixed-length dataset · improves on liteinfer-graphs-mixed | 3,367.3 | 13.2 | 15.2 | 1.00x | 1.00x | 0.77x |
| liteinfer-graphs-mixedCaptured decode with a padded prefill, at a mixed dataset's context budget · reference point | 3,371.2 | 13.2 | 15.2 | — | — | 0.77x |
| vllmvLLM, one sequence at a time · reference point | 188.2 | 0.7 | 272.1 | — | — | — |
| vllm-b4vLLM, up to 4 concurrent sequences · reference point | 721.4 | 2.8 | 71.0 | — | — | — |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 4,396.7 | 17.2 | 11.6 | — | — | — |
| vllm-b128vLLM, up to 128 concurrent sequences · reference point | 10,958.5 | 42.8 | 4.7 | — | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=128 OSL=1024
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 · which is no longer runnable, so this delta spans two engines | 1,240.7 | 1.2 | 165.1 | — | — | 0.30x |
| liteinfer-sdpaContinuous batching, attention through PyTorch SDPA · improves on liteinfer-continuous | 1,265.0 | 1.2 | 161.9 | 1.02x | — | 0.30x |
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 2,018.0 | 2.0 | 101.5 | 1.60x | — | 0.48x |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 3,216.2 | 3.1 | 63.7 | 1.59x | — | 0.77x |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 4,199.0 | 4.1 | 48.8 | — | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=1024 OSL=256
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 · which is no longer runnable, so this delta spans two engines | 720.9 | 2.8 | 71.0 | — | — | 0.25x |
| liteinfer-sdpaContinuous batching, attention through PyTorch SDPA · improves on liteinfer-continuous | 806.4 | 3.1 | 63.5 | 1.12x | — | 0.28x |
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 1,631.9 | 6.4 | 31.4 | 2.02x | — | 0.57x |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 2,162.3 | 8.4 | 23.7 | 1.33x | — | 0.76x |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 2,846.6 | 11.1 | 18.0 | — | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=1024 OSL=1024
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 · which is no longer runnable, so this delta spans two engines | 675.1 | 0.7 | 303.3 | — | — | 0.21x |
| liteinfer-sdpaContinuous batching, attention through PyTorch SDPA · improves on liteinfer-continuous | 700.9 | 0.7 | 292.2 | 1.04x | — | 0.22x |
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 1,912.8 | 1.9 | 107.1 | 2.73x | — | 0.59x |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 2,534.6 | 2.5 | 80.8 | 1.33x | — | 0.78x |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 3,242.6 | 3.2 | 63.2 | — | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=2048 OSL=128
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-sdpaContinuous batching, attention through PyTorch SDPA · improves on liteinfer-continuous | 419.5 | 3.3 | 61.0 | — | — | — |
| liteinfer-paged-attnContinuous batching, decode attention reads the KV pool in-kernel · improves on liteinfer-sdpa | 908.3 | 7.1 | 28.2 | 2.17x | — | — |
| liteinfer-graphsPaged decode replayed from a CUDA graph instead of launched per kernel · improves on liteinfer-paged-attn | 1,000.1 | 7.8 | 25.6 | 1.10x | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=mixed, up to 2048 OSL=16
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-packedPrefill as one flat token run; measure it on a mixed-length dataset · improves on liteinfer-graphs-mixed | 2,422.2 | 151.4 | 1.3 | 4.80x | 4.80x | — |
| liteinfer-graphs-mixedCaptured decode with a padded prefill, at a mixed dataset's context budget · reference point | 504.5 | 31.5 | 6.3 | — | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=mixed, up to 2048 OSL=128
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-packedPrefill as one flat token run; measure it on a mixed-length dataset · improves on liteinfer-graphs-mixed | 3,336.5 | 26.1 | 7.7 | 1.66x | 1.66x | — |
| liteinfer-graphs-mixedCaptured decode with a padded prefill, at a mixed dataset's context budget · reference point | 2,009.2 | 15.7 | 12.7 | — | — | — |
throughput across shapes
| config | 128 / 256 | 128 / 1024 | 1024 / 256 | 1024 / 1024 | 2048 / 128 | mix2048 / 16 | mix2048 / 128 |
|---|---|---|---|---|---|---|---|
| liteinfer-nocache | 60.5 | — | — | — | — | — | — |
| liteinfer-eager | 73.4 | — | — | — | — | — | — |
| liteinfer-native-eager | 72.7 | — | — | — | — | — | — |
| liteinfer-paged | 66.8 | — | — | — | — | — | — |
| liteinfer-eager-b4 | 281.6 | — | — | — | — | — | — |
| liteinfer-native-eager-b4 | 277.6 | — | — | — | — | — | — |
| liteinfer-paged-b4 | 252.9 | — | — | — | — | — | — |
| liteinfer-continuous | 1,688.7 | 1,240.7 | 720.9 | 675.1 | — | — | — |
| liteinfer-sdpa | 1,804.6 | 1,265.0 | 806.4 | 700.9 | 419.5 | — | — |
| liteinfer-paged-attn | 1,997.1 | 2,018.0 | 1,631.9 | 1,912.8 | 908.3 | — | — |
| liteinfer-graphs | 3,374.3 | 3,216.2 | 2,162.3 | 2,534.6 | 1,000.1 | — | — |
| liteinfer-graphs-b128 | 8,060.4 | — | — | — | — | — | — |
| liteinfer-packed | 3,367.3 | — | — | — | — | 2,422.2 | 3,336.5 |
| liteinfer-graphs-mixed | 3,371.2 | — | — | — | — | 504.5 | 2,009.2 |
| vllm | 188.2 | — | — | — | — | — | — |
| vllm-b4 | 721.4 | — | — | — | — | — | — |
| vllm-continuous | 4,396.7 | 4,199.0 | 2,846.6 | 3,242.6 | — | — | — |
| vllm-b128 | 10,958.5 | — | — | — | — | — | — |
latency across shapes
| config | 128 / 256 | 3584 / 128 | 15360 / 128 |
|---|---|---|---|
| liteinfer-nocache | 16.5 | — | — |
| liteinfer-eager | 13.7 | — | — |
| liteinfer-native-eager | 13.7 | — | — |
| liteinfer-paged | 15.3 | — | — |
| liteinfer-eager-b4 | 14.0 | — | — |
| liteinfer-native-eager-b4 | 13.9 | — | — |
| liteinfer-paged-b4 | 15.0 | — | — |
| liteinfer-continuous | 15.4 | — | — |
| liteinfer-sdpa | 14.4 | — | — |
| liteinfer-paged-attn | 13.3 | 13.4 | — |
| liteinfer-graphs | 6.5 | 7.8 | — |
| liteinfer-splitk | 6.5 | 6.7 | — |
| liteinfer-graphs-16k | — | — | 12.0 |
| liteinfer-splitk-16k | — | — | 7.5 |
| vllm | 5.2 | — | — |
| vllm-b4 | 5.2 | — | — |
| vllm-continuous | 5.2 | — | — |