LLM Inference Benchmarking - Measure What Matters

LLM Inference Benchmarking - Measure What Matters Prefill and Decode: The two phases of Inference Metrics Time to First Token (TTFT) Time per Output Token (TPOT) Inter Token Latency (ITL) End to End Latency (E2EL) Token Throughput (TPS) Request Throughput (RPS) The Pareto Frontier Step 1: Establish a baseline Pareto frontier Step 2: Find the operating point Step 3: Push the Frontier Micro-benchmarks Memory Bandwidth (HBM / SRAM) Compute (GEMM) & Attention Kernels (Flash Attention, MHA, MLA etc) Collectives (NCCL / RCCL) Measure What Matters: From the First Principles About the author(s) Try DigitalOcean for free Related Articles Technical Deep Dive: How we Created a Security-hardened 1-Click Deploy OpenClaw Technical Deep Dive: How DigitalOcean and AMD Delivered a 2x Production Inference Performance Increase for Character. ai DoTs SDK Development: Automating TypeScript Client Generation By Piyush Srivastava , Karnik Modi , Stephen Varela , and Rithish Ramesh Updated: February 11, 2026 12 min read Production-grade LLM inference is a complex systems challenge, requiring deep co-designs - from hardware primitives (FLOPs, memory bandwidth, and interconnects) to sophisticated software layers - across the entire stack. Given the hardware variability across GPU providers like NVIDIA and AMD - including generational differences in numeric type performance (FP8, BF16, NVFP4 etc), HBM bandwidth and capacity, peak FLOPs etc - optimal performance is never guaranteed. It depends on the software’s ability to maximize FLOPs utilization during prefill, maximize bandwidth efficiency during decode, optimize expert routing in MoE models, discover optimal parallelism strategies, and more. As inference hardware costs remain high, squeezing maximum performance to improve unit economics is a primary objective for AI teams. We are currently in an era of intense hardware-software co-design that will redefine performance and cost efficiency. Consequently, benchmarking must evolve to track three critical pillars: end-to-end model performance, micro-benchmarking of isolated components and a structured way to go after performance improvements. This article focuses on the LLM performance domain and analyzes the interplay between latency, throughput, concurrency, and cost. LLM Inference works in two-phases: prefill and decode. The prefill phase is where the entire input goes through the model’s forward pass which includes self-attention, add & norm, and pass through the hidden layers of the model’s feed forward network. This phase is extremely compute bound. FLOPs per byte transferred (arithmetic intensity) for the prefill phase is very high.