How LLMs Scale: FLOPs, Bytes, and the Price of a Token

Nish · July 5, 2026

NLP   Fundamentals
⏱️ 25 min read

Table of Contents

Every answer from Gemini, GPT, or Claude feels weightless: text appears on a screen, a few seconds pass, done. Physically, something very different happened. Somewhere, a rack of accelerators streamed hundreds of gigabytes of model weights through silicon for every single token, coordinated over network links carrying terabytes per second, inside a building drawing as much power as a small town. This post is about that physical layer: how frontier models actually run on GPUs and TPUs, and how one deceptively simple ratio, the number of math operations a chip can do per byte it can fetch from memory, ends up explaining training cluster sizes, the pause before the first token, why output tokens cost more than input tokens, and most of what you see on an LLM pricing page.

You do not need a systems background for this. If you know roughly what an LLM is (a transformer predicting one token at a time) and ideally have met the KV cache before, everything else is built up here from two numbers on a spec sheet.

TL;DR

  • An accelerator is characterized by two headline numbers: how fast it does arithmetic (FLOP/s, floating-point operations per second) and how fast it moves data from memory (bytes per second). Their ratio, roughly 295 FLOPs per byte for an H100 and in the low hundreds for most recent chips, is the single most consequential constant in this post.
  • The roofline model says any workload doing fewer FLOPs per byte than that ratio leaves the chip idle, waiting on memory. Big matrix multiplications clear the bar easily; generating one token at a time misses it by two orders of magnitude.
  • Training is the friendly case: it needs about \(6ND\) FLOPs for \(N\) parameters and \(D\) tokens, arrives in huge parallel batches, and keeps chips compute-bound. Its problem is scale: Llama 3.1 405B’s \(3.8 \times 10^{25}\) FLOPs would take a single H100 thousands of years, so the model is sliced across tens of thousands of chips, and the art is paying for all that communication without idling the compute.
  • Inference is the hostile case: decode reloads every weight for every token, so a lone request runs memory-bound at a fraction of a percent of the chip’s potential. Batching many users onto the same weight-reads is the one great lever, and it trades latency for throughput.
  • Cost follows directly. In a simple roofline model of a 70B model on eight H100s, going from batch 1 to the memory limit cuts decode cost from about \$35 to \$0.43 per million tokens, an 80x difference on identical hardware. Batching, prompt caching discounts, and the input/output price gap on API pricing pages are all this arithmetic passed on to you.
  • GPUs and TPUs make different bets (many flexible cores on a switched network vs. a few huge matrix units on a torus), but both obey the same roofline; the constants, not the logos, decide behavior.

Scope. Numbers below are either computed in this post (or its committed figure script) or cited from spec sheets, papers, and the sources at the end. Chip specs are dense (no-sparsity) figures, and simple models deliberately ignore real-world overheads; where that matters, it is flagged.

The machines: matmul factories

Strip away the branding and a modern AI accelerator, whether an NVIDIA GPU or a Google TPU, is a machine for multiplying large matrices next to a stack of very fast memory. That is not a simplification of what LLMs need; it is what LLMs need. A transformer forward pass is overwhelmingly matrix multiplications, or matmuls (the attention projections and feed-forward layers), and a matmul is thousands of independent multiply-and-add steps, which is exactly the kind of work you can spread across a sea of tiny arithmetic units.

Two numbers on the spec sheet tell you almost everything about such a machine:

  1. Compute rate: how many floating-point operations per second (FLOP/s) it can execute, where one FLOP is a single arithmetic step such as multiplying or adding two numbers. An H100 does up to 989 trillion per second in BF16, a compact 16-bit number format (2 bytes per number) that training typically uses.
  2. Memory bandwidth: how many bytes per second it can move between its compute units and its on-package high-bandwidth memory (HBM), a stack of memory dies (the same DRAM technology as ordinary RAM, packaged millimetres from the processor) that holds the model weights. For the H100 that is 3.35 TB/s.
Side-by-side schematic of an NVIDIA H100 GPU and a Google TPU v5e. Both share the same skeleton: a compute die on top, HBM below connected by a fat bandwidth arrow (3.35 TB/s for the GPU, 0.82 TB/s for the TPU), and interconnect links to other chips. The GPU compute die is a grid of 132 small streaming multiprocessors on a switched NVLink network; the TPU die holds two large 128 by 128 systolic arrays plus a 128 MiB scratchpad, wired directly to four torus neighbours.
Two design philosophies, one skeleton. A GPU spreads its arithmetic across 132 small, independently scheduled cores and talks to peers through switches; a TPU concentrates it in a couple of enormous systolic arrays scheduled entirely by the compiler, wired directly to its neighbours in a torus. Both live or die by the fat purple pipe to HBM. Specs from NVIDIA's H100 datasheet and Google's TPU documentation (see Sources).

The two vendors package those ingredients very differently. A GPU is a fleet of small ships: 132 streaming multiprocessors (SMs) on an H100, each with its own tensor cores (small dedicated matrix-multiply engines) and its own scheduler, able to run thousands of threads that the hardware juggles to hide memory delays. A TPU is one giant container vessel: a systolic array, a 128x128 grid of multiply-accumulate units through which data flows in a fixed rhythm, with no per-core scheduling at all; the compiler choreographs every byte’s journey in advance. The GPU bet buys flexibility (irregular workloads, researchers’ custom code); the TPU bet buys density and predictability for the one workload it was built for. For dense matrix multiplication, the difference matters less than the spec-sheet numbers suggest, because both designs are ultimately fed by the same kind of HBM.

Here are those two numbers for the chips you will most often hear about, with their ratio, whose starring role is the next section:

Chip BF16 compute (dense) HBM HBM bandwidth FLOPs per byte
NVIDIA A100 312 TFLOP/s 80 GB 2.0 TB/s ~156
NVIDIA H100 989 TFLOP/s 80 GB 3.35 TB/s ~295
NVIDIA B200 2,250 TFLOP/s 180 GB 8.0 TB/s ~281
Google TPU v5e 197 TFLOP/s 16 GB 0.82 TB/s ~240
Google TPU v6e (Trillium) 918 TFLOP/s 32 GB 1.64 TB/s ~560
Google TPU7x (Ironwood) 2,307 TFLOP/s 192 GB 7.37 TB/s ~313

Figures from NVIDIA datasheets and Google Cloud TPU docs; the last column is just column two divided by column four. “Dense” flags one marketing trap: NVIDIA’s headline numbers assume 2:1 structured sparsity, matrices with half their entries zeroed out, which LLM work generally does not use, so they are halved here (see Sources).

One chip is never the whole story, because no frontier model fits on one chip. The third spec that matters is the interconnect, the wiring between chips: NVLink (NVIDIA’s direct chip-to-chip link) joins 8 GPUs into a node at 900 GB/s per GPU (and NVL72 racks now join 72), with an InfiniBand or Ethernet fat tree (a hierarchy of network switches giving every node a fast path to every other) connecting nodes beyond that; TPUs instead wire each chip straight to its nearest neighbours through the inter-chip interconnect (ICI), forming a 2D or 3D torus, a grid whose edges wrap around, and scaling to pods of 8,960 chips for v5p and 9,216 for Ironwood with no switches between chips at all. Bandwidth falls roughly an order of magnitude at each hop outward: HBM to a chip’s own compute is fastest, links to neighbouring chips next, the general datacenter network slowest. Keep that ladder in mind; the parallelism strategies below are all attempts to match their communication to the rung that can afford it.

The one ratio: rooflines and arithmetic intensity

Now the mental model that carries the entire post, taken from the wonderful scaling book that several sections here lean on. Any computation on an accelerator needs two things to happen: the arithmetic must be executed, and the data must be moved in and out of memory. To a good approximation the chip overlaps them, so the time taken is whichever is slower:

\[T \;\approx\; \max\!\left(\frac{\text{FLOPs}}{\text{compute rate}},\; \frac{\text{bytes moved}}{\text{bandwidth}}\right)\]

Which term wins depends on the workload’s arithmetic intensity: how many FLOPs it performs per byte it touches. Compare that to the chip’s own ratio (compute rate over bandwidth, the last column of the table above) and you know its fate. On an H100, a workload doing more than about 295 FLOPs per byte is compute-bound: the expensive arithmetic units are the bottleneck, which is what you paid for. Below 295, it is memory-bound: the multipliers sit idle while HBM dribbles bytes at them, and adding FLOPs to the spec sheet would change nothing.

Log-log roofline plot for an H100. Achievable FLOP/s rises along a diagonal set by the 3.35 TB/s memory bandwidth until arithmetic intensity reaches the ridge at 295 FLOPs per byte, then flattens at the 989 TFLOP/s BF16 peak. Decode at batch 1 sits far down the diagonal at intensity about 1; decode at batch 64 sits higher but still under the ridge; prefill of a 2,000-token prompt sits on the flat compute-bound roof.
The roofline for an H100 (BF16, dense specs). Left of the ridge, performance is capped by the purple diagonal: memory bandwidth times intensity. Right of it, by the flat blue roof: peak compute. Token generation at small batch lives at the far bottom-left, prompt processing on the roof; the whole economics of LLM serving is the distance between those dots. Computed from the datasheet numbers in the table above.

Why do LLM workloads land where they do on this chart? Take a weight matrix multiplication, the transformer’s staple: multiplying a batch of \(B\) token vectors against a large weight matrix reads the weights once but uses each of them \(B\) times, doing two FLOPs (a multiply and an add) per weight per token. For weight matrices far bigger than the batch, the weight traffic dominates the byte count, so the intensity is roughly \(2B\) FLOPs per \(2\) bytes, i.e. the arithmetic intensity of a transformer’s matmuls is approximately the number of tokens flowing through them together, a count called the batch size from here on. This is the punchline of the whole section. Process 2,000 prompt tokens in one pass and you sit comfortably on the compute roof. Generate one token for one user and your intensity is about 1, a 295th of what the H100 needs: the arithmetic units sit idle roughly 294 of every 295 cycles, waiting on the memory bus.

That ratio is also remarkably stubborn. Compute per chip has grown much faster than memory bandwidth for decades, and as Reiner Pope’s lecture on this exact topic points out, the FLOPs-per-byte ratio of NVIDIA’s data-center line has hovered around 300 for several generations, with no relief in sight. Everything downstream (batch sizes, serving architectures, pricing) is shaped by that constant.

The same roofline, three bandwidths. The formula does not care whether “bytes moved” cross the HBM bus, the NVLink cable, or the datacenter network; swap in the right bandwidth and the same analysis rates every layer of the stack. That is why this one idea explains so much of what follows.

Training: the biggest computations on Earth

Training is, perhaps surprisingly, the easy case for the roofline. The recipe of pre-training is to shovel trillions of tokens through the model in giant batches, and giant batches mean giant matmuls with intensity in the thousands: solidly compute-bound. The challenge is not keeping one chip busy. It is that the total work is astronomical, which forces the computation across tens of thousands of chips, and keeping them all busy at once becomes the game.

How astronomical? There is a classic rule of thumb, from the scaling-laws literature, that training a model with \(N\) parameters on \(D\) tokens costs about

\[C \;\approx\; 6\,N\,D \ \text{FLOPs}\]

and it is worth seeing why, because it is three lines, not a mystery. In the forward pass, essentially every parameter sits in some weight matrix and participates in one multiply and one add per token: \(2ND\). The backward pass does two matmuls of the same size for each one in the forward pass (one to get the gradient flowing to the previous layer, one to get the gradient of the weights themselves): \(4ND\). Total \(6ND\). This counts only the weight matmuls; attention’s score computations add more, but they stay a small correction until contexts get very long (the scaling book puts the crossover around sequence lengths of eight times the model width).

The rule gets a satisfying real-world check. Meta’s Llama 3 paper reports pre-training the 405B model on 15.6 trillion tokens at a total budget of \(3.8 \times 10^{25}\) FLOPs. Run the formula: \(6 \times 405\text{e}9 \times 15.6\text{e}12 = 3.79 \times 10^{25}\). Almost exactly on the nose.

How much is \(3.8 \times 10^{25}\)? A single H100 running flat out at its dense BF16 peak would need about 1,200 years; at the roughly 40% efficiency real training runs achieve (more on that number below), closer to 3,000 years. Meta used up to 16,384 H100s and spent 30.84 million GPU-hours, which works out to a couple of months of calendar time. For scale on the ratio between \(N\) and \(D\): the Chinchilla paper famously found that a compute-optimal training run should use about 20 tokens per parameter; Llama 3 405B used 38, and the gap has since blown out much further, because a model that will serve billions of requests is worth overtraining far past the compute-optimal point to get more quality per serving-FLOP. Pope’s lecture estimates recent frontier models train at as much as a hundred times the Chinchilla ratio.

Why one chip can never do it

Time is one wall; memory is another, and it arrives first. Just storing 405B parameters in BF16 takes 810 GB, ten times an H100’s HBM. Training needs far more: gradients (the per-weight error signals each step produces), plus the running averages of those gradients that the standard Adam optimizer keeps, come to roughly 10 bytes per parameter in the usual mixed-precision setup (a rule of thumb from the scaling book), about 4 TB for the 405B model, or the combined HBM of fifty H100s before a single activation is stored. Frontier training does not use many chips as an optimization. It has no choice.

Four ways to slice a model

So the model and its data must be cut across chips, and every cut creates communication. The four standard cuts differ in what travels over the wires and how often, which determines which rung of the bandwidth ladder can carry them:

Four-panel schematic. Data parallelism: four chips each hold a full copy of the model and process different batches, exchanging gradients via AllReduce. Tensor parallelism: each layer's matrices are split column-wise across the four chips, which exchange activations inside every layer. Pipeline parallelism: consecutive layers live on different chips and activations flow left to right between stages. Expert parallelism: different MoE experts live on different chips and tokens are routed to them and back via AllToAll.
Four ways to slice one model across four chips. Blue is model weights, green is data, orange is the communication each choice signs up for: gradients once per step (data), activations inside every layer (tensor), activations once per stage boundary (pipeline), or routed tokens (expert). Real runs compose several of these at once.
  • Data parallelism: every chip (or group) holds a full model copy and processes its own slice of the batch; once per step, gradients are averaged across copies with an AllReduce (the collective operation that sums a value across chips and hands every chip the result). Communication is rare and can overlap with compute, so this is the workhorse, and its sharded variant (FSDP, or ZeRO, which splits the weights and optimizer state across chips and gathers them just-in-time) fixes the memory cost of full copies for what the scaling book shows is the same total communication volume. The catch: below roughly 2,500 tokens of batch per chip (a threshold the book derives for both v5p pods and H100 clusters, from the ICI and network bandwidths respectively), the gradient exchange stops hiding behind compute and the run goes communication-bound. Batches cannot shrink forever, and models whose state outgrows even sharded memory need other cuts.
  • Tensor parallelism: split every weight matrix itself across chips, so each holds a slab of every layer. Now activations (the intermediate token representations flowing between layers) must be exchanged inside every single layer, squarely on the critical path, which is why this cut lives on the fastest interconnect rungs (within an NVLink node, or a few torus hops) and tops out around 8- to 16-way before communication swamps compute.
  • Pipeline parallelism: give consecutive layers to different chips and pass activations down the line. The communication is tiny (one activation tensor per stage boundary), which makes it the cheap way to span slow links between racks and pods; the cost is bubbles, chips idling while they wait for the pipeline to fill and drain, patched by splitting the batch into micro-batches that chase each other down the pipe and by cleverly interleaved schedules.
  • Expert parallelism: for mixture-of-experts models (whose feed-forward blocks are split into many specialist “experts”, with each token routed to only a few), park different experts on different chips and send each token to the chips holding its experts. The routing is an AllToAll, every chip sending a different slice of its tokens to every other chip, awkward but affordable on switched GPU fabrics, and it is how sparse frontier models keep their enormous parameter counts without paying for them on every token.

Real recipes stack these. DeepSeek-V3, per the scaling book’s GPU chapter, trained on 2,048 GPUs as 64-way expert parallel by 16-way pipeline parallel by 2-way data parallel; Llama 3 405B combined tensor, pipeline, and sharded data parallelism across its 16k GPUs. The design rule underneath is always the roofline with the right bandwidth plugged in: chatty cuts on fast links, quiet cuts on slow ones, and per-chip batch above the threshold that keeps arithmetic, not networking, the bottleneck.

MFU: the scorecard

The honest measure of how well all this slicing worked is model FLOPs utilization (MFU): the fraction of the fleet’s peak FLOP/s actually embodied in the model’s forward and backward passes. Everything lost to communication stalls, pipeline bubbles, per-operation software overheads, and restarts shows up as the gap to 100. The PaLM paper that popularized the metric hit 46.2% on 6,144 TPU v4 chips (and estimated GPT-3’s run at 21.3%); Meta reports 38-43% across Llama 3 405B’s configurations; the scaling book uses 40% as its planning default. That number is worth internalizing: the best engineering teams on Earth, with every trick in this section deployed, keep roughly half the silicon they paid for doing useful math. (Divide Llama 3’s reported \(3.8 \times 10^{25}\) FLOPs by its 30.84M H100-hours at peak rate and you get an average of about 35%, consistent with those steady-state figures once restarts and the later long-context training stages are included.)

Inference: where physics meets product

Training happens once, out of sight. Inference happens every time anyone uses the model, under a user’s impatient gaze, and it is where the roofline turns hostile.

As covered in depth in the KV cache post, generating a response has two phases with opposite characters. Prefill ingests your whole prompt in one parallel pass: thousands of tokens moving through the matmuls together, intensity in the thousands, compute-bound, the chip earning its keep. Decode then produces the response one token at a time, because each token depends on the previous one, and this sequential dependency is the serving problem in miniature: every decode step must stream all the model’s weights (plus the request’s KV cache, the stored attention state of everything in the conversation so far) out of HBM to produce a single token per sequence. One token per weight-read is an arithmetic intensity of about 1 against a ridge of 295.

The two phases map directly onto the two latency metrics every serving team optimizes, latency here meaning simply the time an individual user spends waiting:

  • Time to first token (TTFT): how long before the response starts. Dominated by prefill, so it scales with prompt length and is compute-limited.
  • Inter-token latency, or time per output token (TPOT): the pace of the stream once it starts. Dominated by decode, so it is memory-bandwidth-limited, and (as the KV post shows) degrades as contexts grow and the cache adds to the bytes each step must touch.

A concrete pair of numbers, from a worked example by Hugging Face’s TGI team for a 7B model on an A100: prefill of a 512-token prompt takes about 92 ms of compute against 9 ms of weight-loading (compute-bound, 10x margin), while each decode step takes 0.17 ms of compute against 9.3 ms of weight-loading. In decode, the arithmetic is fifty-five times faster than the memory traffic feeding it. The chip is a Formula 1 car doing its laps in a car park.

Batching: the throughput dial

There is exactly one great lever against this, and it falls straight out of the intensity formula: if the matmul’s intensity is the number of tokens moving through it together, then put more tokens through it together. Decode gives you one token per user per step, so the tokens must come from different users: batch \(B\) concurrent requests and each weight-read now produces \(B\) tokens instead of one. Same bytes, \(B\) times the intensity, \(B\) times the throughput (the total tokens per second the hardware produces across all its users, the number an operator’s costs divide by), until you approach the ridge (around 240-300 concurrent sequences for BF16 weights on these chips, per the scaling book’s inference chapter) or run out of memory for all those users’ KV caches, whichever comes first. One important asymmetry, covered properly in the KV post: batching amortizes the weights, but each sequence still owns a private KV cache whose bytes cannot be shared, which is why attention stays memory-bound even at large batch and why so much architecture work goes into shrinking the cache.

Log-log curve of node throughput in tokens per second versus per-token latency in milliseconds, for a 70B dense model on eight H100s under a roofline model. Throughput climbs steeply from 190 tokens per second at batch 1 to about 11,000 at batch 129 with only modest latency growth from 5.3 to 11.7 ms, then bends: by batch 367 throughput is 15,548 tokens per second at 23.6 ms per token, where a red annotation marks that weights plus KV caches exceed the node's 640 GB of HBM.
The serving frontier for a 70B dense model on an 8x H100 node (BF16, 4k contexts), computed from the roofline model in the figure script: step time is the max of compute time and bytes-over-bandwidth, with each sequence carrying its own KV cache. Early batching is nearly free throughput; past the knee, latency and HBM capacity charge full price. Real engines add overheads, so treat the shape, not the exact values, as the lesson.

Numbers from that curve (all printed by the figure script): batch 1 gives each user 190 tokens/s at 5.3 ms per token, and the node produces 190 tokens/s total. Batch 367 (where the KV caches fill the node’s HBM at these context lengths) gives each user a still-snappy 42 tokens/s at 23.6 ms per token, and the node produces 15,548 tokens/s total. That is 82 times the throughput for 4.5 times the latency. No provider can afford not to take that trade, which is why your tokens always share a GPU with strangers’, and why the scaling book’s applied chapter summarizes serving economics as: accepting about twice the latency can buy up to a hundred times lower cost per token.

Making the trade work in production took one more idea. A naive server batches requests in static groups and waits for the whole group to finish, so one long-winded response holds hostages; Anyscale measured a 13B model crawling at 81 tokens/s this way on an A100. Continuous batching admits and retires requests every step, keeping the batch full at all times, and with paged KV memory on top (the PagedAttention system described in the KV post, which cut cache waste from as much as 60-80% to under 4%), the same benchmark hit roughly 1,950 tokens/s with vLLM: a 23x gain purely from serving software.

More levers

Three other techniques matter enough to name, and all three are roofline plays. Quantization stores weights in 8 or 4 bits instead of 16: decode’s bottleneck is bytes of weights per step, so halving the bytes roughly halves inter-token latency and halves the batch needed to go compute-bound. Speculative decoding attacks the sequential dependency itself: a small, fast draft model proposes several tokens, and the big model verifies them in one batched pass, converting idle compute into accepted tokens (typically worthwhile for a handful of draft tokens per round). And disaggregated serving runs prefill and decode on separate chip pools, because a compute-bound and a memory-bound workload sharing one machine mostly get in each other’s way; the orchestrator ships the freshly built KV cache from a prefill server to a decode server, and capacity planning becomes chairs-to-tables arithmetic (the scaling book’s worked example lands at about three prefill servers per decode server for a typical chat workload). If you have used a reasoning model, note the roofline reading of that trend: thinking tokens are decode tokens, so test-time compute shifts demand toward exactly the memory-bound phase all these tricks exist to cheapen.

The bill

Everything above compresses into money with almost no residue.

Training costs are a rental problem: chips times hours times price. H100s rent for roughly \$2-4 per hour in mid-2026 (big-cloud list prices higher, spot markets lower), so Llama 3 405B’s 30.84M GPU-hours correspond to roughly \$60-120M of rented compute, before staff, storage, networking, and failed experiments. That is consistent with independent estimates: Epoch AI puts GPT-4’s amortized hardware-plus-energy cost around \$40M and Gemini Ultra’s around \$30M, with frontier training costs growing about 2.4x per year. The ceiling on this curve is increasingly physical: xAI’s Colossus cluster stood up 100,000 H100s in 122 days, and SemiAnalysis estimates a cluster of that size draws about 150 MW, five times the largest classical supercomputer of its day. Power and the buildings to house it, not chip supply alone, are becoming the binding constraint on the next order of magnitude.

Serving costs are a throughput problem: dollars per token equals chip-hour price divided by tokens per chip-hour, and tokens per chip-hour is set by the roofline. The frontier figure above makes it concrete: at an illustrative \$3 per H100-hour, that 8-GPU node produces decode tokens at about \$35 per million at batch 1, and \$0.43 per million at batch 367. Same chips, same model, an 80x price difference decided entirely by where on the latency-throughput curve the operator runs. Hold that against a July 2026 pricing page (Claude Opus 4.8 at \$5 in / \$25 out per million tokens; GPT-5.5 at \$5 / \$30; Gemini 3.1 Pro at \$2 / \$12) and the structure snaps into focus:

  • Output tokens cost 5-6x input tokens across all three labs, because input tokens are processed by compute-bound prefill while output tokens are minted one at a time by memory-bound decode. The price gap is the roofline gap, plus the KV-cache burden long generations carry.
  • Prompt caching discounts (cached input at a tenth of the price on Anthropic’s page) pass along prefill the provider skipped by reusing stored KV blocks, exactly as described in the KV cache post.
  • Batch APIs at half price sell the far end of the latency-throughput curve: hand over your latency requirement and the provider fills quiet troughs and fatter batches with your tokens.

None of this required knowing anything about the models’ quality, only where their tokens sit relative to a ratio printed on a spec sheet.

Why both GPUs and TPUs exist

Given identical physics, why two ecosystems? Partly history (TPUs were Google’s 2015 bet that neural network inference deserved its own silicon; GPUs arrived via graphics and kept the generality), and partly a genuine architectural fork. NVIDIA sells flexibility: 132 independently programmable cores, a switched any-to-any fabric, and the CUDA software ecosystem (the programming layer nearly all AI code targets) that runs everyone’s weird new idea on day one. Google builds a vertically integrated pipeline for one customer’s workloads: enormous compiler-scheduled matrix units and switchless torus pods that make 9,000-chip training jobs look, to software, like one big machine. The Gemini technical report describes training across multiple 4,096-chip TPUv4 SuperPods spanning datacenters, and Google’s Ironwood generation is pitched explicitly at what this post calls the hostile case, the “age of inference,” with its 192 GB of HBM at 7.37 TB/s sized for serving. The direction of travel on both sides is the same, and it is the roofline’s direction: interconnect domains keep growing (NVL72 racks; ever-larger pods), HBM stacks keep deepening, and inference-first chip variants keep appearing, because the memory-bound phase is where the tokens, and the money, now are.

What to take away

  • One ratio explains the stack. Chips do hundreds of FLOPs in the time they can fetch one byte. Every workload is either above that line (compute-bound, efficient) or below it (memory-bound, wasteful), and \(T \approx \max(\text{FLOPs}/\text{rate},\ \text{bytes}/\text{bandwidth})\) rates any layer of the system once you plug in the right bandwidth.
  • Intensity is batch size. A transformer matmul’s FLOPs-per-byte is roughly the number of tokens passing through it together. Training and prefill bring thousands and fly; decode brings one per user and starves, so serving is the art of stacking strangers’ tokens.
  • Training’s hard part is coordination, not utilization. \(6ND\) FLOPs at frontier scale means thousands of chip-years, sliced by data, tensor, pipeline, and expert parallelism so each kind of chatter rides a link that can afford it, and even the best runs keep only ~40% of peak.
  • Serving’s hard part is a trade you can draw. Batching buys ~80x throughput for ~4.5x latency in the model above; continuous batching and paged KV memory are what let real systems live near that frontier.
  • Pricing pages are rooflines in a suit. The output/input price gap, prompt-caching discounts, and batch tiers correspond one-to-one to decode’s memory-boundedness, skipped prefill, and the latency-throughput curve.

The next time a response streams in at forty tokens a second, you can narrate the physics: your prompt rode a compute-bound wave through prefill, your tokens are now sharing weight-reads with a few hundred strangers, and the price per token was mostly decided by a ratio two hardware architects chose years before the model existed.

Sources and further reading

Citation Information

If you find this content useful, please cite this work as:

Bhana, Nish. "How LLMs Scale: FLOPs, Bytes, and the Price of a Token". Nish Blog (July 2026). https://www.nishbhana.com/Scaling-LLMs/

Or use the BibTeX citation:

@article{bhana2026scalingllms,
  title   = {How LLMs Scale: FLOPs, Bytes, and the Price of a Token},
  author  = {Bhana, Nish},
  journal = {nishbhana.com},
  year    = {2026},
  month   = {July},
  url     = {https://www.nishbhana.com/Scaling-LLMs/}
}

Related Posts 🔥

The Evolution of Attention: One Mechanism, Four Escape Routes
NLP Jul 2026
Knowledge Distillation: How Frontier Intelligence Gets Cheap
NLP Jul 2026
Diffusion Models: Learning to Create by Learning to Destroy
ML Jul 2026
The KV Cache: Why LLMs Never Do the Same Work Twice
NLP Jul 2026
Reasoning Models: Teaching LLMs to Think Before They Answer
NLP Jun 2026
Mixed Effects Models: Partial Pooling for Data That Comes in Groups
ML Sep 2025

x.com, Facebook