Serving Is a Separate Problem: How vLLM Runs LLMs Fast and Cheap

Running one token through a language model isn't one big GPU operation. It's hundreds to thousands of tiny ones, and each of those has to be launched by the CPU first. During decode, those little operations finish so fast the GPU starves waiting for the CPU to queue the next one. CUDA Graphs fix this by recording the whole repeating sequence once and replaying it with a single launch, so the GPU stays fed. Same math, same result, far less idle time.

The single idea the entire craft rests on is this: the model is a commodity, and the system around it is the job. A raw frontier model is a brilliant intern with amnesia, no hands, and no way to check its own work. The AI engineer gives it hands through tools, a memory across runs, a way to verify itself through evals, and a loop that keeps working until the task is done. That wrapper — not the model weights — is the thing you get paid to build. Learn to build it and you're employable.

Scaled dot-product attention is not a mysterious black box. It's four elementary operations — matrix multiplication, transpose, scaling by a constant, and softmax — plus three learned projections that produce queries, keys, and values. You can write it in about a hundred lines of plain C with no autograd and no libraries. Once you see it as loops instead of magic, you reason about cost, memory, and failure modes from first principles rather than treating the model as a sealed device.

An LLM engine like vLLM exists because serving a model well is its own systems problem, distinct from training it. Text generation is autoregressive and memory-bound, and the key-value cache is the real capacity limit on how many users fit on a GPU. vLLM answers three questions: which requests run next, how a batch executes, and where each request's cache lives. Its headline move is treating GPU memory like virtual memory, which slashes waste and unlocks bigger batches.

Most technical courses go bottom-up: math, then theory, then maybe you build something months later, by which point your motivation is gone. fast.ai flips it. You train a real, state-of-the-art model in lesson one, and only then peel back the layers to see how it learned. The engine that makes this possible for beginners is transfer learning, and that same "inherit capability instead of building it from scratch" move is the single most important pattern in modern AI engineering.

A system called LingBot-Map turns a single moving camera into a live 3D map, a point cloud plus the camera's path through space, at roughly 20 frames per second and staying coherent across sequences of 10,000-plus frames. The magic is two ideas working together. First, it's feed-forward: one fast pass per frame instead of a slow optimization loop. Second, it borrows paged KV-cache attention from LLM serving so its memory doesn't explode over long streams. Seeing a language-model trick reappear in 3D vision is the whole point.

A raw model call is stateless and blind — it can't remember you, can't look anything up, and can't act. An agent is that model wrapped in the parts that fix each gap: tools, instructions, memory, knowledge, storage, and reasoning. A serious agent framework hands you those parts pre-assembled, plus a runtime to deploy them. The single most useful idea is that a framework is really a checklist of what every production agent needs — so reading one teaches you the whole shape of the problem.

Turning one photo into an on-model video clip is not one model doing something clever. It is two models chained together: Whisk makes a consistent still image, and Veo animates it into about eight seconds of motion. The entire skill is managing the handoff between them: matching the aspect ratio, carrying the seed and prompt across the boundary, and conditioning one model on the other's output. The moat is the pipeline, not the button.

The naive scaling law says more tokens equals more intelligence. The smarter move is to scale on three axes at once: token efficiency, context length, and agent swarms. And here's the reframe that ties it together, high-quality training data is finite, so squeezing more learning out of each token doesn't just save compute, it raises the ceiling of how smart a model can get. When data is the bottleneck, efficiency literally is capability.

A single language model, asked to judge a hard decision, quietly averages every perspective it could hold into one cautious "on one hand, on the other hand" answer. An agent swarm refuses that. You spin up several experts, each pinned to a conflicting bias, have them analyze the same decision independently, then reconcile them without smoothing the clash. The disagreement is not noise to be cleaned up. It is the signal that shows you exactly where your decision is genuinely risky.