System Design on the Road podkast

S2 E19 — Design an LLM Inference Service: Batching, Admission Control, and Fairness

0:00
33:03
Do tyłu o 15 sekund
Do przodu o 15 sekund

A short question waits behind a prompt containing an entire book. Daniel and Maya design a shared inference service that uses GPUs efficiently while protecting latency and tenant fairness.


Batch formation, request length, resource budgets, admission, and queueing shape the working path. Slow consumers and worker failures complicate an answer already streaming to a client. The final design explains which throughput gains are worth their waiting time and what can still be promised after partial output.


Chapters

00:00:00 Welcome and a blocked short request

00:02:45 Latency contracts and request ownership

00:06:17 A working inference request path

00:10:01 Batching, long prompts, and latency

00:13:54 Admission budgets and tenant fairness

00:18:22 Worker failure during streamed output

00:22:25 Monitoring and safe service changes

00:29:30 Final inference flow and tradeoffs


Daniel is the interviewer, voiced by Liam. Maya is the engineer, voiced by Bella. AI narration at a calm pace, with three thinking pauses.


Opening music: Stealth Groover by Kevin MacLeod (incompetech.com). Licensed under Creative Commons Attribution 4.0: https://creativecommons.org/licenses/by/4.0/ . Edited excerpt with EQ, volume automation, a smooth fade, and narration.

Więcej odcinków z kanału "System Design on the Road"