
S2 E19 — Design an LLM Inference Service: Batching, Admission Control, and Fairness
A short question waits behind a prompt containing an entire book. Daniel and Maya design a shared inference service that uses GPUs efficiently while protecting latency and tenant fairness.
Batch formation, request length, resource budgets, admission, and queueing shape the working path. Slow consumers and worker failures complicate an answer already streaming to a client. The final design explains which throughput gains are worth their waiting time and what can still be promised after partial output.
Chapters
00:00:00 Welcome and a blocked short request
00:02:45 Latency contracts and request ownership
00:06:17 A working inference request path
00:10:01 Batching, long prompts, and latency
00:13:54 Admission budgets and tenant fairness
00:18:22 Worker failure during streamed output
00:22:25 Monitoring and safe service changes
00:29:30 Final inference flow and tradeoffs
Daniel is the interviewer, voiced by Liam. Maya is the engineer, voiced by Bella. AI narration at a calm pace, with three thinking pauses.
Opening music: Stealth Groover by Kevin MacLeod (incompetech.com). Licensed under Creative Commons Attribution 4.0: https://creativecommons.org/licenses/by/4.0/ . Edited excerpt with EQ, volume automation, a smooth fade, and narration.
Więcej odcinków z kanału "System Design on the Road"



Nie przegap odcinka z kanału “System Design on the Road”! Subskrybuj bezpłatnie w aplikacji GetPodcast.








