Software Engineer Interview Prep Podcast podcast

Kafka in Production: Failure Scenarios for Staff-Level Interviews

0:00
1:00:19
Rewind 15 seconds
Fast Forward 15 seconds

A single misconfigured setting in Kafka doesn't just slow a page down. It can silently vaporize financial transactions. In this deep dive we walk through real production failure scenarios and the exact reasoning interviewers look for in staff-level system design interviews.


You'll learn:

- Why acks=1 loses data, and the four configuration layers for zero data loss

- Consistency vs availability: why a healthy cluster refuses writes on purpose

- Pushing from 400,000 to 2 million events/s with batching, linger.ms and compression

- How retries reorder a debit and a credit, and how idempotent producers fix it

- Rebalance storms, cooperative sticky assignment and static membership

- Hot partitions and the irreducible trade-off between ordering and scale

- How deleting and recreating a topic silently skips hours of data

- A 3 AM role-play: 85 under-replicated partitions, a disk at 96%, and why you never restart the broker

- Exactly-once with Kafka transactions, and exactly-once into PostgreSQL

- Tiered retry topics and dead letter queues

- Split brain, KRaft quorums and vanishing tombstones

- System design at scale: multi-tenant topics, quotas, event-driven sagas and change data capture with Debezium


Ideal for backend and platform engineers preparing for senior and staff-level interviews.


Chapters

00:00 Distributed systems have no X-ray

01:08 Why Kafka interviews matter

02:19 acks=1 and the 200 missing payments

03:27 Four layers of zero data loss

05:21 The CAP theorem trade-off

06:31 Throughput: taxis vs buses

07:42 batch.size, linger.ms and compression

09:38 When retries reorder messages

11:38 Idempotent producers

12:48 Message keys and partition ordering

13:12 Flash sale: buffer exhausted

14:22 Decoupling with a local buffer and circuit breaker

16:18 Rebalance storms

18:13 Eager vs cooperative sticky rebalancing

19:25 Static group membership

20:12 Hot partitions and the hot key problem

23:18 Silent data loss after recreating a topic

25:20 auto.offset.reset and topic versioning

26:08 3 AM: 85 under-replicated partitions

28:54 Why you never restart the broker

29:42 Disk at 96%: the 15-minute response

31:36 Throttled reassignment, Cruise Control, tiered storage

33:36 Duplicates in consume-transform-produce

34:47 Kafka transactions

37:11 Exactly-once into PostgreSQL

39:56 Head-of-line blocking from retries

41:08 Tiered retry topics and DLQs

42:42 Split brain

44:38 KRaft and quorum math

45:27 Log compaction tombstones

48:33 Multi-tenant topic design

50:07 Tenant tiers and client quotas

51:40 Event-driven sagas

53:15 Compensating transactions

54:51 Change data capture with Debezium

57:13 Recap

58:46 Will offsets still matter in five years?


#Kafka #SystemDesign #DistributedSystems #SoftwareEngineering #InterviewPrep #BackendEngineering


Interview Prep Podcast

More episodes from "Software Engineer Interview Prep Podcast"