How Linux is built with Greg Kroah-Hartman

Supported by Our Partners• Graphite — The AI developer productivity platform. • Sentry — Error and performance monitoring for developers.—Reddit’s native mobile apps are more complex than most of us would assume: both the iOS and Android apps are about 2.5 million lines of code, have 500+ screens, and a total of around 200 native iOS and Android engineers work on them. But it wasn’t always like this.In 2021, Reddit started to double down on hiring native mobile engineers, and they quietly rebuilt the Android and iOS apps from the ground up. The team introduced a new tech stack called the “Core Stack” – all the while users remained largely unaware of the changes. What drove this overhaul, and how did the team pull it off?In this episode of The Pragmatic Engineer, I’m joined by three engineers from Reddit’s mobile platform team who led this work: Lauren Darcey (Head of Mobile Platform), Brandon Kobilansky (iOS Platform Lead), and Eric Kuck (Principal Android Engineer). We discuss how the team transitioned to a modern architecture, revamped their testing strategy, improved developer experience – while they also greatly improved the app’s user experience. We also get into: • How Reddit structures its mobile teams—and why iOS and Android remain intentionally separate • The scale of Reddit’s mobile codebase and how it affects compile time• The shift from MVP to MVVM architecture• Why Reddit took a bet on Jetpack Compose, but decided (initially) against using SwiftUI• How automated testing evolved at Reddit • Reddit’s approach to server-driven-mobile-UI• What the mobile platforms team looks for in a new engineering hire• Reddit’s platform team’s culture of experimentation and embracing failure • And much more!If you are interested in large-scale rewrites or native mobile engineering challenges: this episode is for you.—Timestamps(00:00) Intro(02:04) The scale of the Android code base(02:42) The scale of the iOS code base(03:26) What the compile time is for both Android and iOS(05:33) The size of the mobile platform teams (09:00) Why Reddit has so many mobile engineers (11:28) The different types of testing done in the mobile platform (13:20) The benefits and drawbacks of testing (17:00) How Eric, Brandon, and Lauren use AI in their workflows(20:50) Why Reddit grew its mobile teams in 2021(26:50) Reddit’s modern tech stack, Corestack (28:48) Why Reddit shifted from MVP architecture to MVVM(30:22) The architecture on the iOS side(32:08) The new design system(30:55) The impact of migrating from Rust to GraphQL(38:20) How the backend drove the GraphQL migration and why it was worth the pain(43:17) Why the iOS team is replacing SliceKit with SwiftUI(48:08) Why the Android team took a bet on Compose (51:25) How teams experiment with server-driven UI—when it worked, and when it did not(54:30) Why server-driven UI isn’t taking off, and why Lauren still thinks it could work(59:25) The ways that Reddit’s modernization has paid off, both in DevX and UX(1:07:15) The overall modernization philosophy; fixing pain points (1:09:10) What the mobile platforms team looks for in a new engineering hire (1:16:00) Why startups may be the best place to get experience (1:17:00) Why platform teams need to feel safe to fail (1:20:30) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• The platform and program split at Uber• Why and how Notion went native on iOS and Android• Paying down tech debt • Cross-platform mobile development—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• WorkOS — The modern identity platform for B2B SaaS.•⁠ Modal⁠ — The cloud platform for building AI applications• Vanta — Automate compliance and simplify security with Vanta.—What is it like to work at Amazon as a software engineer? Dave Anderson spent over 12 years at Amazon working closely with engineers on his teams: starting as an Engineering Manager (or, SDM in Amazon lingo) and eventually becoming a Director of Engineering. In this episode, he shares a candid look into Amazon’s engineering culture—from how promotions work to why teams often run like startups.We get into the hiring process, the role of bar raisers, the pros and cons of extreme frugality, and what it takes to succeed inside one of the world’s most operationally intense companies. We also look at how engineering actually works day to day at Amazon—from the tools teams choose to the way they organize and deliver work. We also discuss:• The levels at Amazon, from SDE L4 to Distinguished Engineer and VP• Why engineering managers at Amazon need to write well• The “Bar Raiser” role in Amazon interview loops • Why Amazon doesn’t care about what programming language you use in interviews• Amazon’s oncall process• The pros and cons of Amazon’s extreme frugality • What to do if you're getting negative performance feedback• The importance of having a strong relationship with your manager• The surprising freedom Amazon teams have to choose their own stack, tools, and ways of working – and how a team chose to use Lisp (!)• Why startups love hiring former Amazon engineers• Dave’s approach to financial independence and early retirement• And more!—Timestamps(00:00) Intro(02:08) An overview of Amazon’s levels for devs and engineering managers(07:04) How promotions work for developers at Amazon, and the scope of work at each level(12:29) Why managers feel pressure to grow their teams(13:36) A step-by-step, behind-the-scenes glimpse of the hiring process (23:40) The wide variety of tools used at Amazon(26:27) How oncall works at Amazon(32:06) The general approach to handling outages (severity 1-5)(34:40) A story from Uber illustrating the Amazon outage mindset(37:30) How VPs assist with outages(41:38) The culture of frugality at Amazon  (47:27) Amazon’s URA target—and why it’s mostly not a big deal (53:37) How managers handle the ‘least effective’ employees(58:58) Why other companies are also cutting lower performers(59:55) Dave’s advice for engineers struggling with performance feedback (1:04:20) Why good managers are expected to bring talent with them to a new org(1:06:21) Why startups love former Amazon engineers(1:16:09) How Dave planned for an early retirement (1:18:10) How a LinkedIn post turned into Scarlet Ink —The Pragmatic Engineer deepdives relevant for this episode:• Inside Amazon’s engineering culture• A day in the life of a senior manager at Amazon• Amazon’s Operational Plan process with OP1 and OP2—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners•⁠ CodeRabbit⁠⁠ — Cut code review time and bugs in half. Use the code PRAGMATIC to get one month free.•⁠ Modal⁠ — The cloud platform for building AI applications.—How will AI tools change software engineering? Tools like Cursor, Windsurf and Copilot are getting better at autocomplete, generating tests and documentation. But what is changing, when it comes to software design?Stanford professor John Ousterhout thinks not much. In fact, he believes that great software design is becoming even more important as AI tools become more capable in generating code. In this episode of The Pragmatic Engineer, John joins me to talk about why design still matters and how most teams struggle to get it right. We dive into his book A Philosophy of Software Design, unpack the difference between top-down and bottom-up approaches, and explore why some popular advice, like writing short methods or relying heavily on TDD, does not hold up, according to John.We also explore: • The differences between working in industry vs. academia • Why John believes software design will become more important as AI capabilities expand• The top-down and bottoms-up design approaches – and why you should use both• John’s “design it twice” principle• Why deep modules are essential for good software design • Best practices for special cases and exceptions• The undervalued trait of empathy in design thinking• Why John advocates for doing some design upfront• John’s criticisms of the single-responsibility principle, TDD, and why he’s a fan of well-written comments • And much more!As a fun fact: when we recorded this podcast, John was busy contributing to the Linux kernel: adding support to the Homa Transport Protocol – a protocol invented by one of his PhD students. John wanted to make this protocol available more widely, and is putting in the work to do so. What a legend! (We previously covered how Linux is built and how to contribute to the Linux kernel)—Timestamps(00:00) Intro (02:00) Why John transitioned back to academia(03:47) Working in academia vs. industry (07:20) Tactical tornadoes vs. 10x engineers(11:59) Long-term impact of AI-assisted coding(14:24) An overview of software design(15:28) Why TDD and Design Patterns are less popular now (17:04) Two general approaches to designing software (18:56) Two ways to deal with complexity (19:56) A case for not going with your first idea (23:24) How Uber used design docs(26:44) Deep modules vs. shallow modules(28:25) Best practices for error handling(33:31) The role of empathy in the design process(36:15) How John uses design reviews (38:10) The value of in-person planning and using old-school whiteboards (39:50) Leading a planning argument session and the places it works best(42:20) The value of doing some design upfront (46:12) Why John wrote A Philosophy of Software of Design (48:40) An overview of John’s class at Stanford(52:20) A tough learning from early in Gergely’s career (55:48) Why John disagrees with Robert Martin on short methods(1:10:40) John’s current coding project in the Linux Kernel (1:14:13) Updates to A Philosophy of Software Design in the second edition(1:19:12) Rapid fire round(1:01:08) John’s criticisms of TDD and what he favors instead (1:05:30) Why John supports the use of comments and how to use them correctly(1:09:20) How John uses ChatGPT to help explain code in the Linux Kernel—The Pragmatic Engineer deepdives relevant for this episode:• Engineering Planning with RFCs, Design Documents and ADRs• Paying down tech debt• Software architect archetypes• Building Bluesky: a distributed social network—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• Swarmia — The engineering intelligence platform for modern software organizations.• Sentry — Error and performance monitoring for developers.—Why did Meta build its own internal developer tooling instead of using industry-standard solutions like GitHub? Tomas Reimers, former Meta engineer and co-founder of Graphite, joins the show to talk about Meta's custom developer tools – many of which were years ahead of the industry.From Phababricator to Sandcastle and Butterflybot, Tomas shares examples of Meta’s internal tools that transformed developer productivity at the tech giant. Why did working with stacked diffs and using monorepos become best practices at Meta? How are these practices influencing the broader industry? Why are code reviews and testing looking to become even more critical as AI transforms how we write software? We answer these, and also discuss:• Meta's custom internal developer tools• Why more tech companies are transitioning from polyrepos to monorepos• A case for different engineering constraints within the same organization• How stacked diffs solve the code review bottleneck• Graphite’s origin story and pivot to their current product • Why code reviews will become a lot more important, the more we use AI coding tools• Tomas’s favorite engineering metric • And much more!—Timestamps(00:00) Intro(02:00) An introduction to Meta’s in-house tooling (05:07) How Meta’s integrated tools work and who built the tools(10:20) An overview of the rules engine, Herald (12:20) The stages of code ownership at Facebook and code ownership at Google and GitHub(14:39) Tomas’s approach to code ownership (16:15) A case for different constraints within different parts of an organization (18:42) The problem that stacked diffs solve for (25:01) How larger companies drive innovation, and who stacking diffs not for (30:25) Monorepos vs. polyrepos and why Facebook is transitioning to a monorepo(35:31) The advantages of monorepos and why GitHub does not support them (39:55) AI’s impact on software development (42:15) The problems that AI creates, and possible solutions(45:25) How testing might change and the testing AI coding tools are already capable of (48:15) How developer accountability might be a way to solve bugs and bad AI code(53:20) Why stacking hasn’t caught on and Graphite’s work (57:10) Graphite’s origin story (1:01:20) Engineering metrics that matter (1:06:07) Learnings from building a company for developers (1:08:41) Rapid fire round(1:12:41) Closing—The Pragmatic Engineer deepdives relevant for this episode:• Stacked Diffs (and why you should know about them)• Inside Meta’s engineering culture• Shipping to production• How Uber is measuring engineering productivity—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• Graphite — The AI developer productivity platform. • Sonar —  Code quality and code security for ALL code. • Chronosphere — The observability platform built for control.—How do you take a new product idea, and turn it into a successful product? Figma Slides started as a hackathon project a year and a half ago – and today it’s a full-on product, with more than 4.5M slide decks created by users. I’m joined by two founding engineers on this project: Jonathan Kaufman and Noah Finer.In our chat, Jonathan and Noah pull back the curtain on what it took to build Figma Slides. They share engineering challenges faced, interesting engineering practices utilized, and what it's like working on a product used by millions of designers worldwide.We talk about:• An overview of Figma Slides• The tech stack behind Figma Slides• Why the engineering team built grid view before single slide view• How Figma ensures that all Figma files look the same across browsers• Figma’s "vibe testing" approach• How beta testing helped experiment more• The “all flags on”, “all flags off” testing approach• Engineering crits at Figma• And much more!—Timestamps(00:00) Intro(01:45) An overview of Figma Slides and the first steps in building it(06:41) Why Figma built grid view before single slide view(10:00) The next steps of building UI after grid view (12:10) The team structure and size of the Figma Slides team (14:14) The tech stack behind Figma Slides(15:31) How Figma uses C++ with bindings (17:43) The Chrome debugging extension used for C++ and WebAssembly (21:02) An example of how Noah used the debugging tool(22:18) Challenges in building Figma Slides (23:15) An explanation of multiplayer cursors (26:15) Figma’s philosophy of building interconnected products—and the code behind them(28:22) An example of a different mouse behavior in Figma (33:00) Technical challenges in developing single slide view (35:10) Challenges faced in single-slide view while maintaining multiplayer compatibility(40:00) The types of testing used on Figma Slides(43:42) Figma’s zero bug policy (45:30) The release process, and how engineering uses feature flags (48:40) How Figma tests Slides with feature flags enabled and then disabled(51:35) An explanation of eng crits at Figma (54:53) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• Inside Figma’s engineering culture• Quality Assurance across the tech industry• Shipping to production• Design-first software engineering—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• WorkOS — The modern identity platform for B2B SaaS.• Vanta — Automate compliance and simplify security with Vanta.—Linux is the most widespread operating system, globally – but how is it built? Few people are better to answer this than Greg Kroah-Hartman: a Linux kernel maintainer for 25 years, and one of the 3 Linux Kernel Foundation Fellows (the other two are Linus Torvalds and Shuah Khan). Greg manages the Linux kernel’s stable releases, and is a maintainer of multiple kernel subsystems.We cover the inner workings of Linux kernel development, exploring everything from how changes get implemented to why its community-driven approach produces such reliable software. Greg shares insights about the kernel's unique trust model and makes a case for why engineers should contribute to open-source projects. We go into:• How widespread is Linux?• What is the Linux kernel responsible for – and why is it a monolith?• How does a kernel change get merged? A walkthrough• The 9-week development cycle for the Linux kernel• Testing the Linux kernel• Why is Linux so widespread?• The career benefits of open-source contribution• And much more!—Timestamps(00:00) Intro(02:23) How widespread is Linux?(06:00) The difference in complexity in different devices powered by Linux (09:20) What is the Linux kernel?(14:00) Why trust is so important with the Linux kernel development(16:02) A walk-through of a kernel change(23:20) How Linux kernel development cycles work(29:55) The testing process at Kernel and Kernel CI (31:55) A case for the open source development process(35:44) Linux kernel branches: Stable vs. development(38:32) Challenges of maintaining older Linux code (40:30) How Linux handles bug fixes(44:40) The range of work Linux kernel engineers do (48:33) Greg’s review process and its parallels with Uber’s RFC process(51:48) Linux kernel within companies like IBM(53:52) Why Linux is so widespread (56:50) How Linux Kernel Institute runs without product managers (1:02:01) The pros and cons of using Rust in Linux kernel (1:09:55) How LLMs are utilized in bug fixes and coding in Linux (1:12:13) The value of contributing to the Linux kernel or any open-source project (1:16:40) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:What TPMs do and what software engineers can learn from themThe past and future of modern backend practicesBackstage: an open-source developer portal—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• Sentry — Error and performance monitoring for developers.• The Software Engineer’s Guidebook: Written by me (Gergely) – now out in audio form as well.—In today’s episode of The Pragmatic Engineer, I am joined by former Uber colleague, Gautam Korlam. Gautam is the Co-Founder of Gitar, an agentic AI startup that automates code maintenance. Gautam was mobile engineer no. 9 at Uber and founding engineer for the mobile platform team – and so he learned a few things about scaling up engineering teams.We talk about:• How Gautam accidentally deleted Uber’s Java monorepo – really!• Uber's unique engineering stack and why custom solutions like SubmitQueue were built in-house• Monorepo: the benefits and downsides of this approach• From Engineer II to Principal Engineer at Uber: Gautam’s career trajectory• Practical strategies for building trust and gaining social capital • How the platform team at Uber operated with a product-focused mindset• Vibe coding: why it helps with quick prototyping• How AI tools are changing developer experience and productivity• Important skills for devs to pick up to remain valuable as AI tools spread• And more!—Timestamps(00:00) Intro(02:11) How Gautam accidentally deleted Uber’s Java Monorepo(05:40) The impact of Gautam’s mistake(06:35) Uber’s unique engineering stack(10:15) Uber’s SubmitQueue(12:44) Why Uber moved to a monorepo(16:30) The downsides of a monorepo(18:35) Measurement products built in-house (20:20) Measuring developer productivity and happiness (22:52) How Devpods improved developer productivity (27:37) The challenges with cloud development environments(29:10) Gautam’s journey from Eng II to Principal Engineer(32:00) Building trust and gaining social capital (36:17) An explanation of Principal Engineer at Uber—and the archetypes at Uber (45:07) The platform and program split at Uber(48:15) How Gautam and his team supported their internal users (52:50) Gautam’s thoughts on developer productivity (59:10) How AI enhances productivity, its limitations, and the rise of agentic AI(1:04:00) An explanation of Vibe coding(1:07:34) An overview of Gitar and all it can help developers with (1:10:44) Top skills to cultivate to add value and stay relevant(1:17:00) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• The Platform and Program split at Uber• How Uber is measuring engineering productivity• Inside Uber’s move to the Cloud• How Uber built its observability platform• Software Architect Archetypes—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• WorkOS — The modern identity platform for B2B SaaS.• The Software Engineer’s Guidebook: Written by me (Gergely) – now out in audio form as well• Augment Code — AI coding assistant that pro engineering teams love—Not many people know that I have a brother: Balint Orosz. Balint is also in tech, but in many ways, is the opposite of me. While I prefer working on backend and business logic, he always thrived in designing and building UIs. While I opted to work at more established companies, he struck out on his own and started his startup, Distinction. And yet, our professional paths have crossed several times: at one point in time I accepted an offer to join Skyscanner as a Principal iOS Engineer – and as part of the negotiation, I added a clause to my contrac that I will not report directly or indirectly to the Head of Mobile: who happened to be my brother, thanks to Skyscanner acquiring his startup the same month that Skyscanner made an offer to hire me.Today, Balint is the founder and CEO of Craft, a beloved text editor known for its user-friendly interface and sleek design – an app that Apple awarded the prestigious Mac App of the Year in 2021.In our conversation, we explore how Balint approaches building opinionated software with an intense focus on user experience. We discuss the lessons he learned from his time building Distinction and working at Skyscanner that have shaped his approach to Craft and its development.In this episode, we discuss:• Balint’s first startup, Distinction, and his time working for Skyscanner after they acquired it• A case for a balanced engineering culture with both backend and frontend priorities • Why Balint doesn’t use iOS Auto Layout• The impact of Craft being personal software on front-end and back-end development• The balance between customization and engineering fear in frontend work• The resurgence of local-first software and its role in modern computing• The value of building a physical prototype • How Balint uses GenAI to assist with complicated coding projects • And much more!—Timestamps(00:00) Intro(02:13) What it’s like being a UX-focused founder (09:00) Why it was hard to gain recognition at Skyscanner (13:12) Takeaways from Skyscanner that Balint brought to Craft (16:50) How frameworks work and why they aren’t always a good fit(20:35) An explanation of iOS Auto Layout and its pros and cons (23:13) Why Balint doesn’t use Auto Layout (24:23) Why Craft has one code base (27:46) Craft’s unique toolbar features and a behind the scenes peek at the code (33:15) Why frontend engineers have fear around customization (37:11) How Craft’s design system differs from most companies (42:33) Behaviors and elements Craft uses rather than having a system for everything (44:12) The back and frontend architecture in building personal software (48:11) Shifting beliefs in personal computing (50:15) The challenges faced with operating system updates (50:48) The resurgence of local-first software(52:31) The value of opinionated software for consumers (55:30) Why Craft’s focus is on the user’s emotional experience(56:50) The size of Craft’s engineering department and platform teams(59:20) Why Craft moves faster with smaller teams(1:01:26) Balint’s advice for frontend engineers looking to demonstrate value (1:04:35) Balint’s breakthroughs using GenAI(1:07:50) Why Balint still writes code(1:09:44) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• The AI hackathon at Craft Docs• Engineering career paths at Big Tech and scaleups• Thriving as a Founding Engineer: lessons from the trenches• The past and future of modern backend practices—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• WorkOS — The modern identity platform for B2B SaaS.• Graphite — The AI developer productivity platform. • Formation — Level up your career and compensation with Formation.—In today’s episode of The Pragmatic Engineer, I am joined by a senior software engineer and cartoonist, Manu Cornet. Manu spent over a decade at Google, doing both backend and frontend development. He also spent a year and a half at Twitter before Elon Musk purchased it and rebranded it to X. But what Manu is most known for are his hilarious internet comics about the tech world, including his famous org chart comic from 2011 about Facebook, Apple, Amazon, and Microsoft.In today’s conversation, we explore many of his comics, discuss the meaning behind them, and talk about the following topics: • The viral org chart comic that captured the structure of Big Tech companies• Why Google is notorious for confusing product names• The comic that ended up on every door at Google• How Google’s 20% time fostered innovation—and what projects came from it• How one of Manu’s comics predicted Google Stadia’s failure—and the reasons behind it• The value of connecting to users directly • Twitter’s climate before and after Elon Musk’s acquisition and the mass layoffs that followed• And more!—Timestamps(00:00) Intro(02:01) Manu’s org structure comic (07:10) Manu’s “Who Sues Who” comic(09:15) Google vs. Amazon comic(14:10) Confusing names at Google(20:00) Different approaches to sharing information within companies(22:20) The two ways of doing things at Google(25:15) Manu’s code reviews comic(27:45) The comic that was printed on every single door of Google(30:55) An explanation of 20% at Google(36:00) Gmail Labs and Google Stadia(41:36) Manu’s time at Twitter and the threat of Elon Musk buying(47:07) How Manu helped Gergely with a bug on Twitter(49:05) Musk’s acquirement of Twitter and the resulting layoffs(59:00) Manu’s comic about his disillusionment with Twitter and Google(1:02:37) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• How Manu creates comics• Consolidating technologies• Is Big Tech becoming more cutthroat?—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Supported by Our Partners• WorkOS — The modern identity platform for B2B SaaS• CodeRabbit — Cut code review time and bugs in half• Augment Code — AI coding assistant that pro engineering teams love—How do you architect a live streaming system to deal with more load than it’s ever been done before? Today, we hear from an architect of such a system: Ashutosh Agrawal, formerly Chief Architect of JioCinema (and currently Staff Software Engineer at Google DeepMind.)We take a deep dive into video streaming architecture, tackling the complexities of live streaming at scale (at tens of millions of parallel streams) and the challenges engineers face in delivering seamless experiences. We talk about the following topics: • How large-scale live streaming architectures are designed• Tradeoffs in optimizing performance• Early warning signs of streaming failures and how to detect them• Why capacity planning for streaming is SO difficult• The technical hurdles of streaming in APAC regions• Why Ashutosh hates APMs (Application Performance Management systems)• Ashutosh’s advice for those looking to improve their systems design expertise• And much more!—Timestamps(00:00) Intro(01:28) The world record-breaking live stream and how support works with live events(05:57) An overview of streaming architecture(21:48) The differences between internet streaming and traditional television.l(22:26) How adaptive bitrate streaming works(25:30) How throttling works on the mobile tower side (27:46) Leading indicators of streaming problems and the data visualization needed(31:03) How metrics are set (33:38) Best practices for capacity planning (35:50) Which resources are planned for in capacity planning (37:10) How streaming services plan for future live events with vendors(41:01) APAC specific challenges(44:48) Horizontal scaling vs. vertical scaling (46:10) Why auto-scaling doesn’t work(47:30) Concurrency: the golden metric to scale against(48:17) User journeys that cause problems (49:59) Recommendations for learning more about video streaming (51:11) How Ashutosh learned on the job(55:21) Advice for engineers who would like to get better at systems(1:00:10) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• Software architect archetypes https://newsletter.pragmaticengineer.com/p/software-architect-archetypes • Engineering leadership skill set overlaps https://newsletter.pragmaticengineer.com/p/engineering-leadership-skillset-overlaps • Software architecture with Grady Booch https://newsletter.pragmaticengineer.com/p/software-architecture-with-grady-booch—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe