Adventures in DevOps

Share Episode                  We sit down with veteran Principal Engineer Justin Abrahms to discuss why having a clear engineering scoreboard is critical to winning the software delivery game. Drawing on his experience scaling deployment pipelines for payment engineers, platform engineering teams, and globocorps, Justin explains the simple truth: when you deliberately do the right things—embracing continuous delivery, small batch sizes, and tight feedback loops—you get the right outcomes. Yet despite these battle-tested practices existing for decades, our industry suffers from an "Eternal September" fueled by terrible onramps.         We dissect the performative theater of modern engineering, taking aim at the severe lack of intentionality in code reviews. Too many peer reviews devolve into superficial nitpicking rather than educational, inquisitive assessments of long-term maintainability. Justin shares lessons learned from working alongside Trevor Foucher (author of The Art of Readable Code) at Google, demonstrating how high-signal code reviews should actively educate developers and de-risk downstream systems before code ever hits production.         Finally, we confront the reality that raw development speed is profoundly addictive. Teams chase the immediate dopamine hit of generating code faster with LLMs, ignoring what happens when they look just one week or one month out and realize unmaintainable prototypes instantly collapse under the weight of real customer changes         💡 Notable Links:         Book: The Psychology of Software TeamsShort: Rory Sutherland — Slow is safer and better🎯 Picks:         Warren - Malicious domain names researchJustin - Deadworld Isekai

Share Episode                  We welcome David Burns, Head of Open Source at BrowserStack and previously Mozilla, this week to help explain how can browser and mobile app testing be so bad still in 2026. "Even if we had to do it all over again, we probably still wouldn't have gotten it right", might just be the biggest learning. We review some of the core challenges that cause even the most experienced web engineers pain and how some companies are attempting to avoid them.         We also review shares lessons from Mozilla’s 'War on Orange,' demonstrating why chasing a 100% green CI suite at scale is a fool's errand, and why engineering teams must instead manage delta tolerances and adhere to a strict testing pyramid. Additionally, we investigate the physical hardware nightmares of running 30,000 real phones in data centers; from battery swelling and custom Faraday cages to complying with Apple EULA requirements using racks of Mac Minis. And finally, we discuss WebMCP, Project Fugu, and why LLM browser agents still struggle against edge-case browser behaviors like the Back-Forward Cache (BF Cache).         💡 Notable Links:         ✨ Episode: Sam Goto — Security challenges in the Browser World✨ Episode: Actually think about how are your users use your toolsMozilla CI - Tree HerderThe Worst UX site with a perfect 100Project Fugu Browser initiativeWebMCP Project Specification🎯 Picks:         Warren - Paper — The Corporate Bullshit Receptivity ScaleDavid - Anbernic Retro Handheld Game Console

Share Episode                  From your feedback, we know that on-call issues, especially those that arise at the database layer are always the most challenging to solve. This episode builds on our Grafana Observability episode as we go deeper into the challenge of data ownership in data lakes. Apache Druid creator, previous Yahoo Distinguished Engineer, Fellow at Splunk, and currently the Chief Architect as Imply, Eric Tschetter joins us to confirm that there are unsolved problems in data platforms, and it doesn't get better when we throw agents into the mix.         In the true DevOps mindset aligment, Eric reminds us that whoever is operating the system, knows what is going on, they know their service boundaries, and so it's their choice to make. And the sad reality is that at large enough companies, you will always have independent teams making independent decisions, it's never possible to fully align them, and realistically we get down to why it is even a mistake to try.         💡 Notable Links:         ✨ Episode: Grafana — Observability✨ Episode: AST can run anythingBook: Lean Manufacturing — The Toyota WayMonkeys typing the Complete Works of ShakespearePaper: Ironies of Automation (& AI)Jeff Bezos API Rant — 2011Book: The Formula, Science SuccessShow: Burn Notice🎯 Picks:         Warren - The AgencyEric - 継続は力なり — keizoku wa chikara nari

Share Episode                  Sauce Labs CTO Anoop Tripathi addresses the root causes of broken test suites and why teams frequently become numb to intermittent test failures. Anoop shares concrete lessons from his career, detailing how bad practices like embedding fixed sleep routines instead of signal-based waiting ruin test reliability. We discuss the necessity of designing products for testability from day one—such as using pair programming where one developer writes the implementation while the other writes the test suite, or building dedicated test interfaces directly into the application architecture. We also analyze the hidden business costs of poor quality, where engineering debt forces sales teams to spend their time managing upset clients instead of acquiring new revenue.         We explore organizational anti-patterns—such as teams claiming to be 'Agile' with one-year sprint cycles, or shipping debug builds to production because release builds continuously crash, highlighting a pair-programming model where one developer writes the code while another writes the corresponding tests         Finally, we tackle the dangers of chasing vanity metrics, from engineers writing automated assert pass statements to claim fake coverage, to organizations bragging about more code. Anoop distinguishes between output gain and outcome gain, warning that LLMs are creating a massive testing and review crisis by generating 20-page documentation files for minor features.         💡 Notable Links:         Quality Iron Triange of SoftwareBook: Sapiens✨ Episode: DORA Report✨ Episode: Grafana — Managing Open Source🎯 Picks:         Warren - Proof of Quantum Consciousness - PaperAnnop - Book: Genesis: Artificial Intelligence, Hope, and the Human Spirit

Share Episode                  In this episode, we sit down with Nikhil Mungel, Head of AI R&D at Cribl, to discuss how LLMs have dramatically transformed the tech hiring landscape. The conversation opens with an honest look at the flood of AI-generated, low-quality resumes and remote interview cheating tactics, which have pushed many organizations toward in-person whiteboard interviews or closed-network referrals.         We explore how software development has evolved from character-by-character coding into "judgment work," where an engineer's value relies on holistic decision-making, taste, and business alignment rather than raw syntax memorization. Nikhil emphasizes the importance of an "ownership mindset," urging developers to move beyond acting as mere specification-translators and instead evaluate technical trade-offs through a business-focused lens. We also pull back the curtain on modern technical interviews, highlighting why expressing strong convictions, asking intentional career questions, and proving human reasoning without LLM assistance remain essential for standing out.         💡 Notable Links:         Show: Pluribus🎯 Picks:         Warren - Sony made the walkman worseNikhil - Book: Leadership Strategy and Tactics: Field Manual

Share Episode                  Pushkar Gopalakrishna, Senior Staff Software Engineer at Snap, previously Cruise and AWS, joins to explore why engineering organizations pivot away from Terraform and HCL toward Kubernetes-native tools even when they might not be better. We unpack how developer friction, copy-pasted control structures, and misaligned organizational incentives create massive tech debt—often forcing SREs to manually update infrastructure repositories for compliance, resulting in broken pipelines and severe operational friction.         We debate over the mechanics of Crossplane, detailing how its continuous reconciliation loop and Custom Resource Definitions (CRDs) allow teams to express cloud infrastructure as YAML alongside their application manifests at potentially the cost of async validation. Pushkar pulls back the curtain on how Cruise managed infrastructure at scale using a custom internal platform called 'Juno' to bootstrap GCP projects, repositories, and permissions, while leveraging Crossplane for application-level resources. We also dive into the dangers of using CI tools for continuous deployment, detailing a terrifying incident where a pipeline bug accidentally marked three production Kubernetes namespaces for deletion, and how moving to ArgoCD and Argo Rollouts helped prevent future outages for autonomous vehicles.         Finally, we touch on the realities of non-production environment isolation, testing against live APIs, and why platform teams must balance providing a seamless developer experience without stripping away developer accountability.         💡 Notable Links:         CrossplanePodcast Guest Request for Principal Engineer — What work are you doing?Amazon Multi-level fullyment center for drones✨ Episode: Terraform vs OpenTofu🎯 Picks:         Warren - Books: The Murderbot DiariesPushkar - DJI mini drone

Share Episode                  Co-Founder of Grafana Labs, Anthony Woods, joins to share his perspective on how open source solutions are thriving despite the deluge of pull requests being thrown at them through their open source repositories.         Most importantly, he outlines how observability is no longer being done by users looking directly at dashboards. The data rarely makes to LLMs or automation, it rarely makes sense even to humans looking at them, without the context. The context is critical component, and having a model that was built on the semantic concepts relevant to your use cases.         And of course we can't stay away from asking on the record the current state of security of open source repositories from a vendor side. Given how their was an inevitable incident with some of the Grafana open source repos, we dig in to figure out how they are dealing with the real world impacts of malware being spread throughout the ecosystem.         💡 Notable Links:         Coinbase's ridiculous spend on observabilityBook: Crossing the ChasmSilicon Valley Show: Hot Dog or Not Hot DogShai Hulud — Grafana open source compromisePodcast Guest Request for Package Manager Security Expert✨ Episode: Productivity🎯 Picks:         Warren - Best Starfleet Captain: PikeAnthony - The Bitter Lesson

Share Episode                  As it turns out, the entire artificial intelligence boom is essentially running on Wikipedia's free labor, but while knowledge is free, physical server infrastructure definitely is not. We sit down with Moriel Schottlender, Principal Systems Software Engineer at the Wikimedia Foundation, to dissect how public systems survive an endless onslaught of high-volume AI scrapers and aggressive crawlers. Because 65% of the resource-heavy requests originate from automated bots, we explore how Wikimedia navigates this traffic without blocking legitimate users. We skip the approaches of IP-banning which doesn't work in practice and discuss actual mature architectural strategies, by focusing on the users' needs. From structured database dumps and high-volume enterprise APIs to rate-limiting and CDN caching trade-offs.         It's a mind-bogglingly complex ecosystem ­of open-source, a 25-year-old PHP monolith supporting over 900 distinct site instances across 300 languages and 11 unique projects. It's an immense engineering challenge to modernize infrastructure while serving 250,000 active volunteer editors who build custom workflows via Toolforge—Wikimedia's internal, open-source mini-AWS.         Finally, we have to tackle the philosophical divide between artificial statistical models and human creativity. Because LLMs are trained to predict the statistical mean, they inherently miss the edge cases where real human value, internationalization, and accessibility actually reside. And even if they did, we managed to squeeze out every last bit of AI creativity that early models had until what we are actually left with is the most boring result. We also commiserate over the gratuitous low-quality AI pull requests flooding open-source repositories, drawing parallels to the chaotic Hacktoberfest spam of years past.         💡 Notable Links:         Frodo projectImpact of crawlers on Mediawiki's infrastructureBook: The Platform RevolutionMoriel's LLM experiments✨ Episode: 🎯 Picks:         Warren - Video: Are all flags Drawable in PowerPointMoriel - Audiobook: Dungeon Crawler Carl

Share Episode                  We are joined this week by Mark Hay, CTO and co-founder of TextQL and former lead of Text Classification Infrastructure at Meta, to uncover the hidden complexities behind massive-scale machine learning. Mark explains why the most crucial features for identifying abusive behavior, like drug dealers or scammers on Facebook and Instagram, rarely rely on the content itself but instead analyze the underlying behavioral graphs, such as abnormal friend requests or messaging patterns.         Of course we review the adversarial nature of spam detection, where bad actors constantly evolve from simple regex evasion to embedding messages inside images or even utilizing pure symbolic communication, like comparing different sized cucumber emojis to evade text filters. That requires diving into the evolution of database querying and the rise of the semantic layer. Mark unpacks why relying on raw LLMs to write complex SQL is a recipe for hallucinations, and how implementing a "correct by construction" semantic layer guarantees structurally sound queries by restricting outputs to a strictly defined configuration. However, this rigid structure fundamentally stifles the creative flexibility of LLMs.         Lastly, we can't avoid exploring the tension between these approaches and how new tools aim to bridge the gap by dynamically balancing raw SQL generation with structured ontological constraints, providing rapid time-to-value for analytical workflows. Finally, we discuss the controversial philosophical shift occurring within software engineering, particularly the tension between the "Don't Repeat Yourself" principle and "Locality of Behavior".         💡 Notable Links:         ✨ Episode: Semantic Search✨ Episode: Formal Verification✨ Episode: Subjective Model Embeddings🎯 Picks:         Warren - Article: I Left Port 22 Open on the Internet for 54 DaysMark - Hotel Room Exercise: Burpies

Share Episode                           We sit down with Itacama CEO Pia Wiedermayer to discuss the absurdity of siloed QA, the disaster of AI-generated API tests, and why developers hate the word "quality." This time we are asking the age-old question: Who needs testers anyway? Pia and Warren discuss how to dismantle the toxic culture of isolated quality assurance.         We explore how the ghosts of waterfall development still haunt modern teams, creating silos where developers blindly throw unverified code over the wall and expect a separate QA department to magically inject quality. Included is the inevitable discussion on the psychological safety of hiding behind narrow job titles and why refusing to take collective ownership of a product is a guaranteed recipe for architectural failure.         Of course we can't avoid commenting on the terrifying reality of replacing human intuition with automated hype. Pia shares a case study of a scale-up that aggressively pivoted to "full steam AI development," intentionally excluding both their Product Owner and QA from the entire experiment. Predictably, it did not end well, but we were able to laugh at the painful irony that an AI-accelerated project scheduled for four weeks ended up taking eight weeks, proving that simply generating code without human oversight just creates more sophisticated bottlenecks.         🎯 Picks:         Warren - Wason Selection Task on The Rest Is SciencePia - Book: The Culture Map