Inside the DevEx Metrics Compass with André Meyer

Are engineering teams measuring the impact of AI, or just how much it's being used?In this episode of Engineering Enablement, host Brian Houck talks with researcher André Meyer about his DevEx Metrics Compass, why traditional productivity metrics often miss what matters, and how to measure the value AI actually delivers.They also revisit André's Today Was a Good Day study to explore what makes developers' work rewarding, how AI is changing that experience, and practical ways to balance focused work, collaboration, and growing code review demands.Where to find André Meyer-Baron:• LinkedIn: https://www.linkedin.com/in/andrenmeyer • Website: https://andre-meyer.chWhere to find Brian Houck: • LinkedIn: ⁠⁠⁠⁠⁠https://www.linkedin.com/in/brianhouck⁠⁠⁠⁠⁠In this episode, we cover:(00:00) Intro(02:42) Why AI dashboards fail to capture actual value(05:08) What the DevEx Metrics Compass is and why it was created (06:58) Why measuring collaboration matters(09:47) Determining the right metrics for your organization (12:03) Distinguishing measurement gaps from gaps in public reporting(14:35) Why some metrics are harder to act on at scale(18:11) Why customer and business value are difficult to measure(20:03) How to know when it's time to retire a metric(24:20) Hierarchies of metrics (27:31) How to measure AI's impact beyond activity metrics(30:20) André's Good Day study: What makes a good workday for developers (31:56) What observing developers revealed about context switching(34:28) What makes developers' work rewarding in the age of AI(36:27) The tradeoffs of delegating coding to AI(38:01) Balancing collaboration with focused work(42:11) Two ways to improve developers' workdaysReferenced:• CAFE(S): Your Agent Is Only as Good as Its Context• Too Many DevEx Metrics, Too Little Guidance• Today was a Good Day: The Daily Life of Software Developers• DevEx Metrics Compass • Gail Murphy | LinkedIn• Thomas Fritz | LinkedIn• Patrick Meyer | LinkedIn• The SPACE Framework• From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI

In this episode of Engineering Enablement, Brian Houck talks with Andrew Swerdlow, VP of Engineering at Roblox, about how the company is building trust in AI agents to review, test, and deploy code autonomously. Andrew explains how Roblox turns past code review feedback into rules for agents and why engineering is shifting toward specifications and intent. They explore what that means for the skills engineers need, how Roblox measures AI’s impact on productivity, and how it manages costs. They also discuss the guardrails needed to let agents safely ship code with minimal human intervention.Where to find Andrew Swerdlow:• LinkedIn: https://www.linkedin.com/in/andrewswerdlow/ • X:  https://x.com/andrewswerdlow • Website: https://andrewswerdlow.netWhere to find Brian Houck: • LinkedIn: ⁠⁠⁠⁠https://www.linkedin.com/in/brianhouck⁠⁠⁠⁠In this episode, we cover:(00:00) Intro(02:20) Building trust in AI across the software development lifecycle(04:44) When human oversight helps and when it gets in the way(06:50) What AI agents need to know about your organization(08:45) How Roblox turns engineering knowledge into rules for AI agents (12:40) How AI code review compares to human code review(14:59) Capturing engineering knowledge beyond documentation(18:14) How AI changes what engineers need to understand(20:17) The shift to spec-driven development(23:24) How AI changes how we evaluate engineers(26:33) Rethinking engineering interviews in the age of AI(28:23) How Roblox tracks AI-generated code(35:56) Model routing and choosing the right AI tools(39:50) Measuring AI's impact on engineering productivity(44:44) Optimizing AI costs at scale(48:12) Builders at Roblox and the future of software engineering(50:19) Building trust in autonomous code review(53:11) The cultural shift toward autonomous engineeringReferenced:• CAFE(S): Your Agent Is Only as Good as Its Context• Roblox• Tech Leadership by Andrew Swerdlow• Margaret-Anne Storey• Malte Ubl - Vercel | LinkedIn• Grafana Labs• Kushan Mehta | LinkedIn • Diff Risk Score: AI-driven risk-aware software development - Engineering at Meta

As AI agents become more capable, context quality is increasingly the limiting factor in their performance. In this episode of Engineering Enablement, DX researcher Brian Houck by his CAFE(S) is joined by co-authors Max Kanat-Alexander (Capital One), Eirini Kalliamvakou (DX), and Margaret-Anne Storey (University of Victoria) to discuss CAFE(S), a new framework for improving context quality.The panelists introduce the five dimensions of CAFE(S): Clarity, Actionability, Fidelity, Efficiency, and Security. They walk through common context failures, how poor context accrues technical and cognitive debt, what it costs in developer productivity and operating spend, how AI is changing the skills engineers need, and why context review could become as important as code review.Where to find Margaret-Anne Storey:• LinkedIn: https://www.linkedin.com/in/margaret-anne-storey • X:  https://x.com/margaretstorey • Website: https://margaretstorey.comWhere to find Max Kanat-Alexander: • LinkedIn: https://www.linkedin.com/in/mkanat• X: https://x.com/mkanat• Website: https://www.codesimplicity.comWhere to find Eirini Kalliamvakou: • LinkedIn: https://www.linkedin.com/in/eirini-kalliamvakou-1016865 • X: https://x.com/irina_kAl Where to find Brian Houck: • LinkedIn: ⁠⁠⁠⁠https://www.linkedin.com/in/brianhouck⁠⁠⁠⁠In this episode, we cover:(00:00) Intro(01:18) Meet the CAFE(S) authors(03:13) Introducing CAFE(S): Why context quality matters(06:32) Common context failure modes(08:30) The five dimensions of high-quality context(12:33) When model failures are actually context failures(18:24) Why the panel prioritizes clarity in CAFE(S)(22:34) How context quality relates to technical, cognitive, and intent debt(25:35) How human interventions reveal context failures(28:11) How context quality relates to AI fluency(30:28) How AI is changing what makes a good engineer(33:48) The case for context review as a discipline (38:44) Wrap-upReferenced:• CAFE(S): Your Agent Is Only as Good as Its Context• Nicole Forsgren• The Productivity-Experience Paradox - Annie Vella• Startup Studio• From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI• The AI-native developer• CodeScene

Meta researcher Moritz Beller joins Brian Houck to discuss how Meta measures developer productivity. Moritz explains diff authoring time and how Meta uses it to evaluate tools and guide engineering decisions. They explore how AI is changing engineers’ work and revealing gaps in traditional metrics. They also revisit Moritz’s “Mind the Gap” study and discuss the promise and risks of AI agents for testing.Where to find Moritz Beller:• LinkedIn: https://www.linkedin.com/in/inventitech • X: https://x.com/Inventitech • GitHub: https://github.com/Inventitech • Website: https://inventitech.com Where to find Brian Houck: • LinkedIn: ⁠⁠⁠https://www.linkedin.com/in/brianhouck⁠⁠⁠In this episode, we cover:(00:00) Intro(02:10) Moritz’s role at Meta(04:01) Measuring diff authoring time(08:20) Measuring A/B experiments(12:11) What diff authoring time reveals(14:45) Planning and leadership reporting(18:36) Why Meta measures teams, not individuals(22:33) Where developer time goes with AI(26:06) AI’s impact on junior and senior developers(27:18) Capturing invisible work with AI(29:32) The “Mind the Gap” study(33:40) Revisiting “Mind the Gap” in 2026(38:32) AI agents for software testing(41:49) What remains hard to measureReferenced:• State of AI Impact in Engineering Q2 Report 2026• GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis• From Technical Debt to Cognitive and Intent Debt • Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity• The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study• When, how, and why developers (do not) test in their IDEs | Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering• DORA, SPACE, and DevEx: Which framework should you use• A Tale of Two Cities: Software Developers Working from Home During the COVID-19 Pandemic

In this episode of Engineering Enablement, Robert Lucero, Chief Architect at Okta, joins host Brian Houck to discuss how agentic AI is changing identity and access management. They explore how organizations should define agent identities, why authentication and authorization remain foundational, and how sandboxing, fine-grained permissions, and just-in-time access can enable agents to operate autonomously without creating unacceptable risk.Robert shares what Okta has learned from driving AI adoption among security-minded engineers, what makes a repository ready for AI agents, and why strong testing, CI, and review processes matter even more as AI generates more code. They also discuss AI’s role in software validation, the need for human judgment, and whether AI will ultimately give the advantage to security teams or attackers.Where to find Robert Lucero:• LinkedIn: https://www.linkedin.com/in/rlucero Where to find Brian Houck: • LinkedIn: ⁠⁠https://www.linkedin.com/in/brianhouck⁠⁠In this episode, we cover:(00:00) Intro(02:16) Meet Robert Lucero(02:56) Why identity becomes critical as AI agents access more systems(06:07) Governing automated access for non-deterministic agents(08:01) Defining and managing agent identities(10:32) Authentication and authorization for AI agents(13:33) Determining how much autonomy to give AI agents(19:44) Identity systems as the control plane for AI agents(22:40) Driving AI adoption among security-minded engineers(25:48) Why AI’s impact on engineering extends beyond coding(28:44) The three components of AI readiness for repositories(30:55) AI’s role in software testing and validation(34:26) Whether AI gives defenders or attackers the advantage(36:01) Where to find the best food in the countryReferenced:• State of AI Impact in Engineering Q2 Report 2026• Okta• The Hugging Face incident and the road ahead | OpenAI• GitHub Copilot · Your AI pair programmer• Claude Code by Anthropic | AI Coding Agent, Terminal, IDE• Eight Myths on Software Engineering and GenAI | Queue• Adriana Corona

Max Kanat-Alexander is an Executive Distinguished Engineer at Capital One, where he helps lead developer experience and AI-assisted software development.In this episode of Engineering Enablement, Max joins Brian Houck to discuss how AI is changing software development and why strong engineering fundamentals matter more than ever. They explore how engineering skills are evolving, why AI can amplify weaknesses in the development lifecycle, and how teams should rethink code review, quality, and testing.Max also shares how leaders can assess whether their organizations are ready for more advanced AI workflows and why developing the next generation of senior engineers remains one of the biggest open questions in software engineering.Where to find Max Kanat-Alexander:• LinkedIn: https://www.linkedin.com/in/mkanat • X: https://x.com/mkanat • Blog: https://www.codesimplicity.com  Where to find Brian Houck: • LinkedIn: ⁠https://www.linkedin.com/in/brianhouck⁠In this episode, we cover:(00:00) Intro(01:54) Max’s role at Capital One(02:52) Where to invest in engineering organizations  (06:36) The new entry-level engineering skills to pay attention to (10:39) Why deepening your understanding still matters (12:29) The bottlenecks around code review (19:40) Why human code reviews still have value (25:20) Why not all PRs need human review(26:11) The vicious cycle of AI-driven development(30:13) Using LLMs for refactoring (33:58) AI readiness and why fixing engineering systems is so hard(38:42) Why research is needed on creating good senior engineers(41:00) Why AI will increase the need for engineersReferenced:• DX Core 4 Productivity Framework• Capital One • Code Review Guidelines at Google • Code Simplicity by Max Kanat-Alexander• How Microsoft sees engineering bottlenecks changing with AI

AI adoption among software developers is approaching 100%, AI-authored code now makes up more than half of merged code, and developers report saving more time with AI every quarter. But those gains aren’t translating evenly into better outcomes.In this episode of Engineering Enablement, host Brian Houck, Distinguished Scientist at DX, sits down with Justin Reock, Deputy CTO at DX, to unpack findings from DX’s latest AI Impact Report. They explore where AI is improving engineering velocity and developer experience, where concerns are emerging around PR size, change confidence, and failure rates, and why rising AI spend has yet to produce a comparable increase in innovation.They also discuss how AI is changing the meaning of code maintainability and where developers’ AI-driven time savings may actually be going.Where to find Justin Reock:• LinkedIn: https://www.linkedin.com/in/justinreockWhere to find Brian Houck: • LinkedIn: https://www.linkedin.com/in/brianhouckIn this episode, we cover:(00:00) Intro(01:45) How the current AI impact report is tied to Core 4 (03:24) The state of AI adoption(05:12) How much time AI is saving developers and percentage of AI-authored code(07:47) AI’s impact on PR throughput and deployment frequency(11:09) How EMs are shipping more code(13:02) Why larger PRs may be problematic(18:21) The growing gap between code maintainability and change confidence(21:48) How perceived code quality varies by organization size(23:49) The growing volatility in change failure rates(28:07) What the Developer Experience Index reveals(32:20) Cost, dev ramp-up, and innovation ratio (35:38) Where AI time savings are getting lost(37:11) Questions and wrap-upReferenced:• DX Core 4 Productivity Framework• AI Impact report• The AI-native developer - by Brian Houck• GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis• Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools | NBER • The Productivity-Experience Paradox - Annie Vella• EngThrive: Make It Fast and Easy to Do Great Work• The AI efficiency plateau - by Brian Houck• Tradable Quality Hypothesis

Tim Bozarth is a Corporate Vice President in Microsoft CoreAI, where he leads engineering for next-generation developer experiences and Microsoft's Engineering Thrive initiative. Throughout his career at Microsoft, Google, Netflix, and Box, he has focused on developer productivity, engineering systems, and organizational effectiveness.In this episode of Engineering Enablement, Tim joins host Brian Houck to discuss Engineering Thrive, Microsoft's framework for measuring and improving engineering productivity. They explore why AI makes outcome-based metrics more important than ever, where new bottlenecks are emerging in the software development lifecycle, and why verification and confidence may become more valuable than code generation itself. Tim also shares why the purpose of engineering remains the same despite rising levels of abstraction, the skills that remain durable in an AI-driven world, and why engineering leaders should focus on outcomes rather than activity metrics.Where to find Tim Bozarth:• LinkedIn: linkedin.com/in/tbozarth Where to find Brian Houck: • LinkedIn: ⁠https://www.linkedin.com/in/brianhouck⁠In this episode, we cover:(00:00) Intro(02:13) What Engineering Thrive is and the problem it solves(04:46) Why Engineering Thrive isn't specific to Microsoft(09:10) The impact of Engineering Thrive at Microsoft(14:31) Why AI makes outcome-based productivity metrics more important(18:22) Where AI is creating new bottlenecks in the SDLC(24:37) Why more abstraction doesn't change the purpose of engineering(27:25) The durable skills of good engineers (33:03) The changing economics of software development(36:56) Advice for leaders: measure outcomes, not activityReferenced:• DX Core 4 Productivity Framework• EngThrive: Make It Fast and Easy to Do Great Work: Building a durable model for outcome-oriented engineering measurement• Quote by Eliyahu M. Goldratt: “Tell me how you measure me and...”• Jevons paradox - Wikipedia

AI coding agents are changing how software gets built, but they're also forcing organizations to rethink how engineering effectiveness is measured. Traditional developer experience metrics were designed for humans, not AI agents, so how should engineering leaders adapt?In this webinar, Justin Reock, Deputy CTO at DX, is joined by Brian Houck, Distinguished Scientist at DX and co-author of the SPACE framework, to explore the emerging field of agent experience and how it builds on developer experience rather than replacing it. They discuss how organizations can prepare for AI-assisted software development, how the DX Core 4 applies in the age of AI, why metrics like token usage and PR throughput don't tell the whole story, and the growing importance of documentation. They also examine the impact AI-driven pressure is having on burnout and cognitive overload.Throughout the conversation, they share practical guidance for building engineering organizations where both developers and AI agents can do their best work.Where to find Brian Houck: • LinkedIn: https://www.linkedin.com/in/brianhouckWhere to find Justin Reock:• LinkedIn: ⁠https://www.linkedin.com/in/justinreock⁠ In this episode, we cover:(00:00) Intro(01:26) Justin’s new role at DX(03:53) What agent experience is and why engineering leaders should care(08:55) How to improve agent experience at the platform level(11:25) How agent experience and developer experience influence each other(14:41) Preparing engineering teams for agentic work(21:38) Why the DX Core 4 still matters in the age of AI(27:23) What PR throughput actually measures (32:47) The limits of token metrics(37:10) What the data shows about documentation and developer experience(39:32) Improving documentation for AI agents(40:43) AI-washing, burnout, and cognitive overload(45:35) Brian's upcoming research on agent experienceReferenced:• DX Core 4 Productivity Framework• The SPACE of AI | Queue• Sara Chizari on LinkedIn• The Middle Loop - Annie Vella• gastownhall/gastown: Gas Town - multi-agent workspace manager · GitHub• DORA, SPACE, and DevEx: Which framework should you use?• AI Impact Report: Q1 2026• AI-authored code has nearly doubled, but so has PR size • EngThrive: Make It Fast and Easy to Do Great Work: Building a durable model for outcome-oriented engineering measurement• Google's Project Aristotle - Psychological Safety • Zapier

Sejal Amin is the Chief Technology Officer at Priceline, where she leads product engineering, infrastructure, data, and technology operations. Pedro Gutierrez is Senior Director of Software Engineering, where he has helped drive developer experience initiatives and the adoption of Priceline's product operating model.In this episode of the Engineering Enablement podcast, Justin Reock talks with Sejal and Pedro about Priceline's journey from a project-based organization to a product operating model and the role developer experience played in making that transformation successful. They discuss how DX metrics and developer feedback helped identify organizational bottlenecks, guide structural changes, empower engineering managers, and build trust across teams. They also explore the importance of clear communication, creating a dedicated developer experience team, and how their operating model has helped prepare Priceline for AI-driven software development.Where to find Sejal Amin:  • LinkedIn: https://www.linkedin.com/in/sejal-aminWhere to find Pedro Gutierrez: • LinkedIn: https://www.linkedin.com/in/pedro-gutierrez-b6605422Where to find Justin Reock:• LinkedIn: https://www.linkedin.com/in/justinreock In this episode, we cover:(00:00) Intro(01:07) Meet Sejal Amin and Pedro Gutierrez(01:47) How Priceline's developer experience journey began(04:55) Lessons from Priceline's first developer experience surveys(06:55) How DX improved Priceline's developer experience surveys(09:47) Identifying the causes of organizational slowness(12:33) How the product operating model changed the way Priceline works(14:10) Priceline's phased rollout with DX(18:14) How DX insights drove organizational changes(19:33) Why Priceline improved developer experience before org change was complete(22:18) How clear communication builds trust(24:25) Early results from Priceline's Core Four(25:38) Creating a culture of continuous feedback to build trust(27:40) What has changed in the engineering manager role(30:10) Resources for learning about the product operating model(32:40) What Pedro learned from implementing DX(34:51) The developer experience team(35:59) How AI tools have impacted Priceline’s teams (37:20) How the product operating model supports AI-driven development(39:13) Final advice for engineering leadersReferenced:• Measuring developer productivity with the DX Core 4• Transformed: Moving to the Product Operating Model (Silicon Valley Product Group) • Team Topologies• Flow Framework• Project to Product: How to Survive and Thrive in the Age of Digital Disruption with the Flow Framework