Talking Product

In Episode 16 of Talking Product I speak with INSEAD Professor Theos Evgeniou. Theos directs the Executive Education programme Transforming Your Business with AI. He has worked in machine learning and AI for close to 30 years. One of the things that attracted me to Theos' work is his emphasis on the need for CEOs and boards to become intimately involved in a company’s AI strategy and transformation — a theme Collin and I have been exploring over the last few sessions. In this conversation, I give Theos a series of scenarios based on questions I’ve heard from executives, programme managers, investors and founders wrestling with AI adoption and organisational readiness. In this episode, Theos and I explore: Why AI strategy must be co-authored with business strategy. Why asking the right questions may be one of the most important AI capabilities a senior leadership team can develop. How governance can enable experimentation rather than simply restrict it. Is a strong brand enough protection against AI-driven disruption? The questions a board should ask before assuming it is. The challenge for private equity and asset managers: assessing AI maturity, potential upside and disruption risk across portfolio companies. Why stalled AI adoption may be a symptom rather than the problem — and how to identify the underlying constraints across data, infrastructure, skills, incentives and risk appetite. For an AI-native founder, where does defensibility really come from when today’s capability could become tomorrow’s model feature? I really enjoyed the conversation and very much appreciated Theos giving me his time. I found the discussion eye-opening from a number of angles. I hope you enjoy it too. If you like the session, please subscribe, recommend it and leave us a nice review. It helps us build an audience and attract guests to the show. And, as always, feedback is more than welcome. Thanks.   Links to Theos' work: Bio page: https://www.insead.edu/faculty/theodoros-evgeniou INSEAD Knowledge articles: https://knowledge.insead.edu/author/theodoros-evgeniou Transforming your Business with AI: https://www.insead.edu/executive-education/open-online-programmes/transform-business-ai  

Over the last several episodes of Talking Product, Collin and I have been exploring how we increase the probability that our AI investment will translate into business value. Recent news has shown how organisations are having to confront the economics of AI more directly: token budgets being consumed faster than expected, tighter controls on usage, the challenges of moving from prototype to production, how to shape experiments around coherent hypotheses tied to a business outcome, and whether the use of GenAI is actually translating into customer value. In Episode 15, we explore the mindset shift that I have experienced through working with the DSPy framework and becoming familiar with the work of Omar Khattab. This is not a technical discussion. Instead, Collin and I continue to explore the role non-technical senior leaders should play in shaping the AI journey. I believe anyone in an AI leadership role should familiarise themselves with Omar’s work if they haven’t done so already. I have found him to be a grounded, coherent voice in a world inundated by hyperbole and breathlessness. Omar Khattab is an Assistant Professor at MIT. His doctorate is from Stanford. His research focuses on how we program, evaluate and optimise systems built around language models. He created DSPy, an open-source framework for building and optimising LLM applications programmatically, rather than relying primarily on manually engineered prompts. In this episode, Collin and I explore: What belongs to LLM inference and what belongs to good ol’ deterministic programming. Defining tight, measurable evaluations — yes, we’ve been banging on about this since covering Chip Huyen’s book. The power of optimisation — don’t just believe us, go and look at the use cases on the DSPy website. Building on that optimisation, how lower-cost models can sometimes deliver equal and even better performance. The role non-technical senior leaders should play in shaping the AI journey. If you like the session, please subscribe through your preferred podcast service and leave us a smashing five-star review. It helps us build an audience and attract guests to the show. And, as always, feedback is more than welcome. Thanks.   Links referred to in the show: Omar's home page: https://omarkhattab.com/ Two podcasts: How Foundation Models Evolved: A PhD Journey Through AI's Breakthrough Era: https://podcasts.apple.com/us/podcast/how-foundation-models-evolved-a-phd-journey-through/id1740178076?i=1000737229448 DSPy: Transforming Language Model Calls into Smart Pipelines // Omar Khattab: https://podcasts.apple.com/gb/podcast/dspy-transforming-language-model-calls-into-smart-pipelines/id1505372978?i=1000637537521 I also found this post helpful: https://x.com/lateinteraction/status/1921565300690149759  

In this episode of Talking Product, Collin and I discuss a tension we increasingly see inside organisations: digital capability is becoming an enterprise survival issue rather than a technology optimisation issue. Unlike some of our previous episodes, this is less a discussion about a single topic and more a conversation between two practitioners trying to reconcile lived experience with organisational inertia. We believe this tension is surfacing more visibly now because of the disruption caused by generative AI. Collin highlights an article I published on Substack at the end of last year. In it, I argued that the tolerance for organisational inefficiency and limited digital capability is likely to compress dramatically in the years ahead. Organisations that have invested time and energy into improving product delivery practices, architectures, infrastructure, data quality, and operating models are materially better positioned than those that have largely treated such initiatives as secondary concerns. This led me to a broader question: if executives and boards are not asking deeper questions about AI, data, software capability, and operating models, who is? My conclusion is that institutional investors need to engage far more directly with digital capability risk because many portfolios are likely carrying significantly more exposure than is currently recognised. We do not pretend to have definitive answers to many of these questions. In fact, part of the episode is an acknowledgement of just how difficult these topics are to navigate. But we once again argue for the importance of having these conversations inside organisations — because avoiding them does not reduce the risk. It simply delays the moment when those risks become visible. Lastly, we once again argue that getting Chip Huyen’s book AI Engineering and starting conversations inside your organisation about the subjects she raises will be very good value for money — and may save you a great deal of money and grief further down the road. Takeaways: An exploration of why digital capability is an enterprise-level risk rather than a delegated IT concern. A discussion about how generative AI accelerates the consequences of technical debt and organisational underinvestment. Questions around whether executives, boards, and institutional investors are asking sufficiently deep questions about AI, data, and operating models. The Digital Laggard Thesis: https://johnyoung.substack.com/p/towards-an-investment-thesis-on-digital

In this episode, Collin and I continue building our list of questions to help increase the chances of your AI investments delivering a return. We focus in particular on the role non-technical C-level leaders should play in this effort. We use Chip Huyen’s AI Engineering as a practical framework, exploring how it gives digital leaders and non-technical senior managers a structured way to engage more deeply with AI initiatives. In particular, it provides a way into the ecosystem and lifecycle of AI application development—helping leaders ask better questions around things like data, prompt design, fine-tuning, and evaluation—so they can have more meaningful discussions with technical teams about how these applications are built, where the risks sit, and what criteria should be used to define and measure success. We walk through some of the key differences between traditional software and AI systems—particularly the shift from deterministic to probabilistic behaviour, and the central role of data in shaping outcomes. From there, we build on the questions we believe leaders should be asking: What problem are we solving? How are we evaluating outputs? How are we managing risks around data quality, safety, and factual accuracy? What trade-offs are we making between quality, cost, and latency? We spend some time looking at Chip’s section on evaluation criteria, using it as a springboard for non-technical senior leaders to delve deeper into the thinking behind—and expected outcomes of—AI applications. We also introduce the concept of “evals”—ongoing evaluation frameworks that extend beyond traditional testing—and why they require continuous iteration, collaboration, and oversight, even after deployment. This episode continues our exploration of how leaders can better understand what they are funding, engage more effectively with product and delivery teams, and create the conditions for AI investments to deliver real value.   Links to Chip's book & interview on evals referred to in the episode: Chip Huyen’s AI Engineering: https://www.oreilly.com/library/view/ai-engineering/9781098166298/ Lenny’s podcast - Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar: https://www.youtube.com/watch?v=BsWxPI9UM4c

A significant number of AI initiatives are consuming budget and executive attention, while failing to reach production or deliver measurable value. Episode 12 of Talking Product helps you pressure-test your organisation's AI spend today.  In this episode, Collin and I use a Financial Times article as a springboard for a conversation about the state of AI initiatives in large organisations. The article—"AI's awfully exciting until companies want to use it"—captures a familiar pattern in technology: high expectations, significant investment, but limited impact. The article is freely available with FT registration.   In this episode, we explore three themes: Why pilots aren't scaling (often it's your data, not the technology) Whether organisations are bringing experimental rigour to AI adoption, or just buying impressive demos The leadership knowledge gap—understanding not just what AI can do, but what it can't   What you'll get: Questions that will help reveal whether your organisation is truly learning from its AI experiments or just spending Insight into why data governance problems you've been kicking down the road are now becoming existential What Gartner found about hidden costs in AI initiatives—and questions to ask to bring them out into the open Practical guidance on what experimental rigour actually looks like in an AI context #AI #DigitalTransformation #Leadership #ProductManagement

In this episode, Collin shares a case about a leadership team building a hardware and software product. We dig into this case because it highlights familiar product dynamics—great, passionate people, working to tight deadlines, but stuck in semi-siloed patterns and not really operating as one team. Each group pushes its own goals but isn’t fully aligned on the end game. And when the product leaders aren’t working effectively as a team, it has a real impact on outcomes and the ability to deliver at pace. We chose this topic to give product leaders footholds and practical ideas for starting similar conversations in your own organisation. We often call ourselves a team—but are we, as product leaders, really operating as a team? What do we really mean when we use the word “team”? In this session, Collin describes how he asked the leadership group that he was working with this simple, but powerful question: “Are you working as a team?” To their credit, they took the time to work with Collin and reflect on this question, as a collective. One area of the case that I found interesting was when Collin unpacks the complexity of “owning” requirements as new information emerges or thinking shifts about a particular problem. It is this inherent complexity of digital product development that makes siloed, and even semi-siloed, ways of working so dangerous. Collin also highlights a deeper systemic issue: senior and executive leaders often don’t grasp the importance of ways of working in digital product development. Instead, they simply push the silos to deliver faster. I personally feel that this level of naivety at the top is a serious competitive problem. I hope you find this case useful. We welcome recommendations and comments—and we’re looking forward to getting a few more episodes out, sooner rather than later.   At the end of the podcast, I refer to the work of Nonaka & Takeuchi and said I would include a link: 1) The New New Product Development Game  and 2) The Knowledge Creating Company are probably their most well known work. But there is a wealth of other material that is also worth exploring.

In Episode 10, Collin and I dive into a question I was asked during an interview: “We do an offsite with our leadership team, and it’s a great couple of days… but once we’re back in the day-to-day, it all sort of disappears. As a coach, how do you help with that?”   The question has lingered with me for quite some time, and I thought it would be an interesting topic to unpack with Collin. In this episode, we explore: How coaches can help “hold” a team’s desire to change, providing structure and accountability without getting pulled into the daily fires Why working with digital leadership teams means going beyond agile frameworks into the messier world of team dynamics and organizational complexity The importance of examining your own unconscious beliefs about leadership Three common offsite mistakes we often see—and how to avoid them: Senior leaders facilitating instead of participating Agendas so full there’s no space for reflection Avoiding the hard conversations that matter most We share personal experiences to keep the conversation actionable and grounded in the day-to-day realities of organisational life.   I hope you find Episode 10 useful. Thanks to everyone who’s been following the podcast and leaving kind reviews—it means a lot.

What if you could measure what really matters to your customer?  In our latest episode of Talking Product, Collin and I chat with Alex Hidalgo—author of the O’Reilly book Implementing Service Level Objectives. Alex shares how Service Level Objectives (SLOs) can help teams zero in on customer priorities and bring people across the organisation together to define what really matters. SLOs are powerful, but they can also be tricky to implement. I’ve struggled with them myself, so it was great to have the chance to explore this topic in depth with Alex and learn from his expertise. In this session, we cover: Practical steps to get started with Service Level Indicators (SLIs), the foundation of SLOs. How to start small and iterate as you refine your SLOs over time. The cultural challenges of implementing SLOs—and how to navigate them. If you’re just starting out or looking to improve your approach, I hope you find this episode full of insights to help you on your SLO journey. Alex’s book is an excellent resource for making SLOs meaningful and achievable in your organisation. I highly recommend it. I hope you enjoy listening to this session as much as we enjoyed recording it. If you liked the episode, please give us a rating—or go all out and write us a nice review. It makes a big difference! A huge thank you to Alex for making the time to share his insights with us.

In this session Collin & I use Volkswagen’s struggle to build Electric Vehicle (EV) software as a way to explore two themes: the importance of getting a digital transformation right, and the opportunity presented to competitors when large players simply cannot deliver good quality digital products. We use two articles from the Financial Times as a springboard for our conversation. The first article, dated Jun 28, 2024 “Oliver Blume, VW and Porsche boss steering an EV strategy shift”  and the second one, published September 2, 2024 “VW considers closing factories in Germany and cutting jobs”  (both behind a paywall - sorry) The first article describes an investment of $5 billion in the US based EV startup Rivian. The purpose of the investment is to create a joint partnership that allows VW to use Rivian’s EV software. The article quotes VW’s CEO Oliver Blume as saying, “In terms of a big tech transformation, you can’t do it all on your own… It should be a win-win situation . . . The motivation from our side is to speed up software transformation at Volkswagen in all our brands. Rivian has best in class architecture . . . Volkswagen has the scale”. Collin and I explore this thinking of an acquisition / strategic partnership as a means for fixing an internal digital transformation challenge. For those of you who have listened to Collin and I, you probably won’t be surprised to hear that we are not convinced that such an approach will easily remedy digital product development challenges.  As we ask in the show, what mental models will need to change for these two companies - a digital native and a non-digital native, to work effectively together? As Collin highlights, ultimately these transformations are culture changes, which require heavy-lifting. Since recording this session, the FT has come out with more articles on the challenges Volkswagen is facing. I have also started listening to some of the VW YouTube influencers who have been discussing software issues for a number of years. I am finding this to be a fascinating case study, albeit quite sad for the workers of Wolfsburg if the leadership of VW can’t get its digital act together. In this session, I also refer to a 2017 Medium post on how Revolut used the build-measure-learn cycle to develop digital products quickly and acquire customers. Hope you find the session useful.  If you like the show, please give us a nice rating and recommend the show. It helps others find the podcast.

In this last episode of our three part series, Collin and I continue to role-play an imaginary data-heavy product development project. In our last episode, Collin was granted some authority to get more involved in the project in an effort to reduce the risk that Collin felt was inherent in the delivery team’s approach. The delivery teams is a consulting firm that has been brought in by another group in the company. In this episode, Collin elaborates on the types of risks he sees, as well as how he would address those risks through an alternative approach to product development. Collin and I are using this fictitious scenario to explore how ways of working, power dynamics and group-think play out in digital product development. In my opinion, these three forces are always at play in this type of work. Our goal in exploring this is to help listeners become better equipped to work with these forces on their own projects. In this episode, Collin and I discuss the fact that using an approach that pushes risk out into the open earlier in the product-development lifecycle brings reality into the room very quickly. It forces people to stop imagining “happy-path” outcomes through Power Point and Gantt charts. This can often be experienced as “raining on the parade.” In this episode, I ask Collin to help a senior manager identify the risks themselves by role playing the person who procured the consulting firm. In this scenario, a senior manager wants to be able to see the types of risks that Collin is seeing. This manager wants to be able to ask the questions that Collin is asking. And, they want to mitigate some of those risks without having to rely on delivery managers like Collin - “What is it that allows Collin to see digital product development in the way he sees it?” Through the role-play, we discover risks that were hidden in the consulting firm’s Power Point presentation and Gantt charts. Because of the approach the consulting firm was taking, made worse by shrouding it in Agile lingo, these risks would not have revealed themselves until very late in the consulting firm’s engagement. As Collin says, “If the consulting firm had followed that plan they would have delivered late and asking for an extension. That may be good for the bottom-line on the consulting side, but puts the company’s strategic goals at risk.” Hope you enjoy it. As always we would look forward to hearing from people.  If you are liking these podcasts, please do throw us some love in the ratings.  Thanks