Are your experiments too safe?

What does experimentation look like when you’re optimising one of the biggest retail sites in the world?In this episode, we talk with Kevin Washington (former Director of Product Management over Experimentation at Best Buy) about how large e-commerce teams think about testing when revenue comes from multiple directions: purchase conversion, retail media, and marketplace growth.We cover:The “job of the page” mindset (homepage vs. PLP vs. PDP)Choosing primary metrics vs. guardrails (and why “purchase conversion” can mislead)How ad placements can cannibalise core retail performanceWhy short-term revenue wins can quietly damage long-term trust and LTVNon-endemic ads, emergency testing, and what “passes the smell test”Note: This episode includes a screen-share walkthrough — watch the video version on YouTube for the full experience.Bonus: Kevin will be speaking at UNITE Summit.

In this episode of the Conversion Podcast, Matt is joined by Colin, Vice President of North America at Kameleoon, to explore prompt-based experimentation (PBX) and how AI is transforming the way experiments are built and launched.Colin breaks down what PBX is, how it connects AI-driven “vibe coding” with experimentation, and why the build phase has long been the biggest bottleneck for experimentation teams. They discuss how PBX allows teams to create variants, tests, and prototypes directly on live websites using natural-language prompts, cutting build time from weeks or days down to minutes.The episode also covers where PBX fits into the experimentation lifecycle, the importance of keeping humans in the loop, current limitations of the approach, and what adoption looks like for teams just getting started. They close with a discussion of the Prompt-a-thon, an open challenge designed to explore how far prompt-based experimentation can be pushed.What is prompt-based experimentation (PBX) isHow PBX differs from AI coding and vibe-coding toolsWhy building experiments is the biggest friction pointHuman-in-the-loop experimentation with AIAdoption tips and limitationsThe Prompt-a-thon

Setting the scene for this one and diving right in. Tim talks about successful experimentation being cultivated, not designed or manufactured and why building adoption for experimentation (or any organisational change) comes down to a simple question: are you on the soapbox or are you driving the bus?Tim breaks down why correcting people (“you’re doing it wrong”) creates defensiveness, why the “soapbox” approach is the shortcut, and how the “bus ride” approach builds sustainable adoption. He shares an example from Minecraft, walking through a product lifecycle approach, observation, theory, test, bringing teams along for research, insights, solutioning tied to OKRs, prioritisation, experimentation, analysis, and implementation.They then shift into AI and experimentation: where Tim sees real value today (speed from large datasets to insights), where he’s sceptical (AI-generated high-fidelity designs and experiment variations), and where opportunities may be strongest (analytics, projections, predictive modelling, multi-armed bandits, variance reduction, and qualifying/disqualifying users).They close with Tim sharing why he opened time slots for people to chat about experimentation, why he was encouraged by his manager, and why LinkedIn is the best way to connect.What’s covered (from the conversation):“Soapbox vs bus ride” and sustainable adoptionDefensiveness, reactance, and avoiding “shaming” in change effortsMinecraft example: bringing teams through the product lifecycle with experimentationSeparating observation → theory → testTying solutioning to OKRs and prioritising experimentsAI in discovery: summarising large qualitative (and quantitative) datasets quicklySpot-checking + context pitfalls (e.g., “creeper,” profanity interpreted as negative)Prompt templates vs AI agents with guardrailsScepticism on AI for full-fidelity designs and experiment variationsAI/ML in analytics: sample size, test duration, forecasting, bandits, variance reductionCommunity learning + LinkedIn as the main contact channel

In this episode, Matt sits down with Tim Mehta, Experimentation PM at Minecraft, to discuss how one of the world's biggest games approaches growth and testing. Tim shares his journey from agency life to big tech (Microsoft, Lucid) and reveals the inner workings of Minecraft's experimentation program.We dive deep into:• The "Spillover Effect": Unique challenges of A/B testing in a multiplayer game environment and how to solve them.• Delivery vs. Growth Mindset: Why simply shipping features isn't enough and how to shift focus to measurable outcomes.• The Ladder of Evidence: Why tests backed by user research have a 50% success rate compared to the industry average.• 3 Pillars of Experimentation: Trustworthiness, Velocity, and Democratization—and why trust must always come first.Whether you're a Product Manager or Growth Marketer, Tim's insights on building a culture of evidence-based decision-making are invaluable.

Join host Matt Wright with Brent Kostak and David Arbour from Adobe as they reveal how the new Experimentation Accelerator is transforming digital experimentation through AI automation, causal insights, and agentic workflows.Learn how Adobe is helping brands scale testing velocity, uncover deeper experiment insights, and unify teams across marketing, product, and analytics. Featuring early beta results, real-world use cases, and practical advice for adopting AI-driven experimentation successfully.

Are your experiments too safe? In this coffee break episode of The Conversion Podcast, Matt and Steven reveal how to build an experimentation “risk profile” that balances safe, incremental wins with bold bets your competitors are too nervous to run.You’ll learn why treating your test backlog like an investment portfolio unlocks competitive advantage, how A/B testing acts as your safety net, and real-world examples where high-risk experiments drove massive gains, from Domino’s delivery fees to SaaS free trial removals.Whether you’re running CRO programs, growth experiments, or digital product tests, this episode shows you how to reduce fear, maximise ROI, and scale with confidence.

When your paid ads promise one thing and your landing page says another, conversions suffer. In this episode of The Conversion Podcast, Matt and James dig into a classic - yet often overlooked - challenge in digital marketing: maintaining a consistent “scent trail” from ad to landing page.The case study centres on a SaaS brand running heavily paid search campaigns, both branded and non-branded. Their ads emphasised a free trial, but the landing page buried this message at the very bottom. The result? Mismatched expectations and flat conversions, despite expanding the page with more product information and CTAs.By aligning the landing page headline and CTA with the ad promise, “Start your free trial today”, and reinforcing key points like 14-day trial length, easy setup, and cancellation flexibility, the team achieved a 7% lift in leads.This episode is a must-listen for performance marketers, growth teams, and product leaders who want to maximise ROI from paid campaigns and ensure messaging alignment drives results.#paidmedia #saasmarketing #conversionoptimization #digitalmarketing #landingpages #growthmarketing #abtesting #customerjourney #ppc #marketingstrategy #onlinemarketing #performanceadvertising #cx

Sometimes, the “no-brainer” ideas turn out to be… not so smart.In this episode of The Conversion Podcast, Matt and James break down a fascinating experiment run on KitchenAid’s iconic stand mixers — one that proves why you should never skip testing, even when you’re certain the idea will win. The team merchandised a 90-day money-back guarantee on product and listing pages, expecting to reduce buyer anxiety and boost conversions. Instead? Orders dropped by double digits.We explore why this happened, the psychology behind it, and how context changes everything when it comes to persuasion tactics. You’ll learn how guarantees can sometimes introduce doubt instead of removing it, why category leaders need different messaging strategies, and how to structure clean, holistic experiments for reliable insights.If you’ve ever launched a “sure thing” without testing — or almost did — this episode will make you think twice.#conversionrateoptimization #abtesting #digitalmarketing #experimentdesign #productmanagement #growthmarketing #ecommerce #uxdesign #onlinemarketing #conversionoptimization #customerpsychology #contentstrategy #productpage #marketingstrategy #businessgrowth

In this bonus episode of the Conversion Podcast, James shares a fascinating case study about using video to improve product comprehension for a tech product, a cell phone signal booster. We discuss the results of running experiments that tested video placement on landing pages and product pages, revealing unexpected insights about video’s effect on conversions. By iterating and testing in different ways, the team discovered that adjusting video positioning led to a 35% increase in sales despite a drop in video views. Tune in to hear how this experimentation approach helped boost product comprehension and sales, and why understanding user behavior is crucial for conversion optimization. Perfect for marketers, product managers, and business leaders looking for data-driven strategies.

HP’s Instant Ink has revolutionized the ink cartridge industry, offering consumers up to 50% savings and the convenience of never running out of ink. Launched in 2013, it now boasts over 10 million subscribers across 38 countries. In this episode, Sovrin Tolia, HP's Head of Product, and Stephane Bardin, Worldwide Product Manager, dive deep into the secrets of Instant Ink’s rapid growth and success. From overcoming initial consumer behavior challenges to utilizing experimentation and innovative business models, they share key insights into scaling a successful subscription-based service. Learn how HP transformed the printing experience and continues to evolve with the help of data-driven decisions and continuous testing.#HP #InstantInk #SubscriptionBusiness #Innovation #GrowthStrategies #CustomerExperience #BusinessModelInnovation #DataDriven #ABTesting #MarketingStrategy #ProductLedGrowth #ConsumerBehavior #TechGrowth #DigitalTransformation #SubscriptionServices