404 - Why Most Email Marketing A/B Tests Are a Waste of Time (And How to Run One That Actually Works)

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor A client once asked me to A/B test the timing in one of their abandoned cart flows. On paper they had plenty of data - a few thousand people entering the flow every month. But once I mapped out how the flow was actually built, I knew that test would run for weeks, potentially months, and tell us absolutely nothing at the end. That's the trap most people don't see coming. A/B testing sounds scientific and rigorous, but most email marketing tests are set up in a way that makes a real answer essentially impossible. The list gets sliced too thin. The hypothesis is vague. Two versions of a subject line get tested once and someone calls it a win. One result is basically a coin flip wearing a lab coat. Noah Kagan, founder of AppSumo, has said only about one in every eight of their tests actually produces a significant result. So if your tests keep coming back inconclusive, you're in very good company. That doesn't mean you shouldn't test. It means you should set up tests so they have a real shot at telling you something. In this episode I walk through the traps that make most A/B tests a waste of time, how to write a hypothesis that actually means something, and the framework for running a test that gives you a real answer - or at least a useful one. ✨ In this episode, you'll learn: The flow A/B test trap that looks like plenty of data until you map out where people actually go Why knowing when a test can't answer your question is just as valuable as running one that can Why one subject line result on its own is basically meaningless - and what to test instead The types of questions that are actually worth testing consistently over time How to write a hypothesis using the if-then-because framework before you set anything up Why you should only ever change one thing at a time - and why it feels slower but matters How to pick your success metric before you hit send - and why open rate isn't always the right one How many people each version actually needs to see before a result means anything Why flow tests can take weeks or months - and why you should never edit a live test What to do with inconclusive results - and why a tie is sometimes the most useful answer How one client's three-way sign-up form test saved them money even though all three versions performed the same Most A/B tests don't fail because the idea was bad. They fail because they were never set up to answer a question in the first place. Work with Joy Joya: https://joyjoya.com

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor I got an email recently where the sender name was "The Big Fall Sale at Victoria's Secret." In my inbox preview, the "at Victoria's Secret" part got cut off entirely. For a second, I genuinely had no idea who it was from - and I'm a longtime subscriber. Victoria's Secret almost certainly has a deliberate system behind their ever-changing sender names. Two sub-brands, a full CRM team, probably rules about exactly when Pink should appear versus Victoria's Secret versus something more playful. But even with all of that, it still misfired. A real sender name, on a real email, from a real subscriber's perspective - unrecognizable for a moment. And that's the risk. There's also data backing this up. A study tested sender name changes on four real newsletters and found that in every single case, changing the sender name caused a decline in open rate, click to delivery rate, and click to open rate. The part that really stuck with me: even a month after reverting to the original name, engagement still hadn't fully bounced back. Trust is slow to rebuild once it's shaken. But here's where it gets nuanced - because a multi-name system can actually work. It just has to be intentional, small, and consistent from the start. And your sender name and your sender email address are two completely different things with very different rules. In this episode I break down how to think about sender names, when changing them is risky, what a smart system actually looks like, and why your sender email address is the one you really cannot casually mess with. ✨ In this episode, you'll learn: Why Victoria's Secret's sender name strategy is worth studying - and why most brands shouldn't copy it What the data actually says about changing sender names and what it does to engagement Why trust is slow to rebuild after a sender name change - even after you revert The important distinction between unplanned sender name changes and a deliberate multi-name system How to build a small, intentional sender name system that builds recognition instead of breaking it Whether your sender name should be a brand name, a person's name, or a hybrid - and what actually matters Why your sender name and your sender email address have completely different rules Why sending from a personal Gmail or Yahoo address is one of the most common reasons brands land in spam What to check in Klaviyo if you ever do need to update your sender email Your sender name is worth being deliberate about. Your sender email address is a different story entirely - leave that alone unless you have a real reason and a real plan. Work with Joy Joya: https://joyjoya.com

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor Most ecommerce brands send their emails whenever they happen to finish writing them. Or they pick a time that feels right and stick with it forever. Klaviyo actually gives you three different ways to optimize when your emails go out - and most people are only using one of them, if any. There's fixed send time, which most people know about. There's Smart Send Time, Klaviyo's audience-level optimization tool that finds the best time for your whole list. And there's Personalized Send Time, a newer feature that predicts the best moment for each individual recipient based on their own engagement history. One brand saw over a 10% lift in placed order rate after switching to it for non-urgent campaigns. But none of that matters if you're accidentally locking your audience in before your campaign even sends - which is exactly what happens when one easy-to-miss checkbox gets left unchecked. In this episode I walk through all three send time options, when to use each one, what actually drives good send time decisions, and the one setting I want everyone to go check in their account right now. ✨ In this episode, you'll learn: The three send time options Klaviyo gives you - and what each one actually does How Smart Send Time works, what it requires, and why you shouldn't use it for time-sensitive campaigns How to manually test send times if you're under the 12,000 subscriber threshold What actually drives good send time decisions beyond just "when do people check email" How Personalized Send Time works differently from Smart Send Time - and which accounts can access it What a reasonable starting point looks like if you're just getting started with send time testing The "Determine Recipients at Send Time" checkbox that most people miss - and why leaving it unchecked means some subscribers never get your campaign Why being deliberate about send time is a meaningful upgrade even without any advanced features Most brands are underusing what Klaviyo actually gives you here. Even small changes to how you think about timing can make a real difference. Work with Joy Joya: https://joyjoya.com

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor Most ecommerce brands spend months planning their marketing calendar. Kayla Sanders plans hers about a week out - sometimes less - because she never knows what's going to walk through the door. Kayla runs marketing and online sales for The Jewel Box at Accent, an estate and vintage jewelry business in Memphis operating under Accent Jewelers, a family business with over 30 years of history. Every piece is one of a kind. Once it sells, it's gone. And the inventory changes every single week. That constraint creates real marketing challenges - but it also creates something most ecommerce brands spend a lot of energy trying to manufacture: genuine urgency and scarcity. When you feature a piece in an email and it sells out before everyone opens it, that's not a failure. It's proof that the product is worth paying attention to. We worked closely with Kayla to set up The Jewel Box's Klaviyo account, their SMS program, and the segmentation strategy behind it - including how to reach local in-store customers differently from online shoppers across the country. In this conversation we get into how that all came together, what happened when a Valentine's Day giveaway grew the list fast but brought in the wrong subscribers, and the ring she still has nightmares about not buying. ✨ In this episode, you'll learn: How to build a marketing calendar when your inventory is a weekly surprise Why The Jewel Box uses SMS specifically to reach local customers - and how that's different from their email strategy How segmenting by location shaped the entire approach to their marketing The welcome discount challenge that happens when online codes walk into a physical store What happened when a Valentine's Day giveaway grew the list fast - and why list quality matters more than list size Why genuine scarcity is a marketing asset most brands can't replicate - and how to lean into it The reporting challenge of tracking customers across online, in-store, different POS systems, and multiple marketing channels How social, email, and blog content each serve a different role in educating and converting a jewelry customer The one piece Kayla still hasn't gotten over When something is truly one of a kind, urgency doesn't have to be manufactured. It's just the truth. Connect with The Jewel Box at Accent: Website: https://thejewelboxataccent.com Instagram: https://www.instagram.com/thejewelboxataccent/ Work with Joy Joya: https://joyjoya.com

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor My client forwarded me an ad that said something like: 85% of your ecommerce site visitors aren't getting your abandoned cart emails. And her reaction was exactly what the ad was designed to produce - wait, are my flows not working? They were working fine. The ad had nothing to do with whether her Klaviyo emails were sending. It was selling something called identity resolution, and it was using a very real-sounding statistic to make her feel like she was missing out on a massive chunk of revenue. Identity resolution sounds technical, but the core question is simple: is this tool connecting people who already gave you their information, or is it pulling in people from somewhere else entirely? Because those two things are very different - legally, ethically, and practically. The conservative version of identity resolution is something Klaviyo already does: connecting the dots when a subscriber's email, phone number, and purchase history all belong to the same person. Everyone in that picture opted in. The data is just being cleaned up. The other version - the one being pitched to my client - uses IP addresses, device fingerprints, and third-party data networks to identify anonymous site visitors and feed them into your abandoned cart flows as if they'd subscribed. They didn't. They gave their email to some other website or app at some point, and now it's being used to send them emails from your brand. In this episode I break down how this actually works, why it's technically legal in the U.S. even though it feels like it shouldn't be, what I've seen in real accounts using these tools, and why my agency doesn't recommend them. ✨ In this episode, you'll learn: What identity resolution actually means - and why the term covers two very different things How the scare tactic ad targeting ecommerce brands actually works - and what it's really selling The difference between what Klaviyo does with identity resolution and what third-party tools are doing How these tools use IP addresses, device fingerprints, and third-party data to identify anonymous site visitors Why sending to people who never opted into your list is technically legal under U.S. federal law - and why that doesn't mean it's a good idea What I've actually seen in real accounts using these tools and why most clients eventually cancel Why the revenue lift is harder to prove than the pitch makes it sound The questions to ask any vendor selling identity resolution before you spend a dollar on it Just because something is technically allowed doesn't mean it's the smart move for your brand. Work with Joy Joya: https://joyjoya.com

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor When people talk about segmentation, they're almost always talking about who an email is going to. This segment gets this campaign, that segment gets that one. Nobody's really talking about who should not be getting it - and that's a big blind spot. Exclusion segments are exactly what they sound like: groups of people who are intentionally not receiving a specific email, either temporarily or as an ongoing rule running in the background of every campaign you send. And when I audit a Klaviyo account, they're almost never there. This isn't just about keeping things tidy. Every time you send to someone who shouldn't be getting that email, you're increasing the odds of a spam complaint or a low open rate. Inbox providers like Gmail are paying attention to that. Do enough of it and it starts affecting whether your emails land in the inbox at all - for everyone on your list, including the people who genuinely want to hear from you. The reason this gets overlooked is that it's invisible by design. When exclusion segments are working, you don't really notice them. But that's exactly why they matter. In this episode I break down what exclusion segments are, why skipping them is quietly hurting your list health, and exactly which groups are worth building exclusions around right now. ✨ In this episode, you'll learn: What exclusion segments are and why they're almost always missing from Klaviyo accounts Why sending to the wrong people hurts deliverability for everyone - including your best subscribers Why wholesale contacts should almost never be getting your DTC marketing emails The flow recipient overlap problem and how exclusions prevent a subscriber from getting two emails at once Why unengaged subscribers and sunset flow graduates need to be excluded from regular campaigns How to handle bounces and spam complainers with exclusions What role-based email addresses are and why they don't belong on your active list How to set up exclusions mechanically inside Klaviyo - it's simpler than most people think Exclusion segments are unglamorous, invisible, and one of the most important things you can do for your list health. Work with Joy Joya: https://joyjoya.com

Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor Someone at a pool party once asked me, dead serious, if people even look at email anymore. I get some version of this question a lot, so I wanted to actually answer it with real numbers instead of just my gut reaction. Turns out the assumption isn't even close to true. Somewhere around 93 to 99% of email users check their inbox every single day, there are more than 4.5 billion email users worldwide (more than any single social platform), and email marketing consistently returns somewhere between $36 and $42 for every dollar spent. For my own clients doing it well, email drives around 40% of their ecommerce revenue. I also dig into what I think is actually behind that pool party comment: the assumption that Gen Z has abandoned email for texting and social. The data doesn't back that up either - they're just doing it differently. And the part I think matters most: why email keeps winning even as everyone assumes it's dying. It comes down to two things - you own your email list outright, unlike a social following that can vanish with an algorithm change, and for the person receiving it, email is opt-in and still one of the few corners of the internet you actually control. ✨ In this episode, you'll learn: The real numbers behind how often people check email, and how many people use it worldwide Why email marketing's ROI is hard to match with any other channel Whether Gen Z has actually abandoned email (spoiler: no) and how they use it differently The two real reasons email keeps outperforming assumptions - one about who owns the channel, one about the actual user experience Why an opt-in, algorithm-free inbox is genuinely rare online right now Get this free resource: The Front Door Report - an audit of 360+ ecommerce brands' welcome flows → https://www.joyjoya.com/frontdoor Work with Joy Joya: https://joyjoya.com

Get the FREE "Front Door" Report: https://joyjoya.com/frontdoor Jodie Minto started her fashion brand at weekend market stalls with a glass fishbowl, handwritten slips of paper, and a monthly caftan giveaway. By the time she launched her website, she already had hundreds of email subscribers waiting. On day one, she made sales. She scaled that brand to $1 million a year in revenue over ten years, then sold it - and has spent the years since coaching hundreds of ecommerce founders across all different product categories. She's seen what works, what doesn't, and the nuances that make one tactic land brilliantly for one brand and completely miss for another. What I really wanted to dig into with her was the full picture: how lead gen actually works, how email and paid ads need to work together, what product pages are getting wrong, and something that genuinely surprised me - the cultural differences in how SMS and WhatsApp are used for marketing around the world. Her line that stuck with me most: the fortune is in the follow up. You can get someone to sign up, but if you never actually send them anything, none of the work getting them there mattered at all. ✨ In this episode, you'll learn: How Jodie built her first email list with a fishbowl and handwritten slips - and why it worked The lead gen formats she tested over ten years: giveaways, spin the wheel, discount codes, and what actually converted best Why a travel voucher giveaway brought in all the wrong subscribers - and what that teaches you about list quality The most common email mistake she sees when auditing a new client's account How paid ads and email need to work together - and why relying on ads alone leaves money on the table The Klaviyo and Meta integration that lets you retarget your email subscribers with ads What product pages are almost always missing - and why it's quietly killing conversions The marketing mix she recommends for most ecommerce brands and why all three channels have to show up How SMS marketing is perceived very differently in Australia vs. the US - and what WhatsApp has to do with it The fortune is in the follow up. Getting someone to sign up is only half the job. Connect with Jodie: Website: https://jodieminto.com Instagram: https://www.instagram.com/iamjodieminto/ Work with Joy Joya: https://joyjoya.com

Get the FREE "Front Door" Report: https://joyjoya.com/frontdoor I found a client's popup literally firing on top of another popup, one second apart, both on the screen at the same time. Nobody set it up that way intentionally. It's just what happens when you build one popup, then another, then another - and nobody ever goes back to check how they're all interacting. Exit intent is actually a genuinely useful tool when it's set up correctly. The idea is simple: instead of interrupting someone the moment they land on your site, you catch them at the very last moment - right before they leave without converting. A well-configured exit intent popup can recover somewhere between 10 and 20% of visitors who were about to leave anyway. But exit intent done wrong doesn't just fail to convert. It actively makes the experience worse. And the real danger isn't the exit intent popup itself - it's not knowing what you have running, when it fires, and how it's interacting with everything else on your site. In this episode I break down what exit intent actually is, what makes it work, and the popup stack problem that's more common than most brands realize. ✨ In this episode, you'll learn: What exit intent is and how it works differently on desktop vs mobile Why exit intent popups actually tend to convert better on mobile than desktop The two ways to use exit intent - as your only popup or layered alongside your main one Why your exit intent offer needs to be different from your main popup offer What to ask for on an exit intent form - and what to save for later Why design, targeting, and copy all matter more than most brands think Which pages exit intent makes sense on - and where it's counterproductive The real danger of a popup stack nobody's audited in months How to properly inventory your popups and test the actual new user experience on your own site Exit intent is a good tool. But it's not a set-it-and-forget-it one - especially when you're layering it with other popups. Work with Joy Joya: https://joyjoya.com

Get the FREE "Front Door" Report: https://joyjoya.com/frontdoor Ever gotten a marketing text super early in the morning or way too late at night and thought - that's kind of rude? Turns out there are actually laws about that. And if you're texting your customers without knowing what quiet hours are, you could be violating regulations without realizing it. In the U.S., you're legally not allowed to send marketing texts before 8 a.m. or after 9 p.m. But it goes further than that. Some states are stricter - Florida cuts it off at 8 p.m. And if you text internationally, the rules get more complex. France blocks sending on Sundays and public holidays entirely, enforced at the carrier level, meaning the message just won't arrive. This isn't just about being a polite brand. Violations here aren't hypothetical. I've spoken to ecommerce founders - not my own clients, thankfully - who've received lawsuits over quiet hours violations. And in every single case, it wasn't intentional. It was a rogue automation, an edge case nobody caught, or a setting nobody thought to check. The good news is platforms like Klaviyo have built-in safeguards to help. The important thing is knowing what those safeguards actually do - and where they stop covering you. In this episode I walk through exactly what quiet hours are, which rules apply where, how Klaviyo handles this automatically, and why knowing your settings matters even when the platform is doing most of the work. ✨ In this episode, you'll learn: What quiet hours actually are and why SMS is treated differently than email under the law The U.S. federal rule - and the states with stricter requirements you may not know about How international quiet hours work and why France is a particularly important example Why state-level frequency caps matter too - and which states limit you to three promotional texts per day How Klaviyo's built-in quiet hours settings work and why the defaults are more conservative than the law requires The new state-aware safeguards Klaviyo added and what they actually protect you from Why automatic safeguards aren't foolproof - and the scenarios where they can miss What SMS litigators are and why Klaviyo's setting to block them exists Why setting it and forgetting it is exactly how brands end up in trouble SMS can be one of the highest-performing channels you have. It just needs to be treated with the seriousness it deserves. Work with Joy Joya: https://joyjoya.com