Klaviyo Data Science Podcast

Games typically involve some combination of math, chance, strategy, and skill. In this two part mini-series, the Klaviyo Data Science Podcast investigates the role of math in games and how our relationship to them is changing given the increasing number of sophisticated AI Agents that can often beat us at our favorite games.This podcast additionally covers concepts ranging from what makes a game ‘Solved’, to how the early development of Game Theory was influenced by the Cold War.The team also touches on a topic that will be expanded on in the next episode: why AI Agents are so sophisticated at certain games (e.g. Go), while struggling with other games (e.g. no stakes poker or basketball).When they invent the robot LeBron James, it will be the end of human society.- Michael LawsonFor more details, check out the ⁠⁠⁠full writeup on Medium⁠⁠⁠!

All companies start small. The lucky ones grow, and growth necessarily comes with change. This month on the Klaviyo Data Science Podcast, we look at Klaviyo’s growth through the past 8 years and the profound effects that growth has had on R&D, ranging from the tooling we use to the processes we employ to the very culture we strive to cultivate. Listen in to learn:How the day-to-day of some of our most senior engineers has changedWhy common headaches like planning, deployment, and cross-team coordination can get harder as your org grows — and why they can get easier What a single lunch can teach you about a company’s culture For more details, check out the ⁠⁠full writeup on Medium⁠⁠!

This month, we return to a classic Klaviyo Data Science Podcast series: books every data scientist (and software engineer) should read. This episode focuses on the Clean * duology by Robert C. Martin, which teaches the principles of both clean code and clean architecture. We’ve brought on two senior engineers at Klaviyo who’ve learned, practiced, and developed their own opinions on the lessons in these books. Listen in to learn:How to use these books to level up your own skills and the skills of your teamWhy the book’s spiciest opinions make sense, and where you might disagree with them in practice What our panel’s deepest, most intimate thoughts on docstrings areFor more details, including links to these books, check out the ⁠full writeup on Medium⁠!

What should I do next? A common question, one that seems simple on the surface, but the answer, especially a more optimal answer, can be very difficult to uncover. It may involve information that the asker is not aware of, be unintuitive, or even be counter to our instincts.This month, we discuss a new Klaviyo feature: Next Best Action. Utilizing Klaviyo’s knowledge of marketing best practices, we can recommend specific actions that are likely to be highly advantageous

All successful teams have at least one leader, and most have at least one manager. This episode, we dive into how leadership works on highly technical teams, how managing a highly technical team works, and why the two aren’t exactly the same thing. Listen along for more discussion about:The traits of highly effective leaders — and how that might look different on an engineering or data science team How to know that a move into management fits you Our guests’ best recommendations for management books, resources, and experiencesFor more details, including links to the many resources our panel suggest to learn more about leadership and management, check out the full writeup on Medium!

What is agile methodology — and, just as importantly, what is it not? Whether you’re new to agile entirely or you stay up late pondering its most philosophical inner workings, if you want to know more about agile and how organizations can reap its benefits while avoiding its pitfalls, this is the episode for you. You’ll learn about a variety of topics, including:How to effectively compromise between the tenets of agile and the realities of building software — and how not to When agile helps you pivot, and what that means for your customers Our guests’ hottest takes about agileFor the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

This month, the Klaviyo Data Science Podcast welcomes Evan Miller to deliver a seminar on his recently published paper, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations! This episode is a mix of a live seminar Evan gave to the team at Klaviyo and an interview we conducted with him afterward. Suppose you’re trying to understand the performance of an AI model — maybe one you built or fine-tuned and are comparing to state-of-the-art models, maybe one you’re considering loading up and using for a project you’re about to start. If you look at the literature today, you can get a sense of what the average performance for the model is on an evaluation or set of tasks. But often, that’s unfortunately the extent of what it’s possible to learn —there is much less emphasis placed on the variability or uncertainty inherent to those estimates. And as anyone who’s worked with a statistical model in the past can affirm, variability is a huge part of why you might choose to use or discard a model.  This seminar explores how to best compute, summarize, and display estimates of variability for AI models. Listen along to hear about topics like: Why the Central Limit Theorem you learned about in Stats 101 is still relevant with the most advanced AI models developed today How to think about complications of classic assumptions, such as measurement error or clustering, in the AI landscape  When to do a sample size calculation for your AI model, and how to do it About Evan Miller You may already know our guest Evan Miller from his fantastic blog, which includes his celebrated A/B testing posts, such as “How not to run an A/B test.” You may also have used his A/B testing tools, such as the sample size calculator. Evan currently works as a research scientist at Anthropic.  About Anthropic Per Anthropic’s website: You can find more information about Anthropic, including links to their social media accounts, on the company website. Anthropic is an AI safety and research company based in San Francisco. Our interdisciplinary team has experience across ML, physics, policy, and product. Together, we generate research and create reliable, beneficial AI systems. Special thanks to Chris Murphy at Klaviyo for organizing this seminar and making this episode possible!  For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… 2024 Year in Review As the new year starts, we take a look back at 2024. We spoke to data scientists and people who work closely with data scientists, and we asked them all the question we ask every year: what is the coolest data science thing you learned about in 2024? You’ll hear a wide range of answers, including: How a rhyme can topple LLM security Why the Sequential Probably Ratio Test is better at measuring basketball ability than the NBA playoffs  How badly a non-specialized LLM could beat you at chess For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

How we protect invention and ingenuity: Patents Writing software often involves taking known patterns, combining and shaping them, and adding needed context or specialization related to the problem we’re trying to solve. Sometimes, that means writing something that’s effectively been written by someone else before. But sometimes, that means creating something new. What should you do in a case where you’ve genuinely created something new? Perhaps more importantly, how do you know when you’re in that situation?  This month, we explore one of the best tools to help answer both questions: the patent process. Listen along with your fearless co-hosts and a member of Klaviyo’s legal team to learn about what a patent is, why getting them matters, and how to get your own novel work patented, along with: Tips to keep in mind when applying for a patent  Knowing which software you should try to patent How to get your organization to buy into patents  For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.

Welcome to the November episode of the Klaviyo Data Science Podcast for 2024! In years past, November episodes reflected the chaotic Black Friday/Cyber Monday season by examining unique challenges of readiness, scale, and fundamental changes happening with little to no warning, as well as how those challenges were handled; this November is no different. What happens when two of the largest email platforms make sweeping changes to their spam filters, providing a few short months of notice? Stress, uncertainty, and an opportunity for individuals and organization to rise to the challenge. In this month’s episode, we talk with analysts, engineers, and product managers to discuss Klaviyo’s journey to meet Yahoo and Google’s new Email Delivery Requirements — aka Yahoogle, the colloqial name for a new set of rules that must be followed by email senders to have their emails make it to inboxes and not go straight to the junk bin. Listen in to hear more about: Yahoogle. It’s more than just a funny word! The challenges in operating a large-scale email sending system Advice and retrospectives about large, cross-functional projects and tight deadlines For the full show notes, including who's who, see the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Medium writeup⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠.