Your AI Agent Knows More About You Than You Think — Microsoft’s Alyssa Ofstein
Your AI agent is quietly building a profile of you. The real question is what happens to that profile, and who would even notice if something went wrong.In this episode of The AI Frontier Playbook, I sit down with Alyssa Ofstein, Senior Technical Program Manager in Microsoft’s AI Safety and Security team, where she has trained more than 17,000 people on how to build AI agents safely. Alyssa sits right between the red team, the people trying to break AI, and the engineers who build it. Her job is to turn what goes wrong into guidance people can actually follow.Alyssa explains the risk most people never think about: agent memory. Agents don’t just store what you tell them, they make inferences about you, and those inferences, true or not, can spread across every system the agent touches. She is candid about why companies say they want responsible AI but reward speed, why trust and safety teams are still understaffed, and why she agrees that frontier labs should run like nuclear facilities.We also get practical: what makes AI guidance obvious, adaptable, and tangible enough that engineers follow it, how Medusa’s Memory Heist, the open source game she built with Joris de Gruyter, teaches AI security through play instead of fear, the first three things a company should put in place, and the three questions Alyssa asks before she hands any task to AI. She closes with an emotional answer about what a random Tuesday in 2036 could look like if we get this right.You'll learn:How Alyssa turns red team findings and incident reports into guidance engineers followWhy agent memory is the AI risk nobody thinks aboutHow agent inferences about you can spread across tools, networks, and other people’s agentsWhy companies say “responsible AI” but reward speedWhy frontier labs should run like nuclear facilitiesWhat makes AI guidance obvious, adaptable, and tangibleHow Medusa’s Memory Heist teaches AI security through play, not fearThe first three things to put in place: data hygiene, a reward system, and stakeholder relationshipsWhy you can’t rely on your AI provider aloneWhat Alyssa will never hand over to an AI agentHer three-question framework before giving any task to AIWho blows the whistle when every worker is an agent?Why there is a permanence to what we tell an LLMWhat happens when an agent’s goal isn’t your goalTen years out: what AI could mean for a random Tuesday in 2036Whether you’re building AI agents, rolling out AI across your company, or just using AI every day at work, this episode gives you a clear way to think about the risks, and how to stay in control.Resources & LinksAlyssa LinkedIn: https://www.linkedin.com/in/alyssaofstein/Medusa’s Memory Heist (free, open source): https://microsoft.github.io/medusa/site/Our Sponsor Wepoint: https://www.wepoint.com/en📬 Enjoyed this episode?Subscribe to the AI Frontier Playbook: https://samuelboulangerai.substack.com/Connect with me on LinkedIn: https://www.linkedin.com/in/samuelboulanger/Visit the website: https://frontierplaybook.aiNewsletter: https://theaifrontierplaybook.substack.com/