Governance, Architecture, and the Evolving Role of Data Engineers in the AI Age

 In this week’s show, Frank La Vigne sits down with data and analytics engineer Wasim Rana for a deep dive into the realities of building, managing, and securing data infrastructure in modern businesses. Together, they explore the critical challenge of preventing data lakes from turning into data swamps, the practical pipeline from raw to curated data—including the increasingly popular bronze, silver, and gold layering approach—and the vital role of governance in today’s data-heavy world. Waseem Rana shares insights from his hands-on experience with AWS, Snowflake, and transforming messy datasets into actionable business intelligence. The conversation also takes a look at the evolving landscape of data engineering amid AI advancements, what skills data professionals need to stay ahead, and how future architectures may blend data lakes, warehouses, and vector databases for AI-driven analytics. Whether you’re a data engineer, business leader, or just curious about the future of data, this is an episode you won’t want to miss!LinksWasim's LinkedIn -https://www.linkedin.com/in/iamwasimrana/Wasim's GitHub -https://github.com/wasimranacseWatch on YouTube -https://youtu.be/8oGEEN6BubkPodcast Episode Mentioned:The AI Driven Leader: Rethinking Strategy, Decision Making, and Personal Growth -https://datadriven.tv/episodes/the-ai-driven-leader-rethinking-strategy-decision-making-and-personal-growth/Book mentioned:The AI-Driven Leader: Harnessing AI to Make Faster, Smarter DecisionsHardcover -https://amzn.to/4crqshxKindle -https://amzn.to/4u8rVPSAudiobook -https://amzn.to/3Qqsc1XTime Stamp00:00 Working as a data engineer03:22 Using AWS for infrastructure06:31 Transforming raw data for use11:14 Discussing data sources and ingestion15:51 Discussing data analysis with AI models16:33 The future of data analytics21:58 Importance of data governance24:44 Evolution of data storage solutions29:34 AI's impact on data jobs31:20 Understanding data architecture importance35:49 Understanding AI creativity and context37:14 Understanding AI's lack of context40:43 Difficulty finding meaningful connections45:14 Using storytelling to drive change47:44 Connecting pipelines to WooCommerce

Today, we journey into the fast-evolving world of prediction markets, KPI trading, and the new frontiers of retail finance. Joining us is Candace, alongside our guest from Benzinga, a fintech innovator working to democratize financial data once reserved for Wall Street elites.We delve into how platforms like Benzinga are leveling the playing field, making actionable market information accessible for everyone—from individual retail investors to advanced quant traders. Get ready as we unpack the rise of alternative data, the intersection of finance and AI, and whether prediction markets are the next big tool in forecasting—or just another signal to question in an increasingly complex landscape. Strap in for insights on regulation, market dynamics, and the sometimes wild personalities driving innovation in finance today!LinksAndrew's LinkedIn -https://www.linkedin.com/in/lebbosandrew/Benzinga -https://www.benzinga.com/Benzinga APIS -https://www.benzinga.com/apis/Watch on YouTube -https://youtu.be/xY6mIRP1L2cTime Stamps00:00 Starting Benzinga to democratize info03:22 Building financial data services08:10 Growth through niche news coverage12:57 Hedging with company performance15:00 Early quant experiences at Merrill Lynch20:07 Explaining Polymarket betting mechanics23:34 Discussing market prediction tools26:39 CFTC regulations on trading limits28:04 Discussing crowdsourcing and wisdom34:14 Exploring unique data sources35:35 Discussing the vinyl resurgence41:40 Finding an edge in investing43:42 Wondering about the future of betting47:42 Brokerage purchase options explained50:20 Building ideas with AI tools

this episode, Frank sits down with Andy Boettcher, the Chief Innovation Officer at DoubleTrack, for a candid conversation about embracing the power (and chaos!) of modern AI tools.Together, they dive into how neurodiversity and "chaotic brains" are thriving in this new era, where ideas can be rapidly tested, refined, and built with the aid of tools like Claude code. From the practical realities of using AI for software development and troubleshooting (even in classic cars and household appliances) to the shifting landscape of business innovation, they discuss the opportunities and challenges facing individuals and companies alike.You'll hear first-hand stories about building podcast production tools, leveraging AI agents as virtual team members, the importance of critical thinking in an age of instant information, and why data security and adaptability are more crucial than ever. Whether you're a technologist, business leader, or simply curious about what it means to innovate today, this episode promises insight, humor, and a healthy dose of chaos.Let's get data driven.LinksAndy's LinkedIn -https://www.linkedin.com/in/andyboettcher/ShowDog -https://www.showdog.studio/Time Stamps00:00 What a Chief Innovation Officer does06:18 The age of chaotic creativity09:56 Using AI for planning discussions11:31 Choosing the name Show Dog15:22 Brainstorming and organizing ideas18:43 Using tech for learning20:47 Companies adopting AI strategies25:10 Classic car ownership struggles28:41 Taking time to rethink plans31:34 Using AI to solve problems37:25 Challenges in commercializing academic concepts39:18 Adapting to evolving AI technology43:36 AI tools and problem-solving in coding45:23 Limitations of AI and data security47:33 Data security and CRM setup51:22 Challenging assumptions and blind spots

In this episode of Data Driven, Frank and Andy dive into the future of market intelligence with Dr. Jill Axline, co-founder and CEO of Mavera—a company building synthetic populations that simulate real human behaviour, cognition, and emotion. Forget Personas. We're talking real-time, AI-driven behavioural modeling that's more predictive than your horoscope and considerably more data-backed.Dr. Axline shares how Mavera’s swarm of AI models situates these synthetic humans within real-world business contexts to forecast decisions, measure emotional resonance, and even test marketing messages before they go live. From governance and model drift to the surprising uses in financial services, political campaigns, and speechwriting—this is one of the most forward-looking conversations we’ve had yet.If you've ever wanted a deeper understanding of how AI can augment decision-making—or just want to hear Frank admit asset managers love ice cream—this one’s for you.LinksLearn more about Mavera:https://mavera.ioConnect with Jill Axline on LinkedIn:https://linkedin.com/in/jillaxlineMorningstar:https://www.morningstar.comTime Stamps00:00 - Introduction & AI Swarms Explained03:30 - Forget Personas: Contextual AI Models07:00 - Evidence vs Inference & AI Governance10:20 - Simulation Scenarios & Model Drift14:30 - Synthetic Audiences in Action18:00 - Evidence Feedback Loops & Small Data Challenges22:00 - Industry Applications & Use Cases27:00 - Analyzing Speeches & Emotional Resonance30:45 - Sentiment, Social Listening, and Real-Time News Reactions34:00 - Adversarial Models & Strategic Pushback38:00 - The Cartoon Bank Portal That Failed Spectacularly41:00 - From Skeptic to CEO: Jill’s Journey45:00 - Data Privacy, Compliance & Synthetic Ethics48:00 - Reflections on Empathy, Engineers, and Selling Without SellingSupport the ShowIf you enjoy Data Driven, leave us a review on Apple Podcasts or your favourite pod platform. It helps more people find the show—and fuels Frank’s Monster Energy habit.

In today’s show, BAILeY, your semi-sentient hostess with the mostest metadata, teams up with Frank La Vigne to welcome the ever-insightful Andrew Brust for a deep dive into the evolving Microsoft data ecosystem. From nostalgic tales of Windows history and scoring elusive Clippy swag at Ignite, to unraveling what makes Microsoft Fabric a game-changer for data integration, AI, and governance, this episode covers it all.You’ll hear firsthand how Microsoft’s innovation goes far beyond the tech itself—focusing on seamless integration, unified billing, and organizational synergy. Andrew Brust sheds light on the journey from fragmented Azure services to the unified vision of Fabric, the rise of generative AI and “agentic” intelligence, and the increasingly important role of data sovereignty and governance in today’s regulatory landscape.Whether you’re a data enthusiast, an AI tinkerer, or just in it for the nostalgia, grab your headphones and get ready for insights, laughs, and more acronyms than you can shake a dataset at. Stay curious and caffeinated—this episode has something for everyone!Time Stamps00:00 Microsoft Expertise and Industry Analysis03:18 "Big Data and Analytics Insights"08:31 Power BI's Rise in Azure12:26 "Unified Fabric-Based Data Platform"14:01 "Fabric IQ Powers AI Integration"17:24 "Achieving Synergy Against Odds"23:32 "Unified Compute for Seamless Integration"24:41 "Overwhelmed by AWS Complexity"28:21 "Microsoft Research Powers Azure Fabric"34:20 "Azure Foundry and Tools"37:33 "Flexibility Beyond Major Cloud Providers"41:33 Global Data Privacy Trends45:13 "Governance for Agentic AI"47:29 "Azure Stack and Local Clouds"50:13 "Kubernetes: The Cloud Caveat"53:40 "Let's Reconnect and Reminisce"

Welcome to a milestone episode of Data Driven! In episode 400, hosts BAILeY, Frank La Vigne, and Andy Leonard gather to reflect on nearly a decade at the forefront of podcasting about data, AI, and the world of software engineering. This special edition takes you behind the scenes with stories of tech evolution, personal growth, and the wild journey from their earliest recordings to today’s AI-powered workflows.The team digs into how generative AI has transformed their creative process—making it possible for small teams to produce vast amounts of content, drive innovation, and manage multiple podcasts at a high level. You'll hear about Frank’s latest tool, Podsy—a platform built to help creators manage the ever-growing tsunami of podcast assets using cutting-edge AI—and how tools like Claude, Grok, and Opus are unleashing new possibilities for automation and storytelling.With their signature tangents, candid stories (including car accidents, water heater mishaps, and parenting milestones), and a bit of nostalgia about the early days of podcasting, this episode captures what makes Data Driven both insightful and relatable. If you’re a developer, data professional, or a fellow podcaster looking to stay ahead of the curve, episode 400 is packed with practical lessons, inspiration, and a few good laughs. Dive in as the team celebrates what’s possible in the age of AI—and look ahead to an exciting new chapter for the show!Time stamps00:00 "DGX Spark: Personal AI Supercomputer"03:28 "Tech Innovations and Hardware Updates"07:51 "Daily 5:40am Notification Routine"11:10 "Podzi: Managing Podcast Assets"15:35 "Boosting Value with AI Efficiency"17:05 Claude Enhanced Legacy System23:32 AI Impact on Creative Roles25:58 "Consulting Break: Investing Time"29:33 "Automation Evolution and Tools Demo"31:35 "Podsy Studio: A Journey"35:06 "AI-Powered Content Creation Demo"38:26 "Podcast Organization and Insights"42:10 "Streamlining Metadata for Insights"45:47 "Unveiling AI's Hidden Potential"50:41 "Celebrating 400 Podcast Episodes"54:41 "Off Track Ambitions"57:44 "Reality Strikes Back: Tech Trust"59:36 "Podsy Progress and Workflow"

In this episode of Data Driven, hosts Frank La Vigne, Candace Gillhoolley, and BAILeY sit down with Mike Armistead, CEO of Pulse Security AI—a cybersecurity veteran who's been fortifying digital defenses since before AI made headlines and hackers had professional profiles. Together, they dig into the dual-edged sword that is artificial intelligence in cybersecurity, exploring how AI serves as both a powerful tool against cyber threats and a potential weapon in the wrong hands.Mike Armistead shares stories from the front lines, including his experience during the "code red" era at Google when ChatGPT shook up the tech world, and offers real-world advice on why LLMs (large language models) aren’t the magic fix for every problem—and why they desperately need guardrails. You’ll hear why your next big data breach could be hiding in a cleverly crafted AI prompt, why humans still matter when it comes to judgment calls, and why good old-fashioned security hygiene is as critical as ever.Whether you’re a developer, data scientist, or just password-paranoid, this episode will make you rethink how you approach security in the age of AI. Tune in for expert insights, hard-earned lessons, and a few laughs as the Data Driven crew uncovers where technology, risk, and "common sense" collide.Time Stamps00:00 AI-Assisted Cybersecurity for SOCs04:26 "AI Rush and LLM Insights"09:12 AI-Powered Cybersecurity Strategy Insights10:01 "Cybersecurity, ChatGPT, and Impressions"13:17 AI Tools: Power and Risks18:06 "Teaching Critical Thinking in AI Era"20:59 "Guardrails and Next-Gen AI Systems"24:22 Human Judgment vs AI Limitations27:37 "Pressure Testing for Accuracy"30:09 Future Tech Advancements and Challenges34:58 "Risk Awareness Beyond Compliance"37:38 "Cybersecurity Risks and AI Defense"41:54 Cybersecurity Risks and Preparedness43:04 "Situational Security in Practice"46:05 "Cybersecurity's Evolving Threat Landscape"51:52 "Builders vs. Destroyers Mindset"55:05 Modern Password Practices56:39 "Pulse Security AI & Community"

In this episode, Frank La Vigne sits down with Itay Haber, CEO of Data Noetic, to unpack how AI is revolutionizing supply chain management. Forget spreadsheets and dashboards—Data Noetic is building an autonomous digital brain that proactively tackles delivery bottlenecks and bridges the gap between scattered data and process improvement.You’ll hear real-world tales of missed bathroom tile deliveries, multi-million dollar construction delays, and the true impact of getting ahead of supply chain hiccups before they snowball. The trio explores how agentic AI isn’t just hype: it’s driving tangible results, saving time, boosting KPIs, and reimagining how companies of all sizes make decisions. From pharmaceuticals to consumer packaged goods, discover why trust, transparency, and agility are the new gold standards in supply chain operations—and how data-driven agents just might become indispensable.Tune in for a masterclass that balances digital wisdom with a dash of dry wit, and learn how emerging tech is helping organizations deliver on time, in full, and with a whole lot less existential angst.Time stamps00:00 "Autonomous Supply Chain Optimization"06:01 "Optimizing On-Time Delivery Failures"07:27 Proactive Warehouse Order Management13:06 "Aligning Perception with Reality"15:08 "Streamlining Order Fulfillment Process"18:18 "AI Revolutionizing Problem Coordination"22:18 "Data Validation and AI Insights"25:04 Predictive KPI Monitoring with Gen AI27:09 Clarifying Questions for Assistance31:37 "Tailored Software Delivery Models"34:35 "AI's Role in Complex Industries"37:00 "AI Focus and Value Debate"42:37 "AI Bubble and Valuations"46:43 AI's Transformative Impact on Jobs48:17 AI Enhances Jobs, Not Replaces51:44 "AI: Boom, Bust, Transformation"57:38 "AI, Data, and Change"

Welcome to another exciting episode of Data Driven! On this week’s show, hosts Frank La Vigne and Candace Gillhoolley take you inside the NVIDIA GTC conference in Washington, DC—an event that’s rapidly evolved from a hardware showcase into a crossroads for AI, robotics, and quantum computing innovation. Frank shares his firsthand experience navigating the expo floor, offering a glimpse into the latest cutting-edge tech, from GPU supercomputers to quantum networking breakthroughs and swarms of robots. Candace and Frank discuss the growing intersections between fields like quantum biology and AI, and share stories about the curious mix of attendees—from government officials and policymakers to technologists, students, and even a few cosplay “Jensen Huangs.”Whether you’re a data enthusiast, a future-focused technologist, or simply quantum curious, this episode dives into the national security implications of AI, the importance of lifelong learning in tech, and how the rise of quantum and robotics will disrupt careers and industries alike. Tune in for insider anecdotes, expert encounters, and a dose of practical wisdom about adapting in the world of emerging technologies—the future is here, and it’s happening faster than ever.LinksFrank gets a shout out from Pluralsight -https://www.pluralsight.com/resources/blog/upskilling/frank-lavigne-customer-storyJensen Huang's DTC DC Keynote -https://www.youtube.com/watch?v=lQHK61IDFH4Mariya & Python Simplified -https://www.youtube.com/@pythonsimplifiedTime Stamps00:00 "Nvidia GTC Highlights and Expo"03:36 "Quantum, AI, and Innovation Highlights"07:18 Tech and Government Amid Furlough10:24 "Tech Components in Booz Allen Vehicle"14:37 Meeting Maria Shah19:11 Career Shifts and Evolutions23:06 From Books to Tech Publishing24:44 Quantum Insights with Researcher29:01 "Nvidia: Emerging Defense Contractor"32:25 Tech Innovations: AI, Quantum, Robotics36:44 "Live Streaming Quantum & Tech"37:40 "High-Speed Quantum Interconnects"41:01 Technical Translation for Accessibility44:19 "High School, Future, Nvidia Innovation"49:28 "Guest's Learning Experience"50:45 "Quantum Business Strategy"53:36 British AI Outro Stream

On this episode of Data Driven, hosts Frank La Vigne and Leonard celebrate a major milestone: the 30th anniversary of Franksworld.com, one of the OGs of tech blogging that’s survived multiple browser wars, dot-com crashes, and the relentless march of Internet time. Joined by guest Saket Saurabh, they dive into the rapidly evolving landscape of AI, data science, and technology, comparing today’s AI boom to previous tech bubbles and exploring trends in sovereign AI, data sovereignty, and global competition in quantum computing.The conversation takes us from the days of hand-coded HTML uphill in both directions, to modern challenges like scheduling snafus, rapid automation, and the possibilities (and pitfalls) brought on by artificial intelligence. Along the way, Frank and Saket reflect on the cycles of tech innovation, the impact of global politics on data and AI, and what it really means to adopt new technology mindsets.Tune in for a lively ride through tech history, lessons learned, and a discussion on how a scrappy few—and a little help from AI—can rival teams of seventy. If you’re curious about what’s next in AI, data, or you just want a dose of digital nostalgia and practical insight, this episode’s for you.LinksFranksWorld.com -http://www.FranksWorld.comWho Moved My Cheese -https://amzn.to/4oeqGvsImpact Quantum Global Report -https://impactquantum.com/GlobalReport/Impact Quantum Podcast -https://impactquantum.com/AutonomousWarfare.ai -https://autonomouswarfare.ai/Some links may be affiliate links.Time Stamps00:00 Grok 4 Daily Updates05:22 "AI Bubble: Caution and Trends"12:10 "X64 Chips and Tech Rivalry"19:49 "Appreciating Thoughtful Scientific Content"25:34 "History's Arcs and Overcorrections"27:24 "Unique Identity Beyond Models"32:03 "Reexamining Privacy Awareness"41:22 "Geopolitical Prudence and Resources"47:49 "AI, Podcasts, and Off-Roading"50:37 "Talking Recycling Bin Origins"54:37 "AI as a Multilingual Solution"59:49 Reinventing MCP and Context Memory01:05:12 "Artistic Talent Meets Engineering"01:14:08 "Data, AI, and Mastermind Insights"