#128 How Data Engineering Has Changed Because of AI

SQL, Python, and one cloud platform. That's what a post I saw claimed makes a good engineer. I don't fully agree, so in this episode I break down what actually makes you a good data engineer or platform architect.It starts with understanding how a data platform actually works: data sources, processing, storage, visualization, and the goals sitting on top of all of it.Once you understand that, the tools make sense, because you can see which category each tool belongs to and how they fit together (I cover this in my Skills and Tools Guide, linked below).Beyond that, I go through the technical fundamentals that actually separate good engineers from the rest:- Infrastructure (servers, SaaS, containers)- CI/CD- Orchestration- Observability and monitoring- Documentation- Problem-solving (finding the actual root cause, not just patching symptoms)- Identity and access management- Cost managementAnd then there's the part most engineers underestimate: communication. I break down the 5 types of people you'll talk to as an engineer, project management, analysts, customers, related teams, and internal management, and why you need to talk to each of them completely differently.Download the free Skills and Tools Guide!Go to https://learndataengineering.com/ and scroll down to the free learning content section.Let me know in the comments what you think makes a good engineer, and what you'd add to this list.

AI has changed engineering a lot, so in this short solo episode I think through where it's actually landed: development, debugging, optimization, and the AI engineering hype.The honest take: AI writes the code now. Our tests, our docs, root causes, query plans that used to need years of experience. Copilot should just be called Pilot. And data modeling? Not the big deal people make it out to be.So if the doing is handled, what's left? Knowing what to do, not doing it. We're the copilots now.I also get into the RAG, MCP, and agents hype, and why I'm not convinced most companies need that yet.How has AI changed the way you work, and what should I create next? Let me know in the comments.

Is AI actually making us better engineers, or quietly making us worse? I sat down with Dan Beach from Data Engineering Central to talk honestly about what it feels like to build data systems in the age of Cursor, Claude, and the so-called 100X engineer.Dan uses AI every day, but he also admits something a lot of us feel and don't say out loud: he misses coding by hand. We got into the whiplash of loving AI one day and feeling like your skills are atrophying the next, why "vibe coding" gets the ticket done but won't save you when things break, and why the hardest parts of the job, system design, architecture, and business context, are still the parts AI can't do for you.From there we went wide. Why data platforms keep dumbing everything down and what that does to your cloud bill, why on-prem and tools like Exasol aren't going anywhere, and why I think data modeling is one of the most overrated topics in data engineering.▶️ Follow Dan on Instagram: https://www.linkedin.com/in/danieldavidbeach/▶️ And Data Engineering Central on Substack: https://dataengineeringcentral.substack.com▶️ Say hello on Instagram: https://www.linkedin.com/in/andreas-kretz/

In this podcast episode, I’m talking with Vadym Kazulkin, AWS Serverless Hero and Principal Cloud Architect.He’s part of the AWS community for years now, speaking at events, testing new features early, and helping the community grow.We talked about:How serverless is being used in the real worldWhy Java still has a place in modern cloud setupsWhat stops big companies from going all inHow AWS is evolving — and what’s missingAnd what engineers should know before going “cloud-native” If you're building in the cloud or just trying to make smart decisions in a fast-moving space, this episode has a lot for you.Connect with Vadym via LinkedIn: https://www.linkedin.com/in/vadymkazulkin/

In this episode, I’m joined by Christina Stathopoulos, a former Googler who now works independently as a data & AI evangelist, trainer, and advisor.Not only did we talk about her path from data engineering into data science, and eventually into teaching and strategic consulting. We also covered:Why many companies still struggle with data & AI basicsThe hype vs. reality of generative AI (and how to cut through the noise)Why soft skills like communication, critical thinking, and storytelling are underrated in techHow regulation in Europe might be slowing things down, and why AI won’t wait for bordersThe difference between online and in-person learning (and why human connection still matters)and much more!All in all: A thoughtful, honest conversation full of insights on AI, data culture, and the human side of technology.Get in touch with Christina via LinkedIn:https://www.linkedin.com/in/christinastathopoulos/

In this episode, I’m talking with Tom Schamberger from the German consultancy msg. He leads their cloud data platform team and has a super interesting background: started coding Java at 12, co-founded startups, and now helps big companies design scalable data platforms.We talk about:What it really takes to be successful in data engineering consultingWhy soft skills matter just as much as tech skillsHow consulting projects actually work — from Excel chaos to full platformsThe role of tools like Databricks, Snowflake, and Microsoft FabricAnd why being tool-agnostic might be your biggest advantageIf you're curious about consulting, data platforms, or just want to hear what a data engineer's job looks like behind the scenes, dive right into it!

In this episode, I sit down with Mehdi Ouazza - data tinkerer, indie hacker, and content creator - who's always up to something interesting in the world of data and AI.We started with DuckDB but quickly veered off into much more exciting territory: side projects, voice-to-SQL with actual quacks, the power of local models, and why WebGPU might be one of the most underrated browser technologies today.We also talked about how we teach and learn data engineering in 2025: the importance of fun, interactivity, and why we both dream of creating a data engineering game that’s part "Among Us" and part serious skills training.Mehdi shares what tools he's using, where he sees GenAI actually helping—not replacing—engineers, and how he's building courses and meetups that inspire creativity in technical work.Perfect for data folks who like to experiment, educators looking for inspiration, or anyone wondering how far a fun idea can go with the right mix of curiosity and tooling.

In this podcast episode, I’m joined by Simon Späti, long-time BI and data engineering expert turned full-time technical writer and author of the living book Data Engineering Design Patterns. We talk about:His 20-year journey from SQL-heavy BI to modern Data EngineeringWhy switching from employee to full-time author wasn’t planned, but necessaryHow he uses a “Second Brain” system to manage and publish his knowledgeWhy writing is a tool for learning, not just sharingThe concept of convergent evolution in data tooling: when old and new solve the same problemThe underrated power of data modeling and pattern recognition in a hype-driven industry Simon also shares practical advice for building your own public knowledge base, and why Markdown and simplicity still win in the long run.Whether you're into tools, systems, or lifelong learning, this one’s a thoughtful deep dive.***About Simon Späti:Simon is a Data Engineer and Technical Author with 20+ years of experience in the data field. He's the author of the Data Engineering Blog (ssp.sh), curator of the Data Engineering Vault (vault.ssp.sh), and currently writes a book about Data Engineering Design Patterns (dedp.online). Simon maintains an awareness of open-source data engineering technologies and enjoys sharing his knowledge with the community.Socials: Bluesky, LinkedIn, Twitter/X, YouTube

In this episode, I’m joined by Johannes Koch, Principal Engineer and AWS DevTools Hero, to talk about the real DevOps mindset, the evolution of developer experience, and how community work changed his career.Johannes shares how starting in QA and support gave him a unique edge in understanding users, why building a proper CI/CD pipeline should come before writing code, and how the AWS Community Builders program helped him grow into his current role.We also dive into:What DevOps culture actually means beyond automationHis take on GitOps and why CI/CD is still underratedBehind the scenes of the AWS Heroes ProgramCreating content, mentoring others, and avoiding the trap of tech hypeWhether you're starting out in DevOps or deep into cloud architecture, Johannes' insights are packed with value for data and AI professionals.Links:Johannes on LinkedInYouTube: @cicdonawsAWS Community Day DACH

In this episode, I sit down with Deepak Goyal, the founder of AzureLib, to talk all things data engineering, cloud platforms, and how to teach the next generation of engineers.We explore why Azure became his go-to, how real-world projects beat theory every time, and why tools like ChatGPT are great assistants, but no substitute for structured learning and solid fundamentals.Follow Deepak on LinkedIn: https://www.linkedin.com/in/deepak-goyal-93805a17/Check out AzureLib: azurelib.comLearn Data Engineering with me: learndataengineering.com