#22 Why testing data pipelines can be so challenging - and how to tackle it

In this episode of the Plumbers of Data Science podcast, I dive into the challenges of recruiting today, from overwhelming job application volumes to reaching out directly to recruiters. I’m testing new strategies to make the process smoother for everyone involved, focusing on fresh job listings and fostering connections with hiring managers who need skilled engineers. My goal? To secure five job placements in Germany by year’s end! Have thoughts on today’s job market, or tried the Easy Apply feature yourself? Drop a comment below—I’d love to hear your experience!

In this Hero Talk episode, I had the pleasure of chatting with Ben Rogojan, better known as the "Seattle Data Guy." Ben is a data engineer, YouTuber, and freelancer with a background at Facebook. He's become a go-to expert on freelancing for engineers, particularly in the data space. We dive into Ben's journey from being a full-time engineer to making the switch to freelancing, how he built his own business, and the unique challenges freelancers face in this space. We also explore how to break into freelancing, the value of specializing in a specific skill, and practical tips on landing your first freelance clients.

In this episode of the Plumbers of Data Science podcast, I'm sharing some exciting updates about the future of Learn Data Engineering and a big new service we’re launching—recruiting! I explain how this new offering will help engineers find their next career move while connecting companies with top talent. Tune in to hear more about how the Academy, Coaching, and now recruiting fit together into one ecosystem designed to support your career growth. Let me know your thoughts in the comments—are you excited about this new direction?

In this episode of the Plumbers of Data Science podcast, I’m sharing my thoughts on why data modeling isn’t as complicated as people make it out to be. You hear about courses and tutorials that stretch for hours—but is it really that hard? I’ll break down the two main things you need to focus on when modeling data and explain why, once you’ve got those down, the rest falls into place.

In this Hero Talk episode, I talk with Mezue, a seasoned Data Engineer with expertise in Azure Databricks Data Engineering. We cover his journey from Electrical Engineering to Data Engineering and discuss the key skills, like Python, SQL, and Spark, that are essential in the field. Mezue also shares his experience running an Azure Databricks bootcamp and offers advice on how to break into Data Engineering, especially in Cloud environments. We also touch on the challenges of finding junior roles and how to stand out by working on practical projects.

In this Hero Talk episode, I chat with Susan Walsh, the “Classification Guru,” known for her expertise in cleaning and classifying messy data. We dive into her unexpected journey into the data world, starting with a spend analytics job, and how that led to her founding her own business focused on dirty data. Susan shares the unique challenges businesses face with poor data quality, explaining why 99.9% of data problems are actually people problems. We also explore practical ways to deal with these issues, such as finding those "crappy" data cleaning jobs to gain experience, and the importance of consistent data maintenance to prevent future headaches. From addressing dirty CRM systems to battling fraud, Susan’s stories highlight how critical clean data is for business success.

In this Hero Talk episode, I sit down with Paolo Lulli, an experienced Data Engineer, to explore some of the core challenges and decisions in API development and data management. We dive deep into the debate between serverless infrastructure versus traditional servers, discussing the pros and cons of both approaches, particularly in the context of scalability, cost, and maintenance. Paolo also shares his hands-on experience with time series databases, explaining their advantages in handling massive amounts of data from IoT devices. We delve into vendor lock-in issues, highlighting how relying too heavily on cloud providers like AWS or Azure can impact long-term flexibility.

In this episode of the Plumbers of Data Science podcast, I’m diving into why testing can be so challenging for data engineers. The inspiration for this topic actually came from one of my recent Coaching sessions, where the question of test-driven development (TDD) came up during a Q&A. It stuck with me, so I thought it would be a great topic to dive deeper into. I’ll explain the key benefits of TDD, like improved code quality and easier refactoring, and why, despite its advantages, it’s not always widely adopted—especially in fast-paced environments where time constraints dominate. We’ll also talk about the specific challenges data engineers face with TDD, such as handling large, unpredictable data, integrating with external systems, and adapting to ever-changing data.

In this Hero Talk episode, we dive deep into the fascinating world of synthetic data, a critical tool for development, testing, and training Machine Learning models. Joining me is Mario Scriminaci, Chief Product Officer at Mostly AI, who shares his expertise on how synthetic data can revolutionize the way we handle sensitive information, particularly in the context of privacy regulations like GDPR and CCPA. We discuss the real-world applications of synthetic data, how it differs from traditional mock data, and its potential to drive innovation in AI and ML development. Mario also introduces Mostly AI's cutting-edge tools, highlighting how they make it easier than ever to generate realistic, privacy-safe datasets.

In this episode of the Plumbers of Data Science podcast, I’m diving into the debate between bootcamps and coaching programs, especially for those looking to advance in Data Engineering. I’ll break down the pros and cons of each approach - from the structured, intensive nature of bootcamps to the personalized, flexible support of coaching, I’ll share insights to help you choose the right path for your career. I’ll also discuss the experiences of my current coaching students and what I’m focusing on to help them achieve their goals.