DataTalks.Club

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.CHAPTERS00:00 Podcast intro00:08 Tejas's path to AI engineering12:24 Why evals go beyond a manual check17:37 Sourcing evaluation cases and calibrating a judge28:33 Refreshing datasets and pinning severe cases33:10 Judge agreement versus agent performance50:01 Retraining judges and keeping data fresh55:59 Human preferences, sample sizes and audience questions59:46 Evaluating final answers versus the agent's trajectory1:02:46 Faster judging and further evaluation resourcesResourcesAI evals: https://tej.as/blog/ai-evals-llm-judgeCode, demos and slides: https://github.com/TejasQ/basically-agent-evalsTejas: https://tej.asWatch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7twJoin DataTalks.Club: https://datatalks.club/slack.html

In this talk, Dewang Sultania, ML Engineer at Netflix and co-author of Designing AI Systems, shares his technical journey from building early inference platforms at Adobe to deploying multimodal AI models at scale. We explore the complete architecture of enterprise AI platforms, focusing on evaluation datasets, tool calling orchestration, and the practical realities of shipping production agents.You’ll learn about:- The essential components needed to build an enterprise AI platform from scratch- How traditional software engineers can transition into AI platform engineering- The critical technical differences between session management and LLM memory- Strategies for building, versioning, and trusting custom evaluation datasets- Optimizing LLM tool calling and selecting the right performance benchmarks- When to use multi-agent architectures versus single-agent systems- Techniques for overcoming the lost in the middle problem in RAG pipelinesTIMECODES:00:00 Core Components of Enterprise AI Platforms11:46 Career Path to AI Platform Engineering18:33 Session Management Versus LLM Memory Management26:47 Best Practices for Custom LLM Evaluation Datasets32:28 LLM Tool Calling Optimization and Benchmark Selection39:25 Context Window Limits and AI Agent Tool Scaling45:37 Single-Agent Versus Multi-Agent System Architecture52:16 LLM-as-a-Judge Alignment and Human Evaluation Strategies58:11 Solutions for the Lost in the Middle Problem in RAG Systems01:03:27 Designing AI Systems Book AnnouncementThis discussion is perfect for software engineers, machine learning practitioners, and technical leaders building real-world AI applications. It offers practical insights for anyone transitioning into AI platform engineering or tasked with scaling LLM features for enterprise production.Connect with Dewang:- Linkedin - https://www.linkedin.com/in/dewang-sultania/- Github - https://github.com/iron-starkConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Suhas Suresha, Senior Machine Learning Engineer at Adobe and co-author of Designing AI Systems, shares his expertise in building large-scale generative AI infrastructure from deploying real-time ECG analysis models at his previous startup, QALY, to architecting enterprise LLM platforms. We explore the massive engineering effort required to take an AI application from a weekend prototype to a scalable production system, and how to build a unified AI platform that eliminates "AI sprawl."Links: - Book - https://www.manning.com/books/designing-ai-systems- Code - https://github.com/designing-ai-systems/designing_ai_systems_repoYou’ll learn about:- The nine core components every production-ready AI platform needs to succeed.- How to solve "AI sprawl" and prevent fragmented engineering efforts across teams.- Architecting scalable Data Services and Model Services for robust LLM applications.- Implementing comprehensive observability and request tracing to identify system bottlenecks.- Managing long-term session memory, tool registries, and complex agent orchestration.- Navigating the "build versus buy" dilemma for enterprise AI infrastructure.TIMECODES:0:00 AI Prototypes to Production and the Book Origin5:49 AI Sprawl and the Need for a Centralized AI Platform12:42 Definition of an AI Platform and the Build vs Buy Dilemma23:17 AI Platform Engineering Skills and Data Service Architecture31:26 System Observability and Request Flow Bottlenecks37:32 LLM Evaluations, Prompt Experimentation, and Cost Tracking43:57 Session Memory, Tool Registries, and Agent Orchestration50:56 Invocation Modes and Durable Execution for AI Agents56:14 The Nine Core AI Platform Components and Memory ManagementThis episode is designed for machine learning engineers, backend developers, and platform architects looking to transition into AI platform engineering. It is highly recommended for technical leaders and teams who need to scale LLM prototypes into reliable, cost-effective production systems.Connect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/ Connect with Suhas:LinkedIn - https://www.linkedin.com/in/suhas-suresha/Twitter - https://x.com/inceptor92Github - https://github.com/suhas1992Website - https://suhassuresha.com/

In this talk, Sahil Walia, Senior Technical Architect at Snowflake, shares his extensive data engineering expertise from writing simple SQL queries early in his career to architecting enterprise-scale Apache Iceberg lakehouses today. We explore the evolution of the modern data platform, the mechanics of open lakehouses, and how decoupling storage from compute is transforming data architecture.Links:- https://medium.com/@sahil-walia/- Iceberg Spec- https://iceberg.apache.org/spec/- https://ossie.apache.org- https://github.com/Snowflake-Labs/schemachange- https://opensource.guide/how-to-contribute/You’ll learn about:- What defines a modern data platform and the importance of interoperability.- Key differences between traditional storage, Hive metastores, and Apache Iceberg.- How Iceberg metadata simplifies GDPR compliance, data deletions, and file compactions.- When a lakehouse architecture actually makes sense (and when to avoid it).- Standardizing business logic across compute engines using Apache OSI.- Actionable tips for making impactful open-source data engineering contributions.TIMECODES:00:00 Defining the Modern Data Platform11:34 What Exactly is a Lakehouse?17:20 Comparing Hive Metastore and Apache Iceberg22:34 Handling GDPR and Data Deletions in Iceberg27:33 Apache Iceberg vs. Delta Lake34:10 Do You Need a Lakehouse for Small Data?40:11 Understanding Apache OSI (Open Semantic Interchange)49:00 Managing Schema Changes in Data Warehouses54:20 How to Contribute to Open Source Projects01:00:40 The Role of AI in Open Source ContributionsThis talk is designed for data engineers, data architects, and analytics professionals looking to modernize their data infrastructure. It provides essential architectural insights for technical teams evaluating Apache Iceberg, dealing with large-scale data transformation, or aiming to scale their open-source contributions.Connect with Sahil:- Linkedin - https://www.linkedin.com/in/sahilwaliasyracuse- Github - https://github.com/sahil-walia- Website - https://medium.com/@sahil-waliaConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Philip Christos, Associate at M2VC, shares his expertise in frontier AI investing from his early days as a data analyst to evaluating cutting-edge deep tech startups. We explore how AI agents are reshaping modern work, the realities of venture capital in the LLM era, and why the future of software engineering relies more on high-level systems thinking than writing syntax.You’ll learn about:- How venture capitalists formulate AI investment strategies and evaluate startup pitch decks.- The rapid adoption of AI in software engineering compared to traditional sectors like accounting.- The crucial difference between data progress and algorithmic progress in frontier models.- Why LLMs still struggle with high-level software architecture, business logic, and design "taste."- How AI is accelerating iterations and breakthroughs in classical engineering and scientific research.- What specific deep tech and AI infrastructure ideas are actually attracting VC funding today.TIMECODES:00:00 Venture Capital Career Journey05:14 VC Analyst vs Data Analyst Roles10:13 AI Investment Thesis Strategy16:00 Emotion Analytics and Voice AI Startups22:09 LLM Impact on Venture Capital28:19 Software Engineering AI Adoption34:18 Data vs Algorithmic AI Progress40:15 Classical Engineering AI Acceleration46:12 Software Architecture AI Limitations51:11 Future Software Engineering Careers57:55 Systems Thinking in DevOps1:03:00 Pitching Deep Tech StartupsThis talk is perfect for software engineers, AI founders, and technical product managers who want to understand the evolving tech job market and venture landscape. It provides essential insights for anyone looking to build VC-backed AI startups or future-proof their career as AI coding tools become ubiquitous.Connect with Philip- Linkedin - https://www.linkedin.com/in/philipchristos/- Twitter - https://x.com/phil_christos- Website - https://t.me/stacked_channelConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Paul Domanski, founder of Domanski AI, shares his unconventional journey from film directing and affiliate marketing to becoming a vibecoder running dozens of autonomous AI agents. We explore the realities of building an agentic operating system, how non technical builders can orchestrate complex AI workflows, and the exact strategies used to close high ticket B2B clients using AI generated prototypes.Links:- Headroom: https://github.com/domanski-ai/headroom/You will learn about:- How to build and use custom workspaces like Demox and Tmux setups to orchestrate over 50 concurrent AI agents- Using automated research and rapid AI prototyping to close high ticket B2B sales- Balancing usage across multiple Claude and Codex accounts while navigating API limits and context windows- How to design audit and deploy complex software solutions without a traditional computer science background or deep knowledge of tech stacks- How the skills of a film director like vision stubbornness and delegation perfectly translate to managing autonomous AI teams- Why evaluating LLM outputs is the most critical skill for ensuring reliable products for enterprise clientsTIMECODES:00:00 Welcome and Showcasing Demox08:46 Paul's Background in Film and Digital Marketing16:09 Sales Pipeline and Client Onboarding22:20 Token Efficiency and Open Source Models31:19 Curiosity and the Learning Process37:52 Red Teaming and Code Audits44:51 Building Software Without Knowing the Tech Stack51:02 Parallels Between Film Directing and Managing AI56:32 Using AI for 3D Printing and Personal Projects01:04:34 The Importance of AI EvalsThis conversation is perfect for solopreneurs, agency owners, and non technical builders who want to move beyond basic chat interfaces and start leveraging autonomous AI agents for real world business operations. Whether you are transitioning into AI from marketing, project management, or creative fields, this episode offers a practical blueprint for turning AI curiosity into a scalable business.Connect with Paul:- Linkedin - https://www.linkedin.com/in/paul-domanski-ai/- Twitter - https://x.com/domanski_ai- Github - https://github.com/domanski-ai- Website - https://domanski.ai/Connect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Tony Zeljkovic, Founder of Narona Data, shares his unconventional career journey from bioinformatics research and clinical genetics to building a successful AI and data platform consulting business. We explore the essential soft skills, networking strategies, and business mindsets required to thrive as an independent data consultant in today’s rapidly shifting tech landscape.Links:- The book pre-order on packt - https://www.packtpub.com/en-us/product/the-data-consultants-handbook-9781806665266- The book pre-order on amazon - https://www.amazon.com/dp/B0H3TBWHH1?lv=shuf&channelId=500&plpRedirect=mhFallbackYou’ll learn about:- How to smoothly transition from a highly technical or academic role into independent data consulting.- The step-by-step process of building a strong digital footprint and growing a reliable client pipeline from scratch.- Strategic positioning: how to stop selling "data engineering" and start selling business outcomes and ROI.- How leveraging platform implementation partnerships (e.g., Snowflake, dbt) can help you land your first major clients.- Adapting to the AI era and ensuring your consulting services stay relevant as market demands change.- The business mindset of a consultant, including when to invest in tools and outsource tasks like accounting to accelerate your growth.TIMECODES:00:00 Landing the First Consulting Client via Academic Research06:06 Transitioning from Research to Consulting11:42 Becoming a Platform Implementation Partner18:20 Building Your Digital Footprint23:22 Positioning: Selling Business Outcomes vs. Technical Skills31:02 Networking Strategies and Finding Industry Pain Points37:24 Is Consulting Right For You?44:17 Adapting to AI and Changing Market Demands50:16 Business Mindset: Investing in Tools and Outsourcing55:52 Start Early and Pay It Forward1:02:37 The Practical Handbook for Data ConsultingThis talk is perfect for data engineers, analytics professionals, and software developers who are considering the leap into freelancing or starting their own consulting practice. It is highly valuable for technical experts who want to master client acquisition, stakeholder communication, and the business side of the data industry.Connect with Tony:- LinkedIn - https://www.linkedin.com/in/tony-zeljkovic/- GitHub - https://github.com/2tony2- Consulting website - https://www.naronadata.com/​Connect with Juan:- LinkedIn - https://www.linkedin.com/in/jmperafan/- GitHub - https://github.com/jmperafan- Consulting website - https://juanalytics.com/Connect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Radovan Bacovic, Principal Data Engineer and Snowflake/dbt Ambassador, shares his two-decade journey, from traditional database administration to leading modern data platform engineering. We explore the essential building blocks of scalable data architectures, the trade-offs of no-code solutions, and how to effectively integrate AI into your data pipelines while maintaining strict security and governance.You’ll learn about:- The evolution of the full-stack data engineer over the past two decades.- Building scalable, business-driven data platforms without premature optimization.- Efficient data modeling and transformation using DuckDB and dbt.- Leveraging AI to accelerate pipeline development and launch data products.- Implementing enterprise data governance and robust pipeline security.- Balancing no-code integration tools with strict DataOps methodologies.LINKS:https://gitlab.com/radovan.bacovic/wsc26_dataops/-/blob/main/README.md?ref_type=headsTIMECODES:0:00 Data Engineering Career Trajectory and Industry Experience6:37 Full Stack Data Engineer Role Evolution and Responsibilities14:45 Scalable Data Platform Architecture and Business Alignment21:59 DuckDB Integration and Premature Database Optimization Avoidance30:04 Data Pipeline Transformation and Data Modeling with dbt35:31 Modern Data Platform Ecosystem and Infrastructure Orchestration42:13 AI Powered Data Products and LLM Pipeline Automation49:42 Enterprise Data Governance and Pipeline Security Best Practices55:06 No Code Data Integration Tools and DataOps MethodologyThis talk is ideal for data engineers, analytics engineers, and data platform leads looking to modernize their tech stacks and future-proof their infrastructure. It provides highly actionable insights for anyone aiming to build resilient, AI-ready data platforms that prioritize real business value and strict security constraints.Connect with Radovan- Twitter - https://x.com/SvenMurtinson- Linkedin - https://www.linkedin.com/in/radovanbacovic/- Github: https://gitlab.com/radovan.bacovic/- Website: www.bacovic.comConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Paul Iusztin, Creator of Decoding AI and author of the LLM Engineer's Handbook, shares his deep expertise in personal automation from managing a digital life with lightweight data pipelines to architecting autonomous agents for deep research. We explore the mechanics of building personal AI assistants and the critical role of using a "second brain" as a context layer over heavy, over-engineered RAG infrastructure.You’ll learn about:- Organizing your digital life using the PARA method and lightweight data pipelines.- Capturing and retrieving resources effortlessly with Obsidian, Readwise, and custom deep research algorithms.- Leveraging your "second brain" setup as the ultimate context layer for personal AI assistants.- Generating ad-hoc wikis from markdown brain dumps to streamline content creation and research.- Optimizing AI-generated content by deliberately lowering LLM reasoning capabilities for better styling.- Adapting multi-agent workflows and personal wikis to accelerate software engineering and coding tasks.TIMECODES:00:00 Digital life organization using the PARA method and lightweight data pipelines05:21 Seamless resource capture with Obsidian and Readwise09:26 Resource retrieval optimization using a deep research algorithm12:44 High-quality internet curation versus heavy RAG pipelines16:31 Second brain setup as a context layer for personal AI assistants21:40 AI workflow simplification with Anthropic APIs and CLI tools25:26 Ad-hoc wiki generation from markdown brain dumps for content creation29:18 Codebase ingestion and web scraping proxy tool workarounds34:43 Resource reranking and context window management for large texts39:06 Content styling optimization by lowering LLM reasoning capabilities46:18 Multi-agent workflows and personal wikis for software engineering tasks52:32 Personal wiki scaling for enterprise knowledge bases and book writingThis talk is perfect for individual developers, AI engineers, and knowledge workers looking to escape "PoC purgatory" and build practical, low-maintenance personal AI assistants. It offers highly actionable insights for anyone wanting to integrate agentic workflows into their daily productivity systems without over-engineering their tech stack.Connect with Paul- Linkedin - https://www.linkedin.com/in/pauliusztin/- Website - https://www.pauliusztin.ai/Connect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/

In this talk, Maryam Ramezani-Bartsch, Data and AI Leader with over 20 years of experience at companies like adidas and Zalando, shares her extensive career journey from building foundational ML systems at adidas to coaching data experts through the modern AI landscape. We explore the critical intersection of technical strategy and the essential human skills needed to thrive in the AI era.LINKS:- https://maryamramezani.com/designyourdatacareerYou will learn about:- The surprising similarities and critical differences between the current Generative AI boom and the previous Big Data era.- Why the traditional boundaries between data roles are disappearing and the specific T shaped profile companies are actually hiring for today.- The hidden danger of perfectionism in corporate tech, and what a healthy margin of failure actually looks like in practice.- How to stop leaving your career trajectory to chance by applying product Design Thinking to your own life.- A practical framework for navigating industry uncertainty and tech career anxiety without burning out.- Battle tested strategies for regaining your footing and standing out in a highly competitive job market after a layoff.- The specific non technical human skills that will become your ultimate career moat against automation.TIMECODES:00:00 Human Skills in the AI Era07:07 Building ML Systems at Adidas12:49 Generative AI vs Big Data Era18:39 T-Shaped Data Engineering Roles24:04 Overcoming Perfectionism in Tech30:34 Design Thinking for Data Careers38:59 Managing Tech Career Anxiety45:07 Aligning Passion with Tech Skills50:18 Job Search Strategies After Layoffs56:03 Essential Soft Skills for JuniorsThis talk is essential for data professionals, software engineers, and tech leaders looking to future proof their careers in an increasingly automated world. Whether you are a junior developer navigating a tough job market, an engineer bouncing back from layoffs, or a senior professional looking to strategically design your next pivot, this session provides the tools to build a highly resilient career.Connect with Maryam- Linkedin - https://www.linkedin.com/in/maryam-ramezani-bartsch/- Website - https://maryamramezani.com/- Substack - https://maryamramezani.substack.com/Connect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/