
AI Evals Skills: Why Data Scientists Have a Natural Advantage
What are the skills required for AI evals? Why data scientists have a natural advantage in AI evals?
Evaluating AI isn’t just about "vibe coding" with an AI assistant. It actually requires a solid foundation in statistics for picking sample sizes and coding to build your own testing frameworks. Data scientists have a huge head start here because they are already pros at designing metrics and communicating risks.
In the augural episode, we also explain why Evals (pre-launch testing) and Analytics (post-launch user feedback) are two sides of the same coin: one makes sure the AI works, and the other makes sure people actually love using it.
00:00 – Introduction to AI Evals & Analytics
01:31 – Why Data Scientists Have a Natural Advantage
01:59 – Technical Pillar: Statistics
02:48 – Technical Pillar: Coding & Prompt Engineering
05:03 – Technical Pillar: Dataset Generation
08:35 – Soft Skills & Stakeholder Collaboration
11:17 – Domain Expertise in Regulated Industries
15:50 – New Skills for the GenAI Era
19:25 – Why Evals and Analytics Must Come Together
Stella Liu: https://www.linkedin.com/in/wenxingl/
Amy Chen: https://www.linkedin.com/in/amy17519/
More about AI Evals and Analytics -- https://ai-evals.org/
We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.
Powered by Firstory Hosting
More episodes from "AI Evals and Analytics Podcast"



Don't miss an episode of “AI Evals and Analytics Podcast” and subscribe to it in the GetPodcast app.

