AI Evals and Analytics Podcast podcast

AI Evals Skills: Why Data Scientists Have a Natural Advantage

0:00
22:10
Rewind 15 seconds
Fast Forward 15 seconds

What are the skills required for AI evals? Why data scientists have a natural advantage in AI evals?

Evaluating AI isn’t just about "vibe coding" with an AI assistant. It actually requires a solid foundation in statistics for picking sample sizes and coding to build your own testing frameworks. Data scientists have a huge head start here because they are already pros at designing metrics and communicating risks.

In the augural episode, we also explain why Evals (pre-launch testing) and Analytics (post-launch user feedback) are two sides of the same coin: one makes sure the AI works, and the other makes sure people actually love using it.

00:00 – Introduction to AI Evals & Analytics
01:31 – Why Data Scientists Have a Natural Advantage
01:59 – Technical Pillar: Statistics
02:48 – Technical Pillar: Coding & Prompt Engineering
05:03 – Technical Pillar: Dataset Generation
08:35 – Soft Skills & Stakeholder Collaboration
11:17 – Domain Expertise in Regulated Industries
15:50 – New Skills for the GenAI Era
19:25 – Why Evals and Analytics Must Come Together

Stella Liu: https://www.linkedin.com/in/wenxingl/
Amy Chen: https://www.linkedin.com/in/amy17519/

More about AI Evals and Analytics -- https://ai-evals.org/

We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.




Powered by Firstory Hosting

More episodes from "AI Evals and Analytics Podcast"