DataTalks.Club podcast

Building AI Evaluations You Can Actually Trust - Tejas Kumar

23.10.2026
0:00
1:06:23
Spol 15 sekunder tilbage
Spol 15 sekunder frem

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.


CHAPTERS

00:00 Podcast intro

00:08 Tejas's path to AI engineering

12:24 Why evals go beyond a manual check

17:37 Sourcing evaluation cases and calibrating a judge

28:33 Refreshing datasets and pinning severe cases

33:10 Judge agreement versus agent performance

50:01 Retraining judges and keeping data fresh

55:59 Human preferences, sample sizes and audience questions

59:46 Evaluating final answers versus the agent's trajectory

1:02:46 Faster judging and further evaluation resources


Resources

AI evals: https://tej.as/blog/ai-evals-llm-judge

Code, demos and slides: https://github.com/TejasQ/basically-agent-evals

Tejas: https://tej.as


Watch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7tw

Join DataTalks.Club: https://datatalks.club/slack.html

Flere episoder fra "DataTalks.Club"