DataTalks.Club podcast

Building AI Evaluations You Can Actually Trust - Tejas Kumar

23.10.2026
0:00
1:06:23
15 Sekunden vorwärts
15 Sekunden vorwärts

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.


CHAPTERS

00:00 Podcast intro

00:08 Tejas's path to AI engineering

12:24 Why evals go beyond a manual check

17:37 Sourcing evaluation cases and calibrating a judge

28:33 Refreshing datasets and pinning severe cases

33:10 Judge agreement versus agent performance

50:01 Retraining judges and keeping data fresh

55:59 Human preferences, sample sizes and audience questions

59:46 Evaluating final answers versus the agent's trajectory

1:02:46 Faster judging and further evaluation resources


Resources

AI evals: https://tej.as/blog/ai-evals-llm-judge

Code, demos and slides: https://github.com/TejasQ/basically-agent-evals

Tejas: https://tej.as


Watch the original recording: https://www.youtube.com/watch?v=oWVmQDqZ7tw

Join DataTalks.Club: https://datatalks.club/slack.html

Weitere Episoden von „DataTalks.Club“