BrowserStack Talks podcast

500 Red Tests a Day: How Hugging Face Runs CI/CD for AI with Tarek Ziadé

0:00
54:06
Spol 15 sekunder tilbage
Spol 15 sekunder frem

What happens when your CI/CD pipeline can’t give you a simple yes or no? Tarek Ziadé, who leads CI/CD on Hugging Face’s Transformers library (and previously built CI at Mozilla), joins David Burns to talk about testing software where the maths changes every time you run it.

In this episode:

  • Why green doesn’t mean working: non-determinism, GPU variance and out-of-memory leaks across 500+ model architectures
  • Living with 0–500 daily failures out of 13,000 GPU tests, and triaging them with Grafana trends instead of by hand
  • Serge, the overnight agent that reproduces failures, builds patches and opens PRs (170 runs, 19 PRs in two months)
  • Catching reward hacking when the agent “fixes” a test by changing the assertion
  • Cutting agent costs from $1.50 to under 50 cents a review by moving logic out of prompts and into deterministic Python
  • A Ruff-style model linter with 58 rules, tiny overfitted models for CPU-only end-to-end checks, and the Model Integration Contract
  • Sandboxing agents: separating instructions from data, Bubblewrap, network proxies and a human on the merge button

Plus the quickfire round: time-to-signal as the most underrated CI metric, why Tarek’s dream job is running coach, and whether AI will ever replace testing.

Follow Tarek’s writing on the Hugging Face blog.

Subscribe to BrowserStack Talks on Spotify, Apple Podcasts or YouTube so you don’t miss future episodes.

Flere episoder fra "BrowserStack Talks"