Startup Project: Build the future podcast

Why GPUs Aren't the Answer for Every AI Workload | Sid Sheth Founder and CEO of d-Matrix

0:00
48:16
Recuar 15 segundos
Avançar 15 segundos

Sid Sheth, co-founder and CEO of D-Matrix, joins Nataraj on Startup Project to talk about the future of AI inference, why GPUs may not be the answer for every workload, and what it takes to build a semiconductor startup.Key takeaways

  • Training and inference are like education and a career. Sid's view is that inference is the much larger opportunity.
  • D-Matrix focuses on ultra-low-latency inference using SRAM and in-memory compute, not on throughput.
  • Customers now deploy racks and clusters, so D-Matrix moved from chip scale to rack scale.
  • Raptor stacks custom DRAM on compute. Sid says this gives roughly 20–30x more capacity per rack.

Timestamps

  • 0:00 – Intro: from training to low-latency inference
  • 0:43 – Why start a chip company in 2019, before ChatGPT
  • 4:04 – Betting on inference for the data center and cloud
  • 7:08 – Transformers, GPT-3 and finding product-market fit
  • 9:16 – The first chip and the in-memory compute demo
  • 14:25 – How a first chip gets built: partners, TSMC and COVID
  • 17:10 – From a digital in-memory engine to a first product in 2024
  • 20:24 – What "rack scale" means and why the industry moved to it
  • 24:01 – D-Matrix vs. GPUs: throughput vs. interactivity, and no HBM
  • 26:58 – The business model and target customers
  • 29:07 – The Nvidia partnership and the stacked-DRAM product, Raptor
  • 34:32 – The two acquisitions and how they fit
  • 37:06 – The AI hardware investment cycle
  • 40:05 – What success looks like for D-Matrix

Mais episódios de "Startup Project: Build the future"