Eventual Consistency | Your Reality Check on What's Actually Happening in Data podkast

How to Pick the Right Data Warehouse for Your Workload, Backed by a 7-Engine Benchmark

0:00
1:10:49
Do tyłu o 15 sekund
Do przodu o 15 sekund

Seven cloud data warehouses, one open benchmark, and no single winner.

Your warehouse choice probably came down to what someone used at their last job, plus a vendor bake-off with the outcome decided before it started. Then the bill arrived.

James Winegar and Ross Katz from CorrDyn, who ran the benchmark, alongside Dave Yaffe, Co-Founder of Estuary break down what happened when they loaded the same data into Databricks, BigQuery, Redshift, Microsoft Fabric, ClickHouse, MotherDuck and Snowflake. You'll come away with a sharper way to shortlist engines, size compute and challenge vendor claims before you commit budget to your data stack.

The conversation covers warehouse sizing, concurrency, idle time billing, open table formats like Iceberg, and how AI agents change query load on your data infrastructure. It's built for heads of data, CTOs and data engineering leads weighing a warehouse decision. If you want one tool crowned the winner, this benchmark won't give you that.

Key Takeaways

  • Under 10GB, pick the cheapest engine you can run. At 100GB and 1TB, the speed-versus-cost curve flips, and a bigger warehouse can cost you less.
  • "Useful concurrency" combines team size, data volume and a latency target into one sizing number. AI agents can make a ten-person team query like a thousand people.
  • Redshift and Fabric beat their reputations in testing, with caveats that bite if you run 24/7 or juggle many workloads.
  • A second vendor you've tested on your own workload is your best leverage when a three-year contract lands on your desk.

Chapter Markers

00:00 Why build a warehouse benchmark at all

02:17 Why Estuary backed the project

03:33 Data size, concurrency and cost dimensions

04:44 How teams really pick warehouses today

06:02 Why vendor-run benchmarks fall short

10:13 Landing 2.2TB into seven engines

14:11 Query validation for apples-to-apples results

15:21 Sizing compute: speed versus cost curves

19:43 Scan aggregates, ClickHouse and MotherDuck

22:10 Defining useful concurrency

27:34 How idle time drives your bill

32:36 Reputations confirmed and contradicted

39:15 Home field advantage caveats

42:14 Open table formats and managed Iceberg

47:29 Agentic analytics and warehouse load

53:10 AI agents inside the data pipeline

58:06 No clear winner across the board

1:01:35 Why autoscaling was switched off

1:03:21 How to read any vendor benchmark

1:05:43 Before you sign a three-year contract

1:08:57 What could change in a rerun

Useful Links & Resources

  • Data warehouse buyer's guide and companion app (built on this benchmark)
  • Benchmarks referenced: TPC-DS and ClickBench
  • Apache Iceberg (open table format)
  • CorrDyn: https://corrdyn.com

Connect With the Show

  • CorrDyn on LinkedIn: https://www.linkedin.com/company/corrdyn/
  • Ross Katz on LinkedIn: https://www.linkedin.com/in/b-ross-katz/
  • James Winegar on LinkedIn: https://www.linkedin.com/in/james-winegar/
  • David Yaffe on LinkedIn: https://www.linkedin.com/in/davidyaffe/
  • Website: https://corrdyn.com

Which warehouse reputation does your own experience contradict? Tell us where the benchmark matches your workload, and where it doesn't.

#EventualConsistency #DataEngineering #DataWarehouse #DataInfrastructure #Iceberg

Więcej odcinków z kanału "Eventual Consistency | Your Reality Check on What's Actually Happening in Data"