
Lauri Koobas: Data Engineering - from early startup to scaling | Ep. 4
Lauri Koobas, ex-Microsoft and currently Head of Data Platform at Bondora, shed insights on data engineering - from early startup to scaling.
We mostly focused on analytics and building data warehouse - real-world challenges from both data engineering and software engineering sides. We also discussed GDPR and PII challenges when dealing with data.
You can find video version on MapForEngineers YouTube channel: https://www.youtube.com/@mapforengineers
Annotated chapters in timeline:
00:00:00 Sneak peek of episode
00:01:21 Episode overview
00:02:44 Introduction, Lauri's background
00:20:48 Starship robots: huge amount of data there
00:23:37 Data lake, data warehouse, data lakehouse
00:26:44 Devil is in the details: timestamps, texts, character sets...
00:49:44 Moving data from prod to data warehouse
00:53:09 Analytics tools: PostHog, Amplitude, Redash, Databricks
01:00:15 Analytics tools vs real-time monitoring like Prometheus/Grafana
01:04:15 Usability matters: each tool for its job
01:06:38 Startup grows: needs in data analytics
01:11:09 Multiple data sources: when data warehouse really begins
01:19:55 Data and (de-)coupling: software engineers should not be blocked by analytics
01:22:51 Data ETL
01:24:59 Changes in data model: multi-phase migrations
01:29:38 Change data capture, incremental imports
01:34:21 Should analytics have new data in real time? Maybe not?
01:39:02 Importing data into DWH through business events
01:43:37 When DWH subscribes to business events, data model can evolve freely
01:47:16 Quick recap what we discussed so far
01:52:25 GDPR and Data Compliance: start early
01:56:05 PII data: know exactly where you store it, control it well
02:03:37 Lauri's books recommendations on data engineering - Kimball
02:07:18 Lauri's podcast on data engineering, in Estonian
02:08:28 Wrap up
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit log.mapforengineers.com
Otros episodios de "Map for Engineers Podcast"



No te pierdas ningún episodio de “Map for Engineers Podcast”. Síguelo en la aplicación gratuita de GetPodcast.








