Data Systems Requirements: What to Define Before You Build

What happens when multiple transactions hit the same data at the same time? In this episode of Data Engineering Concepts, we dive into transaction isolation levels — a classic interview topic and a must-know for anyone working with transactional systems. We start with the anomalies that isolation levels are designed to prevent: dirty writesdirty readslost updatesnon-repeatable readsread and write skewsand phantom reads.Then we walk up the "ladder" of the four main isolation levels: Read UncommittedRead CommittedRepeatable ReadAnd Serializableand look at how each one removes more anomalies at the cost of performance. Along the way, we touch on MVCC snapshots in PostgreSQL, why isolation levels can behave differently across database systems (hello, DB2's cursor stability), and how Serializable Snapshot Isolation can even roll back your queries.By the end, you'll have a solid foundation for understanding how parallel transactions interact — and how to keep your data consistent.🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠⁠https://deconcepts.com⁠⁠⁠⁠Add me on LinkedIn: ⁠⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

How does database replication work, and why is it so important for modern data systems?In this episode of Data Engineering Concepts, we explore database replication — one of the fundamental techniques for improving high availability, scaling read workloads, and building resilient data architectures.We'll break down the main replication approaches, explain how they differ, and discuss the trade-offs you should understand as a data engineer.In this episode:What database replication is and why we use itPhysical vs. logical replication and Change Data Capture (CDC)Block-level and application-level replicationSynchronous vs. asynchronous replicationSingle-leader, multi-leader, and leaderless architecturesReplication lag, consistency, and potential data lossRecovery Time Objective (RTO) and Recovery Point Objective (RPO)Why replication doesn't replace proper backupsWhether you're preparing for a data engineering interview or looking to strengthen your understanding of database architecture, this episode will help you build a clearer picture of how replication works.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠Add me on LinkedIn: ⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

What is database normalization, and why does it matter?In this episode of Data Engineering Concepts, we explore the fundamentals of database normalization, from understanding data redundancy and modification anomalies to breaking down normal forms, including 1NF, 2NF, 3NF, BCNF, 4NF, and 5NF.We'll also discuss candidate keys, functional dependencies, and the practical trade-offs between normalized and denormalized data.Is normalization always the right choice? And why might denormalization actually be preferable for analytical workloads?Whether you're preparing for a data engineering interview or simply looking to strengthen your database fundamentals, this episode will help you better understand when and why normalization matters.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠Add me on LinkedIn: ⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

What is the CAP theorem, and why does it matter in distributed systems?In this episode, we explore the CAP theorem and its three fundamental properties: Consistency, Availability, and Partition Tolerance. We'll discuss what these properties mean, why distributed systems have to make trade-offs when network failures occur, and how AP and CP systems differ.We'll also look at the limitations of the CAP theorem and introduce PACELC, a more modern approach that considers the trade-offs between latency and consistency even when everything is working normally.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠Add me on LinkedIn: ⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

What is Medallion Architecture, and why is it commonly used in modern data platforms?In this episode, we explore the fundamentals of Medallion Architecture and its three core layers: Bronze, Silver, and Gold. We'll discuss the purpose of each layer, how data moves from raw sources to business-ready datasets, and how this approach relates to traditional data warehouse architectures.We'll also touch on ELT processing and walk through a practical example to see how it all works.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠https://deconcepts.com⁠⁠Add me on LinkedIn: ⁠https://www.linkedin.com/in/dpolovinkin⁠

What should you define before designing a data system? In this episode, I cover the key requirements every data engineer should think about - from data sources, consumers, volume, throughput, and latency to availability, RTO/RPO, retention, compliance, data correctness, and budget.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠Add me on LinkedIn: ⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

Database indexes are one of the most important tools for improving query performance - but there are many different types, and the differences between them are not always obvious.In this episode, I explain how indexes work, what's their benefits and drawbacks, and I go through the main index types and concepts: B-tree and B+ tree indexes, dense and sparse indexes, bitmap and hash indexes, clustered and non-clustered indexes, composite and covering indexes, as well as duplicate and redundant indexes.There also details about why column order matters in composite indexes and how an index-only scan can avoid reading the table altogether - just as an example.This is a solid and practical high-level overview for data engineers who want to better understand indexes for both everyday work and technical interviews.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠Add me on LinkedIn: ⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

Hi, my name is Dmitrii Polovinkin, I'm a data engineer with ~7 years of experience (check out ⁠⁠my LinkedIn⁠⁠) and welcome to the first episode of Data Engineering Concepts, which is a new educational podcast about data engineering, designed to be useful when you're learning on the go.In this episode, I explain what Apache Airflow is, what problems it solves, and when it makes sense to use it in data engineering. We go through the main Airflow concepts, including DAGs and DAG runs, tasks and task instances, operators, sensors, hooks, connections, variables, scheduling, data intervals, XComs, and trigger rules. I also give a high-level overview of Airflow architecture, executors, workers, the metadata database, and some of the key DAG configuration parameters.If you find this episode useful, consider following the podcast and leaving a rating or review!🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠Add me on LinkedIn: ⁠⁠https://www.linkedin.com/in/dpolovinkin⁠⁠

Hi, my name is Dmitrii Polovinkin, I'm a data engineer with ~7 years of experience (check out ⁠my LinkedIn⁠) and welcome to the pilot episode of Data Engineering Concepts, which is a new educational podcast about data engineering, designed to be useful when you're learning on the go.In this episode I explain why I started the podcast, about my experience as a DE, who it is for, what topics future episodes will cover, and how I plan to approach them.If you're a data engineer or interested to become one, work in a related field or preparing for interviews - subscribe, this podcast is for you!This is episode zero is recorded in Da Nang, Vietnam 🇻🇳.🎙️ Data Engineering Concepts — making complex data engineering topics easier to understand, one episode at a time.Podcast website: 🌐 ⁠⁠⁠https://deconcepts.com⁠⁠⁠