Medallion architecture vs. reality - when your Databricks bronze, silver and gold are just folders

Three schemas. Bronze, silver, gold. The diagram got nods in the architecture review. Then two dashboards land in the same leadership meeting with two different active customer counts, and the head of finance asks which one is right.The trail ends at bronze, where rows that failed parsing were quietly dropped. The vendor keeps thirty days. Every label was there. What the medallion was invented to fix was not, and the pipeline had rebuilt the old mess across three schemas.In this episode:- What pipelines looked like before bronze, silver and gold, and the three pains the diagram was drawn to fix- Why dropping bad rows at bronze feels clean and quietly decides how far back you can rebuild- When skipping a layer is the right call, and why seven layers is not a safer bet than three- The question to ask about every gold table before your next architecture reviewThis episode is for Databricks data engineers who build or inherit a medallion architecture and have wondered what silver is actually for. Whether you're designing a new lakehouse or maintaining one that looks textbook, you'll walk away with a way to check that each layer does a job no other layer does, and to defend it with more than "it's best practice."---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Mark's manager asks the room one question: if we move the biggest fact table onto Liquid Clustering, will it fix the slow dashboards? Mark answers first: yes, problem solved. Sarah says what she'd bet on, what she doesn't know yet, and when she'll know. Sarah claimed less. The decision goes to Sarah.Nobody tells Mark why. The instant-answer reflex that got him to mid-level is the same one that quietly keeps him there, and there's a name for what he's paying.In this episode:- Why a fast, confident answer is right early in a Databricks career, and when it stops paying- What the room needs from you when nobody knows for sure- The difference between being decisive and sounding certain- One sentence, two parts and a date, to carry into your next design reviewThis episode is for Databricks data engineers who answer fast and still watch the risky migration and the design review invite go to someone else. You'll walk away with a way to make your "I'm sure" mean something again.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Two in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the other exists, and nothing in Delta would tell them.A lot of design around this runs on superstition. Either "it's Delta, it's ACID, relax." Or "never let two jobs touch that table." Both are guesses, and the tidy middle version, same data collides, different data is fine, falls apart the first time the error shows up.In this episode:- What Delta actually does when the second job tries to commit to a table that moved under it- Why two jobs writing completely different months can still collide, and what that error message is really telling you- Why scheduling jobs hours apart is a lock you built by hand, and the cheaper fix- When a retry quietly does damage instead of fixing thingsThis episode is for Databricks data engineers who own a Delta table with more than one writer: a nightly merge, a backfill, a plain append. Whether you're designing a new pipeline or defending one in a design review, you'll walk away with a way to say which endings each writer can produce, instead of hoping.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

The rollout is done. Every production table migrated to Unity Catalog, every one owned by a team group, and the revenue table carries a certified check mark. The platform lead posts in Slack: fully governed. Monday morning, finance replies in the thread. The revenue on the finance dashboard and the revenue in the sales deck are about three percent apart. Which one's real?The catalog has something to show for every field: an owner, a check mark, a health indicator that says healthy, a beautiful lineage graph. None of them answers that question. Unity Catalog might be the best thing your platform team ever shipped, and it still can't make one person believe a number. The missing piece isn't a feature. It's the habit of treating every trust complaint like one.In this episode:- Why an owner, a certified check mark and a healthy indicator can all look right while the number is still wrong- Why the most-read table in your catalog might be the least trusted one- What a metric view settles when two teams define revenue differently, and what it leaves to humans- The one question to ask before you touch the platform for any governance requestThis episode is for Databricks data engineers who work in Unity Catalog and keep getting governance requests another tag won't fix. Whether you just finished the migration or you're the one answering "which number is real?" in Slack, you'll walk away with a way to sort every request before anyone opens a ticket.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Two proposals sit on a head of data's desk, both about the nightly pipeline that feeds the finance dashboard. One swaps the full reload for incremental loads with Auto Loader: it finishes in a fraction of the time and it's cheaper every month after the first. The other keeps the full reload, puts it on a bigger job cluster and starts it an hour earlier. It costs more. Forever. She asks one question, "What happens if this breaks on quarter close?", and approves the second one.It's tempting to call that ignorance or politics. It's neither. The better proposal was arguing merit to someone who gets asked a completely different question upstairs, in a meeting the engineer who wrote it will never be in.In this episode:- The fear that filters every proposal on your director's desk, and why it isn't fear of change- The question a tech lead, an engineering manager and a director each get asked, and why pitching a director like a tech lead loses- Three kinds of data leader, and how to tell which one you work for from the first question they ask about any proposal- Three things to add to any proposal with a real spread, so it gets funded without getting duller- A two-sentence test to run before anything goes upstairsThis episode is for Databricks data engineers who have watched a better proposal lose to a duller one and couldn't explain why. Whether you're the mid-level engineer writing the pitch or the senior trying to get a platform change funded, you'll walk away able to read your leader's decisions and write proposals they can say yes to.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task carry the whole job, and no amount of extra workers moves that work off it. Sometimes the stuck task turns out to be worse than slow.In this episode:- Why more workers and more shuffle partitions both leave the stuck task exactly where it was- Where heavy keys come from, and the one source that is usually a bug in a performance costume- Why plenty of skewed aggregations never slow down, and where adaptive query execution on Databricks does and doesn't split the load- When salting pays, when a different approach beats it, and when the right call is to leave skew aloneThis episode is for Databricks data engineers whose reflex for a slow Spark stage is a bigger cluster. Whether you're mid-level and tired of paying for compute that changes nothing, or senior and about to be asked "how do you handle data skew?" in an interview, you'll walk away able to tell a skew problem from a capacity problem before you touch the cluster settings.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Sam is four years into a Databricks role, up for senior this cycle. Merged pull requests on the ingestion layer. The nightly job he rescued at three in the morning when the orders table stopped landing. Then his manager asks for one call where the answer wasn't obvious, he framed it, and he lived with the result. Sam starts three different sentences. None of them finish.The packet is full of work and empty of decisions. Sam thinks what he's missing is a chair at the architecture review table. It isn't.In this episode:- What the architecture review was ever supposed to prove, and why the room was never the proof- The one test to run on any call you've made, with no meeting, no invite and no template required- The two paragraphs that turn a design doc from a build plan into a decision somebody can audit nine months later- How to run the whole thing in a Slack thread on a four-person team with no review processFor Databricks data engineers building a case for senior, especially where architecture reviews are rare or don't exist. You'll walk away able to tell work you did apart from a decision you can prove you owned.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Open the workspace. Start a cluster. Get coffee. Come back, and sometimes it's ready. If you've been doing this a few years you can feel that wait in your shoulders, and it's the least interesting part of the story.Before any of it, you had to answer a question you could not possibly answer: how big should this be? Start small and you pay in latency. Start big and you pay for machines that are barely breathing for most of the run. Wait, or waste, and there was never a third option.In this episode:- Why the cluster wait was only the receipt, and what classic Databricks compute never actually solved- Why Databricks can keep compute warm when your own team never could, and what that shows up as in the rate- What "serverless" means here, jobs and notebooks, and why a serverless SQL warehouse is a different conversation- The thing a fixed cluster was quietly doing for your budget that serverless stops doing, and why the policy you're thinking of doesn't cover it- Three tests to run on your own workloads this week, plus the ones that can't move at all no matter what the math saysThis episode is for Databricks data engineers who have serverless sitting in a dropdown and no real argument either way. Whether you're defending a migration in an architecture review or explaining next month's bill, you'll walk away able to say which of your jobs should move, which should stay on classic, and why.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

You cleared the recruiter screen. You cleared the Databricks technical round, and cleared it well. Then the final round, then the email that says strong candidate, tough decision. You go back looking for the mistake and there isn't one. You were good. Somebody was just easier to picture.That sentence never makes it into the feedback. And it isn't a bad panel or a bias against you. It's a belief you carry into the room: that the final round is the last exam, the hardest one. If your last stage is still a coding or SQL round, that one is an exam, go and study. But most of these emails come out of a different loop, where the technical question was settled and the form closed well before you walked in.In this episode:- What that last round is actually deciding, and why it rarely shows up on a rubric- The four places a strong candidate loses it, ordered from the most common to the one that stays invisible longest- How an unnarrated trade-off reads as taste instead of judgment, and the one sentence that fixes it- The single behavior to take into your next final round, and why trying to fix all four is the wrong moveThis episode is for Databricks data engineers who keep reaching final rounds and keep getting the same email. Whether you're three no-offers into a year of interviewing or prepping for a loop that starts next week, you'll walk away with one concrete thing to change in the room, not another weekend of Spark internals.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake

Your org finished the migration. Two systems became one. The nightly copy nobody wanted to own is gone, and so is the quarterly meeting where finance and the data science team each showed up with a real revenue number from tables in different buildings. That pain was real, and it is genuinely dead. So why does your week feel heavier than the deck promised?Because most of what got sold as simplification was a relocation. The copy didn't die, it turned into an argument about which schema in which catalog is the real one. The sync job didn't die, it became a maintenance calendar that pages nobody as an alert and pages you months later as the job that got a little slower every week.In this episode:- Why every architecture in this field gets sold on removal, and what the slide never names- How to tell the work that genuinely died in the lakehouse from the work that only changed address- Why a heavier week after a clean migration is not a verdict on you- The two questions to run on the next architecture pitched to your org, and why the second one is about names, not tasksThis episode is for Databricks data engineers who got the win the lakehouse promised and still can't explain why the calendar filled back up. Whether you ran the migration or inherited it, you'll walk away with a lens you can run on any architecture sold as the thing that finally makes data simple.---Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.Follow The Databricks Data Engineer for new episodes every week.LinkedIn: linkedin.com/in/jrlasakNewsletter: dataengineer.wiki#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake