#562: DuckLake: The Lakehouse That's Just SQL and Parquet

How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first, just to learn which Parquet files matter. DuckLake asks one SQL question instead. The metadata lives in a real database. The data stays in plain Parquet. That's the entire format. Pedro Holanda joined DuckDB in 2018, when it was still a research prototype at CWI. He's the lead DuckLake developer. Guillermo Sanchez Dionis works on DuckLake and the new Quack protocol. With Quack as the catalog, DuckLake handles 200 transactions a second under heavy contention. No other open table format comes close. Episode sponsors Six Feet Up Talk Python Courses Links from the show Guests Pedro Holanda: pedroholanda.org Guillermo Sanchez: linkedin.com PhD on progressive indexes: ir.cwi.nl SQLite: www.sqlite.org Litestream: litestream.io boring hardware: talkpython.fm DuckDB: duckdb.org episode 491: talkpython.fm Iceberg: iceberg.apache.org manifesto: ducklake.select DuckLake: ducklake.select spec: ducklake.select this diagram: blobs.talkpython.fm Data inlining: ducklake.select ducklake-dataframe: github.com Polars course: training.talkpython.fm CSV parser: duckdb.org Zero-copy Arrow: duckdb.org ART index: duckdb.org async I/O: duckdb.org v1.0: ducklake.select Git-like branching: ducklake.select Watch this episode on YouTube: youtube.com Episode #562 deep-dive: talkpython.fm/562 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

How many cores does your machine have, 10, 18? Your async Python code uses just one of them. That isn't a bug in asyncio. That's the design, and optimizing event loops to be faster by 20% doesn't change it. So Giovanni Barillari started over. Joe is the creator of Granian, the Rust-based server that powers Talk Python. His new project is TonIO, an async runtime written from scratch for free-threaded Python. Real threads, a handful of primitives instead of asyncio's pile of them, and it flat out refuses to start if the GIL is on. Episode sponsors Sentry Error Monitoring, Code talkpython26 Python in Production Talk Python Courses Links from the show Guest Giovanni Barillari: github.com Granian: github.com Hyper: github.com Free threaded Python: docs.python.org Sort of: labs.quansight.org did a whole course: training.talkpython.fm uvloop: github.com rloop: github.com TonIO: github.com your EuroPython 2026 talk: www.youtube.com Michael's Cutting Python Web App Memory Over 31% Article: mkennedy.codes Watch this episode on YouTube: youtube.com Episode #561 deep-dive: talkpython.fm/561 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

In 2020, a gastroenterologist in Glasgow did the math on his new research study and came up with 30,000 samples, arriving over two years from three cities and a dozen hospitals. He asked around about how researchers keep track of that. The answer was Microsoft Excel. Shaun Chuah had written some HTML by hand in Notepad back in high school and that was about the whole of his programming experience, so he opened the Django tutorial and started reading. Six years later that app is Foundry120, holding 10 terabytes of clinical and genomics data with an agentic AI running on top of it. Episode sponsors Sentry Error Monitoring, Code talkpython26 Talk Python Courses Talk Python Courses Links from the show Guest Shaun Chuah: github.com Up and Running with Rust Course: talkpython.fm Foundry120: www.foundry120.com Designing Data Intensive Applications: www.oreilly.com Microsoft Foundry: ai.azure.com ChatIBD: www.chatibd.com Blog: shaunchuah.github.io @drshaunchuah: x.com github.com/shaunchuah: github.com Watch this episode on YouTube: youtube.com Episode #560 deep-dive: talkpython.fm/560 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

Your site is down. It's 3am. Is it a bug, a bill, or a breach? You can't tell yet, and everyone is watching you find out. Matt Lea has spent fifteen years being the person companies call when an outage is costing them real money per hour, and his whole argument is that everything you'd want in that moment gets decided months earlier, on ordinary afternoons, when someone chose the convenient thing. We walk his top twelve dos and don'ts in AWS - infrastructure as code, IAM roles instead of access keys, private subnets, no wildcards, no public buckets - and I push on which of them actually matter if you're one person on a small VPS. Then we get to Cloud War Games, where Matt breaks things on purpose so your team's first real incident isn't their first incident. Let's get into it. Episode sponsors Sentry Error Monitoring, Code talkpython26 Talk Python Courses Talk Python Courses Links from the show Guest Matt Lea: linkedin.com Talk Python Certificates: training.talkpython.fm/certificates Schematical: schematical.com CloudWarGames.com: cloudwargames.com Zero to Hero on AWS Security: www.oreilly.com Repo: github.com Custom Wheel Offset: customwheeloffset.com 2012 TechCrunch Disrupt Hackathon: techcrunch.com tech comics: schematical.com shhgit: github.com Zero Trust in 200ms: Implementing Identity-Per-Transaction: us.pycon.org Coolify: coolify.io returned to full GA Nov 2025: aws.amazon.com Signed URLs/cookies: docs.aws.amazon.com Cloudflare: www.cloudflare.com Bunny Shield: bunny.net Cloud War Games One: www.youtube.com Cloud War Games Two: www.youtube.com LinkedIn: linkedin.com YouTube: youtube.com KnocKnoc: knocknoc.io Watch this episode on YouTube: youtube.com Episode #559 deep-dive: talkpython.fm/559 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

Every company has one. The little internal tool that Jane built back in 2021, and then Jane left. Nobody understands it, nobody will touch it. There are two unwritten rules around it: don't change it, it's working. And if you break it, you bought it. That's dark-matter enterprise software. For every app you can actually see, there are ten of these sitting in the shadows, frozen. Michael Booth thinks that just changed. He read my article on hyper-personal software and ran with it, writing about hyper-team software: small teams inside big companies finally building the tools that were never going to get built. We cover where this works, where it quietly goes wrong, and the guardrails that keep it from turning into a mess. Let's get into it. Episode sponsors Sentry Error Monitoring, Code talkpython26 Python in Production Talk Python Courses Links from the show Guest Michael Booth: github.com Talk Python AI Integrations: talkpython.fm/blog From Hyper-Personal to Hyper-Team Software: Small Team-Built, AI-Assisted Tools Inside the Enterprise: www.databooth.com.au What hyper-personal software looks like (MK's article): mkennedy.codes Databooth Site: www.databooth.com.au Wall Street just lost $285 billion because of 13 markdown files: martinalderson.com SaaSpocalypse is real but everyone is panicking about the wrong thing: www.reddit.com Warp Terminal: www.warp.dev Watch this episode on YouTube: youtube.com Episode #558 deep-dive: talkpython.fm/558 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

Security has always been the vegetables of software. Everyone agrees it matters, and somehow it never quite makes it onto the plate. At PyCon US this year, that changed. For the first time ever, security got its own dedicated, day-long track, one of just two at the whole conference, sitting right next to AI. And the room was packed to the back wall. On this episode, I'm joined by the three people at the center of it. Seth Larson, Security Developer in Residence at the Python Software Foundation and, very recently, a CPython core developer. Juanita Gomez, a PhD researcher at UC Santa Cruz in open source security, who co-chaired the track. And Mike Fiedler, PyPI's Safety and Security Engineer, one of the very few people paid full-time to keep the packages you install safe. We use the arc of the track's talks to take the temperature of Python security right now: supply chain attacks, dependency cooldowns, zero trust, SBOMs, and the push to bring Rust into CPython. And why not one of us thinks security is anywhere close to solved. Turns out that's the good news. It's why the room was full. Episode sponsors Sentry Error Monitoring, Code talkpython26 Talk Python Courses Links from the show Guests Juanita Gomez: linkedin.com Mike Fiedler: miketheman.dev Seth Michael Larson: sethmlarson.dev Trailblazing Python Security: us.pycon.org Everything Security at PyCon US 2026 (PSF blog): pyfound.blogspot.com Dependency Cooldowns: cooldowns.dev Anatomy of a Phishing Campaign (Mike Fiedler) Recording: www.youtube.com FedRAMP: www.gsa.gov Zero Trust in 200ms: Implementing Identity-Per-Transaction with Python & Serverless-Tristan McKinnon: www.youtube.com Rust for CPython project: blog.python.org pre-PEP: discuss.python.org Rust for CPython: Making Python Safer and More Robust for Everyone - Emma Smith: www.youtube.com SBOMit: github.com Asleep at the Wheel: Getting your SBOMs to pay attention... - Sanchit Sahay, Abhishek Reddypalle: www.youtube.com Volatility: volatilityfoundation.org Post Incident Runtime SBOM Generation from Python Memory - Hala Ali: www.youtube.com zizmor: docs.zizmor.sh GitHub Actions security in Python packages (Andrew Nesbitt write-up): nesbitt.io andrew/pycon: data & analysis for the GitHub Actions security talk: github.com GitHub Actions Security in Python Packages - Andrew Nesbitt: www.youtube.com gh-profiler: examine a GitHub user's profile to gauge their contributions: github.com PyCon US YouTube channel: www.youtube.com SBOMit: adding verification to SBOMs (OpenSSF): openssf.org Ecosystems: ecosyste.ms Watch this episode on YouTube: youtube.com Episode #557 deep-dive: talkpython.fm/557 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

For years, "Django and async" came with an asterisk. The docs themselves warned you off it. Scary performance notes, a story that felt half-finished. Well, that story just got rewritten, literally, and the person who rewrote it is here to tell you why the old framing was wrong. Carlton Gibson is a former Django Fellow, sat on the security team for eight years, and he's on the steering council. On this episode we get into the async topic doc rewrite, what actually remains versus what was just fear, the new Tasks framework in 6.0, DB-level cascades and fetch modes landing in 6.1, and why free-threading is the bet that's about to pay off big for Django. If you've been told Django's async story isn't ready, this is the episode that puts that myth to bed. Episode sponsors Sentry Error Monitoring, Code talkpython26 Python in Production Talk Python Courses Links from the show DjangoCon Europe: djangocon.eu PyCon Italia: pycon.it Django on the Med: djangomed.eu Django Mantle: noumenal.es PyPI: pypi.org release notes: docs.djangoproject.com on_delete: docs.djangoproject.com Fetch modes: docs.djangoproject.com HttpRequest.multipart_parser_class: docs.djangoproject.com async topic doc: docs.djangoproject.com docs: docs.djangoproject.com DEP 14: github.com django-tasks: github.com django-tasks-local: github.com Celery: docs.celeryq.dev PEP 703: peps.python.org free-threading HOWTO: docs.python.org PEP 779: peps.python.org ASGI: docs.djangoproject.com PGBouncer: www.pgbouncer.org Channels: channels.readthedocs.io sync_to_async / async_to_sync: docs.djangoproject.com noumenal.es: noumenal.es Django Chat: djangochat.com @[email protected]: fosstodon.org Article: Cutting Python Web App Memory Over 31%: mkennedy.codes Watch this episode on YouTube: youtube.com Episode #556 deep-dive: talkpython.fm/556 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

Coding agents have gotten really good at one kind of work. You scope a feature, edit some files, run the tests, ship it. It all happens on disk. But that is not how data work feels. You load something, you look at it, you run a cell, you watch how it responds, and you decide the next move from whatever is sitting in memory. And until now, your agent couldn't see any of that. It only saw the files. Never the live state. This episode, that wall comes down. marimo pair drops a coding agent right inside a running notebook, with full access to every variable Python is holding in memory. The notebook becomes a shared canvas. You point, it runs the code. You tell it to zoom in on the Picasso paintings, and the chart just updates. No MCP tools to wire up, no schema to describe. Just Python, and an agent that can finally see what you see. Trevor Manz is back to walk us through it. Episode sponsors Sentry Error Monitoring, Code talkpython26 Talk Python Courses Links from the show marimo pair: marimo.io/pair Course transcripts announcement: talkpython.fm/blog anywidget: Jupyter Widgets made easy: talkpython.fm marimo: marimo.io blog: marimo.io GitHub: github.com given this: martinalderson.com llms.txt: talkpython.fm mcp: talkpython.fm cli: talkpython.fm open issues: github.com Discord: marimo.io Marimo Pair: marimo.io OpenCode: opencode.ai AI Tooling for Software Engineers in 2026: newsletter.pragmaticengineer.com Watch this episode on YouTube: youtube.com Episode #555 deep-dive: talkpython.fm/555 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

You ask an AI a question and it answers with total confidence. Most of the time, a confidently wrong answer is just an annoyance. But what if the question is medical, and there's a real patient on the other end? In that world, a hallucination isn't a bug, it's a patient-safety event. Sumit Gundawar is a London-based software engineer who builds the clinical platform for a UK longevity and aesthetic-medicine clinic, and his whole argument is that in high-stakes AI, the model is the easy part. Earning trust is the real engineering. We dig into grounding, refusal logic, human-in-the-loop design, and the messy frontier of longevity and biohacking, plus a live demo of an assistant that refuses to answer when it can't back up the claim. Let's get into it. Episode sponsors Six Feet Up Talk Python Courses Links from the show Guest Sumit Gundawar: linkedin.com Course transcripts announcement: talkpython.fm/blog Sumit Gundawar - JAX London Speaker: jaxlondon.com Anthropic: anthropic.com OpenAI Platform: platform.openai.com Anthropic: anthropic.com LangChain: langchain.com OWASP: owasp.org Pydantic: pydantic.dev EU AI Act - Regulatory Framework: digital-strategy.ec.europa.eu HIPAA - HHS: www.hhs.gov NHS: www.nhs.uk Llama: llama.com Qwen - QwenLM on GitHub: github.com OpenAI Platform: platform.openai.com Hugging Face: huggingface.co Llama: llama.com Granola: www.granola.ai HIPAA - HHS: www.hhs.gov CodeRabbit: www.coderabbit.ai Cursor Origin: cursor.com GitHub Status: www.githubstatus.com Midjourney Medical: www.midjourney.com Neko Health: www.nekohealth.com CERN: home.cern ATLAS Experiment: atlas.cern Watch this episode on YouTube: youtube.com Episode #554 deep-dive: talkpython.fm/554 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy

This episode is a fun crossover from our Python news and tips podcast, Python Bytes. We have had some big changes over there. Brian Okken has moved on and Calvin Hendryx-Parker has joined the show as the new co-host. To kick off this new era, we decided to do a longer and more personal episode called "All Our Tools". The idea is both of us talk about some of our most useful day-to-day developer and business owner tools that we think you all would find useful. It was so well received, that I'm bringing it to you all as a crossover episode. Enjoy and we hope you find something new and awesome to help you with your software and data science day to day. Episode sponsors Sentry Error Monitoring, Code talkpython26 Python in Production Talk Python Courses Links from the show @[email protected]: sixfeetup.social @calvinhp.com: bsky.app calvinhp.com: calvinhp.com Original airing on Python Bytes: pythonbytes.fm pi: pi.dev superpowers: github.com Warp.dev: Warp.dev OhMyZSH: ohmyz.sh Commandbookapp.com: Commandbookapp.com Blink: blink.sh kitty: sw.kovidgoyal.net mosh: mosh.org tmux: github.com Claude code: www.anthropic.com Claude.md: Claude.md MacWhisper: goodsnooze.gumroad.com Handy: handy.computer Tailscale: tailscale.com Talk Python episode with Alex: talkpython.fm Telescopo: www.telescopo.app Typora markdown: typora.io formal documentation for many of my open source packages: mkennedy.codes Great Docs: posit-dev.github.io Statement on the US government directive to suspend access to Fable 5 and Mythos 5: www.anthropic.com No second date: x.com Watch this episode on YouTube: youtube.com Episode #553 deep-dive: talkpython.fm/553 Episode transcripts: talkpython.fm Theme Song: Developer Rap 🥁 Served in a Flask 🎸: talkpython.fm/flasksong ---== Don't be a stranger ==--- YouTube: youtube.com/@talkpython Bluesky: @talkpython.fm Mastodon: @[email protected] X.com: @talkpython Michael on Bluesky: @mkennedy.codes Michael on Mastodon: @[email protected] Michael on X.com: @mkennedy