The Pre-Training Wall and the Treadmill After It

I found a comment on the Call of Duty Mobile subreddit. Kevffinn: COMPUTERS CANNOT GENERATE RANDOM NUMBERS AND ANY SUGGESTION OTHERWISE IS FRAUDULENT. I'm not sure about the fraud part, but what does and does not constitute a random number is a surprisingly deep question. So I built a randomness detector, and I bet Don that we would not be able to make a random number. Loser buys lunch. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter

I texted Don about a side gig: €5,000 for every 1% you shave off a zip file. It's a real contest. For 20 years, Marcus Hutter has offered €500,000 to anyone who can losslessly compress a gigabyte of Wikipedia below 110 megabytes. Gzip gets you to 322. After that you're on your own. So we tried. Run-length encoding. Pointing back instead of repeating yourself. Huffman coding worked out by hand. Arithmetic coding, which spends less than one bit per character. Then Claude Shannon's 1951 guessing game, and a paper where an LLM beats every record and still wins nothing, because the model counts against you. Don wanted the euros. I had other reasons for bringing him over. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

I've been trying to understand how machine learning actually works. Not use it, understand it, down to the ifs and loops. How does a program built out of plain conditionals get better on its own? So late one night I sent Don a paper. Three words in the title: reward is enough. The claim is that all of intelligence, the whole thing, comes down to a system maximizing a reward. Don thought that was far too reductive. I wanted to pull it apart and see if it held up. We backed up through the history to find out how far "reward is enough" really goes: B.F. Skinner training pigeons, a backgammon program that taught itself, the Go move no human would have played. It's a story about machine learning, and what that leaves for the rest of us who still do it by hand. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

I've been confusing Don with frontier-lab links late at night for a bit. Ilya Sutskever told a NeurIPS audience that pre-training as we know it would unquestionably end. There's only one internet, and the data isn't growing. The frontier labs call this the pre-training wall. A leaked Google memo from 2023 argued they had no moat. R1 is on GitHub. Llama is on Hugging Face. OpenAI's secondary-market valuation has climbed past $850 billion.  Don was confused. So he came over and we made an episode about it. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

Kate Gregory has been writing C++ for over forty years. Books, keynotes, a consulting firm she built from the ground up. At sixty-three, she's one of the most experienced programmers alive. She surveyed hundreds of software engineers about getting older. What scares you? What's changed? What have you lost? The things people feared most — memory, stamina, keeping up — weren't the real threats. The stuff that was actually breaking down was mostly fixable. A bad knee wasn't aging, it was a torn cartilage. Wrist pain disappeared when she changed how she slept. But buried in the research was something harder to fix. The single factor that predicted whether you'd age well or badly had nothing to do with your body at all. The opponent isn't aging. The opponent is the story about aging. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

Corey told me about his AI cat reel problem. He found these AI-genearted cat videos hilarious. Who makes these? He kept sending them to his wife. Then he tried to stop watching and he couldn't. So I went down the rabbit hole of how social media algorithms actually work. It starts simple. Upvote, downvote, sort by time. But by 2017 Facebook has a metric that quietly reshapes what two billion people see. Then a leaked playbook lands, and a CEO takes the stand in Los Angeles. Today is an investigation into what happens when the algorithm knows you better than you know yourself. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter    

  Multiple VS Code windows. "Agent stopping" in a robot voice. A laptop stand on the treadmill so Claude can keep working while I run. The Big Rich sitting unread by the fireplace while I check if the migration's done. Somewhere along the way, I started reorganizing my life around keeping the machine spinning. Claude Code had become my universal paperclip clicker. This is me trying to figure out the difference between real work and just feeding it tickets.   This is some field notes, a shorter, rougher than a normal epsidoe.  Episode Page Support The Show Subscribe To The Podcast Join The Newsletter

Ron Garret left JPL for a 100-person startup he'd just discovered on Usenet. Four a.m. alarms. Burbank to San Jose on Southwest. A rented room in Susan Wojcicki's house. He expected the search engine engineering and instead he got asked to build ad serving. In Java and with JSPs and no syntax highlighting and no delimiter balancing. Launch week was a stampede and then a window on his screen fills with declines. Numbers he can't explain. Some of them look… real. How do you even name what's happening?    This episode is about creating Google AdWords. Building the machine that prints money, while trying not to get crushed in the gears. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

Imagine facing a problem you can't name, something that feels bigger than any bug you've ever had to fix. How do you debug your own mind when you don't even know what's wrong? Burke Holland's story starts with a college party and a bad trip that leaves a deeper mark than he expects. Sleep gets harder. Fear creeps in. His life starts shrinking. School falls apart, friends drift away, and he ends up back at home trying to understand what's happening to him. He looks for structure in the Coast Guard. Later he discovers computers and realizes he might have found the thing he's meant to do. But the shadow that followed him out of that party doesn't care about career paths. It shows up during college, during work, during marriage, during parenthood. Sometimes it's quiet, sometimes it knocks him completely flat. This is the story of a developer who looks effortless on stage but spent years fighting something no one else could see, and what changed once he finally understood what he was up against.   What do you do when the hardest problem in your life isn't in your code, but in yourself? Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

What do you do when your code breaks and the only fix is to dig into the runtime below? Matt Godbolt lives for that. Tile-based renderers, color-coded scanlines, zero-copy NICs—each story is a clue that leads past the abstraction to the real machine. He shares the rule that guides him: master your layer, learn the one below, and know the outline of the layer under that.  Matt Godbolt's journey proves the real breakthroughs are hideen behind the abstrations where you are comfortable and familiar. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter