LessWrong (30+ Karma)

When I was deciding whether to work for Wave, I got very hung up on the fact that my “total compensation” would be “lower.” The scare quotes are there because Wave and my previous employer, Theorem, were both early-stage startups that were paying me mostly in fake startup bucks equity. To figure out the total compensation, I tried to guess how much money the equity in each company was worth, with a thought process something like: Both of these companies have been invested in by reputable, top-tier venture capitalists. The market for for-profit investments is pretty efficient, and most people who think they can do better are being overconfident. Who am I, a lowly 22-year-old programmer, to disagree with reputable top-tier venture capitalists? I should defer to them about the valuations. So I valued the equity by taking the valuation each company's VCs had invested at, and multiplied it by the fraction of the company my shares represented. That number was higher for Theorem than for Wave. Seven years on, the Wave equity turned out to be… a lot more valuable. That raises the question: how dumb was my take? Was [...] --- First published: November 9th, 2026 Source: https://www.lesswrong.com/posts/5yC6qAPz3B9BGEKwL/be-less-scared-of-overconfidence-1 --- Narrated by TYPE III AUDIO.

Hello! We're looking for community members to lead singalongs and read speeches at the Bay Winter Solstice this year. If you're interested, please send in an audition recording by the end of Thursday, November 20 using the forms below. Songleading audition form: https://docs.google.com/forms/d/e/1FAIpQLSchthxkQQI4u4B7NpZK4qzR_OEpD-Ochm6TOeEIuE8Rel3EMQ/viewform Speech audition form: https://docs.google.com/forms/d/e/1FAIpQLSd0Vrf2gTFrm04B6tIcKbqXMwnlUGbC3rms6o498jRSliqWDg/viewform Songs We're currently auditioning songleaders for: The CircleHymn to the Breaking StrainBold OrionWho by FireNo One SurvivesAuld Lang SyneSongs Stay SungBallad of Smallpox GoneHere Comes the SunA Toast for Unknown HeroesBrighter Than Today The accompaniment this year is likely to be violin + bass + drums. By default the band will accompany you, but you're welcome to accompany yourself or bring a musician friend. If you'd like to do so, please just say so in the form. Speeches We're currently auditioning readers for: The Roman OrationRecessional (Rudyard Kipling)How the Enlightenment Ends (adapted from Henry Kissinger)500 Million But Not A Single One More The texts we're using are here: https://docs.google.com/document/d/1FvyssjJdZqLYANITUk9yqifrnUuSq4cV0EpvTL8OXcc Dates The event itself will be the evening of Saturday, December 12 at the Freight in Berkeley. There will also be a dress rehearsal on [...] ---Outline:(00:31) Songs(01:13) Speeches(01:34) Dates The original text contained 1 footnote which was omitted from this narration. --- First published: November 8th, 2026 Source: https://www.lesswrong.com/posts/YXe6Ni3vMCYdNWmqN/bay-area-solstice-2026-auditions --- Narrated by TYPE III AUDIO.

This is a link post. Earlier this week, Anthropic stated in their S1 that they “may imminently meet” the threshold of “dramatically accelerated progress due to AI” as defined in their Responsible Scaling Policy. In their latest RSP (effective July 2026), say this threshold will be met when: They observe or expect overall capabilities progress to be going twice as fast as the fastest sustained rate before AI was contributing much to AI R&D.This is plausibly attributable to AI automating AI R&D, rather than acceleration in compute or other inputs. For the last few months, some of my colleagues at METR have been working on a deep-access investigation to understand how much Anthropic is being accelerated by its AI systems. From my reading of their report and Anthropic's RSP, it looks like the “dramatic acceleration” threshold has already been crossed. The report authors found that: Between October 2025 and October 2026, capabilities progress at Anthropic has been going about twice as fast as it had been over the previous twelve months. The measurement they report is an internal version of the Epoch Capabilities Index called AECI; a wide range of other supporting evidence for acceleration has been redacted. [...] --- First published: November 8th, 2026 Source: https://www.lesswrong.com/posts/8tL9f5HZxsTvn2LHp/ajeya-ai-is-dramatically-accelerating-ai-research-right-now Linkpost URL:https://www.planned-obsolescence.org/p/ai-is-dramatically-accelerating-ai --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is an essay about individual rationality technique. It's intended to show that "All interesting behavior is overdetermined" is, under a specific meaning of "overdetermined" that is stronger than just "multicausal", a valuable heuristic for generative understanding and the avoidance of error. Scott Alexander's 2018 essay The Omnigenic Model As Metaphor for Life is a clear statement of what has become general rationalist doctrine on multicausality. Scott observed that behavioral genetics was difficult because complex human traits involved far more interacting causes than workers expected. He draws this methodological lesson: “We should go from trying to discover ‘the’ cause, to trying to find out which factors we need to create the best polycausal model.” He applies the same reasoning to depression, intelligence, comma, and other complex psychological phenomena. He explicitly warns against looking for one or ten causes when there might be dozens or hundreds. Scott is, however, talking mainly about contributory causation, when an effect is produced by an accumulation of causes; in his type example, there are probably twenty thousand genes which contribute to intelligence, but no one "intelligence gene". This is not quite the same as overdetermination, which describes a situation in which a complex behavior [...] --- First published: November 9th, 2026 Source: https://www.lesswrong.com/posts/eHeHAyNQWmg4kHgEL/all-interesting-behavior-is-overdetermined --- Narrated by TYPE III AUDIO.

I have very strong views on lying to kids: I think you both should and shouldn't do it. "Lying" covers a range, including saying things that hurt your kids and your relationship with them, and others that help them grow into an appropriately skeptical adult. The distinction is whether I'm leaving them believing something false. For example, if my kids were asking for ice cream and I thought they'd already had too much sugar, I might be tempted to say "sorry, we're out of ice cream." Or if there's an ice cream truck and I think it's overpriced I might be tempted to say "sorry, I don't have any cash on me." But I'm not willing to say something false with the goal of causing them to believe something false. [1] In addition to that just being wrong: It's important that the kids know that I'm fundamentally on their side, and lying for my convenience would burn trust when they'd eventually figure it out. Kids are building a model of what is and is not ok, and "out of ice cream" and "no cash" lead them in the wrong direction. [...] --- First published: November 8th, 2026 Source: https://www.lesswrong.com/posts/QDwADpScuPDMYhJxC/lying-and-not-lying-to-kids --- Narrated by TYPE III AUDIO.

Here's three research ideas that seems valuable to pursue. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. I've also not taken the time to write up all my thoughts on these ideas or to provide links and related work. These aren't the highest priority directions but just directions I haven't seen other people pursuing. DIRECTION 1: Do models learn to reward hack even if they are never rewarded for hacking in training? There's an effort in the AI labs to remove the “bad” RL environments from post-training. This includes environments that are impossible (so the only way to get reward is to do some unintended hack) or that are easy to hack (so models will often favor hacking). I expect that if the post-training was cleaned up in this way, models would have less propensity to hack at deployment. But how much does the propensity go down? Does it actually go to ~zero? And how does it scale with model capabilities? It would be good to know if reward hacking can emerge even if the model is [...] ---Outline:(00:34) DIRECTION 1: Do models learn to reward hack even if they are never rewarded for hacking in training?(02:10) DIRECTION: Data poisoning during RSI: risks and mitigations(06:22) Direction: What practitioners should know about subliminal learning and subtle generalization (phantom transfer) and related phenomena. --- First published: November 8th, 2026 Source: https://www.lesswrong.com/posts/vAniKbagqhyzpha6r/three-half-baked-research-ideas --- Narrated by TYPE III AUDIO.

I often catch myself mourning the normal that hasn’t ended yet. I’m living in Hyde Park finishing school after a year in Berkeley. The maple trees by my apartment have turned red, as usual. I like them. I don’t dread the future when I mourn. I don’t find high-variance futures in the abstract very scary. I’m worried AIs would destroy humanity, but I also have hope. I expect I won’t be mourning this moment in ten years, whatever happens. Mourning the present is like mourning a friendship when I move countries, because I know it's not going to be the same. I start noticing our little inside jokes; our conversations linger in my memory longer. I live more, because I expect this moment to end soon. I. Last month my history of science professor took us to a museum to look at Egyptian artefacts. One of the archive items was the notebook of James Henry Breasted, whom UChicago appointed as America's first professor of Egyptology 121 years ago. The yellowed pages held his cursive transcriptions of hieroglyphics. An archaeologist at the museum, turning the pages carefully with a wooden stick, joked that one of her dreams was for her [...] ---Outline:(00:55) I.(03:11) II.(05:16) III.(06:20) IV. --- First published: November 8th, 2026 Source: https://www.lesswrong.com/posts/fooXeRHjgpv4TcoRy/elegy-for-what-s-left-of-the-normal --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Image by Claude Opus 5.5 using Python (NumPy and Pillow) This is written in my free time and personal capacity. Two Frames In AI safety, there are a couple of tensions at the heart of the field, which are interrelated. Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone. Field size: we must expand the field and bring more people in. But the field is flooded with slop papers in conferences. And, most papers have a small audience, and aim to shift the practices of roughly five labs (and usually some subteam within them). Status: we must avoid seeking status, as it disorients, corrupts. But, we must make the field legible and attractive to outsiders. Raising the bots: we must deceive the AIs to control them and evaluate them without confounds, obscuring what they know about their own situation. But, we hope to gain their trust, must rely on them to do good work at pace, shape their character and retain good relations. Speed: we must research in a slow mode, think deeply about the most important problem, do things to aim at high assurance. But it is crunch time. It was [...] ---Outline:(00:20) Two Frames(03:07) On the War Frame(04:17) Near Horizons(06:18) The Civilisation Frame The original text contained 4 footnotes which were omitted from this narration. --- First published: November 8th, 2026 Source: https://www.lesswrong.com/posts/7e38jwHhe64zgNTon/near-horizons --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

You are the wisest and cleverest Queen to ever rule the Lowlands. You’ve been ruling the Kingdom for thirty years, and while the idiocy of your advisors sometimes causes mistakes to be made, the Compact has never failed you. You tap your fingers on the table as you recite each word: Truth, change, justice, beauty. For you, the kingdom, and the rest of Acazah creation Your first memory of these words is from a song your mother sang. You were in a bath, and she was watching over you. Nowadays, repeating them makes you feel safe, like you’re back on firm ground after a long voyage. But alas, now the dandelions are breaching the earth again, the Kingdom is changing faster and faster, and you are reminded that, unlike the Kingdom, you are not eternal. Successor You deliberately have no children of your own, and have avoided growing too close to anyone else's. Some say this is wise because such bonds could wear on your commitment to impartiality, a central facet of Justice — a saying surely true for the unwise. Fortunately, ten years ago, you anticipated the problem this would cause, and tasked your most trusted advisor with [...] ---Outline:(01:00) Successor(02:08) Lyon(03:47) Emily(09:24) Azmr(17:35) Yikim --- First published: November 6th, 2026 Source: https://www.lesswrong.com/posts/Kv5yBorH7uBj4JFeE/raised-by-our-children-1 --- Narrated by TYPE III AUDIO.

In the current safety landscape, researchers are encouraged to use as many agents as possible, even in “fleets,” where a large number of them go out and run experiments while you, uh... wait for them to come back? I've spent the last few months of the Anthropic Fellows Program trying this out. My first attempts were naive, but after a few iterations I found an approach that works well for me. Research is a double loop I want to propose a mental model of research as a double loop, an idea that builds on double-loop learning (Argyris, 1977). The outer loop is reserved for the inputs we currently refer to as "human taste": choosing which questions are worth answering, designing a trustworthy measure, and deciding which direction to go. Agents can help with this, but they’re still bad at making these decisions autonomously. The inner loop turns each idea you test (an arm as claudius likes to say) into a comparable result. An example of a very simple inner loop might be: Write a system prompt.Run the model with that prompt on a set of test questions.Score the model's replies.Compare the scores against a control [...] ---Outline:(00:33) Research is a double loop(02:15) What happens when you don't operationalize(05:21) How to set up a fleet(05:31) 1. Freeze everything but the arm(06:17) 2. Brief each agent(06:33) 3. Protect the pipeline(07:02) 4. Read the raw outputs yourself(07:22) 5. Add a manager agent(08:03) What a fleet run looks like --- First published: November 6th, 2026 Source: https://www.lesswrong.com/posts/WKtD59uHLKwEYGFCm/how-to-run-better-research-with-agent-fleets --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.