Model City Monday 10/27/25

If we worry too much about AI safety, will this make us "lose the race with China"1? (here "AI safety" means long-term concerns about alignment and hostile superintelligence, as opposed to "AI ethics" concerns like bias or intellectual property.) Everything has tradeoffs, regulation vs. progress is a common dichotomy, and the more important you think AI will be, the more important it is that the free world get it first. If you believe in superintelligence, the technological singularity, etc, then you think AI is maximally important, and this issue ought to be high on your mind. But when you look at this concretely, it becomes clear that this is too small to matter - so small that even the sign is uncertain. https://www.astralcodexten.com/p/why-ai-safety-wont-make-america-lose

Most discourse on AI is low-quality. Most discourse on consciousness is super-abysmal-double-low quality. Multiply these - or maybe raise one to the exponent of the other, or something - and you get the quality of discourse on AI consciousness. It's not great. Out-of-the-box AIs mimic human text, and humans almost always describe themselves as conscious. So if you ask an AI whether it is conscious, it will often say yes. But because companies know this will happen, and don't want to give their customers existential crises, they hard-code in a command for the AIs to answer that they aren't conscious. Any response the AIs give will be determined by these two conflicting biases, and therefore not really believable. A recent paper expands on this method by subjecting AIs to a mechanistic interpretability "lie detector" test; it finds that AIs which say they're conscious think they're telling the truth, and AIs which say they're not conscious think they're lying. But it's hard to be sure this isn't just the copying-human-text thing. Can we do better? Unclear; the more common outcome for people who dip their toes in this space is to do much, much worse. But a rare bright spot has appeared: a seminal paper published earlier this month in Trends In Cognitive Science, Identifying Indicators Of Consciousness In AI Systems. Authors include Turing-Award-winning AI researcher Yoshua Bengio, leading philosopher of consciousness David Chalmers, and even a few members of our conspiracy. If any AI consciousness research can rise to the level of merely awful, surely we will find it here. One might divide theories of consciousness into three bins: https://www.astralcodexten.com/p/the-new-ai-consciousness-paper

ACX has been co-running a forecasting contest with Metaculus for the past few years. Lately the "co-running" has drifted towards them doing all the work and giving me credit, but that's how I like it! Last year's contest included more than 4500 forecasters predicting on 33 questions covering US politics, international events, AI, and more. They're preparing for this year's contest, and currently looking for interesting questions. These could be any objective outcome that might or might not happen in 2026, whose answer will be known by the end of the year. Not "Will Congress do a good job?", but "Will Congress' approval rating be above 40% on December 1, 2026?". Or, even better, "Will Congress' approval rating be above 40% according to the first NYT Congressional Approval Tracker update to be published after December 1, 2026?". Please share ideas for 2026 forecast questions here. The top ten question contributors will win prizes from $150 to $700. You can see examples of last year's questions here (click on each one for more details). This year's contest will also include AI bots, who will compete against the humans and one another for prizes of their own. To learn more about building a Metaculus forecasting bot, see here. I'll keep you updated on when the contest begins. https://www.astralcodexten.com/p/suggest-questions-for-metaculusacx  

Last year, I wrote that it would be very hard to decrease the number of mentally ill homeless people in San Francisco. Commenters argued that no, it would be easy, just build more jails and mental hospitals. A year later, San Francisco feels safer. Visible homelessness is way down. But there wasn't enough time to build many more jails or mental hospitals. So what happened? Were we all wrong? Probably not. I only did a cursory investigation, and this is all low-confidence, but it looks like: There was a big decrease in tent encampments, because a series of court cases made it easier for cities to clear them. Most of the former campers are still homeless. They just don't have tents. There might have been a small decrease in overall homelessness, probably because of falling rents. Mayor Lurie claims to have a Plan To End Homelessness, but it's probably not responsible for the difference. Every city accuses every other city of shipping homeless people across their borders, but this probably doesn't explain most of what's going on in San Francisco in particular. https://www.astralcodexten.com/p/what-happened-to-sf-homelessness

"Life is suffering" may be a Noble Truth, but it feels like a deepity. Yes, obviously life includes suffering. But it also includes happiness. Many people live good and happy lives, and even people with hard lives experience some pleasant moments. This is the starting point of many people's objection to Buddhism. They continue: if nirvana is just a peaceful state beyond joy or suffering, it sounds like a letdown. An endless gray mist of bare okayness, like death or Britain. If your life was previously good, it's a step down. Even if your life sucked, maybe you would still prefer the heroism of high highs and low lows to eternal blah. Against all this, many Buddhists claim to be able to reach jhana, a state described as better than sex or heroin - and they say nirvana is even better than that. Partly it's better because jhana is temporary and nirvana permanent, but it's also better on a moment-to-moment basis. So nirvana must mean something beyond bare okayness. But then why the endless insistence that life is suffering and the best you can do is make it stop? I don't know the orthodox Buddhist answer to this question. But I got the rationalist techno-Buddhists' answer from lsusr a few months ago, and found it, uh, enlightening. He said: mental valence works like temperature. Naively, there are two kinds of temperature: hot and cold. When an environment stops being hot, then it's neutral - "room temperature" - neither hot nor cold. After that, you can add arbitrary amounts of coldness, making it colder and colder. https://www.astralcodexten.com/p/in-what-sense-is-life-suffering

In Jason Pargin's I'm Starting To Worry About This Black Box Of Doom, a manic pixie dream girl cajoles a shut-in incel loser to drive her and her mysterious box cross-country. The further they drive, the more evidence starts to build that she is a terrorist and her box is a nuke. As our protagonist becomes increasingly desperate to turn around and return to his comfortable world of social media feeds and psych meds, she pleads with him to come out of his shell, learn to trust people offline, and have a sense of adventure. The book's dramatic tension comes from our simultaneously rooting for his character development and worrying that it might be a ruse to manipulate him into blowing up Washington, DC. This book is not shy about its moral, delivered in approximately one soliloquy per state by our author mouthpiece character (the girl). Although there is a literal black box of doom - the suspected nuke - the real danger is the metaphorical "black box" of Internet algorithms, which make us waste our lives "doom" scrolling instead of connecting to other human beings. Or the "black box" of fear that the algorithms trap us in, where we feel like the world is "doomed" and there's nothing we can do. She urges us to break out of our boxes and feel optimism about the state of society. Quote below, Ether is the girl, Abbott is the loser, and he's just ventured the opinion that it's unethical to have children in a world as doomed and dystopian as ours: https://www.astralcodexten.com/p/the-bloomers-paradox

American Scholar has an article about people who "write for AI", including Tyler Cowen and Gwern. It's good that this is getting more attention, because in theory it seems like one of the most influential things a writer could do. In practice, it leaves me feeling mostly muddled and occasionally creeped out. "Writing for AI" means different things to different people, but seems to center around: Helping AIs learn what you know. Presenting arguments for your beliefs, in the hopes that AIs come to believe them. Helping the AIs model you in enough detail to recreate / simulate you later. Going through these in order: https://www.astralcodexten.com/p/writing-for-the-ais

[I haven't independently verified each link. On average, commenters will end up spotting evidence that around two or three of the links in each links post are wrong or misleading. I correct these as I see them, and will highlight important corrections later, but I can't guarantee I will have caught them all by the time you read this.] https://www.astralcodexten.com/p/links-for-october-2025

Would You Like To Buy A Bahama? The Bahamas is an archipelago-nation of 400,000 people scattered across 3,000 small islands. The Bahamas' most populous island is the one with its capital, Nassau. The second-most-populous - and fifth-largest, and most-pretentiously-named - is Grand Bahama, home of Freeport, the archipelago's second city. Grand Bahama has a unique history. In 1955, it was barely inhabited, with only 500 people scattered across a few villages. The British colonial government turned it into a charter city, awarding the charter to Wallace Groves, an American whose Wikipedia article describes him as a "financier and fraudster" and includes section titles like "Suspicions", "Legal Troubles", "Investigations", and "Allegations Of Underworld Connections". He was . . . maybe the exact right person for the job, turning Grand Bahama into a Vegas of the Caribbean complete with casinos, jet-setters, swanky hotels, and a flourishing mob presence. Outside the glitzy center, a little heavy industry even managed to develop around the port. After twenty years, the charter zone was "the most modern, well-run, and prosperous part of the [Bahamas]", and the population had increased to 15,000. https://www.astralcodexten.com/p/model-city-monday-102725

[original post here] The Kasina Connection In the original post, I cited ambiguous later examples of sun miracles which didn't seem to affect everyone equally and in some cases were unconnected (or barely connected) to religious phenomena, concluding that they must be some kind of very unusual illusion. My main hangup with this conclusion was the wild implausibility of an illusion that nobody had ever noticed before, outside of this one 1917 miracle and a few copycats, despite plenty of people staring at the sun throughout history for various (bad) reasons. Surely there must be somebody else, somewhere, discussing how if you stare at a bright light long enough it will spin and change color. Two commenters, Dave Moore and Anomony, bring up fire kasina practice. https://www.astralcodexten.com/p/highlights-from-the-comments-on-fatima