#91 - The Uses and Abuses of Statistics (w/ Ben Recht)

Back to basics baby. We're doing a couple introductory episodes on Popper's philosophy of science, following Chapter 10 of Conjectures and Refutations. We start with Popper's theory of content: what makes a good scientific theory? Can we judge some theories as better than others before we even run any empirical tests? Should we be looking for theories with high probability? Ben and Vaden also return to their roots in another way, and get into a nice little fight about how content relates to Bayesianism. We discuss Vaden's skin care routine If you find your friend's lost watch and proceed to lose it, are you responsible for the watch? Empirical vs logical content Whether and how content can be measured and compared How content relates to probability Quotes My aim in this lecture is to stress the significance of one particular aspect of science—its need to grow, or, if you like, its need to progress. I do not have in mind here the practical or social significance of this need. What I wish to discuss is rather its intellectual significance. I assert that continued growth is essential to the rational and empirical character of scientific knowledge; that if science ceases to grow it must lose that character. It is the way of its growth which makes science rational and empirical; the way, that is, in which scientists discriminate between available theories and choose the better one or (in the absence of a satisfactory theory) the way they give reasons for rejecting all the available theories, thereby suggesting some of the conditions with which a satisfactory theory should comply. You will have noticed from this formulation that it is not the accumulation of observations which I have in mind when I speak of the growth of scientific knowledge, but the repeated overthrow of scien- tific theories and their replacement by better or more satisfactory ones. This, incidentally, is a procedure which might be found worthy of attention even by those who see the most important aspect of the growth of scientific knowledge in new experiments and in new observations. - C&R p. 291 Thus it is my first thesis that we can know of a theory, even before it has been tested, that if it passes certain tests it will be better than some other theory. My first thesis implies that we have a criterion of relative potential satisfactoriness, or of potential progressiveness, which can be applied to a theory even before we know whether or not it will turn out, by the passing of some crucial tests, to be satisfactory in fact. This criterion of relative potential satisfactoriness (which I formu- lated some time ago,2 and which, incidentally, allows us to grade the- ories according to their degree of relative potential satisfactoriness) is extremely simple and intuitive. It characterizes as preferable the theory which tells us more; that is to say, the theory which contains the greater amount of empirical information or content; which is logically stronger; which has the greater explanatory and predictive power; and which can therefore be more severely tested by comparing predicted facts with observations. In short, we prefer an interesting, daring, and highly informative theory to a trivial one. - C&R p.294 Let a be the statement ‘It will rain on Friday’; b the statement ‘It willbe fine on Saturday’; and ab the statement ‘It will rain on Friday and itwill be fine on Saturday’: it is then obvious that the informative contentof this last statement, the conjunction ab, will exceed that of its com-ponent a and also that of its component b. And it will also be obviousthat the probability of ab (or, what is the same, the probability that abwill be true) will be smaller than that of either of its components. Writing Ct(a) for ‘the content of the statement a’, and Ct(ab) for ‘thecontent of the conjunction a and b’, we have (1) Ct(a) = Ct(b). This contrasts with the corresponding law of the calculus of probability, (2) p(a) >= p(ab) <= p(b), where the inequality signs of (1) are inverted. Together these two laws, (1) and (2), state that with increasing content, probability decreases, and vice versa; or in other words, that content increases with increasing improbability. (This analysis is of course in full agreement with the general idea of the logical content of a statement as the class of all those statements which are logically entailed by it. We may also say that a statement a is logically stronger than a statement b if its content is greater than that of b—that is to say, if it entails more than b does.) This trivial fact has the following inescapable consequences: if growth of knowledge means that we operate with theories of increasing content, it must also mean that we operate with theories of decreasing probability (in the sense of the calculus of probability). Thus if our aim is the advancement or growth of knowledge, then a high probability (in the sense of the calculus of probability) cannot possibly be our aim as well: these two aims are incompatible. - C&R p.295 Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) How much content does the theory "dish soap is the ultimate face cleanser" have? Send your order of infinity over to [email protected]

We're joined by Jonathan Rauch to discuss what it means to be a radical incrementalist, how to foment revolution on geological timescales, and whether Christianity can be a force for good in politics. Can Jon convince angry-Hitchens-atheist Vaden that Christianity has some benefits? Will both Vaden and Ben be at Sunday prayer? Follow Jonathan on his website (https://www.jonathanrauch.com/about.html), at Brookings (https://www.brookings.edu/people/jonathan-rauch/), at The Atlantic (https://www.theatlantic.com/author/jonathan-rauch/) or on Bluesky (https://bsky.app/profile/jonrauch.bsky.social). We discuss The constitution of knowledge and whether it's holding Norms vs laws, and whether we should introduce more laws to codify norms Popper's paradox of tolerance How should liberals respond to illiberalism? Which tactics, if any, should democrats adopt from MAGA to fight MAGA? Sharp Christianity and Christian nationalism Rauch's plea to Christians References The Constitution of Knowledge: A Defense of Truth (https://www.amazon.com/Constitution-Knowledge-Jonathan-Rauch/dp/0815738862) Cross Purposes: Christianity's Broken Bargain with Democracy (https://www.amazon.com/Cross-Purposes-Christianitys-Bargain-Democracy/dp/0300273541) Errata Jonathan Rauch is the author of nine books, not eight! Socials Follow us on Twitter at @JonRauch, @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) Anyone in Canada have a basement suite Jonathan could rent for a while? Send your address over to [email protected] Special Guest: Jonathan Rauch.

Professor of electrical engineering and computer science Ben Recht joins us to defend Bayesianism, AI doom, and assure us that the statisticians have everything under control. Just kidding. Recht might be even more suspicious of these things than we are. What has statistics ever done for us, really? When was the last time YOU ran a clinical trial after all, huh? HUH? After Ben Chugg defends his life decision to do a PhD in statistics, we talk AI, cults, philosophy, Paul Meehl, and discuss Ben Recht's forthcoming book, The Irrational Decision (https://press.princeton.edu/books/hardcover/9780691272443/the-irrational-decision). Check out Ben's blog (https://www.argmin.net/), website (https://people.eecs.berkeley.edu/~brecht/), and his story about machine learning (https://mlstory.org/). We discuss Ben Recht's theory of blogging Why is Berkeley the epicenter of AI doom? Where the word "robot" came from Is Bayesian reasoning responsible for AI doom? Paul Meehl and his contributions to science Ben Recht's bureaucratic theory of statistics What on earth is null hypothesis testing? What is the point of statistics? "Sweet spots" and "small worlds" Does science proceed by Popperian means? Can Popper get around the Duhem-Quine problem? Errata The z-score for the Pfizer trial was 20, not 12! References Argmin (https://www.argmin.net/), Ben Recht's blog David Freedman, UC Berkeley (https://statistics.berkeley.edu/about/biographies/david-freedman) Paul Meehl's online course (https://www.youtube.com/@michaelmcgovern8633/featured) Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology (https://errorstatistics.com/wp-content/uploads/2015/04/meehl-1978.pdf), Paul Meehl's 1978 paper. Clinical versus statistical prediction: A theoretical analysis and a review of the evidence (https://psycnet.apa.org/record/2006-21565-000), by Meehl On the near impossibility of estimating the returns to advertising (https://ftp.zew.de/pub/zew-docs/veranstaltungen/ICT2013/Papers/ICT2013_Rao.pdf) A Bureaucratic Theory of Statistics (https://arxiv.org/pdf/2501.03457) by Recht The new riddle of induction (https://fitelson.org/confirmation/goodman_1955.pdf) by Goodman Announcing the Irrational Decision (https://www.argmin.net/p/announcing-the-irrational-decision) Patterns, Predictions, and Actions (https://mlstory.org/), textbook by Ben Recht and Moritz Hardt Socials Follow us on Twitter at @BeenWrekt, @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) What's Berkeley's next cult? Send your guess over to [email protected] Special Guest: Ben Recht.

Always the uncool kids at the table, Ben and Vaden push back against the AGI hype domininating every second episode of every second podcast. We react to "We're not ready for superintelligence" (https://www.youtube.com/watch?v=5KVDDfAkRgc) by 80,000 Hours - a bleak portrayal of the pre and post AGI world. Can Ben keep Vaden's sass in check? Can the 80,000 hours team find enough cubes for AGI? Is Agent-5 listening to you RIGHT NOW? Listener Note: We strongly recommend watching the video for this one, available both on youtube and spotify: - https://www.youtube.com/@incrementspod - https://open.spotify.com/show/1gKKSP5HKT4Nk3i0y4UseB We discuss The incentives of superforecasters Arguments by authority Whether superintelligence is right around the corner The difference between model size and data Are we running out of high quality data? Does training on synthetic data work? The assumptions behind the AGI claims The pitfalls of reasoning from trends References Michael I Jordan (https://people.eecs.berkeley.edu/~jordan/) Neil Lawrence (https://en.wikipedia.org/wiki/Neil_Lawrence) Important technical paper from Jordan pushing back on Doomerism (A Collectivist, Economic Perspective on AI) Jordan article talking about dangers of using AlphaFold data (https://news.berkeley.edu/2023/11/09/how-to-use-ai-for-discovery-without-leading-science-astray/) Nature paper showing you can't use synthetic data to train bigger models (https://www.nature.com/articles/s41586-024-07566-y) Paper estimating of when training data will run out (https://arxiv.org/abs/2211.04325v2) (Coincidentally enough, sometime between 2027-2028) Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) But how many cubes until we get to AGI though? Send a few of your cubes over to [email protected] Episode header image from here (https://www.youtube.com/watch?app=desktop&v=0Jsrux_XY8Y&ab_channel=TheAlgorithmicVoice).

Phlogiston? Elan Vital? Caloric? Mention of any of these at a party, and Neil DeGrasse Tyson will be sure to take you out back and kick you in your essences. So why do "essences" have no place in science? In this episode we explore that question (and dive into some of the history behind this debate) by reading Chapter 6 of Conjectures and Refutations: A Note On Berkeley As Precursor Of Mach And Einstein. In one corner, we have the estimable Sir Isaac Newton and Roger Coates (and of course Andre the Giant, upon whose shoulders they are standing), and in the other, we have Bishop Berkeley and Ernst Mach, looking to throw down at the speed of sound. Berkeley can't get Newton and his forces out of his head (literally), and boy oh boy is the fight ever on. We discuss How should teachers address the "students using ChatGPT to write their essay" problem? Can we learn a bit from Stalin here? Is Ben basically Gandhi? (Answer: Yes of course) How can one be both an idealist and an empiricist? WTF is a 'force'??? Instrumentalism and Essentialism The history of the debate between Berkeley and Newton The lifelong feud between Ernst Mach and Ludwig Boltzman What's the difference between essences and unobservables? Is Mach a filthy plagiarist? Who won the essentialism vs instrumentalism debate? (Answer: Neither side won. Popper won.) References Go amuse yourselves and watch some videos of Newton's spinning bucket thought experiment (https://www.youtube.com/watch?v=Jz3mOlUOGoY&t=1093s&ab_channel=Dialect). Boltzmanns Atom: The Great Debate That Launched A Revolution In Physics (https://www.amazon.ca/Boltzmanns-Atom-Launched-Revolution-Physics/dp/1501142445) Quotes Everybody who reads this list of twenty-one theses must be struck by their modernity. They are surprisingly similar, especially in the criticism of Newton, to the philosophy of physics which Ernst Mach taught for many years in the conviction that it was new and revolutionary; in which he was followed by, for example, Joseph Petzold; and which had an immense influence on modern physics, especially on the Theory of Relativity. Popper, C&R Chapter 6 (20) A general practical result—which I propose to call ‘Berkeley’s razor’—of this analysis of physics allows us a-priori to eliminate from physical science all essentialist explanations. If they have a mathematical and a predictive content they may be admitted qua mathematical hypotheses (while their essentialist interpretation is eliminated). If not, they may be ruled out altogether. This razor is sharper than Ockham’s: all entities are ruled out except those which are perceived. Popper, C&R Chapter 6 No attempt was made to show how or why the forces acted, but gravitation being taken as due to a mere "force", speculators thought themselves at liberty to imagine any number of forces, attractive or repulsive, or alternating, varying as the distance,[4] or the square, cube, or higher power of the distance, etc. At last, Ruđer Bošković[5] got rid of atoms altogether, by supposing them to be the mere centre of forces exerted by a position or point only, where nothing existed but the power of exerting a force.[6] https://en.wikipedia.org/wiki/Imponderable_fluid Mach's antipathy to theorizing and to the invocation of "metaphysical" and therefore unprovable notions led him to some extreme opinions. In The Conservation of Energy he remarks: "We say now that water consists of hydrogen and oxygen, but this hydrogen and oxygen are merely thoughts or names which, at the sight of water, we keep ready to describe phenomena which are not present but which will appear again whenever, as we say, we decompose water. David Lindley, Boltzmann's Atom In Mach's world, there was to be no such thing as "explaining" in the way scientists had always understood it. Mach even went so far as to argue that the traditional notion of cause and effect-that kicking a rock makes it move, that heating a gas makes it expand —was presumptuous and therefore to be denied scientific status. David Lindley, Boltzmann's Atom But it was not always so. Well into the latter half of the 19th century, most scientists saw their essential task as the measurement and codification of phenomena they could investigate directly: the passage of sound waves through air, the expansion of gas when heated, the conversion of heat to motive power in a steam engine. A scientific law was a quantitative relationship between one observable phenomenon and another. David Lindley, Boltzmann's Atom Errata Vaden incorrectly said this that this essay was referenced in Mach's wikipedia page. Wrong! Fool! It was Berkeley's wiki page (https://en.wikipedia.org/wiki/George_Berkeley) # Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) Do you have any fluids you'd like us to ponder? Send a sample over to [email protected]

This week we take a break from our regularly scheduled programming to listen to Ben, Rich, and Cam loutishly pontificate on one of the oldest poems in history. That's right, three fiction noobs take on Homer. Ladies, have you ever wondered what your fella is doing when you're out for the evening? Look no further. The podcast you're listening to is Do You Even Lit? (https://doyouevenlit.podbean.com/) which you can find on any podcast platform and on youtube (https://www.youtube.com/@DoYouEvenLit). The hosts are Richard Meadows (https://thedeepdish.org/blog/), Cam Peters (https://falliblepieces.substack.com/), and some third guy. Back to increments in a couple weeks! In the meantime: - find us on twitter at @BennyChugg, @VadenMasrani, and @Incrementspod - come join our discord channel! Send us a message or an email to get a supersecret link - hit those like buttons on youtube to show off your virtuosity Should we switch out Vaden for Rich and Cam? Tell us at [email protected]. Special Guest: Richard Meadows.

Ben and Vaden test their French skills and have Hugo Mercier on the podcast to discuss who we trust and what we believe. Are humans gullible? Do we fall for propaganda and advertising campaigns? Do we follow expert consensus or forge ahead as independent thinkers? Can Vaden go for one episode without bringing up Trump? Hugo Mercier (https://sites.google.com/site/hugomercier/) is a research director at the CNRS (Institut Jean Nicod, Paris), where he work with the Evolution and Social Cognition team. Check out his two books: The Enigma of Reason (https://www.amazon.com/Enigma-Reason-Hugo-Mercier/dp/0674368304) and Not Born Yesterday (https://www.amazon.com/dp/0691208921) . We discuss Mercier's thoughts on the cognitive bias literature Open vigilance mechanisms Criticism of the System 1 vs System 2 dichotomy Why Kahneman misinterpreted the bat and the ball thought experiment Do flat earthers really believe the earth is flat? The Asch conformity experiment Preference falsification vs internalization of professed beliefs How important is social signaling? Trump, MAGA, gullibility, and Tariffs How effective are advertisements? How effective is propaganda? Is social science reforming? References The Enigma of Reason (https://www.amazon.com/Enigma-Reason-Hugo-Mercier/dp/0674368304) by Hugo Mercier and Dan Sperber Not Born Yesterday (https://www.amazon.com/dp/0691208921) Our previous episodes on Not Born Yesterday (https://www.incrementspodcast.com/84) and The Enigma of Reason (https://www.incrementspodcast.com/39) Socials Follow us on Twitter at @hugoreasoning, @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) How much system 2 thinking does it take to misunderstand system 1 vs system 2? Tell us at [email protected] Special Guest: Hugo Mercier.

Go fund me page for Dr. Alaa Al-Najjar Please donate here: https://www.gofundme.com/f/help-dr-alaa-alnajjar-and-her-family-in-gaza Podcast w/ Dr. Ali Al Najjar https://open.spotify.com/episode/2f6twY40sjd0zwT0Eks3NX Further Reading https://www.thesun.ie/news/15281868/sligo-doctor-ali-al-najjar-nieces-nephew-killed-gaza/ https://www.thejournal.ie/brother-woman-nine-kids-gaza-6717437-May2025/ https://amp.rte.ie/amp/1515399/ https://www.oceanfm.ie/northwest-today-show/fundraiser-launched-after-sligo-doctors-9-nieces-and-nephews-killed-in-gaza-attack-504443 https://www.oceanfm.ie/news/dail-hears-of-devastation-caused-to-sligo-doctors-family-in-gaza-503064 https://m.independent.ie/regionals/sligo/news/sligo-doctors-nine-nephews-and-nieces-killed-in-devastating-gaza-attack/a121298480.html Episode Blurb Back with part two of our reaction to What's the most rational way to know? (https://www.youtube.com/watch?v=vNQlmVJxySc&t=3614s&ab_channel=CordialCuriosity), a discussion between Brett Hall and Peter Boghossian on the relationship between confidence and evidence. Listen to part 1 first, availble here: https://www.incrementspodcast.com/85 It seems weird to try to be funny in this blurb, given the introduction, so going to keep the description lean. Back with our usual lighthearted nonsense next episode! Check out more from Brett Hall here (https://www.bretthall.org/) and Peter Boghossian here (https://peterboghossian.com/). We discuss A falliblist's view of confidence Confidence as a form of implicit knowledge Problems with attempting to define "knowledge" Can we "derive" moral facts? Being triggered by the word 'derive' References Paper discussing how it took the wider scientific community over 40 years (after Eddington's experiment!) to become convinced in the truth of general relativity: The 1919 measurement of the deflection of light (https://arxiv.org/abs/1409.7812) Eddington's original paper (https://w.astro.berkeley.edu/~kalas/labs/documents/dyson1920.pdf): Vaden and Brett's blog exchange (https://vmasrani.github.io/blog/2023/predicting-human-behaviour/) Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) Please donate to: https://www.gofundme.com/f/help-dr-alaa-alnajjar-and-her-family-in-gaza

We all knew that Vaden would release his inner Youtube debate bro at some point. Well he finally paid Ben enough to do it, and here we are: our first reaction video. Today we're commenting on the video What's the most rational way to know? (https://www.youtube.com/watch?v=vNQlmVJxySc&t=3614s&ab_channel=CordialCuriosity), a discussion between Brett Hall and Peter Boghossian on the relationship between confidence and evidence. Are we overly confident in our ability to make reaction videos? Evidently. Check out more from Brett Hall here (https://www.bretthall.org/) and Peter Boghossian here (https://peterboghossian.com/). We discuss What is the relationship between confidence and evidence? The "formal apparatus of science" vs the "sociology" of science Eddington's famous experiment Why confidence and belief can't be mathematized (But why they are useful nonetheless) Confidence as a function of falsifying experiments Bayesianism vs critical rationalism References Paper discussing how it took the wider scientific community over 40 years (after Eddington's experiment!) to become convinced in the truth of general relativity: The 1919 measurement of the deflection of light (https://arxiv.org/abs/1409.7812) Eddington's original paper (https://w.astro.berkeley.edu/~kalas/labs/documents/dyson1920.pdf): Vaden and Brett's blog exchange (https://vmasrani.github.io/blog/2023/predicting-human-behaviour/) Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) Where were you last night, and why do you have condoms in your pocket? Tell us at [email protected].

Some thoughts (arguments?) on Hugo Mercier's Not Born Yesterday (https://www.amazon.com/Not-Born-Yesterday-Science-Believe/dp/0691178704), which advances the thesis that humans are not as gullible as is commonly thought. This is our second episode on Mercier's work, and we're as intrigued as ever. But this time we have different interpretations of his thesis, so it's a good thing the man himself is coming on soon to sort us out. We discuss If humans are less gullible than is commonly believed Evolution of Communication Theory Gazelles jumping in the air Are humans too stubborn? Is one of your hosts who shall go unnamed too stubborn? When do humans actually change their minds? Does Mercier's work conflict with Popper? How much of our reasoning is motivated reasoning? How much is social conformity? Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here (https://www.patreon.com/Increments). Or give us one-time cash donations to help cover our lack of cash donations here (https://ko-fi.com/increments). Click dem like buttons on youtube (https://www.youtube.com/channel/UC_4wZzQyoW4s4ZuE4FY9DQQ) Did you know that "gullible" isn't in the dictionary? Tell us at [email protected].