Over the course of four previous articles in this series, we’ve been fine-tuning our ability to notice when something is amiss with the information that’s being presented to us, whether it’s coming from unverified sources on the internet, from friends or coworkers, or from supposed experts through the most trusted channels. Basically, we’ve been building a bullshit detector, along with a whole protocol for what to do when the alarm sounds.
This is worthy work. It’s the core of the critical thought toolset. But there’s a catch. Yes, outright lies and misunderstandings will often come dressed in the clothes of intuitively plausible truths. But, every now and then, the opposite happens. Sometimes we’ll be presented with something that’s so outlandish on the surface that we’re inclined to dismiss it immediately. And yet, buried within, might be a verifiable truth that could reshape our entire city of ideas. What tools do we have when our bullshit detector raises a false alarm?
The world is full of unlikely truths, and this dissonance is one we’ve all dealt with at some point, so we’re not operating entirely without a map. Remember the first time you heard of the placebo effect? Chances are it was hard to believe a sugar pill or an empty gel cap could treat everything from the common cold to insomnia to cancer[1]. Your bullshit detector was undoubtedly going off at full volume.
Probably most of us went through a phase where we imagined the placebo effect was just a matter of a fake treatment curing a fake disease. We believed that we ourselves were personally immune because we were rational human beings with a strong grip on reality. But then we investigated further, and the more we learned about the placebo effect, the weirder it got.
Though some people respond more strongly and reliably to placebos than others, virtually everyone responds to some degree. Some people even get better just from being told they’ve been put on a waiting list for treatment, and some placebos keep working even when people are explicitly told they’re placebos. Weird, right? Placebos can also make people get sicker, especially if they are anticipating side effects[2] . In fact, the placebo effect is so complex and prevalent that controlling for it has radically defined the shape of basically every medical study. And, for some conditions, the effect is so strong that finding a treatment that performs “better than placebo” has become an almost impossibly high bar to clear. This is not just a footnote of science; it’s a startling reminder that our intuitions about cause and effect can betray us completely.
What’s worse, our intuition can be deceived just as badly even when the correct conclusion follows from simple arithmetic. Sometimes the numbers are right in front of us, but our gut adds them up all wrong.
Imagine a deadly and incurable disease so rare that it only affects one person in ten thousand[3]. Now imagine that an affordable test has been developed to diagnose this disease. This test is 99% accurate in both directions, meaning it has a 1% false negative rate and a 1% false positive rate[4]. You’re offered the test as part of a routine blood panel, and it comes back positive. What’s your gut telling you? You’re doomed, right?
Well, no. You’re probably fine. Since only one in 10,000 people have the disease, if we tested a million people, only a hundred would actually have it. And, because there is a 1% false negative rate, ninety-nine of those hundred would get a positive result, correctly identifying the disease (one unlucky person would get a false negative). But, simultaneously, among the 999,900 people who don’t have the disease, 1% (9,999 people) would get a false positive. So of all the people who get a positive result, only a little under 1% will actually have the disease.
The numbers are right there, and when you do the math, it’s irrefutable. But our bullshit detector sees “the test is 99% accurate” and “less than 1% of those who test positive have the disease,” and it freaks out. Surely both can’t be true at once. Our intuition won’t allow it. But both are true. The trick is that the new evidence (a positive result with 99% accuracy) can only be properly evaluated in the context of the base likelihood of the phenomenon the evidence points towards (just 0.01% of people get this disease). In other words, a positive test result doesn’t mean there’s a 99% probability we have the disease, but it does mean the probability of us having the disease is about a hundred times higher than it was before we got tested. It’s a textbook illustration of Bayes’ theorem and the base rate fallacy[5].
The implications of this fallacy, and others like it[6], aren’t confined to textbooks and hypotheticals though. Exactly this sort of mistake is behind all kinds of real-world misunderstandings and problems, from wildly inaccurate beliefs about COVID prevalence early in the pandemic to people losing their jobs as a result of false positives on drug screening tests and people ignoring fire alarms because tests and malfunctions are more common than actual fires[7].
It’s a real catch-22. We need to be skeptical, or we are not thinking critically at all. But the very same tools that help us identify malicious lies, innocent misunderstandings, and rampant hogwash can also lead us to dismiss important truths even when we’re looking straight at them. That’s not better.
But there is a way forward. We can train and refine our intuition. And we can pay attention to it without implicitly trusting it. A gut feeling that not everything adds up is an important clue that we should take a closer look at the information before us, no matter how trusted the source. But we need to actually take that closer look. Sometimes it will turn out that we were just adding up the wrong things.
The bullshit detector we’ve worked so hard at developing is not a source of facts. It’s a warning system. And, just like the test for our hypothetical disease, it can throw false positives. Sometimes the buildings that look the most unstable in our city of ideas are in fact the most incredible marvels of architecture and engineering. But, if we’re diligent, we can become skilled at recognizing the patterns that make false positives more likely. We can learn about Bayes’ theorem and the base rate fallacy. We can learn about survivorship bias. We can learn about regression to the mean. And each time we internalize one of these concepts, our bullshit detector will level up.
But when that alarm goes off, no matter how fine-tuned our instincts have become, we still need to do the work. Our intuition is an excellent scout, but a terrible general. It can tell us when it’s time to arm up with the weapons of critical thought, but it can’t win the battle for us.
[1] Placebo treatments have been shown to be quite effective for controlling cancer-related nausea and pain, and some studies have even shown a small degree of tumour shrinkage, although that may be regression to the mean.
[2] This is called the “nocebo effect.”
[3] This isn’t even really that rare in the greater scheme of rare diseases.
[4] Medical tests frequently have different accuracy rates when it comes to false positives (specificity) and false negatives (sensitivity).
[5] The base rate fallacy occurs when you fail to consider how rare a phenomenon is known to be when you analyze evidence suggesting that phenomenon. Bayes’ theorem is the statistical rule that provides the correct framework for avoiding this fallacy.
[6] For another striking example of a mathematical pitfall that commonly triggers the bullshit detector, take a look at Simpson’s Paradox, in which something that is true of every individual subset of a population can be demonstrably false when considering the population as a whole.
[7] This last example is actually a case of people learning to correctly adjust for the base rate, but fail to adjust for the relative consequences of being wrong.