We’ve seen that people are surprisingly ignorant, more ignorant than they think. We’ve also seen that the world is complex, even more complex than one might have thought. So why aren’t we overwhelmed by this complexity if we’re so ignorant? How can we get around, sound knowledgeable, and take ourselves seriously while understanding only a tiny fraction of what there is to know?
The answer is that we do so by living a lie. We ignore complexity by overestimating how much we know about how things work, by living life in the belief that we know how things work even when we don’t. We tell ourselves that we understand what’s going on, that our opinions are justified by our knowledge, and that our actions are grounded in justified beliefs even though they are not. We tolerate complexity by failing to recognize it. That’s the illusion of understanding.
The myth of individual thought and the power of collective wisdom
by Steven Slomana and Philip Fernbach
What We Know
The Allure of Illusion
The human mind is both brilliant and pathetic. We have mastered fire and have stood on the moon, and yet every one of us is fundamentally ignorant, irrational and prone to making simple mistakes every day. ‘In The Knowledge Illusion, the cognitive scientists Steven Sloman and Philip Fernbach hammer another nail into the coffin of the rational individual . . . positing that not just rationality but the very idea of individual thinking is a myth.’ Yuval Harari, bestselling author of Sapiens and Homo Deus In this groundbreaking book, cognitive scientists Steven Sloman and Philip Fernbach show how our success as a species is down to us living in a rich community of knowledge where we are drawing on information and expertise outside our heads. And we have no idea that we are even doing it. Utilizing cutting-edge research, The Knowledge Illusion explains why we think we know more than we do, why beliefs are so hard to change and why we are so prone to making mistakes. Providing a blueprint for successful ways to work in collaboration to do amazing things, it reveals why the key to human intelligence lies in the way we think and work together.
“So why don’t we realize the depth of our ignorance? Why do we think we understand things deeply, that we have systematic webs of knowledge that make sense of everything, when the reality is so different? Why do we live in an illusion of understanding?
**
Thought is for action. Thinking evolved as an extension of the ability to act effectively; it evolved to make us better at doing what’s necessary to achieve our goals.
**
We will see that humans specialize in reasoning about how the world works, about causality. Predicting the effects of action requires reasoning about how causes produce effects, and figuring out why something happened requires reasoning about which causes are likely to have produced an effect. This is what the mind is designed to do.
**
This makes human ignorance all the more surprising. If causality is so critical to selecting the best actions, why do individuals have so little detailed knowledge about how the world works? It’s because thought is masterful at extracting only what it needs and filtering out everything else. When you encounter a complicated causal system, you similarly extract the gist and forget the details.
**
The mind is not built to acquire details about every individual object or situation. We learn from experience so that we can generalize to new objects and situations. The ability to act in a new context requires understanding only the deep regularities in the way the world works, not the superficial details.
**
Our minds need to be designed to treat information that resides in the external environment as continuous with the information that resides in our brains. Human beings sometimes underestimate how much they don’t know, but we do remarkably well overall. That we do is one of evolution’s greatest achievements.
**
You now have the background you need to understand the origin of the knowledge illusion. The nature of thought is to seamlessly draw on knowledge wherever it can be found, inside and outside of our own heads. We live under the knowledge illusion because we fail to draw an accurate line between what is inside and outside our heads. And we fail because there is no sharp line. So we frequently don’t know what we don’t know.
**
Appreciating the communal nature of knowledge can reveal biases in how we see the world. People love hero. We glorify individual strength, talent, and good looks. Our move and books idolize characters who, like Superman, can save the planet all by themselves. TV dramas present brilliant but understated detectives who both solve the crime and make the climactic final arrest after a flash of insight. Individuals are given credit for major breakthroughs. Marie Curie is treated as if she worked alone to discover radioactivity, Newton as if he discovered the laws of motion in a bubble. All the successes of the Mongols in the twelfth and thirteenth century are attributed to Genghis Khan, and all the evils of Rome during the time of Jesus are often identified with a single person, Pontius Pilate. The truth is that in the real world, nobody operates in a vacuum.
**
Landauer was a pioneer of cognitive science, holding academic appointments at Harvard, Dartmouth, Stanford, and Princeton and also spending twenty-five years trying to apply his insights at Bell Labs. He started his career in the 1960s, a time when cognitive scientists took seriously the idea that the mind is a kind of computer. Cognitive science emerged as a field in sync with the modern computer. As great mathematical minds like John von Neumann and Alan Turing developed the foundations of computing as we know it, the question arose whether the human mind works in the same way. Computers have an operating system that is run by a central processor that reads and writes to a digital memory using a small set of rules. Early cognitive scientists ran with the idea that the mind does too. The computer served as a metaphor that governed how the business of cognitive science was done. Thinking was assumed to be a kind of computer program that runs in people’s brains. One of Alan Turing’s claims to fame is that he took this idea to its logical extreme. If people work like computers, then it should be possible to program a computer to do what a human being can. Motivated by this idea, his classic paper “Computing Machinery and intelligence” in 1950 addressed the question Can machines think?
**
But this is only shocking if you believe the human mind works like a computer. The model of the mind as a machine designed to encode and retain memories breaks down when you consider the complexity of the world we interact with. It would be futile for memory to be designed to hold tons of information because there’s just too much out there.
**
Cognitive scientists don’t take the computer metaphor so seriously anymore. There is a place for it; some models of how people think when they’re thinking slowly and carefully – when they are deliberating step-by-step as opposed to being intuitive and less careful – look like computer programs. But for the most part these days, cognitive scientists point to how we differ from computers. Deliberation is only a tiny part of what goes on when we think. Most of cognition consists of intuitive thought that occurs below the surface of consciousness. It involves processing huge quantities of information in parallel. When people search for a word, for example, we don’t consider one word at a time sequentially. Instead, we search our entire lexicon – our mental dictionary – simultaneously, and the word we’re looking for usually rises to the top. That’s not the kind of computation that von Neumann and Turing had in mind in the early days of computer science and cognitive science.
**
There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.
**
Stephen Jay Gould explained how chaos introduces complexity to the study of history: “little quirks at the outset, occurring for no particular reason, unleash cascades of consequences that make a particular future seem inevitable in retrospect. But the slightest early nudge contacts a different groove, and history veers into another plausible channel, diverging continually from its original pathway. The end results are so different, the initial perturbation so apparently trivial.” Gould’s observation that events seem inevitable in retrospect is a deep insight about human ignorance. We just don’t appreciate what it takes to make things happen.
**
The answer is that we do so by living a lie. We ignore complexity by overestimating how much we know about how things work, by living life in the belief that we know how things work even when we don’t. We tell ourselves that we understand what’s going on, that our opinions are justified by our knowledge, and that our actions are grounded in justified beliefs even though they are not. We tolerate complexity by failing to recognize it. That’s the illusion of understanding.
**
Eventually we’ll address a deeper question. Rather than asking how we tolerate complexity, we’ll ask how we manage it. How can humanity achieve so much when people are so ignorant? It turns out we have been very successful at dividing up our cognitive labor. But to understand how we share our knowledge without communities, we must first understand how we think as individuals.
**
Garcia’s study suggests that rats are predisposed to learn causally meaningful relations, not arbitrary links. Even rats engage in a kind of simple causal reasoning, figuring out the likely causes of their distress.
**
If rats are causal thinkers and don’t rely only on simple associations, the same is presumably true of dogs. Pavlovian associations don’t occur between arbitrary pairs of stimuli, they happen only the association has some possibility of making causal sense. So we apologize for defaming Cassie’s cognitive abilities. We have great respect for dogs and their ability to think causally.
**
What we do excel at is reasoning about how the world works. We’re gifted causal reasoners, and rats, as it happens, are too. What could be more useful if you’re an animal who has evolved to operate in the world?
**
We saw that the purpose of thinking is to choose the most effective action given the current situation. That requires discerning the deep properties that are constant across situations. What sets humans apart is our skill at figuring out what those deep, invariant properties are. It takes human genius to identify the key properties that indicate if someone has suffered a concussion or has a communicable disease, or that it’s time to pump up a car’s tires.
**
We may be better at causal reasoning than other kinds of reasoning, but the illusion of explanatory depth shows that we are still quite limited as individuals in how much of it we can do.
**
Just as people don’t think only associatively (as Pavlov thought we do), people do not reason via logical deduction. We reason by causal analysis. People make inferences by reasoning about the way the world works. We think about how causes produce effects, what kinds of things disable or prevent effects, and what factors must be in place for causes to have their influence. Rather than thinking in terms of propositional logic, the logic that tells us whether a statement is true or false, people think in terms of causal logic, the logic of causation that incorporates knowledge about how events actually come about in order to reach conclusions.
**
Taking the time to learn a useful technique or art makes sense only if you can see far enough into the future by reasoning about the causal mechanisms that govern social change, like death.
**
We excel at causal analysis not just when dealing with physical objects and social change, but also when confronted by problems in the psychological realm. Imagine that someone – let’s say your spouse – refuses to talk to you. Now you have a problem to solve. You need to engage in causal reasoning to identify the problem and to figure out what to do about it.
**
Backward reasoning also takes longer than forward reasoning. Backward reasoning from effect to cause may be hard, but it’s also what makes humans special; it’s not clear that any other organism has the capacity or interest to figure out the causes of what has happened.
**
Even though we’re not great at diagnostic reasoning, our ability to do it may be what makes us human. There’s hardly any evidence that any other animal can do it. Animals may be able to respond to their environments in very sophisticated ways, and we saw earlier that rats are sensitive to causal considerations, but no animals have been shown to exhibit diagnostic reasoning from effect to cause.
**
Perhaps the most common way that people pass causal information to one another is by storytelling. Consider the old Yiddish story about the shopkeeper who arrive at his shop only to find abusive and derogatory graffiti spray-painted all over his store window. He cleaned the window, but the same thing happened again the next day. So he hatched a plan: On the third day, he waited until the local ruffians showed up and did their dirty work and then paid them $10 to thank them for their effort. The next day, he thanked them again but only paid them $5. He continued to pay them to deface his property but the amount kept decreasing so that soon they were only getting $1. They stopped coming. Why both doing all that work se the shopkeeper for so little money?
**
This apocryphal tale is really a cause a lesson. It’s about what causes people to act and how you can modify their motivations, to make them think they’re doing something for a different reason than they initially thought.
**
Fairy tales and urban legends tend to teach us about what we should avoid, what’s dangerous, and how we determine whom to trust. Stories about heroic acts tell us about the surprising extent of our own potential.
**
Storytelling is our natural way of making causal sense of sequences of events. That’s why we find stories everywhere. In one of the classic demonstrations in social psychology from the 1940s, Fritz Heider and Marianne Simmel showed people a simple animated film starring a circle and two triangles moving around a screen. That’s it: no sound, no text. Sometimes two of the geometric figures would get close to each other; sometimes one would appear to chase another; sometimes they would appear to fight. People inevitably saw more than circles and triangles; they saw a romantic drama play out. People see stories everywhere.