Despite our certainty about the potential of individual candidates, our forecasts were largely useless. The evidence was overwhelming. Every few months we had a feedback session in which we could compare our evaluations of future cadets with the judgments of their commanders at the officer-training school. The story was always the same: our ability to predict performance at the school was negligible. Our forecasts were better than blind guesses, but not by much.
We were downcast for a while after receiving the discouraging news. But this was the army. Useful or not, there was a routine to be followed, and there were orders to be obeyed. Another batch of candidates would arrive the next day. We took them to the obstacle field, we faced them with the wall, they lifted the log and within a few minutes we saw their true natures revealed, as clearly as ever. The dismal truth about the quality of our predictions had no effect whatsoever on how we evaluated new candidates and very little effect on the confidence we had in our judgments and predictions.
I thought that what was happening to us was remarkable. The statistical evidence of our failure should have shaken our confidence in our judgments of particular candidates, but it did not. It should also have caused us to moderate our predictions, but it did not. We knew as a general fact that our predictions were little better than random guesses, but we continued to feel and act as if each particular prediction was valid. I was reminded of visual illusions, which remain compelling even when you know that what you see is false. I was so struck by the analogy that I coined a term for our experience: the illusion of validity.
I had discovered my first cognitive fallacy.
Decades later, I can see many of the central themes of my thinking about judgment in that old experience. One of these themes is that people who face a difficult question often answer an easier one instead, without realizing it. We were required to predict a soldier's performance in officer training and in combat, but we did so by evaluating his behavior over one hour in an artificial situation. This was a perfect instance of a general rule that I call WYSIATI, "What you see is all there is." We had made up a story from the little we knew but had no way to allow for what we did not know about the individual's future, which was almost everything that would actually matter. When you know as little as we did, you should not make extreme predictions like "He will be a star." The stars we saw on the obstacle field were most likely accidental flickers, in which a coincidence of random events - like who was near the wall - largely determined who became a leader. Other events - some of them also random - would determine later success in training and combat.
I see this every day: "We had made up a story from the little we knew but had not way to allow for what we did not know." And leaders are the best creators of stories out of this limited information - whether or not they are true. They are also powerful and persuasive. This is how delusion expands from individuals, to groups, to entire corporations.