Tools for Evaluating Information

Farnam Street has a very good article explaining some of the techniques used by Nobel Laurette Richard Feynman for evaluating information. The topic is relevant as ever.

A large part of wisdom is knowing what to ignore. A large part of expertise is knowing where to place your attention.

Feynman's recommendations are distilled in 7 points:

1. Decide whether the other person truly knows their stuff or is mimicking others. When asked a direct, honest, intelligent question, and the person doesn't know the answer, does he admit that he doesn't know the answer? If he doesn't know it, can he explain how he thinks he could find it?

2. Learn to deal with uncertainty. Few ideas in life are actually either true or false. What you want is to get as close to the truth as you can with the information available. Put things on a gradient from "probably true" to "probably false" ("Grey Thinking").

3. Knowledge is not static. We need to be open to continually evaluating what we think we know. As we investigate whether something is true or not, new evidence and new methods of experimentation should show the effect of getting stronger and stronger, not weaker.

4. Learn to ask the right question. Don't ask Could this be the case?, but Is this actually the case? For example, don't ask if you can disprove that this could be a flying saucer. Instead, make a judgment about whether it is a flying saucer.

5. Do not use the same data that gave you the clue to make the conclusion. Don't judge the probability of something happening after it has happened.

6.The plural of anecdote is not data. Use proper statistical sampling to determine if we know what we are talking about.

7. Many errors come from a lack of information. Realize that we are missing information that we don't know we are missing.