The Nonsense Detection Kit 2.0 is a revision of The Nonsense Detection
Kit. The contents of the full kit can be found in In Evidence We Trust 2nd Edition.
The impetus
for writing the Nonsense Detection Kit was previous suggestions made by Sagan
(1996), Lilienfeld et al. (2012) and Shermer (2001). The Nonsense Detection Kit
is referring to nonsense in terms of “scientific nonsense”. So, nonsense as it
is referred to here refer to “nonscientific information” that is often
perpetuated as scientific, when in fact it is not scientific.
The Nonsense
Detection Kit provides guidelines that can be used to separate sense from
nonsense. There is no single criterion for distinguishing sense from nonsense,
but it is possible to identify indicators, or warning signs. The more warnings
signs that appear the more likely that claims are nonsense.
Below is a
brief description of indicators that should be useful when separating sense
from nonsense. These indicators should be useful when evaluating claims made by
the media, on the Internet, in peer-reviewed publications, in lectures, by
friends, or in everyday conversations with colleagues.
Nonsense indicator-
claims haven’t been verified by an independent source
Nonsense perpetuators often claim special
knowledge. That is, they have made specific discoveries that only they know
about. Others lack know how, or do not have the proper equipment to make the
finding. These findings are often reflected in phrases such as, “revolutionary
breakthrough”, “what scientists don’t want you to know”, “what only a limited
few have discovered”, and so on. These findings are not subject to criticism or
replication. That is not how science works. When conducting studies it is
imperative that researchers operationalize (provide operational
definition- precise observable operation used to manipulate or measure a
variable) variables so the specifics can be criticized and replicated.
Non-scientists are not concerned with others being able to replicate their
findings; because they know attempted replications will probably be
unsuccessful. If a finding cannot be replicated this is a big problem, and it
is unreasonable to consider a single finding as evidence. It is also
problematic when only those making the original finding have replicated
successfully. When independent researchers using the same methods as those used
in the original study are not able to replicate this is a sign that something
was faulty with the original research.
Nonsense
indicator- claimant has only searched for confirmatory evidence
The
confirmation bias is a cognitive error (cognitive bias) defined as tendency to
seek out confirmatory evidence while rejecting or ignoring non-confirming
evidence (Gilovich, 1991). Confirmation bias is pervasive, and may be the most
common cognitive bias. Most people have a tendency to look for supporting
evidence, while ignoring or not looking very hard for disconfirmatory evidence
(showing a dislike for disconfirmatory evidence). This is displayed when people
cherry pick the evidence. Of course, when you’re a lawyer this is what you need
to do. You don’t want any evidence entering into the case that may be incongruent
with the evidence you present. However, as a scientist it is important to look
for disconfirming evidence. In fact, it has been suggested that a good
scientist goes out of their way to look for disconfirmatory evidence. Why look
for disconfirmatory evidence? Because when discovering reality is the objective
it is necessary to look at all the available data, not just the data supporting
one’s own assertions. Confirmation bias occurs when the only good evidence,
according to the claimant, is the evidence that supports their claim. Often,
perpetuators of nonsense may not even be aware of disconfirmatory evidence.
They have no interest in even looking at it.
A study by
Frey & Stahlberg (1986) examined how people cherry-pick the evidence. The
participants took an IQ test and were given feedback indicating their IQ was
either high or low. After receiving feedback participants had a chance to read
magazine articles about IQ tests. The participants that were told they had low
IQ scores spent more time looking at articles that criticized the validity of
IQ tests, but those who were told they had high IQ scores spent more time
looking at articles that supported the claim that IQ tests were valid measures
of intelligence.
Scientific thinking should involve an effort to
minimize confirmation bias. However, science does involve confirmation bias to
a degree; this if often demonstrated in publication bias and forms of myside
bias. The late Richard Feynman (Nobel Laureate, Physics) suggested that science
is a set of processes that detects self-deception (Feynman, 1999). That is,
science makes sure we don’t fool ourselves.