Jamie Hale

Jamie Hale

Monday, April 26, 2021

Nonsense Detection Kit 2.0

The Nonsense Detection Kit 2.0 is a revision of The Nonsense Detection Kit. The contents of the full kit can be found in In Evidence We Trust 2nd Edition.

The impetus for writing the Nonsense Detection Kit was previous suggestions made by Sagan (1996), Lilienfeld et al. (2012) and Shermer (2001). The Nonsense Detection Kit is referring to nonsense in terms of “scientific nonsense”. So, nonsense as it is referred to here refer to “nonscientific information” that is often perpetuated as scientific, when in fact it is not scientific.

The Nonsense Detection Kit provides guidelines that can be used to separate sense from nonsense. There is no single criterion for distinguishing sense from nonsense, but it is possible to identify indicators, or warning signs. The more warnings signs that appear the more likely that claims are nonsense.

Below is a brief description of indicators that should be useful when separating sense from nonsense. These indicators should be useful when evaluating claims made by the media, on the Internet, in peer-reviewed publications, in lectures, by friends, or in everyday conversations with colleagues.

Nonsense indicator- claims haven’t been verified by an independent source

Nonsense perpetuators often claim special knowledge. That is, they have made specific discoveries that only they know about. Others lack know how, or do not have the proper equipment to make the finding. These findings are often reflected in phrases such as, “revolutionary breakthrough”, “what scientists don’t want you to know”, “what only a limited few have discovered”, and so on. These findings are not subject to criticism or replication. That is not how science works. When conducting studies it is imperative that researchers operationalize (provide operational definition- precise observable operation used to manipulate or measure a variable) variables so the specifics can be criticized and replicated. Non-scientists are not concerned with others being able to replicate their findings; because they know attempted replications will probably be unsuccessful. If a finding cannot be replicated this is a big problem, and it is unreasonable to consider a single finding as evidence. It is also problematic when only those making the original finding have replicated successfully. When independent researchers using the same methods as those used in the original study are not able to replicate this is a sign that something was faulty with the original research.

Nonsense indicator- claimant has only searched for confirmatory evidence

The confirmation bias is a cognitive error (cognitive bias) defined as tendency to seek out confirmatory evidence while rejecting or ignoring non-confirming evidence (Gilovich, 1991). Confirmation bias is pervasive, and may be the most common cognitive bias. Most people have a tendency to look for supporting evidence, while ignoring or not looking very hard for disconfirmatory evidence (showing a dislike for disconfirmatory evidence). This is displayed when people cherry pick the evidence. Of course, when you’re a lawyer this is what you need to do. You don’t want any evidence entering into the case that may be incongruent with the evidence you present. However, as a scientist it is important to look for disconfirming evidence. In fact, it has been suggested that a good scientist goes out of their way to look for disconfirmatory evidence. Why look for disconfirmatory evidence? Because when discovering reality is the objective it is necessary to look at all the available data, not just the data supporting one’s own assertions. Confirmation bias occurs when the only good evidence, according to the claimant, is the evidence that supports their claim. Often, perpetuators of nonsense may not even be aware of disconfirmatory evidence. They have no interest in even looking at it.

A study by Frey & Stahlberg (1986) examined how people cherry-pick the evidence. The participants took an IQ test and were given feedback indicating their IQ was either high or low. After receiving feedback participants had a chance to read magazine articles about IQ tests. The participants that were told they had low IQ scores spent more time looking at articles that criticized the validity of IQ tests, but those who were told they had high IQ scores spent more time looking at articles that supported the claim that IQ tests were valid measures of intelligence.

Scientific thinking should involve an effort to minimize confirmation bias. However, science does involve confirmation bias to a degree; this if often demonstrated in publication bias and forms of myside bias. The late Richard Feynman (Nobel Laureate, Physics) suggested that science is a set of processes that detects self-deception (Feynman, 1999). That is, science makes sure we don’t fool ourselves.


Wednesday, December 30, 2020

Critical Thinking is Rational

The sub title of In Evidence We Trust is The need for science, rationality and statistics. The subtitle could have been The need for critical thinking. Critical thinking / rational thinking includes scientific thinking, and statistical / probabilistic thinking is a component of scientific thinking. Critical thinking as conceptualized by cognitive scientists has been examined in an array of contexts, and the work of various cognitive scientists led the Stanovich Research Lab to develop the first comprehensive measure of rationality: Comprehensive Assessment of Rationality (CART- 2016).

Rationality consists of two broad categories- instrumental and epistemic rationality. Rational thinking skills are important. They are as important as intelligence. Intelligence and rationality are often dissociated. Research demonstrates that intelligence is often a weak predictor of rationality. This has been shown over a wide range of studies. Intelligence is important, but there is more to good thinking than intelligence. Intelligence reflects reasoning abilities across a wide variety of domains, particularly novel ones. In addition, intelligence reflects general declarative knowledge acquired through acculturated learning. Rationality reflects appropriate goal setting, goal optimization, and holding evidence-based beliefs.

Myths About Critical Thinking

Critical thinking (rational thinking) is good thinking; it involves forming appropriate goals, goal optimization and forming evidence based beliefs. Two common myths associated with critical thinking are emotion prevents critical thinking and critical thinking is synonymous with formal logic. Full article  

Critical Thinking in Modern Society

Educators often pay lip service to the idea of teaching “critical thinking.” But, when asked to define critical thinking, answers are often weak and ambiguous. Common responses to the defining critical thinking include: “teaching them how to think,” “teaching them formal logic,” “teaching them to be thinkers,” “teaching them how to think for themselves,” or “teaching them how to solve problems.” They already know how to think; logic is only a portion of what is needed to increase critical thinking, independent thinking doesn’t necessarily imply critical thinking and teaching them how to solve problems are hard to measure assertions. Full article 

Chapter 2 from In Evidence We Trust features short articles on critical thinking. Some of the articles focus on the rationality intelligence dichotomy. Also included in this chapter are interviews with Keith Stanovich and the Stanovich Research Lab (Keith Stanovich, Richard West and Maggie Toplak). In the interviews with Stanovich, he discusses the development of an RQ Test. In the interview with the Stanovich lab, rationality and intelligence are discussed.

Purchase In Evidence We Trust 2nd Edition (Hale 2019)

Wednesday, November 25, 2020

Replication Studies: Another Perspective

 Replicable (reproducible) findings are important to science; they are a sub-component of converging evidence. When referring to the replication crisis it is important to understand that what is meant- is lack of replicating statistically significant findings. It would be more precise to say there is a "statistically significant replication crisis." Consider replication from another perspective; the original study failed to detect stat...sign.. (using criteria NHST prevalent with use of frequentist stats), but additional studies detect statistical significance.  What would the implications be??  College instructors should make an effort to address this condition- non-significant precedes significant findings. Students are often advised no need to try to replicate non-significant findings, but sign..findings should be replicated. This implies that the non-sign....findings must be accurate (if they occurred first), even though all studies are susceptible to flaws.   

Gelman asserts the time-reversal heuristic needs consideration: "One helpful (I think) way to think about this episode is to turn things around. Suppose the Ranehill et al. experiment, with its null finding, had come first. A large study finding no effect. And then Cuddy et al. had run a replication under slightly different conditions with a much smaller sample size and found statistically significance under non-preregistered conditions. Would we be inclined to believe it?"    

What are some different types of replication studies?

There are least 3 general types of replication studies- direct replication, conceptual replication and replication-plus-extension. In direct replication, researchers attempt to conduct research using methods that are as close as they can to those used by original researchers. The more transparent the original research the easier it will generally be to directly replicate....

Replication is an important part of science. Non-sign..and sign...studies may be flawed. An array of variables determine the value of the findings- publication source, funding sources, study replication, study design, sample size, conflicting interest, sampling error, different measures of reliability and validity, reporting limitations, and other possible criticisms of the study. 

The top-tier of scientific evidence is converging evidence.   

Thursday, September 10, 2020

So Many Brain Myths

Discussions on the brain are ubiquitous. Magazines, books and T.V. are saturated with  information related to the brain. Lots, if not most of it is wrong.

How Many Neurons Are in the Human Brain?

When I was an undergraduate in graduate school, I learned the human brain consists of 100 billion neurons (Kolb and Whishaw 2009). This number was reported in scholarly journals, textbooks, and in college lectures. It was accepted as fact. I never saw a citation of an original source to support the claim, nor did I ever hear anyone question whether or not there was evidence to support it. I just assumed it was common knowledge and must be supported by a large body of data. Even the general public knew that the human brain consists of 100 billion neurons. In addition to academia’s dissemination of the supposed fact, popular media embraced and promoted the 100 billion neuron idea...


The Allure of Brain Science

The media have become fascinated with brain images—and the use of those images to explain almost everything. Neuroscience (the scientific study of the nervous system, in many cases focusing only on the brain) has made a mark in mainstream media and everyday conversation. You have probably seen headlines such as “This Is Your Brain on Sugar,” “The Brain’s Evil Spot,” or “Brain Based Learning.” These phrases, and the stories associated with them, generally hold some truth but at the same time are misrepresented and often fuel false beliefs and misconceptions. As an example, consider the implications of the so-called “Sugar Brain.” Proponents claim that consumption of sugar can activate the same brain reward mechanisms (dopamine pathway referred to as mesolimbic dopamine system) as those activated when consuming addictive drugs. Some of the same brain areas are activated (varying in strength and intensity) when consuming sugar and drugs, but other stimuli also activate the mesolimbic dopamine system. The mesolimbic dopamine system is rich in dopaminergic neurons. Dopamine cell bodies (parts of brain cells where dopamine is synthesized) are located in the brainstem...


The Truth About Nootropic Supplements

Nootropic substances—from the Greek words meaning “mind-bending” –are ingestible chemicals often promoted for their cognitive enhancing properties (Jasanoff 2018). According to companies selling nootropic products, benefits of using the products include prevention of cognitive decline, enhanced memory, increased learning, improved concentration, and rapid cognition. Nootropic drugs include stimulants like amphetamine and methylphenidate, marketed under the names Adderall and Ritalin, as well as sleep suppressants like Modafinil. Nootropics also include a range of dietary supplements...





Monday, August 17, 2020

Replication Studies in Science

Ideally, scientific research should be replicable (reproducible). The research should use processes that can be used by others wanting to conduct a similar or the same study. When referring to the replication crisis it is often understood that what is meant is lack of replicating statistically significant findings. It would be more precise to say there is a "statistically significant replication crisis." It is possible that original studies that fail to show significance may demonstrate a type 2 error- missing an effect. This could occur do to a number of methodological or statistical issues. As an example, when I conducted a study on expectations influence on food liking the finding was insignificant; when I ran a statistical power analysis it revealed I needed a larger sample, considering effect size and p-value to find significance. Statistically significant and insignificant finding should be replicated, and they should involve different type of replications using samples with varying characteristics.

What are some different types of replication studies?

There are least 3 general types of replication studies- direct replication, conceptual replication and replication-plus-extension. In direct replication, researchers attempt to conduct research using methods that are as close as they can to those used by original researchers. The more transparent the original research the easier it will generally be to directly replicate. In conceptual replication researchers address same topics, questions, but use different methods. Variables are manipulated and measured using different strategies, but conceptualization remains intact. In a replication-plus-extension study, researchers replicate original studies, but also add variables, that may include different operationalizations.

What are the implications of replication studies?

Extra weight is often given to studies that are replicated (also find significance) outside of the original lab, or when conducted by researchers other than the ones making the original findings. A red flag is indicated if only a specific group or lab is able to make a finding. Why is it others can't make the finding? It is essential that researchers are transparent with their methods and all relevant research materials. Strong evidence is the result of various studies; not a single study, or series of studies that can only be found by one research group. To reiterate, scientific progress is cumulative; it develops as a product of the work, of sometimes many people. In some cases it is necessary to repeat studies that didn't find significance. The original study might be flawed. The Apex of evidence is converging evidence. Various research methods, stats, models and inferential strategies have limitations- it is the preponderance of evidence from various lines of inquiry that converge to produce the highest level of evidence.

For further discussion on issues with scientific methods refer to -   In Evidence We Trust 2nd Edition  

Various articles on replication from Andrew Gelman's site