Jamie Hale

Jamie Hale

Wednesday, May 24, 2017

Don't Forget About The Stats: Quantitative Research

In the context of everyday language statistics (numbers, quantitative representations) are used to represent basketball player's free throw average, death rates, life spans and so on. In science statistics are tools used in describing, organizing, summarizing and analyzing data. Learning about stats will help you think in terms of probabilities, and allow you to gain a better understanding of research data. Statistics are not easy, but with some effort the basics can be learned by most people. Research methods and statistics are often taught together in college courses. Quantitative research uses stats, and these stats are essential in an effort to represent the data; stats are required for making sense of the research.          

Descriptive statistics are numerical measures that describe a population by providing information on the central tendency of the distribution, the width of distribution (dispersion, or variability), the shape of distribution (Jackson, 2009). Inferential statistics are procedures that allow us to make an inference from a sample to the population. That is, we are able to make generalizations about a population based on the information derived from the sample.

A key reason we need statistics is to be able to effectively interpret research. Without statistics it would be very difficult to analyze the collected data and make decisions based on the data. Statistics give us an overview of the data and allow us to make sense of what is going on. Without statistics, in many cases, it would be extremely difficult to find meaning in the data. Statistics provides us with a tool to make an educated inference.

Most scientific and technical journals contain some form of statistics. Without an understanding of statistics, the statistical information contained in the journal will be meaningless. An understanding of basic statistics will provide you with the fundamental skills necessary to read and evaluate most results sections. The ability to extract meaning from journal articles, and the ability to evaluate research from a statistical perspective are basic skills that will increase your knowledge and understanding of the article of interest. To reiterate, quantitative research uses stats, and to assess statistical validity, at least a basic understanding of stats is essential.

When researchers question a study’s statistical validity they are questioning issues relevant to how well the conclusions coincide with the results, represented as statistics. Interrogating statistical validity may include some of the following questions: If the study found a difference what is the probability that the conclusion was a false alarm?  If the study’s finding found no difference what is the probability that a real relationship went unnoticed?  What is the effect size?  Is the difference between groups statistically significant? Are the finding practically significant? What type of inferential stats were used to assess predictions? Could different statistical procedures have been used?

Gaining knowledge in the area of statistics will help you become a better-informed consumer. Statistics are difficult for many people. Students often cringe when they hear the word - statistics. Learning about statistics requires the same strategies as learning about other topics (strategies to improve learning and memory). Once an individual learns theoretical aspects and calculations used for basic statistical procedures the learning of more complex statistics become much easier. Everyone benefits from learning the basics of statistics. Statistics is not an easy subject compared to many other subjects, but the subject is much easier when one doesn't have negative expectations and realizes that with the appropriate cognitive effort and understanding of some rather basic mathematical principles the subject is learnable.  Being knowledgeable in the area of statistics will be beneficial across domains of scholarly and everyday life. 

Recently I asked Dr. Jonathan Gore (from Eastern Kentucky University) the following question- Why is a basic understanding of stats important for the public? He gave the following answer:
"My answer to why stats is important is that pretty much everything operates based on probability. Even some of the "hard" sciences are starting to realize that phenomena that used to only require a basic equation are now having to factor in probability to account for all that they observe."

If the objective is to thoroughly analyze the study, don't skip over the "Results" section when reading the paper. A key guideline for the Results section is a presentation of numerical findings that should be stated clearly, concisely and accurately. The methodology provides detailed information regarding processes used in the collection of data, while statistical procedures provide information on detecting meaningful signals among the noise: making sense of the data collected.    

 The book contains 76 questions and answers regarding scientific research methods and stats. It also contains practice problems involving statistical procedures. 
References are available upon request

Saturday, January 14, 2017

Better Study Strategies

Studying should be cognitively challenging, persistent and structured. Strong memory is not built easily or overnight. This article consists of links to articles relevant to learning / memory and key points from my seminar- Strategies To Maximize Learning.

How To Study

The effort required to form strong memory is often intense for students. Students often spend hours trying to master new information. Of course, methods to enhance memory are important for everyone, not just students. For example, when a friend recommends a new shoe store we want to remember the name of of it, or when going to the grocery it is important to remember the items we need to pick up. What are some strategies that can be used to strengthen memory?

Trying To Remember

In one study researchers investigated the role of intentional-encoding instructions and task relevance at study on visual memory performance (Varakin & Hale, 2014). Task relevance was manipulated by having participants keep a running tally of either the objects they were attempting to remember or an irrelevant category of objects during study. Half of the participants within each level of task relevance were further instructed to remember one category of objects for a subsequent recognition memory test (intentional memory group) , and the other half of the participants were not informed of a memory test (incidental memory group). Intentional-encoding instructions improved recognition discrimination only when participants were not already keeping a tally of the to-be-remembered objects. This result suggests that intentional-encoding instructions may improve visual memory due

Building a Better Memory

Are learning and memory completely distinct?  No; both are experienced based.  “[M]emory is the consequence of learning from an experience- that is, the consequence of acquiring new information” , asserts James McGaugh (memory researcher, author of Memory and Emotion).  Learning is a process of memory formation.  There are 2 general categories of memory: explicit and implicit.

Key Points from- Strategies to Maximize Learning (Hale, 2014):

      Memory is the product of learning
      Memory formation = brain change
      All cognition, emotion, feeling, perception  and learning emanate from the brain
      Healthy brain is imperative to maximize learning / memory
      Mind- body is a unit- not separate
      All cognition, emotion, feeling, perception  and learning emanate from the brain Healthy brain is imperative to maximize learning / memory   Mind- body is a unit- not separate
      Foundations of memory include: brain health, focused attention, elaborative encoding, spaced rehearsal and testing
      Understanding is imperative for strong memory
      Studying should be structured: progressive, organized, spaced over multiple sessions and involve accurate evaluation

Wednesday, November 16, 2016

Rationality Quotient: Comprehensive Assessment of Rational Thinking

Stanovich and colleagues recently developed a prototype of the first comprehensive assessment of rational thinking. The test is discussed, and presented in detail in the new book,  titled- The Rationality Quotient.

Up until publication of- The Rationality Quotient - components of rational thinking had been tested using various tasks, but a comprehensive test was not available. I first discussed the development of such a test with Stanovich, in 2013- interview here.

In the following interview (conducted in November, 2016) Stanovich provides detailed answers to important questions about the test. 
What are some of the initial reactions, regarding the RQ, from academics?

Uniformly positive so far, and I believe that is because we were careful in the book to be explicit about two things.  First, we were clear about what our goals were and the goals were circumscribed.  Secondly, we included an entire chapter contextualizing our test (the Comprehensive Assessment of Rational Thinking, CART) and discussing caveats regarding its use as a research instrument or otherwise. In fact, I think we have already entirely achieved our aims.  We have a prototype test that is a pretty comprehensive measure of the rational thinking construct and that is grounded in extant work in cognitive science.  Now, this is not to deny that there is still much work to be done in turning the CART into a standardized instrument that could be used for practical purposes.  But of course a finished test was not our goal in this book.  Our goal was to show a demonstration of concept, and we have done that.  We have definitively shown that a comprehensive test of rational thinking was possible given existing work in cognitive science.  This is something that I have claimed in previous books but had not empirically demonstrated with the comprehensiveness that we have here by introducing the CART.  As I said, there are more steps left in turning the CART into an “in the box” standardized measure, but that is a larger goal than we had for this book.

I think that, at least so far, most academics have understood our goals and the feedback has been good.  We wrote a summary article on the CART in a 2016 issue of the journal Educational Psychologist (51, 23-34) and the feedback from that community has been good.

Are there components of the RQ that can be expected to show a strong positive correlation with intelligence?

The CART has 20 subtests and four thinking dispositions scales (the latter are not part of the total score). Collectively they tap both instrumental rationality and epistemic rationality.  In cognitive science, instrumental rationality means behaving in the world so that you get exactly what you most want, given the resources (physical and mental) available to you.  Epistemic rationality concerns how well beliefs map onto the actual structure of the world.  The two types of rationality are related.  In order to take actions that fulfill our goals, we need to base those actions on beliefs that are properly calibrated to the world.

The CART assesses epistemic thinking errors such as: the tendency to show incoherent probability assessments; the tendency toward overconfidence in knowledge judgments; the tendency to ignore base rates; the tendency not to seek falsification of hypotheses; the tendency to try to explain chance events; the tendency to evaluate evidence with a myside bias; and the tendency to ignore the alternative hypothesis.
Additionally, CART assesses instrumental thinking errors such as:  the inability to display disjunctive reasoning in decision making; the tendency to show inconsistent preferences because of framing effects; the tendency to substitute affect for difficult evaluations; the tendency to over-weight short-term rewards at the expense of long-term well-being; the tendency to have choices affected by vivid stimuli; and the tendency for decisions to be affected by irrelevant context. 
Importantly, the test also taps what we call contaminated mindware.  This category of thinking problem arises because suboptimal thinking is potentially caused by two different types of mindware problems.  Missing mindware, or mindware gaps, reflect the most common type—where Type 2 processing does not have access to adequately compiled declarative knowledge from which to synthesize a normative response to use in the override of Type 1 processing.  However, in the book, we discuss how not all mindware is helpful or useful in fostering rationality.  Indeed, the presence of certain kinds of mindware is often precisely the problem.  We coined the category label contaminated mindware for the presence of declarative knowledge bases that foster irrational rather than rational thinking.  Four of the 20 subtests assess contaminated mindware.

My purpose in digressing here to describe the CART is to point out that given the number and complexity of rational thinking skills, it is likely that the subtests will have correlations with intelligence that are quite variable.  The four subtests with the highest correlations are: the Probabilistic Reasoning Subtest; the Scientific Reasoning Subtest; the Reflection Versus Intuition Subtest; and the Financial Literacy Subtest.  Correlations with these subtests tend to .50or higher.  Most of the subtests of the CART correlate with intelligence in the range of .25 to .50 (a few have even lower correlations). Some very important components of rational thinking do show considerable dissociation from intelligence.  Overconfidence (measured by the Knowledge Calibration Subtest of the CART) shows only a .38 correlation with intelligence.  This represents a substantial amount of dissociation for a key component of rational thinking.  Kahneman, for example, devoted substantial portions of his best-selling book to this component of rational thinking.  Myside bias (measured by our Argument Evaluation Subtest) likewise shows a correlation of .38, indicating a substantial dissociation.  This thinking bias is at the center of many discussions of what it means to be rational.  Some of the subtests that most directly measure the components of the axiomatic approach to utility maximization show relatively mild correlations with intelligence.  For example, the Framing Subtest shows a fairly low .28 correlation.  Framing measures a foundational aspect of rational thinking according to the axiomatic approach.

Finally, some subtests of immense practical importance show very low correlations with intelligence in the CART.  The skill of assessing numerical expected value shows a correlation of only .21, and the ability to delay for greater monetary reward shows a correlation of only .06.  The tendency to believe in conspiracies shows a modest correlation of .34.

Do you think rationality will acquire the same high level status as intelligence in the near future?

Not in the near future, no.  Our goal with the book was more modest—to simply raise awareness of the importance of rational thinking and the ability of modern cognitive psychology to measure it.  The result of our efforts will, we hope, redress the imbalance between our tendency to value intelligence versus rationality.  In our society, what gets measured gets valued.  Our aim in developing the CART was to draw attention to the skills of rational thought by measuring them systematically.  In the book, we are careful to point out that we operationalized the construct of rational thinking without making reference to any other construct in psychology, most notably intelligence.  Thus, we are not trying to make a better intelligence test.  Nor are we trying to make a test with incremental validity over and above IQ tests.  Instead, we are trying to show how one would go about measuring rational thinking as a psychological construct in its own right.  We wish to accentuate the importance of a domain of thinking that has been obscured because of the prominence of intelligence tests and their proxies.  It is long overdue that we had more systematic ways of measuring these components of cognition, that are important in their own right, but that are missing from IQ tests.  Rational thinking has a unique history grounded in philosophy and psychology, and several of its subcomponents are firmly identified with well-studied paradigms.  The story we tell in the book is of how we have turned this literature into the first comprehensive device for the assessment of rational thinking (the CART).

Why does society need a comprehensive assessment of rational thinking?

To be globally rational in our modern society you must have the behavioral tendencies and knowledge bases that are assessed on the CART to a sufficient degree.  Our society is sometimes benign, and maximal rationality is not always necessary, but sometimes, in important situations, our society is hostile.  In such hostile situations, to achieve adequate degrees of instrumental rationality in our present society the skills assessed by the CART are essential.  In Chapter 15 of The Rationality Quotient we include a table showing that rational thinking tendencies are linked to real life decision making.  In that table, for each of the paradigms and subtests of the CART, an association with a real-life outcome is indicated.  The associations are of two types.  Some studies represent investigations where a laboratory measure of a bias was used as a predictor of a real-world outcome.  Others are reports of real-world analogues of biases that were originally discovered in the lab.  Clearly more work remains to be done on tracing the exact nature of the connections—that is, whether they are causal.  The sheer number of real-world connections, however, serves to highlight the importance of the rational thinking skills in our framework.  Now that we have the CART, we could, in theory, begin to assess rationality as systematically as we do IQ.  If not for professional inertia and psychologists’ investment in the IQ concept, we could choose tomorrow to more formally assess rational thinking skills, focus more on teaching them, and redesign our environment so that irrational thinking is not so costly.  Whereas just thirty years ago we knew vastly more about intelligence than we knew about rational thinking, this imbalance has been redressed in the last few decades because of some remarkable work in behavioral decision theory, cognitive science, and related areas of psychology.  In the past two decades cognitive scientists have developed laboratory tasks and real-life performance indicators to measure rational thinking tendencies such as sensible goal prioritization, reflectivity, and the proper calibration of evidence.   People have been found to differ from each other on these indicators.  These indicators are structured differently from the items used on intelligence tests.  We have brought this work together by producing here the first comprehensive assessment measure for rational thinking, the CART.

Tuesday, October 25, 2016

Science and Rationality in Modern Society

Science and rationality are important in modern, technologically advanced, industrial societies. Science is a large enterprise consisting of multiple components. Science, although fallible, is the great reality detector. Rationality, in this context, refers to rationality as it is conceptualized in cognitive science. Rationality is concerned with judgment and decision making. Rationality consists of two main categories- instrumental and epistemic. Instrumental rationality reflects goal optimization, and epistemic reflects evidence based beliefs. There is overlap between the two categories of rationality. In my most recent book- In Evidence We Trust: The need for science, rationality & statistics- I provide information on various aspects of science, rationality and mathematical procedures (statistics) used in describing and making inferences in the context of scientific research. 

In Evidence We Trust

 It is often said we live in the information age, but we also live in the mis-information age.  How do we decide what constitutes knowledge and what constitutes nonsense?  Maybe there are no wrong or right answers, and just opinions?  This notion is fallacious.  There are facts and opinions, right and wrong answers.  There is a reality that extends beyond personal comforts and opinions (Mitchell & Jolley, 2010).  In the context of science  facts are tentative.  They are assertions that are supported by the preponderance of evidence.  Facts in the context of science (primary concern in this book) are based on levels of certainty, but absolute certainty is never attained.  Scientific findings are presented in terms of probabilities and data (e.g. laws, principles, theories, etc.) is revised in accordance to findings.

Testimonials, anecdotes, they-says, wishful thinking and so on do not count for evidence.  If  these types of claims and feelings are labeled  as evidence then any discussion of evidence is vacuous.  Testimonials exist for almost any claim you can imagine.  That does not mean that claims of this sort have no value.    Experiences are confounded (confused by alternative explanations). Experiences may be very important in some contexts, and they may serve as meaningful research questions.  However, a meaningful question or a possible future finding is not synonymous with evidence. Scientific evidence is drastically different than evidence as it relates to everyday discourse.  As Joy Victoria points out- it should be obvious from the book's title that the type of evidence I am referring to in the book is derived from scientific findings (paraphrased). 
The content in chapter one includes short-articles (old, new & revised), a science discussion roundtable (featuring individuals from various fields) and a nonsense detection kit. Some of the short articles presented in chapter one have been published on various internet sites, and some of the same or similar information may be discussed in across different articles.  There are at least two key benefits that can occur when presenting similar information across different articles (in different contexts): strengthening of memory connections, and each article can be read as a stand-alone article.  In the science discussion roundtable participants are asked two questions.  One) Do you have any tips for people that are interested in enhancing their ability to read scientific research?  Two) What is the biggest (or at least one of the biggest misconceptions) misconception about science? The Nonsense Detection Kit is presented at the end of chapter one.  The impetus for designing the Nonsense Detection Kit was similar kits devised by Sagan, Shermer, and Lilienfeld.

Chapter two features short articles on rationality.  Some of the same or similar information is contained across different articles.  There are at least a couple of advantages to presenting information in this manner (refer to previously mentioned advantages in chapter one).  Many of the articles focus on the rationality intelligence dichotomy.  Also included in this chapter are interviews with Keith Stanovich and the Stanovich Research Lab (Keith Stanovich, Richard West and Maggie Toplak).  In the interview with Stanovich, he discusses the development of an RQ Test. In the interview with the Stanovich lab, rationality and intelligence are discussed. Since the publication of the book Stanovich, West and Toplak have designed the first comprehensive test for rational thinking

Chapter three features frequently asked questions about research methods and statistics. Many of the questions are questions I have received in the past from my students.  Some of the questions address basic research and statistics problems, while other questions are more complex.  At the end of the chapter recommended sources are provided for readers that are interested in furthering their studies on research methods and statistics. 
The book ends with an appendices section. Practice problems, and guidelines regarding APA citations and reference lists are given.

The content in this book may be difficult for some to comprehend. However, with some effort and patience the content is learnable for most people. In the words of Albert Einstein “Things should be made as simple as possible, but not any simpler.” Science, rationality and statistics can be simplified to a degree, but relative to most other topics these topics are difficult.  This book is not written for cognitive misers (the cognitively lazy).  This book is written for individuals that are interested in separating knowledge and nonsense, and are willing to put forth at least a moderate level of cognitive effort.  This book is not written in the format often used by pop science writers.
I would like to thank Joy Victoria, Kitty Mervine, Jason Silvernail and Coert Visser for the review articles of IEWT they have written. 
RecommendedResources: In Evidence We Trust by Joy Victoria 

In Evidence We Trust (Review) by Coert Visser