Rationality Quotient: Comprehensive Assessment of Rational Thinking

Stanovich and colleagues recently developed a prototype of the first comprehensive assessment of rational thinking. The test is discussed, and presented in detail in the new book,  titled- The Rationality Quotient.

Up until publication of- The Rationality Quotient - components of rational thinking had been tested using various tasks, but a comprehensive test was not available. I first discussed the development of such a test with Stanovich, in 2013- interview here.

In the following interview (conducted in November, 2016) Stanovich provides detailed answers to important questions about the test. 

 

What are some of the initial reactions, regarding the RQ, from academics?

Uniformly positive so far, and I believe that is because we were careful in the book to be explicit about two things.  First, we were clear about what our goals were and the goals were circumscribed.  Secondly, we included an entire chapter contextualizing our test (the Comprehensive Assessment of Rational Thinking, CART) and discussing caveats regarding its use as a research instrument or otherwise. In fact, I think we have already entirely achieved our aims.  We have a prototype test that is a pretty comprehensive measure of the rational thinking construct and that is grounded in extant work in cognitive science.  Now, this is not to deny that there is still much work to be done in turning the CART into a standardized instrument that could be used for practical purposes.  But of course a finished test was not our goal in this book.  Our goal was to show a demonstration of concept, and we have done that.  We have definitively shown that a comprehensive test of rational thinking was possible given existing work in cognitive science.  This is something that I have claimed in previous books but had not empirically demonstrated with the comprehensiveness that we have here by introducing the CART.  As I said, there are more steps left in turning the CART into an “in the box” standardized measure, but that is a larger goal than we had for this book.

I think that, at least so far, most academics have understood our goals and the feedback has been good.  We wrote a summary article on the CART in a 2016 issue of the journal Educational Psychologist (51, 23-34) and the feedback from that community has been good.

Are there components of the RQ that can be expected to show a strong positive correlation with intelligence?

The CART has 20 subtests and four thinking dispositions scales (the latter are not part of the total score). Collectively they tap both instrumental rationality and epistemic rationality.  In cognitive science, instrumental rationality means behaving in the world so that you get exactly what you most want, given the resources (physical and mental) available to you.  Epistemic rationality concerns how well beliefs map onto the actual structure of the world.  The two types of rationality are related.  In order to take actions that fulfill our goals, we need to base those actions on beliefs that are properly calibrated to the world.

The CART assesses epistemic thinking errors such as: the tendency to show incoherent probability assessments; the tendency toward overconfidence in knowledge judgments; the tendency to ignore base rates; the tendency not to seek falsification of hypotheses; the tendency to try to explain chance events; the tendency to evaluate evidence with a myside bias; and the tendency to ignore the alternative hypothesis.

Additionally, CART assesses instrumental thinking errors such as:  the inability to display disjunctive reasoning in decision making; the tendency to show inconsistent preferences because of framing effects; the tendency to substitute affect for difficult evaluations; the tendency to over-weight short-term rewards at the expense of long-term well-being; the tendency to have choices affected by vivid stimuli; and the tendency for decisions to be affected by irrelevant context. 

 

Importantly, the test also taps what we call contaminated mindware.  This category of thinking problem arises because suboptimal thinking is potentially caused by two different types of mindware problems.  Missing mindware, or mindware gaps, reflect the most common type—where Type 2 processing does not have access to adequately compiled declarative knowledge from which to synthesize a normative response to use in the override of Type 1 processing.  However, in the book, we discuss how not all mindware is helpful or useful in fostering rationality.  Indeed, the presence of certain kinds of mindware is often precisely the problem.  We coined the category label contaminated mindware for the presence of declarative knowledge bases that foster irrational rather than rational thinking.  Four of the 20 subtests assess contaminated mindware.

My purpose in digressing here to describe the CART is to point out that given the number and complexity of rational thinking skills, it is likely that the subtests will have correlations with intelligence that are quite variable.  The four subtests with the highest correlations are: the Probabilistic Reasoning Subtest; the Scientific Reasoning Subtest; the Reflection Versus Intuition Subtest; and the Financial Literacy Subtest.  Correlations with these subtests tend to .50or higher.  Most of the subtests of the CART correlate with intelligence in the range of .25 to .50 (a few have even lower correlations). Some very important components of rational thinking do show considerable dissociation from intelligence.  Overconfidence (measured by the Knowledge Calibration Subtest of the CART) shows only a .38 correlation with intelligence.  This represents a substantial amount of dissociation for a key component of rational thinking.  Kahneman, for example, devoted substantial portions of his best-selling book to this component of rational thinking.  Myside bias (measured by our Argument Evaluation Subtest) likewise shows a correlation of .38, indicating a substantial dissociation.  This thinking bias is at the center of many discussions of what it means to be rational.  Some of the subtests that most directly measure the components of the axiomatic approach to utility maximization show relatively mild correlations with intelligence.  For example, the Framing Subtest shows a fairly low .28 correlation.  Framing measures a foundational aspect of rational thinking according to the axiomatic approach.

Finally, some subtests of immense practical importance show very low correlations with intelligence in the CART.  The skill of assessing numerical expected value shows a correlation of only .21, and the ability to delay for greater monetary reward shows a correlation of only .06.  The tendency to believe in conspiracies shows a modest correlation of .34.

Do you think rationality will acquire the same high level status as intelligence in the near future?

Not in the near future, no.  Our goal with the book was more modest—to simply raise awareness of the importance of rational thinking and the ability of modern cognitive psychology to measure it.  The result of our efforts will, we hope, redress the imbalance between our tendency to value intelligence versus rationality.  In our society, what gets measured gets valued.  Our aim in developing the CART was to draw attention to the skills of rational thought by measuring them systematically.  In the book, we are careful to point out that we operationalized the construct of rational thinking without making reference to any other construct in psychology, most notably intelligence.  Thus, we are not trying to make a better intelligence test.  Nor are we trying to make a test with incremental validity over and above IQ tests.  Instead, we are trying to show how one would go about measuring rational thinking as a psychological construct in its own right.  We wish to accentuate the importance of a domain of thinking that has been obscured because of the prominence of intelligence tests and their proxies.  It is long overdue that we had more systematic ways of measuring these components of cognition, that are important in their own right, but that are missing from IQ tests.  Rational thinking has a unique history grounded in philosophy and psychology, and several of its subcomponents are firmly identified with well-studied paradigms.  The story we tell in the book is of how we have turned this literature into the first comprehensive device for the assessment of rational thinking (the CART).

Why does society need a comprehensive assessment of rational thinking?

To be globally rational in our modern society you must have the behavioral tendencies and knowledge bases that are assessed on the CART to a sufficient degree.  Our society is sometimes benign, and maximal rationality is not always necessary, but sometimes, in important situations, our society is hostile.  In such hostile situations, to achieve adequate degrees of instrumental rationality in our present society the skills assessed by the CART are essential.  In Chapter 15 of The Rationality Quotient we include a table showing that rational thinking tendencies are linked to real life decision making.  In that table, for each of the paradigms and subtests of the CART, an association with a real-life outcome is indicated.  The associations are of two types.  Some studies represent investigations where a laboratory measure of a bias was used as a predictor of a real-world outcome.  Others are reports of real-world analogues of biases that were originally discovered in the lab.  Clearly more work remains to be done on tracing the exact nature of the connections—that is, whether they are causal.  The sheer number of real-world connections, however, serves to highlight the importance of the rational thinking skills in our framework.  Now that we have the CART, we could, in theory, begin to assess rationality as systematically as we do IQ.  If not for professional inertia and psychologists’ investment in the IQ concept, we could choose tomorrow to more formally assess rational thinking skills, focus more on teaching them, and redesign our environment so that irrational thinking is not so costly.  Whereas just thirty years ago we knew vastly more about intelligence than we knew about rational thinking, this imbalance has been redressed in the last few decades because of some remarkable work in behavioral decision theory, cognitive science, and related areas of psychology.  In the past two decades cognitive scientists have developed laboratory tasks and real-life performance indicators to measure rational thinking tendencies such as sensible goal prioritization, reflectivity, and the proper calibration of evidence.   People have been found to differ from each other on these indicators.  These indicators are structured differently from the items used on intelligence tests.  We have brought this work together by producing here the first comprehensive assessment measure for rational thinking, the CART.