Jamie Hale

Jamie Hale

Monday, May 16, 2016

Defending- In Evidence We Trust

In 2005 I had an idea to write a book about scientific and rational thinking. I started writing the book that year, but progress was slow. In 2006 and 2007 I wrote a few books on different subjects; however I continued to work on the 2005 idea.  In 2010 I completed another book; this one was on exercise and nutrition myths.  Over the next few years I was occupied with various projects, and my ideas about the book contents changed often.  January 1, 2014, after changing the contents many times, my idea- In Evidence We Trust- came to fruition. 

The majority of books sold have been to college students (undergraduate & graduate).  Surprisingly, the book hasn't done as good as I thought it would with science writers and others that perpetuate science to the public.  Before publication of the book I thought those promoting science to the general public would be my main audience.  In accordance with reviews and feedback from readers there are 2 major criticisms aimed at the book- it is redundant and too complex.

Firstly, I would like to respond to the redundancy criticism.  To reiterate, the book was written over a period of 9 years, so overlap can be expected with contents regarding similar topics. Some of the short articles presented in chapter one and two have been published on various internet sites, and some of the same or similar information may be discussed in different articles. In chapter three some of the questions overlap. Many of those questions were questions asked by students from courses on research methods and statistics.  There are at least two key benefits that for presenting similar information across different articles (in different contexts): strengthening of memory connections, and each article can be read as a stand-alone article.  Elaborating on the information and repeating consistently over time will assist in enhancing memory (2 of foundations of memory).  I suspect some of this information will be new to readers, thus redundancy will be beneficial to an even higher level.  Novel information is processed differently than routine information.  With the appropriate strategies novel information moves from being cognitively difficult to much easier (requires few cognitive resources and is retrieved with less effort) to process, once it becomes learned (discussed in detail in my seminars - Strategies To Maximize Learning and Exploring Memory). Stand-alone articles have the benefit of allowing the reader to read that section only; no need to read entire chapter if only interested in specific topic.

In addressing the second major criticism- complexity- it is important to point out that science, rationality and statistics are difficult.  A comprehensive understanding of science and its applications requires general knowledge in the meta-sciences (philosophy of science, history of science, sociology of science, psychology of science) research methodology and statistics.  In Evidence We Trust- is not a complete guide to the meta-sciences (consists of phil. of science and psy of science information- concise), but it is focused on research methodology and statistics.  It is important to recognize different areas of science use the same words differently, and they often use different types of methodologies. The research methods and statistics discussed in the book may not apply to some areas of science.  This topic is mentioned in the book.  Is the book too complex?  The content in this book may be difficult for some to comprehend. However, with some effort and patience the content is learnable for most people. In the words of Albert Einstein “Things should be made as simple as possible, but not any simpler.” Science, rationality and statistics can be simplified to a degree, but relative to most other topics these topics are difficult.  This book is not written for cognitive misers (the cognitively lazy).  This book is written for individuals that are interested in separating knowledge and nonsense, and are willing to put forth at least a moderate level of cognitive effort.  If you sale science (science writer, science educator, science based -often synonymous with evidence based provider) you need to  understand research methods and statistics. If you are not willing to learn at least the basics you should stop selling science. 

Mervine, in a review of In Evidence We Trust published in Skeptic Briefs, points out that "Hale states, 'after learning the information provided in this chapter [chapter 3], you should have the skills necessary to read scientific reports.'  I come from a non-science background, and I do not feel confident that I could tackle a scientific report after reading this chapter alone."  A few others have pointed out that they agree with Mervine on this point.  A more appropriate statement is, after learning the information provided in this chapter and thoroughly reviewing the recommended sources (at end of chapter 3), references and appendices, you should have the skills necessary to read scientific reports. 

Overall, the response to the book has been positive.  Almost all of the conversations I have had, regarding he book, with students have been good.  A few students have said the book helped with their school work and assisted them in distinguishing between science and pseudo-science.  In addition, some top intellectuals have had some good things to say about the book.

“A great introduction to scientific thinking, useful for the student and the general reader alike”  …Keith E. Stanovich, Emeritus Professor, University of Toronto, author of How To Think Straight About Psychology 

“A useful, informative, and engaging compendium of critical thinking tools.  Should come in handy for novices and experts alike.  I recommend it!”  …Scott O. Lilienfeld, Ph.D., Professor, Department of Psychology, Emory University, Atlanta, Georgia, co-author of 50 Great Myths of Popular Psychology.

"Jamie Hale has written an engaging introduction to the importance of scientific and critical thinking and how useful and needed it is in everyday life. On top of that, he throws in a nice primer on basic statistical topics and offers useful and insightful answers. This book would be useful to any student of science and rationality as well as any person interested in these topics."
...Gregory Feist, Ph,D., Associate Professor of Psychology, San Jose State University, San Jose, CA, author of The Psychology of Science and the Origins of the Scientific Mind.

In the context of In Evidence We Trust evidence is synonymous with scientific evidence. Testimonials, anecdotes, they-says, wishful thinking and so on do not count for evidence.  If we consider these types of claims and feelings as evidence then any discussion of evidence is vacuous.  Testimonials exist for almost any claim you can imagine.  That does not mean that claims of this sort have no value.  However, experiences are confounded (confused by alternative explanations).  Experiences may be important in some contexts, and they may serve as meaningful research questions.  However, a meaningful question or a possible future finding is not synonymous with evidence; although, in the future they could become evidence.  

Recommended Readings:

Review Article: In Evidence We Trust
Mervine, B. (2014).  Trusting Evidence.  Skeptic Briefs, 24(2), p.7. 


Thursday, March 10, 2016

Project: thinking about science

The word science is derived from a latin word scientia meaning an organized body of theoretical knowledge.  Giving a precise definition of science is difficult as there is little consensus in the scientific community as to what that definition should be.   When discussing science what is often discussed are products of science or scientific methodology.   That is, making use of scientific findings (scientific products).  Even though this description provides part of the picture it is an incomplete description regarding the enormous enterprise of science. Science in various contexts has huge implications and a comprehensive study should be approached using a multi-disciplinary approach.   A comprehensive understanding of the implications of science requires more than the procedural skills to conduct research. Acquiring general or domain specific knowledge or citing studies is important, but doesn't indicate scientific cognition, or a comprehensive understanding of science.
Project: thinking about science involves pilot studies, assessments, a wide range of samples, full studies, scale design, univariate, bivariate and multivariate stats.  The project is a long term commitment with an emphasis on examining the association among various measures of science and finding predictors that may be useful regarding scientific achievement in an array of domains. Science is hard work.  The project will involve a number of people from various disciplines. 
Scientific literacy vs. Scientific cognition (thinking, reasoning)

Scientific literacy has been conceptualized  in various ways (see; Norris & Phillips, 2002).  Examples include understanding science and its applications, knowledge of what counts as science, knowledge of risks and benefits of science, etc.  In the current research scientific literacy is synonymous with general scientific knowledge that involves various domains.  This form of literacy is sometimes referred to as derived scientific literacy.  Scientific cognition is not the same thing as scientific literacy; it involves multiple components and sub-components (Feist, 2006).  Deanna Kuhn asserts that the essence of scientific thinking is coordinating belief with evidence (2001).  At the very least scientific cognition involves philosophy of science, scientific methodology, quantitative reasoning, probabilistic reasoning and elements of deductive and inductive logic.
Various scales have been developed to measure scientific thinking / reasoning / cognition.  Kahan developed a scale called the Ordinary Science Intelligence Scale (OSI_2.0, Kahan, 2014).  Drummond and Fischhoff (2015) developed the Scientific Reasoning Scale.  Drummond and Fischhoff found that measures of scientific reasoning were distinct from measures of scientific literacy.  Kevin Dunbar (2000) and Zimmerman (2005) have also conducted research on scientific thinking.  Dunbar's research mostly involves examining cognitive processes underpinning thinking during the research process, while Zimmerman's research is broader, examining various scales, and development patterns of scientific thought.  Fugelsang et al. (2004) have examined strategies that scientists and non-scientists use to evaluate data that is consistent or non-consistent with expectations. 

Stage 1: Current research

Sam Sloss and I are working on a paper that involves a large number of outcomes, including measures of general scientific knowledge, attitudes towards science and confidence in science.  The paper also involves an array of stats; univariate and multi-variate statistical procedures are used.

The development of a new scientific literacy scale and scientific cognition scale are also in the process.  Soon, the scales will be administered, in a pilot study, to University students.  Cronbach's alpha will be utilized once data is collected.  The scale might need to be modified  before using in studies. 
We are in the early stages of the project and hope to advance at moderate pace.  I plan on making progress reports consistently.  In the end my hope is that a large amount of data will be collected and can be utilized to further advance science and provide everyone interested in science useful information.  

References available upon request