Jamie Hale

Jamie Hale

Monday, September 29, 2014

Bad Evidence


When discussing evidence it is important to point out that Common Evidence (evidence in the context of everyday discussion) is drastically different than Scientific Evidence (evidence derived from scientific processes).   Common Evidence, generally consists of proof or testimony.  Webster’s New Dictionary of The English Language (2006) provides the following definitions for evidence:  “1: outward sign  2: proof or testimony.”  An outward sign, proof or testimony are ambiguous, and can mean almost anything.  From a scientific perspective, testimonials, anecdotes, they-says, wishful thinking and so on do not count for evidence.  Testimonials exist for almost any claim you can imagine.  That does not mean that claims of this sort have no value. However, they have little value in the context of science.  Experiences are confounded (confused by alternative explanations). Experiences may be important in some contexts, and they may serve as meaningful research questions.  However, a meaningful question or a possible future finding is not synonymous with evidence; although, in the future either could become evidence (Hale, 2013).  Scientific evidence is derived from scientific studies.  All scientific evidence is not created equal.  Many bad studies get published and many good studies do not get published. 

The contents of this article address scientific evidence. I address Common Evidence in a different article- Testimonials Aren't Real Evidence
  
Understanding research methods & statistics

Reading and understanding research methods and statistics are not easy.  For most people formal training may be necessary to gain a firm understanding of these relatively difficult subjects.  There are people that lack formal training, in these areas, that have exceptional knowledge on these topics. 

Scientific methods are the most powerful methods we have for discovering reality.  Statistics allows us to organize, summarize, and interpret research data collected from samples. In order to fully appreciate and apply the knowledge that has been acquired through the scientific process it is imperative to have a basic understanding of scientific research methodology.  Scientific Methodology-scientific techniques used to collect and evaluate data. 
    
It is important to understand that all research methods play an important role in leading us to tentative conclusions concerning how things work in the observable universe.  But, it also important to realize different types of research should be interpreted and applied in a different manner. As an example, the primary goal of correlation research is prediction, while the primary goal of experimental research is explanation / understanding (determining cause and effect relationships). 

Quantitative research is different than qualitative research.  With quantitative research the results are presented as numbers or quantities; qualitative research presents the results in words (Patten, 2004).  Knowledge of statistics is required if one is interested in understanding quantitative research.  For a detailed discussion on quantitative vs. qualitative research refer to - Understanding Research Methods by M.L. Patten and Health Psychology by L. Brannon and J. Feist. 

An understanding of research methods and statistics is attainable by most people.  However, it requires a lot of effort for most.   Understanding research requires more than reading an abstract, glossing over the Discussion section of a paper, or repeating what your favorite guru said about the results of a study. 

Bad Evidence

When considering the value of evidence, reliability and validity must be considered.  The type of study also needs to be considered. In addition, other factors should be considered when evaluating studies.  A concise discussion regarding reliability and validity is addressed here: Reliability & Validity  Refer to In Evidence We Trust (Hale, 2013) to learn more about evaluating research. 

It is imperative to recognize that not all scientific journal articles are quality articles.  Journals often publish poor studies.  And, good studies are sometimes not published.  Students in research methods and stats courses know there are a lot of bad studies published.  Students are often required to critique bad journal articles.  I hated doing this in graduate school.  However, it was great learning experience, and my ability to spot bad evidence was enhanced.   

In terms of evidence bad evidence can be thought of as no evidence.  Deciding the value of evidence is an intense intellectual activity and becomes increasingly difficult with complex studies. 

Experimental Research Fallacy

It is a fallacy that experimental research is always good research.  This fallacy is not generally explicitly stated, but may be suggested when only experimental research seems to count in regards to the discussion or topic being discussed.  As with other research methods, the reliability and validity must be considered along with additional factors that may impact the outcome or inferences regarding the outcome.  Considering internal validity (in addition to external, construct, and statistical validity) is important when evaluating experimental results.  Research methods, other than experiments, can provide valuable information, contrary to what some appear to think.  As an example, epidemiological studies were the first to detect a relationship between the behavior of smoking and heart disease (Brannon & Feist, 2010). 

If the goal is determining causation true experimentation is required. Some researchers suggest that some level of causation can determined using methods other than just experiments (Stanovich, 2007; Gore, 2013).  True experiments require tedious work and high levels of control.  However, experiments are not always practical or ethical.  Thus, one of the reasons other types of methods are needed.  If we didn’t have other research methods in addition to experiments many questions couldn’t be examined.   

Evidence Based Practice

Some people in the medical, educational, and fitness industry consider themselves evidence-based practitioners.  What is an Evidence Based Practitioner?  Consider the following illustration in relation to the medical field:  Research utilization (RU) overlaps with some of the same principles of an evidence based practice (EBP). However, EBP extends beyond the steps taken with scientific research.  Research utilization refers to the review and critique of scientific research, and then the application of the outcomes to clinical practice (Estabrooks, 1998).  Evidence-based practice (EBP) represents a wider concept. When clinicians utilize the EBP approach, they go beyond the expertise of clinicians and researchers, and consider the patient's preferences and values to guide patient care (Ingersoll, 2000).  This sort of ambiguous definition is problematic.  How can this concept be operationalized?  

Operationism (using operational definitions) removes the concept from the feelings and intuitions of an individual and allows it to be tested by anyone with the resources to carry out the measures (Stanovich, 2007).
When clinical recommendations are incongruent with statistical recommendations one should generally prefer statistical recommendations.  Refer to the following article to learn more about clinical vs. statistical prediction- When Experts Are Wrong  
 
Procedures for enhancing medical adherence may involve not only scientific findings.  Adherence to the program or treatment plan should be considered. The treatment provider and patient relationship should also be considered in terms of efficacy.  Research indicates a positive relationship between clinician and patient is often associated with a positive outcome (Benedetti, 2011)  Does one practicing medicine need to be well read in science to perform well as a clinician?  Of course, in order to understand and explain how things work a scientific understanding is needed.  But, recipe knowledge may work fine in regards to successful clinical practice.  
 
I have asked many of the self-proclaimed evidence based fitness crowd to provide a definition, or at least an approximate definition of evidence based fitness, but I haven’t even received an answer.  If there are no guidelines, criteria, or approximations of the concept- Evidence Based Fitness (EBF)- the concept is weak.  If this concept is hard to test or as some have suggested non-testable, then it is a non-scientific matter.  If what is implied by EBF is that some elements of the training program adhere to scientific findings then it is reasonable to suggest most successful programs are evidence based.  Just as successful diet programs are successful due to some scientific principles, whether the proponents of the diet programs are aware or not of what those principles are.  In addition to lacking a definition, other problems exist regards EBF- logical inconsistency, and no evidence indicating that clients of evidence based fitness practitioners has better outcomes.    

The following questions need to be addressed:

Do trainers need to be well read in science and statistics to be good trainers?
Isn’t a large part of fitness training artistic in nature?
Should evidence based practitioners ridicule people for making non-evidence based fitness claims, and then proceed to make irrational claims associated with other domains of knowledge?
What is the ultimate objective of the EBF crowd?  To be well read in science, to design quality training programs or both?
How can EBF be operationalized?  

Do those that call themselves evidence based fitness trainers understand research methods and statistics?

The evidence based fitness movement can have some positive implications- highlighting the importance of science, encouraging people to learn more about science, and encouraging thorough evaluation of popular fitness claims.

Conclusion  

Bad evidence is often published in science journals.  Understanding that not all scientific journal articles are created equal is imperative.  To reiterate, experimental research is one of many scientific methods.  Other methods can contribute to an understanding of the universe.