Measures in Science

 An instrument can provide accuracy and preciseness but lack value if the measurement is non-valid. When determining the validity of the measurement one must ask does the measurement really measure the concept in question?

The key aspects concerning the quality of scientific measures are reliability and validity (Hale, 2011). Reliability is a measure of the internal consistency and stability of a measuring device. Validity gives us an indication of whether the measuring device measures what it claims to.

Internal consistency is the degree in which the items or questions on the measure consistently assess the same construct. With an internally consistent measure items are positively correlated with each other. This measure of internal consistency is particularly important regarding self-report measures. It isn't as important when considering performance based measures, tests or surveys. Each question should be aimed at measuring the same thing. Stability is often measured by test / retest reliability. The same person takes the same test twice and the scores from each test are compared. Interrater reliability is sometimes used in assessing reliability. With interrater reliability different judges or raters (two or more) make observations, record their findings and then compare their observations. If the raters are reliable then the percentage of agreement should be high.

When asking if a measure is valid we are asking if it measures what is supposed to. Validity is a judgment based on collected data; it is not a statistical test. Two primary ways to determine validity include: existing measures and known group differences.

The existing measures test determines if the new measure correlates with existing relevant valid measures. The new measure should be similar to measures that have been recorded with already-established valid measuring devices. Known group differences determine whether the new measure distinguishes between known group differences. An illustration of known group differences is seen when different groups are given the same measure, and are expected to score differently. As an example, if you were to give Democrats and Republicans a test assessing the strength of certain political views, you would expect them to score differently. Various sub-categories of validity (external, internal, statistical and construct) are also important in some contexts. Validity rating is not overly objective; in fact, it is relatively subjective in some areas. There isn't a perfect validity.

It is possible to have a reliable but not valid measure. However, a valid measure is always a reliable measure,

Often, when using unsystematic (non-scientific) approaches to knowledge measures are not reliable or valid. That is, they do not measure the trait or characteristic of interest consistently nor do they measure what they are intended to measure. Quality scientific approaches generally make great efforts to ensure reliability and validity.

What about Replication in Science??

Replicable (reproducible) findings are important to science; they are a sub-component of converging evidence. When referring to the replication crisis it is important to understand that what is meant- is lack of replicating statistically significant findings. It would be more precise to say there is a "statistically significant replication crisis." Consider replication from another perspective; the original study failed to detect stat...sign.. (using criteria NHST prevalent with use of frequentist stats), but additional studies detect statistical significance.  What would the implications be??  College instructors should make an effort to address this condition- non-significant precedes significant findings. Students are often advised no need to try to replicate non-significant findings, but sign..findings should be replicated. This implies that the non-sign....findings must be accurate (if they occurred first), even though all studies are susceptible to flaws.  Read more 

Learn more about the need for science, rationality and statistics  - In Evidence We Trust