Tag: validity

Categories
Tutorials

How to Understand Clinical Research, Part II: Quality of Evidence

The ability to critically understand & judge the data from a study is crucial in making decisions on whether a new drug is safe & effective. As we will see with studies regarding nootropics, the answer is not always clear. Understanding concepts of validity, bias, & limitation can help in the evaluation of any study.

Internal validity

Internal validity is analogous to the inner workings of a clock
Internal validity is analogous to the inner workings of a clock– a study with strong internal validity will produce results that truly reflect what the investigators sought to explore, like how a clock with well-adjusted inner workings will accurately display the time.

Internal validity is the strength of the study’s purported causal or associative relationship.

Higher level studies, such as randomised controlled trials & meta-analyses, seek to demonstrate a causal relationship (e.g. drug A causes improved cognitive function). Lower level studies, such as cohort studies & case-control studies provide evidence that demonstrates an association between a cause & effect (not as strong of an assumption: drug A is associated with improved cognitive function). The tighter the study’s internal validity, the more reliance we can have that drug A does indeed cause or is associated with improved cognitive function, rather than any other conclusion (i.e. has no effect on, worsens cognitive function).

Bias

Bias comprises confounding factors which may compromise a study’s internal validity.

For example, consider a study with 2 treatment groups testing the effect of a new drug on cognitive function. The group that receives the drug is generally more educated, while the group that receives placebo is generally less educated.

How much faith would you have if the investigators concluded that the drug significantly improved cognitive function?

This is an example of sample selection bias. We will introduce more forms of bias later that could impair internal validity & thus the ability to truly believe that a study’s results are relevant to the study question.

External validity

External validity is the generalisability of a study’s findings to populations beyond the study sample.

External validity should only be assessed after a study is found to be internally valid. If a study is not found to be internally valid, then its findings could not be said to truly answer the study question; & thus there would be no reason to evaluate whether its results should be generalised to others.While internal validity is susceptible to bias, external validity is counterbalanced by limitation. These are characteristics of the study sample which add to & restrict the population to whom the results may be generalised.

Always consider the type of people who were enrolled in the study
Always consider the type of people who were enrolled in the study & whether what worked (or didn’t work) for them would work for you.

Consider the previous example of a study, but with both groups comprising generally older subjects otherwise comparable at baseline. Absent other confounding variables, the study could be said to be internally valid: if the investigators reported a significant improvement in cognitive function, then this result would be probably accurate. However, whether we could assume that this drug would work for younger individuals would be up for question as it has not been tested in this population. This limitation of generalisability applies to other demographic information such as race, sex, & even geographic location, & may include comorbidity (the presence of other health conditions), diet, & other factors depending on how the data is to be used. Including more diverse individuals within a study may decrease limitations & increase external validity, but possibly at the expense of internal validity (without randomisation). Conversely, creating a more uniform sample could increase internal validity but introduce more limitations.

Other forms of bias

Other forms of bias that could impair internal validity include:

  • Sample selection bias is when the treatment groups are not equal at baseline due to demographic differences.
  • Intervention selection bias is present if different forms of the experimental variable are used. This is a risk when the study protocol is ambiguous; for this reason study protocols are usually very detailed so as to prevent deviation. Consider the previous study with 2 treatment groups. Assume the groups are balanced at baseline. However, among the individuals in the group receiving the study drug, two different manufacturers of the drug are used. This could potentially produce variation in the results, providing an imperfect picture of how well the drug actually works. To limit this type of bias, it would be more prudent to select one manufacturer or have two treatment groups (one for each manufacturer).
  • Measurement bias is when there exist variations in how outcomes are measured. If the study drug group took an easier cognitive test than the placebo group, then the results would show that they performed better, when in fact the comparison was not equal.
  • Outcome bias occurs if the selected endpoints do not correspond to the desired outcome of interest. For instance, if the researchers claiming to measure cognitive performance instead administered a personality test.
  • Attrition bias is when more subjects from one group leave the study than in the other. Although the two groups may have been equal at baseline, attrition may result in unequal groups later on in the study which can result in confounding due to imbalanced characteristics (e.g. if all the young subjects left from one group) or simply due to number (sample size too small to detect a difference).

When cognitive tests are administered to a group of subjects, two possible biases could confound the results:

  • Statistical regression to the mean is a type of bias wherein on the second administration of the same test to the same subjects, the worst performers from the first administration tend to perform better & the best performers tend to perform worse on the second exam. They regress to the mean.
  • The testing effect is when subjects who take the same test become familiar with the style & better at taking the test. Although they might perform better on subsequent applications of the same cognitive test, it may not be because the study drug resulted in cognitive improvement.
  • Other forms of biases which are less commonly implicated but could still undermine a study’s findings include maturation bias & history bias.

Summary

  • Validity, bias, & limitations are key aspects of study designs to consider when researching & evaluating clinical data. The strength of evidence is best with high internal & external validity, & low risk of bias.
  • Internal validity is the strength of a study design to determine a causal or associative relationship. Studies with highly controlled experimental methodology (which we will discuss in depth later) exhibit tight internal validity minimizing the effect of biases.
  • External validity is the extent to which findings from internally valid studies may be generalized to populations beyond the study sample. Being aware of limitations to external validity guides the extrapolation of study data.

If you are interested in reading more about validity & bias, & how to apply them when reading an article, I highly recommend the Cochrane Foundation’s tool for bias risk assessment. It has since been widely used in meta-analyses when deciding whether to include articles.