article

The ability to critically understand & judge the data from a study is crucial in making decisions on whether a new drug is safe & effective. As we will see with studies regarding nootropics, the answer is not always clear. Understanding concepts of validity, bias, & limitation can help in the evaluation of any study.

Internal validity

Internal validity is the strength of the study’s purported causal or associative relationship.

Higher level studies, such as randomised controlled trials & meta-analyses, seek to demonstrate a causal relationship (e.g. drug A causes improved cognitive function). Lower level studies, such as cohort studies & case-control studies provide evidence that demonstrates an association between a cause & effect (not as strong of an assumption: drug A is associated with improved cognitive function). The tighter the study’s internal validity, the more reliance we can have that drug A does indeed cause or is associated with improved cognitive function, rather than any other conclusion (i.e. has no effect on, worsens cognitive function).

Bias

Bias comprises confounding factors which may compromise a study’s internal validity.

For example, consider a study with 2 treatment groups testing the effect of a new drug on cognitive function. The group that receives the drug is generally more educated, while the group that receives placebo is generally less educated.

How much faith would you have if the investigators concluded that the drug significantly improved cognitive function?

This is an example of sample selection bias. We will introduce more forms of bias later that could impair internal validity & thus the ability to truly believe that a study’s results are relevant to the study question.

External validity

External validity is the generalisability of a study’s findings to populations beyond the study sample.

External validity should only be assessed after a study is found to be internally valid. If a study is not found to be internally valid, then its findings could not be said to truly answer the study question; & thus there would be no reason to evaluate whether its results should be generalised to others.While internal validity is susceptible to bias, external validity is counterbalanced by limitation. These are characteristics of the study sample which add to & restrict the population to whom the results may be generalised.

Always consider the type of people who were enrolled in the study & whether what worked (or didn’t work) for them would work for you.

Consider the previous example of a study, but with both groups comprising generally older subjects otherwise comparable at baseline. Absent other confounding variables, the study could be said to be internally valid: if the investigators reported a significant improvement in cognitive function, then this result would be probably accurate. However, whether we could assume that this drug would work for younger individuals would be up for question as it has not been tested in this population. This limitation of generalisability applies to other demographic information such as race, sex, & even geographic location, & may include comorbidity (the presence of other health conditions), diet, & other factors depending on how the data is to be used. Including more diverse individuals within a study may decrease limitations & increase external validity, but possibly at the expense of internal validity (without randomisation). Conversely, creating a more uniform sample could increase internal validity but introduce more limitations.

Other forms of bias

Other forms of bias that could impair internal validity include:

Sample selection bias is when the treatment groups are not equal at baseline due to demographic differences.
Intervention selection bias is present if different forms of the experimental variable are used. This is a risk when the study protocol is ambiguous; for this reason study protocols are usually very detailed so as to prevent deviation. Consider the previous study with 2 treatment groups. Assume the groups are balanced at baseline. However, among the individuals in the group receiving the study drug, two different manufacturers of the drug are used. This could potentially produce variation in the results, providing an imperfect picture of how well the drug actually works. To limit this type of bias, it would be more prudent to select one manufacturer or have two treatment groups (one for each manufacturer).
Measurement bias is when there exist variations in how outcomes are measured. If the study drug group took an easier cognitive test than the placebo group, then the results would show that they performed better, when in fact the comparison was not equal.
Outcome bias occurs if the selected endpoints do not correspond to the desired outcome of interest. For instance, if the researchers claiming to measure cognitive performance instead administered a personality test.
Attrition bias is when more subjects from one group leave the study than in the other. Although the two groups may have been equal at baseline, attrition may result in unequal groups later on in the study which can result in confounding due to imbalanced characteristics (e.g. if all the young subjects left from one group) or simply due to number (sample size too small to detect a difference).

When cognitive tests are administered to a group of subjects, two possible biases could confound the results:

Statistical regression to the mean is a type of bias wherein on the second administration of the same test to the same subjects, the worst performers from the first administration tend to perform better & the best performers tend to perform worse on the second exam. They regress to the mean.
The testing effect is when subjects who take the same test become familiar with the style & better at taking the test. Although they might perform better on subsequent applications of the same cognitive test, it may not be because the study drug resulted in cognitive improvement.
Other forms of biases which are less commonly implicated but could still undermine a study’s findings include maturation bias & history bias.

Summary

Validity, bias, & limitations are key aspects of study designs to consider when researching & evaluating clinical data. The strength of evidence is best with high internal & external validity, & low risk of bias.
Internal validity is the strength of a study design to determine a causal or associative relationship. Studies with highly controlled experimental methodology (which we will discuss in depth later) exhibit tight internal validity minimizing the effect of biases.
External validity is the extent to which findings from internally valid studies may be generalized to populations beyond the study sample. Being aware of limitations to external validity guides the extrapolation of study data.

If you are interested in reading more about validity & bias, & how to apply them when reading an article, I highly recommend the Cochrane Foundation’s tool for bias risk assessment. It has since been widely used in meta-analyses when deciding whether to include articles.

A major part of the intrigue surrounding nootropics has to do with the fact that many of these compounds have not been largely studied. In effect, we are gambling with our neurochemistry in order to gain some benefit in our mental functions. But we are not without resources which can improve our chances of using nootropics safely & effectively. While the body of evidence behind nootropic agents is not large, it is growing, & will likely continue to grow with increasing rapidity as public interest in nootropics increases. Drawing upon this background of research can help us understand how nootropic agents work, in whom they work, & what the risks are. To this end, I will be launching a multi-part tutorial on how to understand & interpret clinical trials, designed for both novices & more advanced users. Topics we’ll cover include:

Validity & bias
Types of studies & hierarchy of evidence
Reading results (graphs & tables)
Methodology
Basic biostatistics

How to access clinical research

A Pubmed search with limitations for clinical trials, free full text, & publication within the last 5 years.

First of all, clinical research is generally accessible by online databases through universities or hospitals. If you have access to some of these databases through your institution, such as Medline or Ebscohost, I highly recommend familiarizing yourself with them. For those without institutional database access, Pubmed offers a large open-access catalog of many journal articles. Google Scholar is another option for finding studies, but does not offer advanced search functions.

Pubmed screenshot 2 — The abstract is shown, as well as full-text links on the right side.

Here are some other tips when searching for journal articles:

I would generally recommend limiting one’s search to full text articles, as abstracts do not often reveal the full story of a study.
Searching by MeSH terms (analogous to tags for topics) usually provides more relevant results than a basic keyword search. This involves searching for a MeSH term, selecting it, then running the search.
More recent results (preferably within the last 5 years) are preferable, as scientific research can move fast. Depending on the topic area, however, you might find yourself stretching your search to include up to 10 years.
Be aware of the country of origin of the article, as standards for publication may vary.
Authors who write many articles on the same topic may be biased &/or highly knowledgeable.

Pubmed screenshot 3 — Select the desired MeSH term, add it to your search on the right side, then run your search.

Structure of a journal article

So you have located a journal article of interest. Fortunately, every journal article generally follows a similar structure. This organisation is designed to present the details of the study in an intuitive order.

The abstract is a short summary of study’s methods & results.
The background section reviews the current state of understanding in the topic area of interest. The investigators conducting the study also explain what they are trying to show.
In the methods portion, key details of how the study works are defined:
- Endpoints or outcomes are what is being measured, such as performance on a cognitive test
- Experimental variables are what is being tested, such as the study drug & the control against which it is compared (placebo or standard treatment)
- The type of individuals the investigators wanted to analyse in their study as test subjects
- The allocation or assignment of enrolled individuals to either treatment groups (who receive the study drug) or control groups is typically visualized in a flowchart
- Statistical tests used to analyse the data
The results section is where authors list their findings only (without interpretation). These include:
- Baseline characteristics– a description of the final sample. Most often, this is summarized in table labelled as Table 1. When reading this section, think about the age, race, geography, & health status of the sample, & whether they are similar to you.
- Outcomes– how did people who took the drug do in comparison to those who took the control? These data will be presented in tables, charts, graphs, & text.
Considered to be the most important section, the discussion area is where investigators interpret the results- what they mean, whether they are significant, where there could be error, the weaknesses of their study, & areas for further research. What is stated in this section can sometimes be highly contentious.

Some tips for reading an article:

The background section is not usually necessary unless if the topic area is new to the reader- if you are extensively researching a drug by reading multiple articles, you will find that many of their background sections are similar. However, if you don’t understand what’s covered in the background section, bring yourself up to speed with other resources such as Wikipedia.
Some prefer to read the abstract first to obtain a rapid summary of the study, then the discussion second for a detailed look at how the authors felt about the findings.
Always compare the raw numbers from the results section against the authors’ interpretation in the discussion section. Never take what the authors state at face value. Do the numbers actually show what they claim is happening?
Yes, the word «data» is plural.
I personally prefer to print out the pdf article & write comments on the hard copy as I read.
It’s not uncommon to read the same article several times. These subjects are quite advanced, & many details are important.
Check the articles cited in the bibliography for other studies that might be related to your topic.

Table 1. — A (very short) table 1. More meticulous studies will list more baseline characteristics of test subjects.

Summary

In review, using clinical data can provide a powerful edge when making decisions about nootropics. The informed nootropic user is better able to discern which nootropics are safe & effective.

Clinical research is accessible within databases which are offered through institutions. The general public can access some research through resources such as Pubmed or Google Scholar.
All journal articles follow the same general format consisting of an abstract, background, methods, results, & discussion sections. Knowing where to find what information you need within an article can make reading articles faster.