Tag: clinical

Categories
Tutorials

How to Understand Clinical Research, Part II: Quality of Evidence

The ability to critically understand & judge the data from a study is crucial in making decisions on whether a new drug is safe & effective. As we will see with studies regarding nootropics, the answer is not always clear. Understanding concepts of validity, bias, & limitation can help in the evaluation of any study.

Internal validity

Internal validity is analogous to the inner workings of a clock
Internal validity is analogous to the inner workings of a clock– a study with strong internal validity will produce results that truly reflect what the investigators sought to explore, like how a clock with well-adjusted inner workings will accurately display the time.

Internal validity is the strength of the study’s purported causal or associative relationship.

Higher level studies, such as randomised controlled trials & meta-analyses, seek to demonstrate a causal relationship (e.g. drug A causes improved cognitive function). Lower level studies, such as cohort studies & case-control studies provide evidence that demonstrates an association between a cause & effect (not as strong of an assumption: drug A is associated with improved cognitive function). The tighter the study’s internal validity, the more reliance we can have that drug A does indeed cause or is associated with improved cognitive function, rather than any other conclusion (i.e. has no effect on, worsens cognitive function).

Bias

Bias comprises confounding factors which may compromise a study’s internal validity.

For example, consider a study with 2 treatment groups testing the effect of a new drug on cognitive function. The group that receives the drug is generally more educated, while the group that receives placebo is generally less educated.

How much faith would you have if the investigators concluded that the drug significantly improved cognitive function?

This is an example of sample selection bias. We will introduce more forms of bias later that could impair internal validity & thus the ability to truly believe that a study’s results are relevant to the study question.

External validity

External validity is the generalisability of a study’s findings to populations beyond the study sample.

External validity should only be assessed after a study is found to be internally valid. If a study is not found to be internally valid, then its findings could not be said to truly answer the study question; & thus there would be no reason to evaluate whether its results should be generalised to others.While internal validity is susceptible to bias, external validity is counterbalanced by limitation. These are characteristics of the study sample which add to & restrict the population to whom the results may be generalised.

Always consider the type of people who were enrolled in the study
Always consider the type of people who were enrolled in the study & whether what worked (or didn’t work) for them would work for you.

Consider the previous example of a study, but with both groups comprising generally older subjects otherwise comparable at baseline. Absent other confounding variables, the study could be said to be internally valid: if the investigators reported a significant improvement in cognitive function, then this result would be probably accurate. However, whether we could assume that this drug would work for younger individuals would be up for question as it has not been tested in this population. This limitation of generalisability applies to other demographic information such as race, sex, & even geographic location, & may include comorbidity (the presence of other health conditions), diet, & other factors depending on how the data is to be used. Including more diverse individuals within a study may decrease limitations & increase external validity, but possibly at the expense of internal validity (without randomisation). Conversely, creating a more uniform sample could increase internal validity but introduce more limitations.

Other forms of bias

Other forms of bias that could impair internal validity include:

  • Sample selection bias is when the treatment groups are not equal at baseline due to demographic differences.
  • Intervention selection bias is present if different forms of the experimental variable are used. This is a risk when the study protocol is ambiguous; for this reason study protocols are usually very detailed so as to prevent deviation. Consider the previous study with 2 treatment groups. Assume the groups are balanced at baseline. However, among the individuals in the group receiving the study drug, two different manufacturers of the drug are used. This could potentially produce variation in the results, providing an imperfect picture of how well the drug actually works. To limit this type of bias, it would be more prudent to select one manufacturer or have two treatment groups (one for each manufacturer).
  • Measurement bias is when there exist variations in how outcomes are measured. If the study drug group took an easier cognitive test than the placebo group, then the results would show that they performed better, when in fact the comparison was not equal.
  • Outcome bias occurs if the selected endpoints do not correspond to the desired outcome of interest. For instance, if the researchers claiming to measure cognitive performance instead administered a personality test.
  • Attrition bias is when more subjects from one group leave the study than in the other. Although the two groups may have been equal at baseline, attrition may result in unequal groups later on in the study which can result in confounding due to imbalanced characteristics (e.g. if all the young subjects left from one group) or simply due to number (sample size too small to detect a difference).

When cognitive tests are administered to a group of subjects, two possible biases could confound the results:

  • Statistical regression to the mean is a type of bias wherein on the second administration of the same test to the same subjects, the worst performers from the first administration tend to perform better & the best performers tend to perform worse on the second exam. They regress to the mean.
  • The testing effect is when subjects who take the same test become familiar with the style & better at taking the test. Although they might perform better on subsequent applications of the same cognitive test, it may not be because the study drug resulted in cognitive improvement.
  • Other forms of biases which are less commonly implicated but could still undermine a study’s findings include maturation bias & history bias.

Summary

  • Validity, bias, & limitations are key aspects of study designs to consider when researching & evaluating clinical data. The strength of evidence is best with high internal & external validity, & low risk of bias.
  • Internal validity is the strength of a study design to determine a causal or associative relationship. Studies with highly controlled experimental methodology (which we will discuss in depth later) exhibit tight internal validity minimizing the effect of biases.
  • External validity is the extent to which findings from internally valid studies may be generalized to populations beyond the study sample. Being aware of limitations to external validity guides the extrapolation of study data.

If you are interested in reading more about validity & bias, & how to apply them when reading an article, I highly recommend the Cochrane Foundation’s tool for bias risk assessment. It has since been widely used in meta-analyses when deciding whether to include articles.

Categories
Tutorials

How to Understand Clinical Research, Part IV: Sample Data

In previous articles, we have learned how to access & read journal articles, how to interpret validity as a measure of the strength of evidence, and the various types of studies. With this foundation, we should have a basic idea of what to look for in a study. The next step would be developing our skills of interpreting the actual data from the article, which can be expressed as charts, tables, graphs, or text. The next two articles will be focused on the two major types of results in studies: sample data & endpoint data.

Sample Data

people-312122_1280

Sample data are information about the subjects who participated in the study. Although not what most people think of when they think of study results, sample data can profoundly influence one’s interpretation of endpoint data.

The first thing to consider in sample data is the sample size (n), or how many people participated in the study. Ideally, studies should enrol enough individuals to detect a treatment effect. For example, drug A may slightly improve cognitive function, but perhaps only in 15% of people. If only 50 patients are enrolled in the study, & 25 receive the study drug while the other half receives placebo, then only 3.75 individuals will likely experience the slight improvement in cognitive function. This small change is unlikely to be detected, especially when tested against a placebo effect.

How many subjects are needed, then, in order to truly detect a treatment effect? There exist statistical tests which guide investigators in all aspects of study design, including calculating the minimum sample size needed to detect a treatment effect. This process usually involves:

  1. estimating the magnitude of the treatment effect based upon previous studies,
  2. estimating the number of subjects needed to truly see that treatment effect, &
  3. minimizing the effects of chance & randomness

The first two parts should seem fairly intuitive after the previous example. The third requirement is a new concept, & is based on the following components (This is an area of biostatistics that gets a little complicated with over-thinking, but can be understood with enough time. Comment if you have questions!):

  • The hypothesis (H1) is what is predicted to happen in the experiment. It should be explicitly defined, such as: drug A shortens rapid recall time in comparison to placebo.
  • The null hypothesis (H0) is the opposite of the hypothesis- it is what is predicted to happen if there is no treatment difference: drug A does not affect rapid recall time in comparison to placebo.

Next we have the concepts of true positives/negatives & false positives/negatives. This is usually discussed in terms of the null hypothesis, which can sometimes be confusing if you overthink it.

Screen Shot 2015-09-21 at 18.39.04

The leftmost column represents the real result. If drug A really shortens rapid recall time, then in the best case scenario, the researchers will detect this difference & claim a treatment difference. This means rejecting the null hypothesis & accepting the hypothesis. If they fail to detect this difference (maybe because they did not enrol enough participants) then we would see the false negative scenario.

On the other hand, if drug A really is ineffective, then we would either see the true negative scenario or a false positive scenario. Both the false positive & false negative scenarios emerge out of chance & confounding (e.g. some subjects on the placebo have a good day while taking the recall test, or some start taking phenylpiracetam without telling the investigators, or some subjects in the treatment group take the recall test after a night of heavy drinking; these are all examples of biases that could compromise internal validity).

  • A false positive is called a type I error. The chance of a type I error happening is denoted as α (alpha).
  • A false negative is called a type II error. The chance of a type II error happening is denoted as β (beta).
  • When β is subtracted from 1 (1 – β), this difference is called the power of a study. The power is the chance of finding a true treatment effect (true positive). To bring the discussion back to prior to this talk on hypotheses, true & false results, & errors, power is a key component in determining what sample size we need to detect a treatment effect. The standard power in a study is 80% (1 – β = 0.80). At 80% power, we have an 80% chance of correctly rejecting the H0 & claiming that there is a treatment difference when, in fact, there is one. We achieve sufficient power by enrolling enough subjects, as power & sample size are proportionally related (⊕ n → ⊕ 1 – β ).

*Although power higher than 80% is possible, usually it is not done because:

  • Initial increases in sample size lead to higher increases in power than further increases, as shown in the following graph.
  • Enrolling many subjects requires a lot of funding.
  • α & β are inversely proportional- that means that as you increase your power (1 – β) & β decreases, then α will increase- meaning there is a higher risk of a false positive. At 80% power, there is a 20% chance of a false negative & a 5% chance of a false positive. 80% power is considered a balance between the type I error rate & the type II error rate.
Initial increases in sample size lead to more substantial increases in power than later increases. Select a sample size where power ≥ 0.8 for the best chance of detecting the treatment effect- here it is a little less than 120 patients. Enrolling fewer patients would lead to a higher risk of a false positive.

So, we have established that power is a key driver of how many patients to enroll. We need to enroll enough patients so that the treatment effect does emerge & we can reliably detect it, with minimal concern for confounding & chance.

A few other odds & ends to consider in sample data:

  • Ideally, all treatment groups will be equal or similar in size
  • All treatment groups should remain similar throughout the study- if more subjects drop out of one group than another, that could lead to attrition bias. If a disproportionate amount of patients drops out from the study drug group, that should be a warning sign– investigate why they dropped out (side effects?).
  • Baseline characteristics are traits of the subjects in the study at the beginning before the study drug or placebo has been administered. These data, as previously mentioned, are typically presented in a standard table 1. It is important to skim this table for (1) significant differences between the groups & (2) generalisability/external validity- whether the results could be applied to you.
modafinil vs placebo
Discussion of power in a small RCT comparing modafinil vs. placebo. [1]

Summary

  • Sample data describe the characteristics of participants in a study.
  • Sample size (n) is the amount of subjects that are enrolled in a study. It is far from an arbitrary number– the determination of how many people are needed in a study is predicated upon the anticipated magnitude of the treatment effect, the number of subjects needed to have that effect emerge, & minimising «statistical noise» by reducing the risk of false positives (α) & false negatives (β).
  • Power (1 – β) is the chance of a true positive. Researchers typically prefer 80% power as it balances the chance of a false negative with the chance of a false positive (both error rates cannot be controlled simultaneously). Power is achieved by increasing the number of patients in a study. For those interested, power & its determinants can be mathematically expressed as: ⊕ Δ , – σ, ⊕ n → ⊕ δ → ⊕ ( 1 – β ) where Δ is the true difference between groups, σ is the standard deviation, n is the sample size, & δ is the non-centrality parameter. An equation form would be δ = ( Δ ÷ σ ) √ ( n ÷ 2 )
  • Other key sample data to consider are the baseline, interim, & final sample sizes between the groups as well as the baseline characteristics.

References   [ + ]

Categories
Tutorials

How to Understand Clinical Research, Part III: Types of Studies

In clinical research, there exist different types of studies which serve particular purposes. These studies are distinguished based on their experimental design (how the study is conducted) & the kind of data they produce– from the way a study is designed, we can draw certain expectations about the grade of evidence it produces.

Experimental designs can be described in several ways. A basic division of study designs can be made on how test subjects are enrolled, which significantly determines the study’s strength in describing a relationship between a cause (an experimental variable such as a drug to be tested) & an effect (an outcome such as cognitive performance). This particular way of classifying studies results in two main families of studies: observational studies & assignment studies.

Observational Studies

Observational studies are conducted in order to determine associations between certain prior exposures (e.g. a drug) & outcomes of interest (e.g. death).

Here, participants are selected based on exposure or outcome, depending on the type of observational study. These are more prevalent in research on nootropics as randomised controlled trials (RCTs) are generally conducted with larger samples requiring more funding. Small observational studies build up a body of evidence which provide the grounds for an RCT, a process called «hypothesis-generating» (as opposed to hypothesis-testing). Cohort studies & case-control studies are two major types of observational studies, both of which involve following a group of patients over a period of time.

Subjects in cohort studies are selected based on their having received a particular exposure, then they are followed prospectively (forward in time) until a certain outcome of interest (e.g. death) occurs. RCTs are also prospective.

Depiction of observational studies. Prospective cohort studies work from the perspective of the left viewer, while retrospective case-control studies work from the perspective of the right viewer.

In a case-control study, subjects are selected based on their exhibiting a certain outcome & tracing their history back (retrospectively) to find out whether they have had a certain exposure (e.g. used a particular drug). Retrospective case-control studies are especially useful when studying rare diseases.

Assignment Studies

Assignment studies enroll subjects to either test or control groups.

Assignment studies are subdivided based on (1) whether the allocation of subjects into test groups is randomised & (2) whether a control group is present.

Randomised controlled trials (RCTs) are generally considered to be the gold standard of clinical evidence for their strong internal validity, & are used to demonstrate causal relationships between experimental variables & outcomes. Randomly assigning patients to either treatment or control groups theoretically establishes equal groups, as any differences in age, race, comorbidity, or other features are equally distributed (eliminates sample selection bias). Prospective follow-up & the presence of a control group allows for comparison of the experimental variable (e.g. a new drug) against a standard treatment (to demonstrate a better treatment effect) or placebo (to demonstrate a treatment effect).

Mohamed AD, Lewis CR. Modafinil increases the latency of response in the Hayling Sentence Completion Test in healthy volunteers: a randomised controlled trial. PLoS One. 2014 Nov 12;9(11):e110639.
Standard CONSORT diagram depicting enrolment, allocation, follow-up, & analysis of subjects from an RCT comparing modafinil vs. placebo.

Synthetic Studies

Systematic reviews & meta-analyses critically evaluate the literature by consolidating the results of several studies focused on the same topic.

Synthetic studies are more recent study designs that have been developed out of a need to draw from the existing evidence on a topic. Prior to the rise of systematic reviews & meta-analyses, studies were selectively cited which led to bias (e.g. selecting only the studies which supported the use of a drug & either intentionally or unwittingly omitting the others which found significant side effects). Nowadays, both are considered the highest forms of clinical evidence, producing strong inferences of treatment effects. Synthetic studies are part of the trend of comparative efficacy analyses (CEAs): gathering data on several major drugs used for the same purpose & determining which are superior. Collecting findings from multiple RCTs & observational studies can produce a more complete picture of a drug’s safety & efficacy- in other words, considering the ‘big picture’. However, before drawing conclusions, one must be cognisant of differences between the studies that have been gathered (e.g. study protocol, different doses used, different sample characteristics).

Systematic reviews present findings from a pre-defined, reproducible search of the literature- that is, the authors exhaustively describe the methods they used to search databases & how they selected which studies to include in their systematic review, usually with a pre-defined criteria set. The importance of reproducibility is to reduce bias from selective inclusion of studies– this is a weakness of narrative reviews, in which the author performs a search & simply chooses which studies to include.

Meta-analyses are systematic reviews where the gathered data is then combined, producing an estimate of the true treatment effect from the pooled data. This is commonly expressed in what’s called a Forest plot, which shows the individual trials included in the systematic review as well as a diamond representing the estimate of the true efficacy or safety measure of the drug (how to read & interpret different tables & graphs will be covered later!).

An example Forest plot, with included trials listed on the left & their findings on the right. The findings are also plotted, with the diamond representing the composite of the studies’ results.

Summary

As we have seen, the design of a clinical trial can provide a quick way to judge its findings.

  • Observational studies produce evidence of associations by following patients over a period of time. Patients are selected based on a specific previous exposure in the case of prospective cohort studies or for a certain outcome in the case of retrospective case-control studies.
  • Assignment studies produce evidence of causal relationships by assigning patients to multiple groups including a comparator arm. The most prominent example of an assignment study is the randomised control trial, which has become the standard for clinical data.
  • Synthetic evaluations of the literature, such as systematic reviews & meta-analyses, draw on existing studies to better approximate treatment effects of drugs of interest.
  • Other experimental designs which are less commonly relevant to the area of nootropics include cross-sectional observational studies & non-randomised controlled trials.
Categories
Tutorials

How to Understand Clinical Research, Part I: Accessing and Reading Research

researchA major part of the intrigue surrounding nootropics has to do with the fact that many of these compounds have not been largely studied. In effect, we are gambling with our neurochemistry in order to gain some benefit in our mental functions. But we are not without resources which can improve our chances of using nootropics safely & effectively. While the body of evidence behind nootropic agents is not large, it is growing, & will likely continue to grow with increasing rapidity as public interest in nootropics increases. Drawing upon this background of research can help us understand how nootropic agents work, in whom they work, & what the risks are. To this end, I will be launching a multi-part tutorial on how to understand & interpret clinical trials, designed for both novices & more advanced users. Topics we’ll cover include:

  • Validity & bias
  • Types of studies & hierarchy of evidence
  • Reading results (graphs & tables)
  • Methodology
  • Basic biostatistics

How to access clinical research

Pubmed search
A Pubmed search with limitations for clinical trials, free full text, & publication within the last 5 years.

First of all, clinical research is generally accessible by online databases through universities or hospitals. If you have access to some of these databases through your institution, such as Medline or Ebscohost, I highly recommend familiarizing yourself with them. For those without institutional database access, Pubmed offers a large open-access catalog of many journal articles. Google Scholar is another option for finding studies, but does not offer advanced search functions.

Pubmed screenshot 2
The abstract is shown, as well as full-text links on the right side.

Here are some other tips when searching for journal articles:

  • I would generally recommend limiting one’s search to full text articles, as abstracts do not often reveal the full story of a study.
  • Searching by MeSH terms (analogous to tags for topics) usually provides more relevant results than a basic keyword search. This involves searching for a MeSH term, selecting it, then running the search.
  • More recent results (preferably within the last 5 years) are preferable, as scientific research can move fast. Depending on the topic area, however, you might find yourself stretching your search to include up to 10 years.
  • Be aware of the country of origin of the article, as standards for publication may vary.
  • Authors who write many articles on the same topic may be biased &/or highly knowledgeable.
Pubmed screenshot 3
Select the desired MeSH term, add it to your search on the right side, then run your search.

Structure of a journal article

So you have located a journal article of interest. Fortunately, every journal article generally follows a similar structure. This organisation is designed to present the details of the study in an intuitive order.

  • The abstract is a short summary of study’s methods & results.
  • The background section reviews the current state of understanding in the topic area of interest. The investigators conducting the study also explain what they are trying to show.
  • In the methods portion, key details of how the study works are defined:
    • Endpoints or outcomes are what is being measured, such as performance on a cognitive test
    • Experimental variables are what is being tested, such as the study drug & the control against which it is compared (placebo or standard treatment)
    • The type of individuals the investigators wanted to analyse in their study as test subjects
    • The allocation or assignment of enrolled individuals to either treatment groups (who receive the study drug) or control groups is typically visualized in a flowchart
    • Statistical tests used to analyse the data
  • The results section is where authors list their findings only (without interpretation). These include:
    • Baseline characteristics– a description of the final sample. Most often, this is summarized in table labelled as Table 1. When reading this section, think about the age, race, geography, & health status of the sample, & whether they are similar to you.
    • Outcomes– how did people who took the drug do in comparison to those who took the control? These data will be presented in tables, charts, graphs, & text.
  • Considered to be the most important section, the discussion area is where investigators interpret the results- what they mean, whether they are significant, where there could be error, the weaknesses of their study, & areas for further research. What is stated in this section can sometimes be highly contentious.

Some tips for reading an article:

  • The background section is not usually necessary unless if the topic area is new to the reader- if you are extensively researching a drug by reading multiple articles, you will find that many of their background sections are similar. However, if you don’t understand what’s covered in the background section, bring yourself up to speed with other resources such as Wikipedia.
  • Some prefer to read the abstract first to obtain a rapid summary of the study, then the discussion second for a detailed look at how the authors felt about the findings.
  • Always compare the raw numbers from the results section against the authors’ interpretation in the discussion section. Never take what the authors state at face value. Do the numbers actually show what they claim is happening?
  • Yes, the word «data» is plural.
  • I personally prefer to print out the pdf article & write comments on the hard copy as I read.
  • It’s not uncommon to read the same article several times. These subjects are quite advanced, & many details are important.
  • Check the articles cited in the bibliography for other studies that might be related to your topic.
Table 1.
A (very short) table 1. More meticulous studies will list more baseline characteristics of test subjects.

Summary

In review, using clinical data can provide a powerful edge when making decisions about nootropics. The informed nootropic user is better able to discern which nootropics are safe & effective.

  • Clinical research is accessible within databases which are offered through institutions. The general public can access some research through resources such as Pubmed or Google Scholar.
  • All journal articles follow the same general format consisting of an abstract, background, methods, results, & discussion sections. Knowing where to find what information you need within an article can make reading articles faster.