Internal Medicine Portal: Archived - Journal Club Blog

Block 2 RCT Wrap-Up - Significance (9/2020)

RCT Wrap-up (Block 2) - Significance

In this block, we looked at critically appraising RCTs with a special focus on bias & confounding, power, and clinical vs statistical significance. Residents used a variety of well-known trials (ACCORD-BP, SPRINT, ANDROMEDA, and HYVET) to put these concepts to practice. Below is a summary of concepts related to significance. 

One of the common problems faced by readers (and authors!) of medical articles is in the interpretation of the word “significance.” The term “statistical significance” is often misinterpreted as a “clinically important” result. The confusion stems from the fact that many people equate “significance” with its literal meaning of “importance,” whereas in statistics, it has a far more restrictive connotation

Ranganathan 2015 (PMID: 26229754)

Statistical significance - implies that the observed result, or a more extreme result, is unlikely to occur by chance alone and that the groups are therefore likely to truly differ

• Lower significance levels indicate that you require stronger evidence before you will reject the null hypothesis.
• The significance level, also denoted as alpha or α, is the probability of rejecting the null hypothesis when it is true. A significance level of 0.05 indicates a 5% risk of concluding that a difference exists when there is no actual difference.

Clinical significance -  refers to the magnitude of the actual treatment effect, which will determine whether the results of the trial are likely to impact current medical practice

• In clinical research, study results, which are statistically significant are often interpreted as being clinically important. While statistical significance indicates the reliability of the study results, clinical significance reflects its impact on clinical practice.
• Treatment effect - difference between the intervention and control groups,

4 options of significance:

1. Statistically significant (p<0.05), not clinically significant
   1. "Statistical significance is heavily dependent on the study's sample size; with large sample sizes, even small treatment effects (which are clinically inconsequential) can appear statistically significant"
2. Not statistically significant (p>0.05) but clinically significant
3. Both
4. Neither

To avoid falling in the trap of thinking that because a result is statistically significant it must also be clinically important, you can look out for a few things…

1. Look to see if the authors have specifically mentioned whether the differences they have observed are clinically important or not.
2. Take into account sample size: be particularly aware that with very large sample sizes even small, unimportant differences may become statistically significant.
3. Take into account effect size. In general, the larger the effect size you have, the more likely it is that difference will be meaningful to patients.