When reading papers or doing research and hitting a non-significant result, I always remember what my statistics teacher said: “Not significant means we cannot reject the null hypothesis, not that the two are the same.” We’ve also learned that the ability to detect a difference between the two is called statistical power. So, if we calculate the power of the data, and if this “ability to detect a difference” is greater than 80%, but we still get a non-significant result, does that mean the two groups of data are really similar?

[EBM] Should I calculate study power when results are not significant? Misleading post-hoc power

I used to believe that, but the more I calculated, the weirder it felt

Until I calculated it a few more times myself and realized that power is actually deeply tied to the p-value. Both power and p-value are affected by sample size, standard deviation, and mean difference. When the mean difference between the two groups is very close and p-value > 0.05, it indicates a high degree of overlap in the data distributions of the two groups. Using these results to calculate power will, of course, yield a very low power.

(To clarify further, power calculated after the experiment is done is called post-hoc power, which is our topic today. The power we use when calculating the required sample size to collect is simply called power, and that’s an important and necessary step.)

Post-hoc power is actually affected by the p-value

I started reading more articles and found that calculating post-hoc power is indeed influenced by the p-value (after all, they are calculated using the same sample size, standard deviation, and mean difference). When p-value > 0.05, if we calculate post-hoc power hoping to reach 0.8, we actually need a p-value less than 0.0065 to get there. Isn’t this self-contradictory!

So if we don’t calculate post-hoc power, what else can we do?

First, we can calculate partially post-hoc power, a method proposed in this paper. Sure, the name is strange—how can something be partially post-hoc? What it mentions is that the mean difference we use to calculate power should be replaced by the minimal clinically important difference (MCID). This detaches it from the p-value. When interpreting, instead of saying, “We have post-hoc power % confidence to detect a significant difference between these two groups,” it becomes, “We have post-hoc power % confidence to detect a significant difference of at least the MCID between these two groups.”

Second, we can look at the confidence interval of the results alongside the “minimal clinically important difference” (MCID). Take height, for example: let’s assume a meaningful MCID is 5 cm. If we want to know the height difference between two classes, and we get a 95% confidence interval of [-2cm, 3cm] for the difference, we can determine that it includes 0, so it’s a non-significant result. Moreover, the range of the confidence interval doesn’t include +5cm or -5cm, meaning even if there is a difference, it doesn’t reach a meaningful magnitude. But if we get [-10cm, 20cm], then not only is it non-significant, but it also indicates the difference could be anywhere from 10 cm shorter to 20 cm taller. In that case, we wouldn’t really dare to say there’s no difference between the two groups.

Third, we can use a Bayesian model. This is less common in clinical research, and people are generally less familiar with interpreting it. Statistics falls into two broad categories. The first is traditional statistics, which calculates probability by looking at distributions; the t-tests and ANOVAs we usually use belong here. The second is Bayes’ theorem, where you first subjectively assign a likely probability or value (a prior), and then, after collecting cases, calculate what the likely probability becomes. This is a major category distinct from traditional statistics, and my personal understanding isn’t deep enough either, so if you’re interested, please just Google it yourself~

Three alternatives: which method is better to use?

According to the methodology paper I found, partially post-hoc power is still considered to be based on post-hoc power calculations, so it will still be affected by the collected data. A confidence interval combined with MCID is very easy to interpret and very versatile, making it a good option. A Bayesian model is probably the best way to express the actual situation, but most people aren’t familiar with it.

Looking at it this way, interpreting the confidence interval ends up being the most practical. And this actually aligns nicely with the latest approach to rating imprecision proposed by the Cochrane GRADE guidelines in September 2022. For this item, GRADE doesn’t look at the p-value; instead, it looks at whether the 95% confidence interval crosses the upper or lower limits of the MCID to make a judgment. If you’re interested, you can check out this newly published video.

Ref:

  1. John J. Dziak, Lisa C. Dierker, and Beau Abar. The Interpretation of Statistical Power after the Data have been Gathered. Curr Psychol. 2020 Jun; 39(3): 870–877.
  2. Linan Zeng, Gordon Guyatt. Updated GRADE guidance for imprecision rating using a minimally contextualized approach. YouTube. Cochrane Training. 2022 Sep.