As the ANU community is starting to think about slowly returning to campus, this might be a good time to check back on those research studies that were interrupted or haven’t started yet and take a minute to consider whether they are powerful.
The majority of research studies are conducted within a Null Hypothesis Significance Testing (NHST) framework and as such there are some key concept that come at play. Researchers often put a lot of emphasis on their significance level, alpha, and their p-values but maybe tend to downplay the other side, namely the power of their study.
As a refresher, power is the probability to correctly reject the null hypothesis. Or, in other words, the likelihood of finding an effects (e.g. change in means) when it is truly there.
Let’s look a bit closer at what a powerful study is made out of.
Power and sample size guesstimation
When researchers plan their studies, they are prompted to do a power analysis at the time that they need to provide a sample size when they submit their ethics application or grant proposal.
Power and sample size go hand in hand. But sample size is not the only factor that influences power. There is also the variation in the measurements as well as the expected change in the measurements (often termed effect size). So in order to determine the sample size of a study for a given power, we will need to provide values for those.
Power also depends on the design and the statistical method that will be used to analyse the data. The more complex the design, the harder the sample size calculation and it is quite possible that online available calculators are not appropriate.
So how to avoid that your sample size estimation becomes a guesstimation?
The more realistic your values for the variance and effect size the better. Best case scenario is to have some pilot data from which you can derive those. Another option is to refer to the literature and find similar studies.
If determining an effect size is difficult, it could be worthwhile thinking about what the smallest change could be that would be considered of importance. For examples, in nutrition interventions dieticians often look for a 5% change in body weight which they deem clinically significant (Donelly et al., 2009). Consider whether your field has a similar minimal change.
Given that the power is contingent on the analysis performed, it is crucial that you know how you will analyse your data. If in doubt, consult an expert.
Keep in mind that a powerful study doesn’t guarantee statistical significance. It is more about creating the optimal conditions to indeed find that effect if it is present.
“Accepting” the null hypothesis
On a slight side note, researchers often write in their paper that they accept the null hypothesis when their p-value exceeds the significance level. Whilst that makes for nicer writing, the correct conclusion here is that they failed to reject the null hypothesis.
Failing to demonstrate a significant difference is not the same as proving that there is no difference (at least in statistical terms). If you are looking for proof that populations/parameters are equal, what you need is equivalence testing and consequently any power analyses and sample size estimations would be based on this type of testing.
A note on post-hoc power analyses
In some fields it is common practice for reviewers to ask about the observed power of the experiment or for researchers to attempt to justify their non-significant effects by conducting a post-hoc power analysis. Whilst statistical software packages allow you to obtain such power, do not be tempted to oblige the reviewer.
Let’s be quite clear, step away from calculation of post hoc power!
For some inspiration on how to reply to the reviewer and a thorough explanation on the why, check out this blog post by Daniel Lakens.
Besides, by conducting a power analysis a priori there is no need for a reviewer to ask this question anyway.
Donnelly JE, Blair SN, Jakicic JM, Manore MM, Rankin JW, Smith BK. Appropriate Physical Activity Intervention Strategies for Weight Loss and Prevention of Weight Regain for Adults. Med Sci Sports Exerc. 2009;41:459–471.
Marijke joined the Statistical Consulting Unit in May 2019. She is passionate about explaining statistics, especially to those who deem themselves not statistically gifted.