Discussion
First usable version of the assurance sample size tool I'm building as part of my thesis:
So far, only the part for t-tests is implemented, but that's most of the heavy lifting done.
https://01a1c292-5832-6be9-e0ea-ff6d8eaec555.share.connect.posit.cloud/
@zwol the easiest would be to wait until I get this thesis finished and maybe published because it's very much aimed at that audience :)
Alternatively here's a paper by the authors who introduced the concept where they already simplify it a bit: https://onlinelibrary.wiley.com/doi/10.1002/pst.175
@zwol the very short version:
Power is a conditional probability of finding a suspected effect (rejecting the null hypothesis), dependent on a specific effect size.
Assurance is the unconditional probability of rejecting the null hypothesis. It treats effect size not as a fixed value but as a probability distribution in itself. It's a hybrid concept between frequentist and Bayesian statistics.
It's actually nice to play around with, and the results are so much worse than I thought.
Good for me though, because that's the whole point of my thesis, but damn, if you have a small effect size AND uncertainty about it, your sample size from power analysis might be REALLY far off
Context: as a researcher conducting a clinical trial you want to be on the green line. But if you do everything "right" as your basic statistics class has taught you, you're more likely on the black line and underestimating your required sample size, so that you're more likely to get a false negative result than you aimed to be.
The risk here is not primarily that your study makes an unjustified claim, but rather that you're wasting time and resources doing a trial that is less informative than you thought it would be.
Would you rather collect 10% more participants or find out when it's too late that you should have done so because your expensive and time consuming study turned out informative?
@aesthr Do you know offhand where I could read up on what "assurance" means in this context and how it differs from power analysis? Not urgent - I have no immediate need to do this kind of study - but it feels like something that could easily come up for me in the future.