The minimum effect is my power threshold so they cancel each other out. How can I do this? Preferably with linear models in r (I like emmeans and simr.
@lakens you wrote that more power is needed for minimum effects compared to null tests, so you might know.
@lakens exactly. In my example in the forum post, I say my control is 100 mg, the minimum effect I expect of a treatment is 105 mg. So the minimum effect is 5. For a null hypothesis, I calculated the sample size (6 samples with 90% power). But I want to test the hypothesis that the effect is at least 5. So when I simulate 100 and 105, they are basically both 100
@raoulvanoosten I might miss something but it sounds like you want to test if the difference is 0, or if both effects are 100? If so, do an equivalence test of the difference. It is a range prediction. The observed value should be within say 90 and 110. Does that work?
@raoulvanoosten and you expect it to be 5. So, if you are right, this test can not detect it. You should predict it IS 5 (or better, between 0 and 10) or just that it is larger than 0.
@lakens not exactly. I expect my treatment to have a positive effect, and to be falsifiable I consider at least 5 a meaningful effect. So I test the hypotheses >5 and <5. But before I conduct the experiment, I would like to know how many samples I need for either hypothesis to be falsified. Indeed, if the true effect is exactly 5 I cannot falsify either hypothesis, which is where my issue with the power lies.
@raoulvanoosten yes, this is the same problem as with have NHST and a null effect. No amount of data can show it. You would need a second test. If the true value is the value you test against, no amount of data can reject it. Same as with NHST and an effect of 0. You have recreated the problem, just moved the test value to 5 instead of 0.
@raoulvanoosten A NHST test of the effect is > 0 or < 0. You just replaced 0 with 5. But then you still have the same problem that you can never falsify the null (you just have a non-nill null)
@lakens so thát's a non-nil null 💡. Indeed I shifted the problem, and the core of the matter is that a null can never be falsified. Thanks, problem "solved" 🙏.
@lakens would the most sensible solution be to test for effects larger than zero and calculate power accordingly (for an effect of 5), together with a test for equivalence (at <5)? Otherwise I could/would calculate power for an arbitrary effect between 0 and 5.
@raoulvanoosten Yes to the first part - that is also what I recommend for the NHST + Equivalence combi. I do get your desire for the test you proposed - with sufficient power, and if the effect is not exactly 5, you would also have an informative answer. But if the effect is 5 or close to, power is too low to make it a useful (only) test.
Add comment