Thought about hypothesis testing as an approach to doing science. Not sure if new, would be interested if it's already been discussed. Basically, hypothesis testing is inefficient because you can only get 1 bit of information per experiment at most.
In practice, much less on average. If the hypothesis is not rejected you get close to 0 bits, and if it is rejected it's not even 1 bit because there's a chance the experiment is wrong.
One way to think about this is error signals. In machine learning we do much better if we can have a gradient than just a correct/false signal. How do you design science to maximise the information content of the error signal?
In modelling I think you can partly do that by conducting detailed parameters sweeps and model comparisons. More generally, I think you want to maximise the gain in "understanding" the model behaviour, in some sense.
This is very different to using a model to fit existing data (0 bits per study) or make a prediction (at most 1 bit per model+experiment). I think it might be more compatible with thinking of modelling as conceptual play.
I feel like both experimentalists and modellers do this when given the freedom to do so, but when they impose a particular philosophy of hypothesis testing on each other (grant and publication review), this gets lost.
Incidentally this is also exactly the problem with our traditional publication system that only gives you 1 bit of information about a paper (that it was accepted), rather than giving a richer, open system of peer feedback.
Choose hypotheses where both possibilities are exciting (i feel like a lot of null hypotheses are borderline stupid, and that makes them bad null hypotheses. If you look at things like engineering in space, when something goes wrong in a spacecraft and you only have a 2 min window of communication with 2 days to get an answer... people find way to maximize the info out of their null hypotheses)
@jonny well the first one seems to be arguing for longer spent before doing a hypothesis test so arguably that's an even lower information rate overall. 😉 The second one seems closer but I haven't read past first page yet. Does it talk about how we could have a richer error signal?
@neuralreckoning
Not just longer, but with a more firm derivation chain from theory to result which includes a lot of the exploratory work that youre talking about - the hypothesis test should only happen once we have established the rich background for it to be genuinely informative in, so substantially greater than the "1 bit per experiment"
If youve never read that meehl paper I wont spoil it for you, its a true gem. Stick with it even in the places where he seems like he's teaching a 101 course, its worth an attentive read (in part bc his writing voice is incredible).
For strictly the bools -> floats as experimental outputs argument, this is the gist of a lot of "abandon NHST, embrace effect sizes" lit of like 2011-2020
@jonny will definitely read, looked interesting and that's a strong recommendation.I think my point is maybe something like: if the real value is not the output of the experiment but the exploratory work, shouldn't we be teaching this and valuing it more highly rather than denigrating it as fishing trips and rejecting grants on this basis?
@neuralreckoning oh i am definitely on the same page as you there. needless to say i think the insistence on having everything be hypothesis testing and discrete "results" rather than a tapestry of observations is an artifact of the journal article form.
another book along these lines is "a new kind of science" by stephen wolfram, but i don't recommend reading that because he is one of the worst writers i have ever read even if the ideas are interesting between all the chaff. arguing for pure exploration, "let the system speak for itself" using cellular automata as the illustrative system.
@deboraha @neuralreckoning
The meehl paper? It would definitely make my list of papers everyone should read regardless of discipline. I mean there are more lyrical meehl papers but this is one that like once you see it you cant see normal lit the same way
Add comment