"Randomized trials cannot address all causal questions of importance in medicine and health policy and may have limited generalizability; thus, investigators may need to use observational studies as a source of evidence to address causal questions. The challenge, then, is to balance the importance of addressing the causal questions for which observational studies are needed with caution regarding the reliance on strong assumptions to support causal conclusions."
Folks outside of science may not be familiar with this but there is an absolutely massive struggle across so many fields to think about whether our scientific conventions around evidence from the 1900s are suitable to this century. I am firmly on the side of observational causal inference; the principle of "randomization" is almost never ACTUALLY implemented in the real world and our ecological validity suffers. We need all forms of evidence. RCTs are not possible for MOST questions.
it is also necessary to acknowledge the deep sociological realities that keep us from advancing: grad students who are massively underserved and undertrained in new methods, the devaluing of "practical statistics" education (despite all the hype about data); if we had healthier more inclusive pathways of learning for trainees of course we would have healthier methods. Many of us out here doing applied science have to entirely self-teach and un-learn poor statistics and poor methods training
It's really interesting to see health, achingly and terribly slowly, BEGIN to acknowledge the scale of the problem. Ironically in many ways health research has gone backwards in terms of not respecting or investing in high quality observations, as MDs churn out salami-sliced papers with stats they farm out to somebody else. The med school<>med publishing cycles are really something else. Can't tell you how many friends I've had consult on med papers' stats and get treated like shit by MDs.
Oh, it's not just a massive intellectual struggle within the scientific community. It's become a hot "identity politics" issue in the general population. (… of those who argue online.)
"Deniers" shout "Double Blind Randomized Controlled Trials are The Gold Standard of Science! I will not accept any 'evidence' that 'falls short' of that!!!"
I think what really gets me about this stuff is remembering how deeply I was interested in and motivated by the possibilities of large observational datasets in the world, and how soundly that was discouraged and made to feel lesser by the scientific establishment. How I was always more interested in properties of observed data and confounding and interactions, how it was never recognized that these are deep mathematical and technical skills I had, just because it wasn't in vogue posturing
We have to somehow thread this needle: we need to be hard on the way we do science but gentle as hell on the scientists because I know -- I KNOW -- we won't change or fix anything by making a scared grad student who's being yelled at by their PI feel lesser and like they're messing up their analyses. Instead we need to create pathways and acceptance for learning, for welcoming people who genuinely have been trying really hard to create evidence in this world.
Finally it's VERY freeing to embrace the reality that all scientific evidence is useful-but-wrong in some way and that this is the nature of it and that we are not defined as scientists by Being Right but by Trying To Be More Right and that is an incredibly human-centered philosophy that opens up your mind and heart to non-extractive, collaborative science, to seeing yourself as a collaborator not a ruler in this world and ecosystem of which we are all such a tiny part. So many fears fall away.
@grimalkina I regularly point out to people that the right way to think about science training is that it is more about teaching to be ok with saying "I Don't Know". Science is more about accepting that there are some answers we cannot adequately answer right now and living with it.
It is not about being right, it is about being ok with saying that we may not have a right answer rn.
@grimalkina This might be really dumb or something that’s already being done but is it possible to separate the statistics part and the science part? Like, the scientist comes up with the hypothesis and the experimental design and makes the observations, then collaborates with a statistician who crunches the numbers? Hell, maybe even assign a statistician to the scientist to ensure they’re disinterested in the outcome.
@ratkins It's not dumb at all, this is very frequently a model that's used and absolutely exists for some labs/teams/fields HOWEVER there are lots of different ways to be a "scientist" and for many types of work you also want scientists to know the research statistics and make research statistical choices (it is a large part of our authorship. Disinterest is not the only way we safeguard results because informed knowledge and expertise in the area can also be really important. So...
@ratkins ...it is better to have pre-registration of analysis plans and other things like that that ensure we commit to a path rather than "hack" the results; I don't personally believe good science comes from just reducing all the pieces to vendors and contractors. This is kind of how business school research works a lot of the time 🙃 which many of us feel VERY negatively about because it's like uuuh, you're just paying for someone else to do your science and then getting authorship??
@ratkins But for departments, labs, and teams to sometimes or often have a dedicated "statistician" role? That is a model I believe in (assuming it is supported, respected, well staffed etc) and in fact one that I just created for my own lab. And I have a number of friends who are "staff statistician" kind of roles within scientific departments. It can be really good to have this as a dedicated team member!
@grimalkina I agree outsourcing sounds like a terrible model. I more had in mind the “balanced team” approach we used at Pivotal Labs where engineers, designers and product managers worked together daily—often pairing across disciplines—in service of the project. I’m more interested in the things a scientist doesn’t find in the data (because they find the stats painful and want to get back to “the science”) than the p-hacking to get the result they thought they should.
@grimalkina I suppose it’s Sturgeon’s Law, 90% of everything is crap like any other field. Once in a while you get a team who fights against the overwhelming external incentives and engineers themselves an environment where everyone is respected and valued and they can do great work.
@grimalkina oh man this was just too real a thread for Friday morning! It always bummed me out that I would want to study some question that actually mattered but it wasn’t “worth” it cause it could never be published .
Depends on which ones. They are not all the same, but the ones that submit to academic, empirical and mathematical rigor fight an up hill battle for air time. Consider that the tech bro guys in Silicon Valley consider obeisance to neoclassical and Austrian economics as a prerequisite to vote.
There is a different discipline out there punching to get through.
"Many of us out here doing applied science have to entirely self-teach and un-learn poor statistics and poor methods training."
So true.
I see recent graduates with the same faulty NHST-based statistical education that I received decades ago. It's disappointing how poorly education has kept up with new and better statistical methods.
@grimalkina Do you have any recommendations for which statistical methods to learn? I have the space right now to learn more, and I'd love to spend that effort usefully.
@grimalkina I am a data scientist/librarian working in healthcare, but I spend most of my time cleaning data and building pipelines - but I would love to know more about how to analyze data to see latent signals. The kind of things you can't build a trial for, like progression of untreated delirium, or pressure ulcers. If you knew the condition existed, you would act; but you don't always know.
Add comment