Here's the logical structure of what you will be taught in terms of #statistics as a masters student in pretty much any #science field.
If MY DATA is a sample from two random number generators of PARTICULAR TYPE, and MY TEST has a small p value then MY FAVORITE EXPLANATION FOR THE DIFFERENCES IS TRUE.
This is, quite simply, a logical fallacy. The first thing wrong is that your data IS NOT a sample from a random number generator of that particular type. So we can ignore the rest logically.
The very first basic logical assumption is invalidated... But even if we grant that's true and that you go around studying random number generators, a class of things so notoriously hard to build that they are repeatedly the cause of major security flaws in cryptographic software between 1990-2010, the existence of a small p value in a test says NOTHING about the explanation for WHY the test fails, it particularly says nothing about whether your favorite explanation is true.
So the very basic stuff you are taught in masters level #biostats or #socialsciences stats or whatever is built on two layers of logical fallacy.
So that's why poor stats practices are absolutely more common than good stats practices. That's why the kind of thing posted on the Reddit post I mentioned in my recent posts is actually everywhere in science. Students are literally taught to do things wrong in the textbooks.
Having hypothesized a mechanism for how MY DATA comes about, and explained that in terms of a simplified model with CERTAIN PARAMETERS and collected data from these processes, does the mechanism and the data imply different parameters for my different experimental conditions? Given the results of those analyses, do the models consistently correctly predict future data?
This is the logic of quantitative model building and #Bayesian statistics.
So I promise you if you name a prominent journal in your field where people publish data based analyses with standard statistical results I will find multiple papers published in the last 3 months where the logical fallacies implied by the paper's analysis absolutely demolish the scientific merit of the conclusions and the correct conclusion from the paper will be "hunh interesting data but we learn virtually nothing reliable from the analysis"
And that's even before the problems of people choosing which statistical test they will do based on which one gives them statistical significance or throwing out "outliers" or trying 5-10 model specifications and publishing the one they like best, or manufacturing data wholesale, all of which are shockingly more common than anyone wants to believe. But those aren't even the biggest problems just the bog standard "I swear I'm doing it just like the stats textbook said" results are already wrong.
I don't do stats like a textbook in general, because that's all too shiny (and linear) for the real world. It's best to work from the Scientific Method itself and the nature of the data (like ordinality, etc.)
Either way, you're describing statistical fishing, which I don't see as research so much as digging for gold.
Still, I live in a car, so maybe being true to form is a detriment in today's world. Good guys finish quite last, I guess.
@rythur
I think we're in agreement? I mean I'm certainly not advocating that stuff of testing random number generator outputs versus a textbook table of possible p-value generating hypothesis tests. What I'm arguing for is to start like you said with the nature of the experiment that you are running and the way your particular type of science works (biology ecology flow of money from between agents chemical bonds etc)
@rythur
To me it's very telling that somebody who starts from that position in today's modern academia winds up being drummed out of teaching and living in a car. I am really sorry to hear that in your case and I hope you find some way to do something productive that you don't feel is a sham, Lord knows it's really hard to find.
Well, it's partly a decision to value time much more than money, but yes, I don't feel as though I fit in anywhere that things are done incorrectly as a norm, which rules out a lot.
I won't work on weapons, or banking. In general, I won't do any work that directly harms another. As such, I remain poorer than I'd like to be.
Imagine a much more moral Einstein. History would throw him away.
1/ You are not alone with such thoughts. Many possibilities run through my head, which I then discard. First, however, I have to solve the problem with the apartment, which I still need for the time being. I don't want to do any programming, computer or telecommunications service anymore, nor do I want to do any networking or electrical engineering, it's all boring to me.
2/ I would have fun with a kind of mobile repair café, small crafts, small repairs, housing adaptations for people with disabilities and things like that. I prefer to work with natural materials such as wood and clay. Maybe everything from a living and working van. All of these activities should be creative. A bit like carpenters when they go on a journey and work here and there.
3/ My peers tell me I'm too old for such thoughts and for such a life, but I don't think so. It doesn't get any easier as the years go by, but I don't want to be a cog in the wheel just because I'm supposedly no longer young enough.
I knew you and I had a lot in common. I could feel it.
I too am a firm believer in do it when you can, even if that means taking classes from a nursing home. Others won't easily understand this. The programming is just too strong.
We need meaningful work that can be done with pride and a lack of crummy management. Need. There are probably too many folks like us, not being used efficiently. The world could be so much better, easily. Change but a few things!
This is a subject I desperately want to know more about.
May I drop in?
I have noticed these anti patterns in venture-funded Internet startups since 1997 and I lacked the math knowledge to be able to describe them more formally.
I do have some observations about the relationship of the money flow the level of rigor. As @sennoma
suggests, it's about funding but also how accounting treats investments as expenses.
🧵
the level of rigor is related to the distance between the author and the first accountability event.
Accounting doesn't have a way to depreciate investments in maintenance or efficiency, it treats them as overhead (expense).
So there's an unexamined relationship between the level of rigor and the consequences of failure.
If quality is "too expensive" it's easy to cut. If failure can be delegated, it's easier to take risks.
(Boeing, etc al)
🧵 @rythur@kalfatermann@dlakelan@sennoma
@blaise
I think of accounting as one of the earliest mathematical models we have as a species. Good accounting should connect measurements of causes to their effects... I think you're talking about a disconnect. But I'm not an expert in this topic. @rythur@kalfatermann@sennoma
@blaise
For example, suppose we stop maintaining certain ship subsystems for many years there may be no visible consequence. The accounting shows a major savings. Then one year the ship loses power and crashes into a bridge killing tens of people and shutting down traffic for months or years. Here we put some cost somewhere on the books, but do we connect the causality? Do we debit the savings on maintenance with the tens of billions loss? @rythur@kalfatermann@sennoma
Probably not. so we don't learn, particularly the people who don't look at things other than "the accounts" don't learn. So these are important ways that mathematical models used by everyone can fail. @rythur@kalfatermann@sennoma@blaise
Good example, and it is timely.
In fact, the cost for the recovery will show up as an expense on someone else's balance sheet.
So the only financial incentive is to save money on insurance premiums.
The example that brought me to this discovery was trying to answer the question
"How much can we spend to test the service before we release to the customer?"
The answer was "we can't spend more than our overhead budget for any single project."
I call this pattern "hire 9 women to have a baby in a month."
Investors keep doing this because every now and then they get a baby. Usually, they don't.
Accounting has no way to record that one of the women is 8 months pregnant.
Add comment