Breaking Down Cass Review Myths and Misconceptions: What You Need to Know – The Quackometer

Seen the “98% of studies were ignored!” one doing the rounds on social media. The editorial in the BMJ put it in much better terms:

“One emerging criticism of the Cass review is that it set the methodological bar too high for research to be included in its analysis and discarded too many studies on the basis of quality. In fact, the reality is different: studies in gender medicine fall woefully short in terms of methodological rigour; the methodological bar for gender medicine studies was set too low, generating research findings that are therefore hard to interpret.”

Cogency, (edited )
@Cogency@lemmy.world avatar

A better breakdown advocate.com/…/hilary-cass-nhs-report-debunked#to…

And edited this one too www.tandfonline.com/doi/…/26895269.2024.2328249

Streetlights, (edited )

Ms Reed is not an objective source. Nor does it appear she has much experience with systematic reviews. Indeed she repeats many of the “myths” such as the one about 98% of studies being thrown out.

"A closer inspection of the reviews released alongside the Cass report reveals that 101 out of 103 studies on gender-affirming care were dismissed for not being of “sufficiently high quality,” "

That is a lie.

Lastly, it seemingly endorses restrictions on transgender people under the age of 25, stating that they should not be allowed to progress into adult care clinics.

This is another lie from Ms Reed. I am beginning to think this “debunking” article you’ve shared is actually the original source of most of the myths being peddled on social media.

Cogency,
@Cogency@lemmy.world avatar

It’s not a lie, they were mostly dismissed even according to your own article, they were dismissed and synthesized into one conclusion. That is still dismissal.

Streetlights,

101 of 103 studies were not dismissed. All systematic reviews classify their source studies based on the quality of the work. Of the 103, two were classed as high quality, 58 as moderate quality and the remaining 43 as low quality. For synthesis, only high and moderate quality studies were drawn on. That’s more than half, not 2%.

So yes, Erin is lying.

Cogency,
@Cogency@lemmy.world avatar

You can’t say she’s lying until we do a systemic review of why the Cass study dismissed everything but 2 studies for the numbers it used to reach its conclusion. You can’t say she’s lying without that review no more than I can support Erin by reading each study that was dismissed. What I can tell you is that dismissing that many studies is not normal scientific analysis. It reeks of bias.

Streetlights,

You can’t say she’s lying until we do a systemic review of why the Cass study dismissed everything but 2 studies

This is the lie. They didn’t dismiss all but two studies, they actually included 60. More than half of the 103 studies identified for the review.

So yes, Erin, and now yourself, are peddling a lie.

What I can tell you is that dismissing that many studies is not normal scientific analysis.

It’s key part of synthesising multiple sources into a meta-analysis. Including poor quality studies dilutes the quality of the overall analysis.

It reeks of bias.

By design, it’s biased towards higher quality research.

Cogency,
@Cogency@lemmy.world avatar

Synthesis is a paragraph summary inclusion ONLY, it means they didn’t use data from the study, it is dismissal. I’m done arguing that with you.

Streetlights,

They have absolutely used the data from those 60 studies. You can read where they say explicitly that in the report if you cared to.

You are utterly mistaken and firm on your conviction, these are not the qualities of skepticism.

“Don’t seek refuge in the false security of concensus”

Cogency, (edited )
@Cogency@lemmy.world avatar

That’s not what synthesis means. I’ve written synthesis reports before and the data you include from those reports once you have dismissed them as inaccurate, it is an entirely selective process of whatever you want to include from them. We even have a phrase for it in law, Summarily dismissed.

Streetlights, (edited )

And of the 103 reviewed they included data from 60. It is a lie to say they “dismissed all but two.”

Legalese is irrelevant. A systematic review of scientific literature is a different beast to “writing a few synthesis reports”.

Cogency, (edited )
@Cogency@lemmy.world avatar

So you don’t know what you are talking about. Gotcha.

Synthesis reports in a scientific study when presenting data, are the parts of the report where you explain why you are dismissing data, so in this case ~98% of the data or studies. So what you just said is ~98% of the data was included in the synthesis report. that’s not inclusion of the data. That’s selective inclusion to support a conclusion. A normal scientific study can’t dismiss 98% of available data. That reveals bias.

Streetlights, (edited )

You can read the reviews for yourself

t.co/82Rjs2L1pA

adc.bmj.com/content/…/archdischild-2023-326669

Let me know where you find the bit where they dismiss 101 out of 103 papers.

Hint, they didn’t. It is lying to say they dismissed 98% of the studies they looked at.

Cogency, (edited )
@Cogency@lemmy.world avatar

Read it. Their only inclusion in the report is to half explain why the were discluded, exactly what I said. Most of the dismissals are unscientific, not supported by a statistical analysis of why it was discluded. Data doesn’t become unreliable just because it is incomplete.

That report is absolutely rife with white washing and selection bias, I’d expect a scientific review of trans literature and studies to be a book at this point not 32 pages dismissing 98% of the data. It’s frankly insulting to anyone that’s read or written any number of scientific studies.

Streetlights,

They did not dismiss 98% of the data.

Cogency, (edited )
@Cogency@lemmy.world avatar

Putting 98% of the relevant available data in a supplementary table like 4 is not including the data.

Streetlights,

Supplementary Table 4 (from the first review) is a list of each of the 53 studies included in the review and how they were scored based on the Newcastle-Ottawa scale.

The “data” is in supplementary tables 3, 5, 6 and 7. Only studies that were scored as low quality were excluded from the synthesis.

“They dismissed 98% of data” is a lie.

Cogency,
@Cogency@lemmy.world avatar

No it’s not. None of the dismissals are statistically/ scientifically supported, and the data they present is blurbed and incompletely presented in a way that isn’t inclusive of what those studies actually say.

Streetlights,

Nothing was dismissed at all (and “statistics” has nothing to do with it so curious to mention it).

Studies were scored for quality on the well established Newcastle-Ottawa Score. High and Moderate quality studies were included in the synthesis. Low quality studies were not, but their outcomes are still reported.

Outcomes from each study were included in tables 3, 5, 6 and 7.

'They dismissed 98% of the data" remains a lie.

Cogency, (edited )
@Cogency@lemmy.world avatar

You can’t remove a study from a scientific paper without having statistical analysis to back it up. Each of those removed studies all had a statistical analysis of how confident they remained in their data even with the gaps. Because there aren’t completed 100% studies in science it just doesn’t happen so you use the data you have and test it for a confidence value you obtain using statistics. And the idea that some trans people don’t make it to the completion of a study due to personal reasons or even suicide isn’t that rare. Not using 98% of the data because of that would be stupid.

Streetlights,

You can’t remove a study from your a scientific paper without having statistical analysis to back it up.

You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality.

Each of those removed studies all had statistical analysises of how confident they remained in their data even with the gaps.

Studies are also scored low on quality if, for example, they don’t control for important sociodemographic confounders. Study that do control these, will have more reliable results.

You can read how the scoring works in supplementary material 1.

Not using 98% of the data because of that would be stupid.

“They dismissed 98% of the data” remains a lie. Repeating it doesn’t change anything.

Cogency, (edited )
@Cogency@lemmy.world avatar

“You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality”.

That’s wrong when you are trying to be scientifically correct. A science paper without that math isn’t science my dude. And comparing trans healthcare data to Facebook polls is ridiculous

Streetlights, (edited )

It’s remarkably common in systematic reviews, a feature even. You give the impression that this is a new or foreign concept to yourself and are just encountering these ideas for the first time.

Search on pubmed or the bmj or the Cochrane library for other systematic reviews using the Newcastle-Ottawa score. You’ll trip over them.

And comparing trans healthcare data to Facebook polls is ridiculous

One of the studies reviewed recruited patients over Facebook and polled them.

“They dismissed 98% of the data” remains a lie.

Cogency, (edited )
@Cogency@lemmy.world avatar

Again I’ve written these reports. It is absolutely not common practice to disclude data without scientific reason and analysis. It is explicitly taught not to do it that way in college. And it is not scientific to do that without a statistical threshold and confidence analysis of your reasoning.

Streetlights,

Again I’ve written these reports.

I am forced to strongly doubt this given your whole misunderstanding of the basic concepts on assessing methodical quality…

Certainly, you’ve never authored a systematic review for a reputable medical journal.

But don’t take my word for it…

…cochrane.org/…/13_5_2_3_tools_for_assessing_meth…

It is absolutely not common practice to disclude data without scientific reason and analysis.

You mean such as using a method like the Newcastle-Ottawa score to assess data quality?

It is explicitly taught not to do it that way in college.

If your college course covered systematic reviews and didn’t include a review of study assessment methods, ask for a refund.

And is not scientific to do that without a statistical threshold

Statistics are not required to assess that a study without a comparator is weaker than one with.

“They dismissed 98% of data” remains a lie.

Cogency,
@Cogency@lemmy.world avatar

The Newcastle method is not seen as a scientific basis for dismissal on its own.

98% of the data was dismissed in the synthesis and was not used to reach the conclusion that there wasn’t enough scientific evidence to support transition when 98% of the science says that is wrong.

And every scientific paper is expected to be comprehensive on its subject matter and/or thesis.

Streetlights,

It’s not used for “dismissal” it’s used to score studies on their likelihood of bias. Studies without appropriate controls for example are more susceptible to bias than those with.

98% of the data was dismissed in the synthesis

Demonstrably false, only low quality studies were excluded from the synthesis which account for less than half of the 103 reviewed. A lie is a lie no matter how often repeated.

and were not used in the conclusion that there wasn’t enough scientific evidence to support transition when 98% of the science says that is wrong.

That’s not what the conclusions say, for example:

Synthesis of moderate-quality and high-quality studies showed consistent evidence demonstrating efficacy for suppressing puberty

And

Evidence from mainly pre–post studies with 12-month follow-up showed improvements in psychological outcomes

“They dismissed 98% of data” remains a lie.

Cogency,
@Cogency@lemmy.world avatar

www.tandfonline.com/doi/…/26895269.2024.2328249

Streetlights,

That was published a month before Cass came out and so hasn’t anything to do with the two systematic reviews being discussed above. It doesn’t even mention them.

I’m uncertain what expertise a business graduate can bring to assessing the quality of a systematic review in medicine.

Readers are free to Google the author and subsequently make a judgement on their objectivity on the subject matter.

Cogency, (edited )
@Cogency@lemmy.world avatar

And yet you have no scientific reason other than an ad hominem fallacy with the author with which to dismiss the criticism with. That like the Cass report are not scientifically sufficient reasons to disclude the criticism or the data respectively.

And I can garuntee you that the Cass report was not peer reviewed like all of the studies they dismissed were because it would have been torn apart. That’s the real litmus test of scientific debate.

Streetlights,

And yet you have no scientific reason other than an ad hominem fallacy with the author with which to dismiss the criticism with.

If they made a scientific argument about these review papers under discussion I might but this is just a polemic using unscientific language like “cis-supremacy” in a low impact obscure journal.

That like the Cass report are not scientifically sufficient reasons to disclude the criticism or the data respectively.

Newcastle-Ottawa scoring is a scientific method for weighting the methodical quality of scientific studies.

And I can garuntee you that the Cass report was not peer reviewed like all of the studies they dismissed were because it would have been torn apart.

It was peer reviewed since thats BMJ policy, unless you have evidence to the contrary. There is even a link on the online edition of both reviews for you to submit a rapid response pointing out all their flaws which I would encourage you to do.

That’s the real litmus test of scientific debate.

Interestingly some nice fellow DM’d me with a link to “Patient Zero” of the “they dismissed 98% of the data” myth.

twitter.com/benryanwriter/…/1779671152148857212

And of course, everyone has doubled down rather than admit they read the wrong paper. A better “litmus test” of scientific debate is humbly correcting yourself when shown to be wrong.

“They dismissed 98% of the data” remains a lie.

Cogency, (edited )
@Cogency@lemmy.world avatar

98% of the data could be summarized in one sentence. Trans healthcare and hrt works. 98% of the data comes to that conclusion with vast consensus across multi disciplines and fields comes to that conclusion and that was ignored. 98% of the data was discarded. Most of those studies discarded already had a statistical analysis backing up their efficacy while the Cass report doesn’t. Nor does the Cass report include a nearly mandatory implicit bias report.

Those peer reviews are most likely selected and not randomized selections or contestations as most peer reviews are required to be, they are ok for initial release irc. But it is an outgoing process that doesn’t have an endpoint. They were most likely provided prior to release and the normal peer review process won’t be completed for years to undo the damage. But it is not considered peer reviewed yet.

Again you have not proven that the new castle Ottawa scale has any efficacy or scientific merit as a disqualifying tool No one has as far as I know.

Streetlights, (edited )

98% of the data was discarded

Liar.

Those peer reviews are most likely selected and not randomized selections or contestations

So now they were peer reviewed but by people you don’t trust based on the same evidence you used to assert it wasn’t peer reviewed in the first place I.e. zero.

Again you have not proven that the new castle Ottawa scale has any efficacy or scientific merit as a disqualifying tool No one has as far as I know.

You’d better tell the Cohcrane library to bin every systematic review they’ve ever done which used this system then. I’d be eager to hear their reply to you

Cogency, (edited )
@Cogency@lemmy.world avatar

98% of the data was discarded

No I’m just explaining the process and why it isn’t complete yet. Or even valid yet

And show me that the Cochrane library ever discarded a study using the criteria even once yet alone with the same level as the Cass report and I’ll write them

For something that illustrates the problem with the Cass report read www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/

Streetlights, (edited )

98% of the data was discarded

It was not. All studies that scored high or moderate quality made it into the synthesis. That’s 60 out of the 103 looked at, that’s not 2%.

No I’m just explaining the process and why it isn’t complete yet. Or even valid yet

You are speculating, based on nothing.

And show me that the Cochrane library ever discarded a study using the criteria even once yet alone with the same level as the Cass report and I’ll write them

Here’s one I found in <7 seconds

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

If you want to find more simply search the Cochrane library for reviews with “Newcastle-Ottawa” in the main body of text. It seems like this is new to you.

For something that illustrates the problem with the Cass report read www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/

The relevance of a joke paper from 2003 to a systematic review published last week is certainly questionable but it seeks you’re trying to imply that Cass discarded anything except RCT’s. The didn’t and that’'s myth #2 from the original Quackometer article.

Where will the goalposts move next?

Cogency, (edited )
@Cogency@lemmy.world avatar

Goal posts haven’t moved and I’ve already pointed out a dozen of so methodological flaws around the Cass report that you are choosing to ignore.

That’s on you

Streetlights,

Goal pays haven’t moved and I’ve already pointed out a dozen of so methodological flaws around the Cass report that you are choosing to ignore.

You haven’t pointed out, let alone substantiated, any. If you truly believe you have then I implore to use the rapid response function on the bmj site and communicate these catastrophic flaws to the editorial team immediately. I’d be eager to know what their reply is.

Cogency,
@Cogency@lemmy.world avatar

I don’t need to, it is already happening within the scientific community of which I am merely a part.

Streetlights,

Don’t abdicate responsibility to someone else, you’ve clearly got a firmer grasp of the issue than the editorial board of the British Medical Journal. You would be neglecting your duty as “part of the scientific community” to abdicate responsibility on such an important matter.

Indeed the whole medical establishment must be told about the critical flaws in the Newcastle-Ottawa scoring system before other medical scandals are allowed to happen. Imagine having that on your conscience.

Cogency, (edited )
@Cogency@lemmy.world avatar

Lol first sign that you might actually be human.

And it has already been widely criticized before that’s why there was the parachute joke report. Hence it is already the brunt of jokes to use that scoring scale.

Streetlights,

So strange that everyone waited over 20 years and 100’s of systematic reviews in medicine and science before, serendipitously, discovering that the Newcastle-Ottawa Scale was infact no good during these two particular reviews into trans care in the UK.

Just what are the odds?

Cogency, (edited )
@Cogency@lemmy.world avatar

No the Cass report is just misusing the scale. It’s not a disqualifying tool and the scale still has uses which just means further analysis into the subject matter. Which is why the Cass report needed to be books longer, it’s not comprehensive.

Streetlights,

That’s a new goalpost. It’s being used by Cass exactly the way it’s supposed to by scoring studies based on their susceptibility to bias.

If you’d bother to read that similar systematic review on postoperative inflammatory bowel disease you would have seen the exact same usage.

Cogency,
@Cogency@lemmy.world avatar

Again that’s a joke to do that.

Streetlights,

Damn here’s another “joke” about contraceptives and bone fractures

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

And one another abput yellow fever and HIV

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

And influenza vaccines in cancer patients

www.cochranelibrary.com/cdsr/doi/10.1002/…/full

And there’s another 96 on the first search tab alone!

Just what are those clowns at the Cochrane Library up to eh?

Cogency,
@Cogency@lemmy.world avatar

Relevance?

Streetlights,

They all use the same Newcastle-Ottawa system to score studies based on their likelihood of bias in the exact same way the Cass reviews do. The method you described as a joke.

Cogency,
@Cogency@lemmy.world avatar

It’s not an indicator of bias, no causal study has been done to show that there is a relationship between bias and the Newcastle Ottawa scale

Streetlights,

Studies that self select their cohort and don’t include adequate controls are more susceptible to bias than those that do otherwise. Evaluating studies based on their susceptibility to bias is a vital part of the systematic review process.

You can read more about it here www.ohri.ca/programs/…/oxford.asp

Cogency,
@Cogency@lemmy.world avatar

But not actually proof of bias.

Streetlights,

Ah young padawan, there is no such thing as proof of bias. There is merely the risk of susceptibility of it.

Cogency,
@Cogency@lemmy.world avatar

Exactly which is why the Ottawa whatever standard is not sufficient to discard a study. You have to do more.

Streetlights,

Again, you really need to feed this startling discovery back to the medical community which has been using NOS for over 20 years. What a scandal.

Cogency,
@Cogency@lemmy.world avatar

No the medical community largely respects the short comings and uses of the Ottawa protocol. That’s what made Class’s report so insulting.

Streetlights,

Blimey, you’re speaking for the medical community itself as a whole now, pray tell then why haven’t they binned every systematic review ever carried out using the NOS system?

Why after 20 years of use is this system only being rubbished after two reviews into gender affirming healthcare in the UK were published?

Why are you the only person complaining about the Newcastle-Ottawa system when everyone else online is making up lies like “98% of data was dismissed”?

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • skeptic@lemmy.world
  • ethstaker
  • thenastyranch
  • GTA5RPClips
  • everett
  • Durango
  • rosin
  • InstantRegret
  • DreamBathrooms
  • magazineikmin
  • Youngstown
  • mdbf
  • slotface
  • cisconetworking
  • kavyap
  • JUstTest
  • normalnudes
  • modclub
  • cubers
  • ngwrru68w68
  • khanakhh
  • tacticalgear
  • tester
  • provamag3
  • Leos
  • osvaldo12
  • anitta
  • megavids
  • lostlight
  • All magazines
    •