exactly how hard did beer person have to try to miss the point when they read a thread about how an AI confidently provided a wrong diagnosis and warning about how we shouldn’t always trust AI and proceeded to write a reply accusing Misha Saul of being a tech bro who believed an AI over a human doctor
I don’t think we should ignore AI diagnosis just because they are wrong sometimes. The whole point of AI diagnosis is to catch things physicians don’t. No AI diagnosis comes without a physician double checking anyway.
For that reason, I don’t think it’s necessarily a bad thing that an AI got it wrong. Suspicion was still there and physicians double checked. To me, that means this tool is working as intended.
If the patient was insistent enough that something was wrong, they would have had them double check or would have gotten a second opinion anyway.
Flaming the AI for not being correct is missing the point of using it in the first place.
I don’t think it’s necessarily a bad thing that an AI got it wrong.
I think the bigger issue is why the AI model got it wrong. It got the diagnosis wrong because it is a language model and is fundamentally not fit for use as a diagnostic tool. Not even a screening/aid tool for physicians.
There are AI tools designed for medical diagnoses, and those are indeed a major value-add for patients and physicians.
I think it’s being framed wrongly for the narrative by the guy posting the screenshot.
A friend sent me MRI brain scan results
Without more context I have to assume guy was still convinced of his brain tumor, knew a friend who knew and talked about Claude, had said friend run results through Claude and told guy who’s brain was scanned that Claude gave a positive result, and friend went to multiple doctors for a second, third, fourth opinion.
In America we have to advocate hard when there is an ongoing, still unsolved issue, and that includes using all tools at your disposal.
“AI is nowhere near to being ready to replace you at your job. It is, however, ready enough to convince your boss that it’s ready to replace you at your job.”
I remember reading an article or blog post years ago that persuasively argued that the danger of AI is not going to be that it ends up doing things better than humans, but that it causes a lot of harm when entrusted with tasks it actually isn’t good at. I think that thesis seems much more plausible now, watching people respond to clearly flawed AI systems.
That reminds me of a fairly recent article about research around visualisation systems to aid with interpretable or explainable AI systems (XAI). The idea was that if we can make AI systems that explain their reasonings, then they can be a useful tool, especially in the hands of domain experts.
Turns out that actually, the fancy visualisations that made it easier to understand how the model had come to a conclusion actually made subject matter experts less accurate in catching errors. This surprised researchers and when they later tried to make sense of it, they realised that they had inadvertently dialled up people’s likelihood to trust the model because it looked legit.
One of my favourite aphorisms is “all models are wrong, some are useful.” Seems that the tricky part is figuring out how wrong and how useful.
This is nothing new though. For decades, managers have fallen for “solution in a box” sales pitches even though front line workers know it’s doomed to fail as soon as they set eyes on it. This time the solution just happens to be “AI.”
Seems to me that a lot of the world’s problems start with “well, the managers think…” They all seem extremely bad at the whole managing thing, good thing we don’t overpay them or anything like that.
It’s worse now than ever though, many managers have been steeped in tech optimism their whole working careers. The failures of “revolutionary new systems” have been forgotten about while the success of other things are lauded.
They’ve been primed to jump on any new “innovation” and at the same time B2B marketing has started adopting some of the most manipulative practices that used to be only used on consumers. They’ve crafted a narrative that shapes discourse so the main objections that appear are irrelevant to the actual issues managers might run in to.
Stuff like “but what if it is TOO good?!” and “what if the wrong people get their hands on this AMAZINGLY POWERFUL new tech?!”
Instead of “but does this actually understand anything or is it just giving output that looks correct?” or “ Wait, so, how was this training data obtained? Will there be legal issues from deliverables made with this?”
The average manager has been primed by the zeitgeist to ask the sales rep the kinds of questions they want to answer.
I need help finding a source, cuz there are so many fluff articles about medical AI out there…
I recall that one of the medical AIs that the cancer VC gremlins have been hyping turned out to have horribly biased training data. They had scans of cancer vs. not-cancer, but they were from completely different models of scanners. So instead of being calibrated to identify cancer, it became calibrated to identify what model of scanner took the scan.
I am failing to find source, but there is also a story about an older predictive model that worked great at one hospital, but failed miserably at the next. There was just enough variation in everything that the model broke.
(I think the New England Journal of Medicine podcast, but I am not finding the episode.)
Wasn’t there something about CV’s for job applications and the AI ended up figuring out that black people or women are less likely to get the job so adjusted accordingly? Or how in England during COVID, poorer schools got lower predicted grades while more upper schools got higher, even against the Teacher’s grade, regardless of the work done
I am glad that "I googled why I was coughing and it said I had cancer and would die in 7 days so farewell you are a good friend" will live on for more years.
Maybe consider a tool made for the task and not just some random Claude, which isn’t trained on this at all and just makes up some random impression of what an expert could respond in a dramatic story?!
I know of at least one other case in my social network where GPT-4 identified a gas bubble in someone’s large bowel as “likely to be an aggressive malignancy.” Leading to said person fully expecting they’d be dead by July, when in fact they were perfectly healthy.
These things are not ready for primetime, and certainly not capable of doing the stuff that most people think they are.
To be honest, it is not made to diagnose medical scans and it is not supposed to be. There are different AIs trained exactly for that purpose, and they are usually not public.
Exactly. So the organisations creating and serving these models need to be clearer about the fact that they’re not general purpose intelligence, and are in fact contextual language generators.
I’ve seen demos of the models used as actual diagnostic aids, and they’re not LLMs (plus require a doctor to verify the result).
Add comment