AI vs Doctors Competition (RESULTS)
1,053,885
Published 2024-05-31
There were 4 groups: Large language models, groups of expert eye doctors, eye doctors in training, and unspecialized junior doctors. Everyone had to answer 87 questions from a standard test that doctors in the UK need to pass before becoming an eye specialist. Here's how they each did.
The intention of this study is NOT to see if any AI tool can replace any doctors. This study just showed that in this one test, one model’s performance was almost as good as experts. We could imagine a world where more generalized doctors who don’t have easy access to a specialist could ask an AI for a second opinion or help make a referral or even help with a diagnosis. It’s one way researchers are exploring how to improve medical care for more people.
If you like optimistic science and tech stories, subscribe for more of our show Huge If True.
If you want to read the full study, search this: DOI:10.1371/journal.pdig.0000341
#science #sciencefacts #education #study
All Comments (21)
-
Asking a doctor for a second opinion from chat gpt would be wild
-
Google AI be like: "Yeah, you can safely apply sulphuric acid on your retina to see better 🤓"
-
As a med student, i can confirm that chat gpt 4 can answer about 90% of the standard questions I give to it correctly. However, when i tried to test it on an actual case that presented to us in the hospital, or when a professor tweaks the question a bit to make it more difficult for us, the rate dropped to almost 40% - 50%. I guess what I'm trying to say is that if you feel sick, dont try to figure out what's wrong by asking AI. Please go to a hospital (even though i know going to a hospital kinda sucks)
-
Answering a multiple choice checklist is very different than looking at a real world situation and applying high judgement to come up with the best solution.
-
One of the first successful AI applications was with medicine. Doctors make decisions based on symptoms, but there are a load of rare diseases that often slipped through. AI was able to suggest the obscure diseases and suggest tests that should be performed to make a better diagnosis.
-
Something else to consider, is that medicine is very hypothesis driven. If you don’t look for something, you won’t find it. Actually interviewing a patient and coming up with a differential diagnosis is much more difficult than answering a practice question.
-
Google AI be like yeah you can safely remove your eye to clean it from grains of sand under tap water and put it back on
-
One of the most important skills good doctors have is interacting with patients to obtain an accurate picture of all the symptoms and other important information required for diagnosis. If this is not done correctly, no computer will ever be able to do an adequate diagnosis.
-
If you want to read the full study, search this! DOI:10.1371/journal.pdig.0000341
-
"Doc are you sure removing my eye is the right way to go?" Doctor: "Yep, I even got a second opinion"
-
If my doctor starts asking an ai for advice I'm out
-
I'm not sure how exactly but if AI can take some of the easier or time consuming jobs away from experts, but still have critical results be reviewed, then it might help with doctor shortages.
-
This tends to happen when you train on the entire internet. Data spoilage is a big problem in this case - those questions were definitely online.
-
“These questions aren’t really available on the internet.” How do we know that?
-
The prompting technique plays a huge role.
-
The thing is, AI is theoretically capable of doing all of these things as a tool. It just isn’t as reliable as it would be to make it viable. Same way you probably wouldn’t feel comfortable sleeping in a self driving car
-
Lets play the Is this hallucinations or not 😂
-
Please state the nature of the medical emergency.
-
This highlights the need for improvement in testing methods today more than the advancement level of modern LLMs.
-
ok, but were the doctors allowed to look up the answer if they were in doubt. Cause I would expect a doctor to know when and how to lookup things they do not feel sure about. I don't need them to know everything from on the op of there head.
