The popular artificial intelligence (AI) chatbot ChatGPT had a diagnostic error rate of more than 80 percent in a new study looking at the use of artificial intelligence in pediatric case diagnosis.<\/p>\n\n\n\n
For the study published in JAMA Pediatrics this week, texts from 100 case challenges found in JAMA and the New England Journal of Medicine were entered into ChatGPT version 3.5. The chatbot was then given the prompt: \u201cList a differential diagnosis and a final diagnosis.\u201d<\/p>\n\n\n\n
These pediatric cases were all from the past 10 years.<\/p>\n\n\n\n
The accuracy of ChatGPT\u2019s diagnoses was determined by whether they aligned with physicians\u2019 diagnoses. Two physician researchers scored the diagnoses as either correct, incorrect or \u201cdid not fully capture diagnosis.\u201d<\/p>\n\n\n\n
Overall, 83 percent of the AI-generated diagnoses were found to be in error, with 72 percent being incorrect and 11 percent being \u201cclinically related but too broad to be considered a correct diagnosis.\u201d<\/p>\n\n\n\n
Despite the high rate of diagnostic errors detected by the researchers, the study recommended continued inquiry into physicians\u2019 use of large language models, noting it could help as an administrative tool.<\/p>\n\n\n\n
\u201cThe chatbot evaluated in this study\u2014unlike physicians\u2014was not able to identify some relationships, such as that between autism and vitamin deficiencies. To improve the generative AI chatbot\u2019s diagnostic accuracy, more selective training is likely required,\u201d the study said.<\/p>\n\n\n\n
ChatGPT\u2019s available knowledge is not regularly updated, the study also noted, meaning it doesn\u2019t have access to new research, health trends, diagnostic criteria or disease outbreaks.<\/p>\n\n\n\n
Physicians and researchers have increasingly looked into ways of incorporating AI and language models into medical work. A study published last year found that GPT-4 from OpenAI was able to provide an accurate diagnosis of patients over the age of 65 better than clinicians. This study, however, only had a sample size of 6 patients.<\/p>\n\n\n\n
Researchers in this earlier study noted the chatbot could potentially be used to \u201cincrease confidence in diagnosis.\u201d<\/p>\n\n\n\n
The use of AI diagnostics is not a novel concept. The Food and Drug Administration has approved hundreds of AI-enabled medical devices, though none that use generative AI or are powered by large language models like ChatGPT have been approved so far.<\/p>\n","protected":false},"excerpt":{"rendered":"
The popular artificial intelligence (AI) chatbot ChatGPT had a diagnostic error rate of more than 80 percent in a new study looking at the use of artificial intelligence in pediatric case diagnosis. For the study published in JAMA Pediatrics this week, texts from 100 case challenges found in JAMA and the New England Journal of […]<\/p>\n","protected":false},"author":2,"featured_media":22829,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_seopress_robots_primary_cat":"none","_seopress_titles_title":"","_seopress_titles_desc":"","_seopress_robots_index":"","footnotes":""},"categories":[2],"tags":[],"class_list":["post-22827","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"rttpg_featured_image_url":{"full":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",960,540,false],"landscape":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",960,540,false],"portraits":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",960,540,false],"thumbnail":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12-150x150.jpg",150,150,true],"medium":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12-300x169.jpg",300,169,true],"large":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",960,540,false],"1536x1536":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",960,540,false],"2048x2048":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",960,540,false],"post-thumbnail":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",747,420,false],"graptor-sq-xs":["https:\/\/web3unplugged.io\/blog\/wp-content\/uploads\/2024\/01\/Untitled-12.jpg",100,56,false]},"rttpg_author":{"display_name":"Admin CG","author_link":"https:\/\/web3unplugged.io\/blog\/author\/admin-cg\/"},"rttpg_comment":0,"rttpg_category":"news<\/a>","rttpg_excerpt":"The popular artificial intelligence (AI) chatbot ChatGPT had a diagnostic error rate of more than 80 percent in a new study looking at the use of artificial intelligence in pediatric case diagnosis. For the study published in JAMA Pediatrics this week, texts from 100 case challenges found in JAMA and the New England Journal of…","_links":{"self":[{"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/posts\/22827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/comments?post=22827"}],"version-history":[{"count":1,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/posts\/22827\/revisions"}],"predecessor-version":[{"id":22830,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/posts\/22827\/revisions\/22830"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/media\/22829"}],"wp:attachment":[{"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/media?parent=22827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/categories?post=22827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/web3unplugged.io\/blog\/wp-json\/wp\/v2\/tags?post=22827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}