含羞草研究所 Downstate Researchers Publish Groundbreaking AI Study on the Clinical Accuracy of Chatbots
By Office of Communications & Marketing | Aug 24, 2023
As Consumers Increasingly Utilize AI for Personal Medical Questions, Researchers Study Responses for Accuracy and Misinformation
Study to be Published in Prestigious JAMA Oncology Journal Concludes AI, While Accurate, Should be Used Only as Supplementary Tool for Information
BROOKLYN, NY鈥揜esearchers from 含羞草研究所 Downstate鈥檚 Department of Urology today published a study in the prestigious American Medical Association journal JAMA Oncology. The study, 鈥�Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries 含羞草研究所 Cancer,鈥� is the first of its kind to evaluate the quality of information about the most common cancers generated by artificial intelligence (AI) chatbots. The peer-reviewed research was spearhead by Alexander Pan, BS, David Musheyev, BA, Daniel Bockelman, BS, Abdo E. Kabarriti, MD, and Stacy Loeb MD, MSc, PhD (Hon).
Because consumers are increasingly using AI as a source of information despite the results not being validated for clinical accuracy, the study鈥檚 objective was to evaluate for misinformation and characterize the quality of responses from popular AI chatbots. In a cross-sectional study, the researchers assessed AI chatbots鈥� text responses to inquiries regarding the five most common types of cancer: skin, lung, breast, colorectal, and prostate. Search data were then extracted from the publicly available Google Trends platform and identical prompts were used to generate responses from four AI chatbots: ChatGPT version 3.5, Perplexity, Chatsonic, and Bing AI.
The study found that the AI chatbots generated high quality responses, and no misinformation was identified. In addition, the results indicated that three of the four chatbots cited reputable sources, such as the American Cancer Society, Mayo Clinic, and Centers for Disease Control and Prevention.
鈥淎I chatbots have very quickly become one of the most popular ways for people to receive information, and this study is the first of its kind to use validated instruments to evaluate the responses from chatbots when prompted with cancer-related questions,鈥� said Abdo E. Kabarriti MD, FACS. 鈥淚t cannot be overstated just how important it is to ensure that the vital information that these chatbots are generating is backed by solid, trustworthy sources. I am proud of our team鈥檚 work, and look forward to continuing to study the burgeoning use of AI in the medical field.鈥�
Overall, findings of the study suggest that AI chatbots generally produce reliable and accurate medical information however responses are not readily actionable and are written at an advanced reading level. Because of the limitations identified in the research, the study ultimately concludes that AI chatbots should be used supplementarily and not as a primary source for medical information.
The study will be published online August 24, 2023, by JAMA Oncology, and can be accessed
###
Contact: Dawn S. Walker
347.533.2071 (Cell)
dawn.walker@downstate.edu
含羞草研究所 含羞草研究所
含羞草研究所 is the borough鈥檚 only academic medical center for health education, research, and patient care. It is a 342-bed facility serving the healthcare needs of New York City and Brooklyn鈥檚 2.6 million residents. University Hospital at Downstate (UHD) is Downstate鈥檚 teaching hospital; it is backed by an outstanding medical school鈥檚 expertise and world-class academic medical center research facilities. Over 800 physicians, representing 53 specialties and subspecialties鈥攎any of them ranked as tops in their fields鈥攃omprise Downstate鈥檚 staff.
In addition to high-risk neonatal and infant services, pediatric nephrology, and dialysis (kidney diseases)鈥攁nd offering the only kidney transplantation program in Brooklyn, among many other distinctive programs. Downstate comprises a College of Medicine, a College of Nursing, a School of Health Professions, a School of Graduate Studies, a School of Public Health, and a multifaceted biotechnology initiative, including the Downstate Biotechnology Incubator and BioBAT for early-stage and more mature companies, respectively. For more information, visit www.downstate.edu or follow us on Twitter at .