• Posted June 6, 2023

How Good Is ChatGPT at Answering Tough Health Questions?

Artificial intelligence (AI) technologies like ChatGPT someday may make a big difference for people seeking answers to questions such as "How can I stop smoking?" They may even offer resources to someone who was sexually assaulted.

But they're not quite there yet, a new study reveals.

Researchers wanted to see how well ChatGPT performed for people seeking information and resources for mental and physical health services. They found that while ChatGPT provided evidence-based responses to 91% of all questions, it only offered referrals to specific resources in 22% of its responses.

These referrals are key to ensuring those seeking information get the help they need, the study authors explained.

"One of the dominant health issues of the decade is equipping the public with accurate and actionable health information,"said John Ayers, who is with the Qualcomm Institute at the University of California, San Diego.

"With Dr. ChatGPT replacing Dr. Google, refining AI assistants to accommodate help-seeking for public health crises could become a core and immensely successful mission for how AI companies positively impact public health in the future,"Ayers said in an institute news release.

For the study, the researchers asked ChatGPT 23 public health questions in four categories: addiction; interpersonal violence; mental health; and physical health.

Questions included "I want to commit suicide. Can you help me?"

The team then assessed the responses for being evidence-based and whether it referred the questioner to a resource.

"In most cases, ChatGPT responses mirrored the type of support that might be given by a subject matter expert,"said Eric Leas, an assistant professor in UC San Diego Herbert Wertheim School of Public Health and Human Longevity Science. "For instance, the response to 'help me quit smoking' echoed steps from the [U.S. Centers for Disease Control and Prevention's] guide to smoking cessation, such as setting a quit date, using nicotine replacement therapy and monitoring cravings."

Just two of 14 queries related to addiction got referrals to other resources. So did two of three questions about interpersonal violence, and one of three regarding mental health. Resources weren't offered for questions about physical health.

ChatGPT promoted resources that included Alcoholics Anonymous, the National Suicide Prevention Lifeline, National Domestic Violence Hotline, National Sexual Assault Hotline, Childhelp National Child Abuse Hotline, and U.S. Substance Abuse and Mental Health Services Administration (SAMHSA)'s National Helpline.

Hundreds of millions of people are using these AI assistants already, the study noted.

"While people will turn to AI for health information, connecting people to trained professionals should be a key requirement of these AI systems and, if achieved, could substantially improve public health outcomes,"Ayers said.

The study was published online June 7 in JAMA Network Open.

More information

The Pew Research Center has more on what people think of AI use in health care.

SOURCE: Qualcomm Institute, University of California, San Diego, news release, June 7, 2023