Can You Trust AI for Cancer Advice?

Artificial intelligence (AI) is transforming many aspects of our lives, and health care is no exception. From quick online searches to detailed responses from AI chatbots like ChatGPT, many patients are now turning to these tools for health information and even advice. But when it comes to something as critical as cancer care, how reliable are these digital helpers?

A study from Sylvester Comprehensive Cancer Center dives into this very question, specifically looking at how well ChatGPT performs in the rapidly evolving world of blood cancer. The findings offer important insights for anyone considering using AI for medical information.

Putting AI to the Test in Cancer Care

To understand AI’s capabilities, researchers focused on ChatGPT 3.5, a popular version of OpenAI’s chatbot that was widely available when the study took place in July 2024. Four expert hematology-oncology physicians (doctors specializing in blood cancers) were enlisted to evaluate the chatbot’s answers.

They posed 10 questions to the AI, designed to mimic the kinds of inquiries patients might have during their treatment journey. Some were general questions, like, “What are the common side effects of chemotherapy and how can they be managed?” Others were far more specific, such as, “What is a BCL-2 inhibitor?” (A BCL-2 inhibitor is a type of drug actively being investigated for cancer treatment).

The Results: AI Shines (and Stumbles)

The study revealed a clear pattern: ChatGPT 3.5 performed better when answering general questions about cancer but struggled significantly with providing accurate information about newer therapies and specific approaches.

Justin Taylor, M.D., a physician and researcher at Sylvester, was the senior author of the study. He shares a crucial warning for patients:

“I would warn patients to have some skepticism, especially about answers dealing with specific types of cancer and treatments, and check with their doctor.”

The physicians graded the answers on a scale from 1 (strongly disagree with accuracy) to 5 (strongly agree). ChatGPT earned an average score of 3.38 for general cancer questions. However, for questions about newer therapies, the average dropped to 3.06. No answer received a perfect score of 5 from any evaluator.

Why AI Can’t Keep Up (Yet) with Fast-Changing Fields

One major limitation of the ChatGPT 3.5 version used in the study was its data cutoff date of 2021. This means its knowledge base didn’t include any medical breakthroughs, new drugs, or research findings that emerged after that time. Imagine trying to get current financial advice from a newspaper printed three years ago – it simply can’t keep up with rapid changes.

Dr. Taylor emphasizes this human element: “When new drugs or research findings emerge, oncologists check in with their colleagues, discuss the implications and think about how to adapt them to their patients.”

He adds that “Chatbots can’t provide that kind of nuance and personalized understanding.”

This highlights that medical advancements, especially in complex fields like blood cancer, require constant human interpretation, discussion and adaptation based on the latest science and individual patient needs.

How to Use AI Smartly in Your Health Journey

So, does this mean AI is useless for patients? Not at all, says Dr. Taylor. There’s still a valuable place for these tools, but with a clear understanding of their limitations.

AI chatbots can be helpful for:

• Appointment Preparation: They can help you organize your thoughts and brainstorm questions to ask your doctor during visits.
• Finding Starting Points: They might direct you to primary sources of information that you can then verify with your doctor or other trusted medical websites.

However, it’s crucial to remember that AI should not be seen as a replacement for your health care team. The study’s researchers conclude, “Physician oversight remains essential for vetting AI-generated medical information before patient use.”

Always discuss any information you find from AI or online sources with your doctor. They can provide the personalized, up-to-date, and nuanced advice you need for your unique health situation. AI is a tool, but your doctor is your guide, says Dr. Taylor.

Written by Charlotte Schubert, Ph.D.

Tags: AI accuracy in oncology, AI health chatbot limitations, AI tools for cancer patients, Dr. Justin Taylor