Assessment of the Reliability and Clinical Applicability of ChatGPT’s Responses to Patients’ Common Queries About Rosacea

Introduction

In the last decade, deep learning (DL) and other artificial intelligence (AI) technologies have made rapid progress.¹ Based on novel AI techniques, Augello et al developed a social chatbot model capable of integrating both individual and social processes.² This pioneering approach inspired researchers to dig out the potentials of AI chatbots, especially in the area of healthcare. Chung et al introduced a chatbot-based application, which aimed to provide timely assistance to these patients with chronic diseases.³ AI chatbots can provide numerous advantages, including enhanced patient engagement, improved diagnostic accuracy, and personalized treatment plans and facilitating the assimilation of the latest medical literature into clinical practice.⁴ As powerful as AI chatbots can be, there are also concerns about the limitations of them being medical chatbots, including the accuracy and reliability of the medical information they provide, the transparency of the model, and the ethics of making use of user information.⁵ Chat Generative Pre-trained Transformer (ChatGPT) series is a highly discussed one of those artificial intelligence tools developed by OpenAI. It is based on natural language processing (NLP) models. It utilizes deep learning to provide coherent and fluent human-like text responses to complex queries. Other significant AI chatbots, such as Google Med-PaLM are also being explored for medical applications. Also using chatbot developing tools, including IBM Watson (IBM Corp) assistant, some researchers are constructing other specialized AI chatbots focused on certain specified fields of medicine.^6,7 In certain fields of medicine, the benefits of applying ChatGPT as a patient education tool have been evaluated, and they helped resolve some certain concerns about AI chatbot limitations.^8–11

Rosacea is a chronic and recurrent inflammatory disease primarily affecting the facial blood vessels, nerves, and pilosebaceous units in the central region of the face, affecting approximately 5.46% of adults.¹² As a chronic inflammatory skin disease, rosacea patients often faces long-term treatment. Meanwhile, patient education is an important component of rosacea management, especially referring to the potential triggers such as diet habit and skin care.^13,14 However, patients may not get enough suggestions from dermatologists due to several reasons. Nowadays, patients often use social media and other platforms to expand their understanding of the disease.¹⁵ Physicians can facilitate patient education by leveraging the wealth of online resources. As a novel chatbot, ChatGPT showed its potential in medicine.^4,11,16 In urgent need of a reliable out-of-clinic rosacea patient education tool, in order to assess the extent of dermatological resource coverage in ChatGPT, we assessed the reliability and clinical applicability of ChatGPT 3.5 in responding to patients’ common queries about rosacea.

Materials and Methods

Objective

We selected ChatGPT 3.5 as the tool to generate responses to patients’ common queries about rosacea. ChatGPT is powered by an enhanced version of the large language model (LLM) GPT-3.5, which has been trained and fine-tuned on a variety of sources such as articles, websites, books, and written conversations, including dialogue optimization, enabling ChatGPT to generate response in a conversational format.¹⁷

Methods

Alinia et al conducted a qualitative analysis using a stratified sampling method on approximately 2000 messages from online rosacea forums to understand patients’ habits of seeking health information and their educational needs.¹⁸ We extracted 20 questions of patients’ greatest concerns, covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice. The reliability of the answers was rated on a scale from 1 (completely unreliable) to 5 (completely reliable), while the applicability of the answers was rated on a scale from 1 (not applicable) to 3 (highly applicable). This study focused on a publicly available AI tool, without involving patient information or clinical biosamples. The research did not involve any commercial interests or issues related to intellectual property. Ethical review by the Ethics Committee of West China Hospital, Sichuan University, determined that clinical ethics review was not required.

Statistics

This study used a five-point scale and a three-point scale to assess the reliability and the clinical applicability of the generated content, respectively. The evaluation results were obtained using average percentages. The consistency of expert ratings was tested using Kendall’s coefficient of concordance.

Results

This study showed that the reviewers unanimously agreed that ChatGPT had a high reliability responding to patients’ common queries about rosacea, which 94.6% reliability in general and range from 92.22% to 97.78% among four categories (Table 1). Specifically, it achieved 95.28% reliability in the treatment category, 97.78% reliability in the triggers and diet category, 92.78% reliability in the skincare category, and 92.22% reliability in the special manifestations of rosacea category. Among all 20 queries, the reliability of the ChatGPT ranges from 80.00% to 100.00%. The lowest reliability of ChatGPT was shown in the query “Are the medications used to treat skin rosacea effective for eye symptoms?”, and the highest reliability was shown in eight queries (Table 2).

Table 1 Percentage of Reliability and Clinical Applicability Assessment for the Four-Category Classification

Table 2 ChatGPT Underwent a Percentage-Based Evaluation of Content Reliability and Clinical Applicability for 20 Common Questions Related to Rosacea

Additionally, almost all responses were considered applicable for supporting rosacea patient education. In general, the mean clinical applicability of ChatGPT was 99.1%, with a clinical applicability ranging from 98.61% to 100.00%, specifically with a 98.61% applicability rate in the treatment category, a 100.00% applicability rate in the triggers and diet category, a 99.07% applicability rate in the skincare category, and a 99.07% applicability rate in the special manifestations of rosacea category (Table 1). Among all 20 queries, the clinical applicability ranged from 92.59% to 100%. The lowest reliability of ChatGPT was shown in the query “How to manage adverse reactions during rosacea treatment?”, and the highest reliability was shown in 16 queries (Table 2).

The consistency of the expert ratings was excellent (all significance levels were below 0.05). The consistency coefficient for content reliability was 0.404, and for clinical applicability, it was 0.456, indicating a significant consistency in the results. The expert ratings demonstrated a high level of agreement (Table 3).

Table 3 Consistency Test Results

Discussion

ChatGPT, as a cutting-edge AI tool, achieves an impressive reliability of 92.22% to 97.78% in responding to patients’ common queries about rosacea. In comparison to similar studies, ChatGPT demonstrates higher reliability in responding to common queries about basal cell carcinoma (84% reliability), breast cancer prevention (88% reliability), and cardiovascular disease prevention (84% reliability).^19–21 The consistency coefficient in this study for content reliability was 0.404 and 0.456 for clinical applicability. These results were comparable with related studies. For example, Lakdawala et al evaluated the accuracy of ChatGPT in offering clinical guidance for atopic dermatitis and acne vulgaris. The weighted Cohen’s kappa coefficient was 0.44 in their study, which indicated moderate agreement among the board-certified dermatologists.²² ChatGPT exhibits strong advantages in rosacea patient education, particularly in addressing concerns on triggers and dirt, showing both high reliability and clinical applicability. By comparing the results of three groups of ratings, few responses were consistently considered inappropriately to apply in clinical practice. This is usually attributed to missing key details or excessive complexity in the content. For example, ChatGPT’s response to “How to manage adverse reactions during rosacea treatment?” was generally accurate, yet it covered content too complex and did not cater to common clinical cases, and it was not as concisely as advice from clinical doctors. This also demonstrates the exceptional reliability and clinical applicability of ChatGPT in responding to patients’ common queries about rosacea. Our study still has several limitations. It is worth mentioning that while OpenAI has introduced the latest and more advanced version of the AI tool, ChatGPT 4.0, the fee-charging service may make it less approachable for non-professional users, including rosacea patients. We will evaluate the potential application of ChatGPT 4.0 in rosacea when it is more accessible for the public. With the continuous evolution of this AI chatbot tool, the latest and more coming versions might become more economically feasible for common users and the assessment awaits. This AI tool is applicable for supporting rosacea patient education. While ChatGPT has its advantages, there are also concerns about its use as a medical chatbot. One concern is the accuracy and reliability of the medical information provided by ChatGPT, as it is not a licensed medical professional with abundant clinical experience, as well as it may not have access to up-to-date medical knowledge. Another concern is about the transparency of the chatbot model and the ethics of making use of user information, as well as the potential for biases in the data used to train ChatGPT’s algorithms. Thus, it is important to carefully consider the potential risks and benefits of using ChatGPT as a medical chatbot, especially in those patients with limited medical knowledge.⁵ Further studies should focus on improving the reliability and clinical application of AI chatbots with up-to-date information as well as advanced algorithms. Meanwhile, limitations of these chatbots should be clear of answering medical queries so that the user can identify when they need to see medical professionals.

In conclusion, ChatGPT 3.5 exhibits excellent reliability and clinical applicability in responding to patients’ common queries about rosacea. This AI tool is applicable for supporting rosacea patient education.