ChatGPT: Is This Patient Education Tool for Urological Malignancies Readable for the General Population?

Background

Cancer incidence and prevalence is rising steadily over time, with only 47,414 cases diagnosed in 1982 in Australia, rising to 162,163 in 2021.¹ Correspondingly, diagnosis of new cases of urological malignancies have increased as well, from 6,545 cases in 1982 to 31,988 cases in 2022.¹ This sharp increase in the diagnosis of cancer can be attributed to improved health awareness through public health campaigns, increased use of screening services, technological advancements and proliferation, as well as an aging population and modifiable lifestyle factors.² However, rural and remote communities have consistently suffered from poorer health outcomes, with high disease burden, low population screening, delayed treatment, and low overall 5-year cancer survival rates compared with metropolitan populations.³ Part of the reason for this disparity in health outcomes is attributed to lower education levels, poor health literacy, and lack of access to timely specialist healthcare services. Western Australia is also unique amongst the countries and states of the world, being the largest and most rural state, with 28% of the population living in rural and remote areas.⁴ Therefore, access to healthcare is vital to improve the overall general health of these populations. The advent of the worldwide web and increasing adoption of virtual technology has “closed the gap” dramatically in the last decade, as can be witnessed during the COVID-19 pandemic.⁵ Utilizing Telehealth conferencing, online resources, multi-way conversations over platforms such as Microsoft Teams and Zoom to name but a few equips the modern day health professional with a multitude of methods to communicate with patients effectively.⁶ Similarly, patients can now have access to a large volume of information about their health conditions at the click of a mouse, without needing to seek medical advice or explanations from a medical professional.

ChatGPT is an online, open access platform that utilizes semi-supervised or self supervised learning tools called large language models (LLMs) to generate human-like responses to routine conversational inputs.⁷ In essence, it is a condensed search engine used to answer a specific targeted question posed by the user and relay the answers in an eloquent, contextually appropriate, conversational manner. Since the release of ChatGPT 3.5 in November 2022, there have been numerous papers on ChatGPT of late, detailing the benefits and pitfalls of its use in medicine, especially in the area of patient information provision.^8–11 The current ChatGPT-4 has even been shown to be able to generate literature papers that would have otherwise been successfully published in reputable journals, being able to bypass traditional plagiarism software.¹² The number of recorded users have increased dramatically as well, with more than 100 million registered users within 2 months of its launch. The potential of tapping on ChatGPT to aid in providing useful, up to date information across a variety of medical conditions to all patients and improve access to healthcare information remains very enticing despite emerging issues with its ethical consideration, credibility, and limitations.

Aim

The aim of this study is to determine if information provided by ChatGPT for urological malignancies is at an appropriate readability level for the target audience. The average Australian according to a census from the Australian Bureau of Statistics in 2021 would have received mandatory education to year 10 equivalent in secondary school, with an average Australian having a reading ability equivalent to year 8, or between age 13–14.¹³ Therefore, a reasonable level at which medical information should be pitched for the majority of the population should be at year 8. The Flesch-Kincaid reading ease score developed in 1975 for the United States Navy remains one of the most widely used validated readability tests for English texts.¹⁴ Although very simplistic in nature, this tool allows for a validated evaluation of the texts in question without use of complex linguistic metrics and hence has been adopted in this study.¹⁵ This assessment tool takes into account the length of sentences and the number of syllables per word to determine the level of reading difficulty of a text. It is scored from 0–100, with a higher score reflecting a text that is easier to read. The reading ability for a Year 8 individual corresponds to a Flesch-Kincaid score of 60.

Study Design

A number of questions were pre-selected by two Urology consultants from Royal Perth Hospital whilst taking into consideration existing patient information sheets to ensure comparability for the three most common primary urological cancers, including prostate, bladder, and renal cancers (Table 1). The questions were written based on a questionnaire databank collected from patients seen in clinics to determine the most frequent questions that patients have regarding their own illnesses. Questions were posed at a generic population level as well as at the level of year 8 repeatedly with answers recorded and assessed for ease of readability with the Flesch-Kincaid reading ease score. Three other sources of cancer-related information were collected as a comparison; information from the Cancer Council Australia as well as the predominantly American databases UpToDate and the Mayo Clinic. Cancer Council Australia (CCA) is the pre-eminent national organization in undertaking cancer research, patient support and education, and development of a cancer control policy.^16–18 UpToDate and the Mayo Clinic, on the other hand, are clinician authored resources that provides current, evidence based statements about a variety of medical conditions to aid clinicians in making point-of-care decisions.^19–29 Content was taken from the patient information section of the databases, with “basic” and “beyond basic” discussions from UpToDate being evaluated independently. The information was collected from existing online resources made available to the general public.

Table 1 Readability comparison between various information sources using Flesch-Kincaid readability score

Content of information provided was assessed independently by two separate Urology specialists to determine if it was adequate and accurate. Each question was posed five times to ChatGPT with Flesch-Kincaid score counted as an average and used for comparison between all three sources. A further inclusion criteria on questions posed on ChatGPT whereby the target audience was selected at year 8 education level was also evaluated as a comparison. Statistical significance was calculated using a two sample T-test to determine if there was a difference in readability level between the different sources of information.

Results

Figure 1. Readability comparison between different sources using Flesch-Kincaid readability score.

The average readability score for information provided by ChatGPT was 31.2 ± 12.6, compared with the 41.3 ± 5.5 (p = 0.26) for UpToDate Beyond Basics, 47.5 ± 5.1 (p = 0.17) for CCA, 53.7 ± 3.5 for the Mayo Clinic (p = 0.11), and 64.9 ± 12.6 (p = 0.09) for UpToDate Basics (Figure 1). This meant that, overall, the difference in readability of information provided by chatGPT was not statistically significant comparable to those provided by CCA, the Mayo Clinic, and UpToDate.

Figure 1 Readability comparison between different sources using Flesch-Kincaid readability score.

However, when individual cancer information was looked into in greater detail, apart from questions regarding general information about prostate cancer, all other questions posed with the general public as the target group were at a more difficult readability level when compared to basic patient information sheets provided on UpToDate. When comparison was made with the more in-depth (Beyond Basics) information sheet on UpToDate, readability between the two were comparable. Ease of use between information provided on ChatGPT and CCA was comparable for information regarding prostate and bladder cancer, with the readability of information on renal cancer being significantly easier for information provided on CCA.

When the additional criteria was included for information to be pitched to a 16-year-old audience, we found that the information provided was made significantly more readable, with an average improvement of 16.25 ± 12.05 on the readability scale. In most cases, this made the information provided easier to understand when compared to information on CCA or the Beyond Basics section on UpToDate.

Information provided for each category was examined to determine if it was accurate and sufficient to allow an average person without medical knowledge to learn more about the cancer in question. Through rigorous reviews independently by two specialist urologists, it was deemed to contain the basic information required for an understanding of each condition without being overwhelming or misleading. However, the consensus was that information provided by CCA and the Mayo Clinic were more patient focused and covered more aspects of the urological malignancies, including peri-operative care, surveillance, and patient support information, which is a vital aspect and very useful in aiding patients in understanding their own condition.

Discussion

The instant acceptance and proliferate use of ChatGPT makes it difficult to regulate and monitor or validate information that would otherwise be detrimental to patients receiving care or impact on patient–doctor relationships. In addition, information provided by government bodies, health organizations, and medical institutions have been specially designed for use as educational tools by patients, and have been screened and amended to suit a variety of target audiences, taking into consideration health literacy of the population, whereas information provided by artificial intelligence (AI) softwares may not exhibit the same motivation. To date, this is the first analysis of the readability of AI software in providing urological medical information to patients in the literature.

This study demonstrates the remarkable versatility and potential of ChatGPT when used appropriately. The instantaneous difference in language use, medical jargon, and resultant ease of readability with a single modification of age of the target audience is presented above. Gender, occupation, and ethnicity, among other non-modifiable personal information included in the question posed, would elicit even more targeted and improved individualized responses. The information included in the responses hase been demonstrated to be non-inferior to that presented by established healthcare governing bodies in terms of readability and lends weight to the usefulness of similar AI programs as adjuncts in dissemination of health literature and increasing health literacy.

The difficulty in the employment of intuitive AI technologies in medical practice stems from the inability to control and regulate what is being presented when questions are posed. Indeed, even when the same question is posed a different answer is generated with varying content and readability with consistency. Furthermore, the constant inundation of information on a daily basis means that information need to be constantly updated. The current ChatGPT-4 model is still based on formation from September 2021, meaning that current medical advancements are not included in the generated responses.

There are several limitations to this study. Firstly, a direct comparison of readability was made based on information acquired and does not look in depth into separate content of the various sources. It is difficult to qualitatively assess information provided to patients based on a retrospective cross-sectional study, though further studies in the future can be performed in this regard, looking at patient satisfaction and understanding when presented with the various sources.

Secondly, the Flesch-Kincaid reading ease score does not incorporate the impact that diagrams, images, audiovisual content, and indeed webpage design and layout have on patient understanding. The CCA and Mayo Clinic information sheets employ these tools extensively and this helps to further improve ease of understanding. To our knowledge, there is no tool that reliably assesses non-text content of digital or print information, thereby introducing bias into our study.

This is the first study to date looking at readability of information generated by ChatGPT when prompted about urological malignancies. Our study confirms that the majority of information provided by medical institutions around the world is above the average reading ability of an Australian layperson, ChatGPT included. The complex nature of urological malignancies necessitate a higher education level to be able to understand and assimilate the concepts presented. Therefore, a clinician is vital in helping patients to understand about their disease, available treatment options, and how it impacts on their activities of daily living so that patients can make an informed choice regarding shared treatment decision. Open access information systems like ChatGPT should only be used as an adjunct to clinical consults until further improvements can be made in the future.

A potential development in the future may include certain aspects of ChatGPT in the creation of a natural language chatbot with a medical journal database so that information extracted include only validated and peer reviewed data rather than indiscriminate crawling through the worldwide web. This database could be updated periodically to include the most up to date information so that the responses generated are current.

Conclusion

ChatGPT, when used appropriately, can revolutionize healthcare by providing a useful platform or tool with which to improve health literacy within the community. In its current state, ChatGPT cannot be considered as a source of factual information as the generated responses require further evaluation and validation. Further studies can be performed to determine how technology can make healthcare even more accessible and affordable to rural and remote communities, which traditionally suffer from worse healthcare outcomes.