ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students

Introduction

The emergence of artificial intelligence (AI) has revolutionized various fields, including healthcare and education. Among the latest innovations, the introduction of the Chat-Generative Pre-trained Transformer (ChatGPT) by OpenAI in November 2022 marks a significant stride in AI technology’s evolution. This model has harnessed the power of AI and natural language processing to simulate human-like conversation, offering both transformative potential and challenges across scientific domains, particularly in healthcare.¹ While ChatGPT has shown promising capabilities in various medical applications, including some positive outcomes in medical licensing examinations, recent studies, such as the one conducted on the pharmacist licensing examination in Taiwan, highlight the need for careful consideration of its limitations in specific assessment contexts.^2,3

Within healthcare, ChatGPT has showcased its ability to generate formal discharge summaries swiftly in response to brief patient profiles. It has also found applications in enhancing radiological decision-making, risk prediction, and even drug discovery processes.^4,5 Beyond the technological dimension, ChatGPT extends medical information access to patients, promising personalized healthcare enhancements across diverse clinical disciplines, such as cardiology, endocrinology, and gynecology.^6–8

In the sphere of pharmacy, AI-powered systems have captured attention, with potential applications encompassing prescription review, drug counseling, adverse drug reaction (ADR) monitoring, and patient medication education.¹ This resonates profoundly with clinical pharmacists, who play a pivotal role in optimizing medication therapy, ensuring patient safety, and offering patient-centered care. The integration of AI-powered tools into clinical pharmacy practices could bolster efficiency, decision-making, and patient outcomes.⁹ However, despite the promise, the accuracy, reliability, and practical viability of ChatGPT in real-world clinical pharmacy settings have yet to be comprehensively explored.^1,10 Previous studies in the field have shown promising results for ChatGPT, but the reliability and accuracy need to be thoroughly validated in real-world healthcare settings. Some studies have highlighted the importance of evaluating ChatGPT’s performance across diverse clinical scenarios to ensure its practical viability.^11–13 The investigations have undertaken an exploration of ChatGPT accuracy through diverse methodologies. Notably, one of these inquiries employed a set of 20 pharmacotherapy cases to scrutinize both the accuracy and consistency of ChatGPT. The findings from this study revealed discernible variations in accuracy rates across different temporal intervals.^1,14

Recent research endeavors have shed light on ChatGPT’s medical education potential, ranging from assisting students’ understanding of complex topics to providing tutorials and homework aid.² Yet, the integration of ChatGPT to pharmacy practice training remains to be investigated. This study aimed to assess Pharm-D students’ perceptions, concerns, and ‎practices of integrating ChatGPT into pharmacy practice training. ‎

Method

Study Design and Setting

This cross-sectional study was conducted between April and May 2023 to assess students’ perceptions, concerns, and ‎practices related to the integration of ChatGPT into clinical pharmacy education. ‎The study included PharmD students studying at the two public pharmacy schools in Jordan. A convenient sampling method was used to select the participants by distributing the study questionnaire using an online platform (Facebook and WhatsApp). This sampling technique was chosen for its broad accessibility among pharmacy students and alignment with their digital communication habits, facilitating efficient data collection. Additionally, considering that the final-year students were involved in ward rotations, it may not be feasible to gather all of them in one place to distribute the questionnaires. Those who expressed interest in participating were provided with a link to view the study’s instructions and to provide their electronic consent before proceeding to the survey.

Survey Development

The initial study questionnaire was generated and evaluated for content and face validity by two survey experts. Minor modifications were made based on the notes and feedback provided. The final version of the questionnaire consisted of four parts. The first part comprised demographic characteristics of the study population. The second part included statements to assess students’ perceived benefits of ChatGPT incorporation into their clinical training. The third section included statements to evaluate students’ concerns related to ChatGPT incorporation into their pharmacy training. The last part evaluated students’ experience with the use of ChatGPT in clinical training. A 5-point Likert scale (1: strongly disagree, 2: disagree, 3: neutral, 4: agree, and 5: strongly agree) was utilized to evaluate the second and third sections. A benefit and concern scores were calculated for each student. A 5-point Likert scale ranging from strongly disagree to strongly agree was utilized to evaluate the students’ perceived benefits and concerns, with scores ranging from 1 to 5, respectively. So, the scores for perceived benefits (16 items) ranged from 16 to 80, and the overall score for perceived concerns (8 items) ranged from 9 to a max score of 45.

Sample Size Determination

The sample size was estimated using Raosoft ^® sample size calculator for online survey using the following formula: n = P × (1−P) × z2/d2. With a margin of error (d) of 10%, a confidence level of 95%, a population size of 800 PharmD students, and a response distribution (P) of 50%, the minimum recommended sample size was 86 participants.

Ethical Considerations

Ethical approval for the study was granted by the Institutional Review Board at the Jordan University Hospital (Approval number: 10/2023/16,283). The current study was conducted following the Declaration of Helsinki of the World Medical Association.

Statistical Analysis

The data analysis was conducted using SPSS software, version 25 (IBM Corp., Armonk, NY, USA). Frequency and percentages were computed for categorical variables. The data normality was examined using the Kolmogorov–Smirnov test and histogram visualizations, revealing a non-normal distribution. The test yielded a significant result (p < 0.05), affirming this non-normal distribution. Mann–Whitney U-test was used to assess the differences in perceived benefit and concern median scores between ChatGPT users and non-users. A p-value of less than 0.05 was deemed to signify statistically significant results.

Results

Sociodemographic Characteristics

Table 1 provides an overview of the sociodemographic characteristics of the 211 students recruited in this study. The median age of participants was 23 years (IQR= 1.0), and the majority were males (n= 163, 77.3%). For the academic qualification, over half of the respondents were sixth-year students (n= 120, 56.9%). The sample is nearly evenly split, with 50.2% (n= 106) from the University of Jordan and 49.8% (n= 105) from the Jordan University of Science and Technology. Delving into their technological familiarity, over two-thirds of participants (n= 144, 68.2%) had prior experience with artificial intelligence. Proficiency in computer and digital technology usage varied, with the majority (n= 180, 85.3%) indicating they have good to excellent proficiency. Notably, over two-thirds of the participants (n= 145, 68.7%) were aware of ChatGPT.

Table 1 Sociodemographic Characteristics of the Study Sample (n=211)

Students’ Perceived Benefits of Incorporating ChatGPT into Clinical Training

Table 2 describes the students’ perceived benefits of incorporating ChatGPT into clinical training. A significant percentage of participants (n= 139, 65.9%) believed that PharmD students could benefit from using ChatGPT. However, when it comes to ChatGPT’s ability to provide accurate clinical practice information, only 39.8% (n= 84) believed so. Over half of the respondents believed ChatGPT could expedite the process of identifying and managing treatment-related problems (n= 121, 57.3%). Moreover, 54.5% (n= 115) perceived ChatGPT as a tool that could readily identify and manage drug-drug interactions. Additionally, half of the participants (n= 106, 50.2%) trusted ChatGPT’s ability to identify and manage adverse drug reactions.

Table 2 Students’ Perceived Benefits of ChatGPT Integration in Clinical Training (N= 221)

When it came to patient education, 46.9% (n= 99) believed that ChatGPT could assist in that task. The platform’s potential was also recognized in medication reconciliation, with over two-fifths (n= 94, 44.5%) believing in its utility for such purposes. On the other hand, under a third of participants (n= 69, 32.7%) agreed or strongly agreed that ChatGPT can be of assistance in determining appropriate dosage regimens for ‎patients. Regarding medication prescription, 38.9% (n= 82) believed ChatGPT could identify wrongly prescribed medications, 36.5% (n= 77) agreed/strongly agreed it could recognize unnecessary medications, and 35.5% (n= 75) believed it could pinpoint additional medications required for patients.

Regarding the development of pharmaceutical care plans, 43.6% (n= 92) and 48.3% (n= 102) of participants believed ChatGPT could aid in crafting pharmacological and non-pharmacological plans, respectively. Less than half of students (n= 97, 46%) agreed on ChatGPT’s potential in therapeutic drug monitoring, and a similar percentage (n= 98, 46.4%) believed in its efficacy in clinical pharmacokinetic calculations. Overall, while a significant proportion of students see the potential benefits of ChatGPT in clinical training, a noteworthy percentage remains neutral or skeptical about its capabilities.

Students’ Concerns About the Utilization of ChatGPT in Clinical Training

Table 3 illustrates the students’ concerns regarding the application of ChatGPT in clinical training. Nine areas of concern have been identified. First and foremost, 61.1% of the respondents (n= 129) echoed concerns on three aspects: the potential for over-reliance causing a deficiency in essential skills such as critical thinking, concerns about ChatGPT’s accuracy and reliability, and the potential pitfalls in ChatGPT’s contextual understanding which could lead to inadequate responses. Even more notably, the digital divide’s implications for learning stood out, with 58.8% (n= 124) apprehensive that disparities in technological accessibility might jeopardize equal learning opportunities.

Table 3 Students’ Concerns for the Utilization of ChatGPT in Clinical Training (N= 211)

Additionally, there is a palpable apprehension about the potential compromise in personal interactions, as a significant percentage of students (n= 123, 58.3%) felt that ChatGPT might not provide the degree of personal interaction and feedback crucial for their training. Around half of the students (n= 117, 55.5%) voiced that ChatGPT might have a limited scope, potentially not encapsulating all essential topics for pharmacy training. Ethical concerns also surfaced, with 51.2% of the students (n= 108) worried about potential breaches in patients’ privacy and confidentiality. Lastly, the integrity of ChatGPT’s training data was raised, with 50.2% (n= 106) believing that this could lead to potential biases in its responses.

Students’ Adoption of ChatGPT in Clinical Training

Table 4 shows the students’ adoption of ChatGPT in clinical training. Almost half of the students (n= 98, 46.4%) did not use ChatGPT in their clinical training, with only a minority (n= 27, 12.8%) having used it frequently. When participants were asked about the likelihood of incorporating ChatGPT into their future clinical practice, over half (n= 112, 53.1%) indicated that they were somewhat likely or very likely to do so. In terms of recommending ChatGPT’s use for pharmacy students and preceptors, opinions varied: 63% (n= 123) would recommend it, 25.6% (n= 54) remained undecided, whereas 11.4% (n= 24) would not recommend it.

Table 4 Students’ Adoption of ChatGPT in Clinical Training (n=211)

Table 5 provides insights into the students’ utilization of ChatGPT in clinical training. Among those who have ever used ChatGPT (n= 113), participants’ engagement with ChatGPT for various clinical tasks is noteworthy. Just over a third of the participants (n= 72/113, 64.6%) acknowledged employing ChatGPT to evaluate drug-drug interactions. Similarly, a slightly higher percentage of participants (n= 82/113, 72.6%) reported employing ChatGPT to acquire information about medications and diseases and aiding in the identification and management of adverse drug reactions (n= 74/113, 65.5%).

Table 5 Students’ Experience with the Use of ChatGPT in Clinical Training (n=113)

Furthermore, the utilization of ChatGPT extended to other critical clinical functions as well. Approximately one-third of participants expressed employing ChatGPT for medication reconciliation (n= 66/113, 58.4%) and determining appropriate dosage regimens (n= 67/113, 59.3%). Moreover, 55.7% of the participants (n= 63/113) utilized ChatGPT to aid in the development of pharmacological pharmaceutical care plans. Notably, a higher proportion of respondents (n= 80/113, 70.8%) had utilized ChatGPT for the development of non-pharmacological pharmaceutical care plans. Interestingly, ChatGPT’s utility extended to aiding students in therapeutic drug monitoring (n= 67/113, 59.3%) and clinical pharmacokinetic calculations (n= 64/113, 56.6%), highlighting its potential in supporting complex clinical decision-making processes.

Figure 1 shows the results of Pharm-D students’ perceptions regarding their experience with ChatGPT across various facets of pharmacy training, while considering the efficiency and accuracy compared to established methods. In terms of developing pharmaceutical care plans, students indicated a higher level of perceived usefulness for non-pharmacological care plans, with 46.2% finding ChatGPT equally helpful and 27.7% finding it more helpful compared to human-made decisions. This contrasts with pharmacological care plans, where 41.5% reported equal helpfulness and 21.3% reported more helpfulness.

Figure 1 Students’ perception about how they found ChatGPT when they used it in various pharmacy training aspects. Notes: Percentages were calculated out of those who had used it as presented in Table 5.

For clinical pharmacokinetic calculations, approximately 39.1% of participants found ChatGPT to be “equally helpful”, while 30.9% considered it “more helpful” compared to other calculation tools. In contrast, the perceived usefulness of ChatGPT in determining appropriate dosage regimens presents a different trend. Only 13% of participants reported finding ChatGPT “more helpful”, while a larger percentage (55.6%) indicated that it was “equally helpful” compared to other standard resources. When considering therapeutic drug monitoring, a notable 48.1% of participants found ChatGPT to be “equally helpful”, 32.1% reported finding it “less helpful”, while 19.8% found it “more helpful”.

The results of Figure 1 also underscore the remarkable perceived usefulness of ChatGPT in the domain of checking Drug-Drug Interactions (DDI). This aspect stands out as one of the most positively received functionalities, with a substantial majority of participants expressing high levels of perceived helpfulness. Remarkably, over 80% of respondents indicated that ChatGPT’s utility in this domain was either “equally helpful” or “more helpful” compared to other resources typically utilized for assessing drug interactions.

When comparing previous experience in using ChatGPT with their perceived benefit and concern scores (Table 6), previous users showed higher perceived benefit perception scores and lower perceived concerns scores compared to non-users, but these differences were not statistically significant (P>0.05 for both). However, participants more inclined to use ChatGPT in future clinical practice tended to exhibit lower concern scores, showing a statistically significant correlation (p-value of 0.038 for those “Somewhat likely” to use). Additionally, those more likely to recommend ChatGPT (“Yes, definitely”) presented higher perceived benefit scores (3.8 with a p-value < 0.001). These correlations suggest a link between users’ attitudes towards ChatGPT, their perceived benefits, and concerns, potentially influencing their inclination to use or recommend it in clinical practice.

Table 6 Correlation Analysis of Participants’ Concerns and Benefits Scores with Prior Usage, Future Intent, and Recommendation for ChatGPT Implementation in Clinical Practice

Discussion

The implementation and application of the large-scale language model, ChatGPT, is increasing tremendously in clinical training and education. It offers capabilities that, in some respects, exceed the limitations of humans, but in others, it is surpassed by human´s versatility, emotional intelligence, and comprehending complex intersecting concepts.¹⁵

Results revealed that two-thirds of the participants believed that ChatGPT offers them benefits, especially in the identification of treatment-related problems, drug-drug interactions, and adverse drug reactions participants. Many other applications of ChatGPT were studied, such as differential diagnosis,¹⁶ clinical decision-making,¹⁷ and clinical decision support.¹⁶ The perceived benefits of incorporating ChatGPT in education were echoed in a mixed-method approach study conducted in Ghana. Bonsu and Koduah investigated the students’ perceptions and their willingness to use ChatGPT in education. Most participants expressed being comfortable utilizing ChatGPT, which guarantees a satisfying experience as they take it up in their academic careers. The students also agreed that the answers generated by artificial intelligence were accurate. The researchers recommended adopting these innovative technologies in higher education accompanied by regulatory measures concerning ethical challenges such as creativity, misuse, and copyright issues.¹⁷

The accuracy of the data provided by the ChatGPT was questionable, and less than half of the participants believed it could produce accurate information. This reflects the concerns regarding the accuracy of this AI model, which is not surprising since ChatGPT is trained on certain databases, and it follows instructions rather than enrolling in interactions and investigations. The reported “hallucinations”, which is content generated by the model that is not based on reality, in the AI model can restrict its usefulness in aspects where accuracy is required.¹⁸

These concerns were also reflected in medical students in Saudi Arabia. The students acknowledged many limitations of ChatGPT, despite its numerous benefits, such as lack of trust in the produced results, obscurity of references, and inability to conduct analysis that requires critical thinking. The participants emphasized the need for policies and regulations to detect and manage any academic misconduct.¹⁹ Another study that was conducted in Korea investigated the perceptions of ChatGPT as a feedback tool in medical school. The majority of students agreed that it was useful in providing answers and summarizing information, but it did not provide correct evidence.²⁰ In Saudi Arabia, a large survey was conducted among 1057 healthcare workers to assess their perceptions of the advantages of ChatGPT and intentions for future utilization. Two-thirds of the participants believed that although it could positively contribute to healthcare provision, concerns regarding the accuracy and legal issues hinder its current use and should be accompanied by human supervision.²¹

Kusunose et al examined the ability of ChatGPT to accurately answer questions related to the management of hypertension. ChatGPT exhibited an overall accuracy rate of 64.5% and a higher accuracy for clinical questions of 80% compared to limited evidence-based questions, 80% and 36%, respectively.²² Another study that was conducted in the US revealed that ChatGPT responded appropriately to 21 of 25 questions (84%) related to cardiovascular disease prevention recommendations.²³ A recent study by Johnson et al explored the accuracy and reliability of AI-generated responses to 284 medical questions developed by thirty-three physicians across 17 specialties. The accuracy was determined using a 6-point Likert scale (range 1 - completely incorrect to 6 - completely correct). The median accuracy score for all questions was 5.5, between almost completely and completely correct.²⁴

Many participants agreed that ChatGPT can be helpful in providing patient education. Communication with patients seems to be an area where ChatGPT can be very helpful.²⁵ However, the comprehension of online information by the patients may be an impediment and could create disparities among patients with different education levels.²⁶

The views of the students concerning the utilization of ChatGPT in their clinical training and the peril of equal opportunities concur with the debate on its role in education and the learning process. Many educators consider it a threat to the training to acquire knowledge and problem-solving skills and augment the inequality in education.²⁷

Ethical doubts that the participants expressed regarding the patient’s privacy are a legitimate concern. These echo a major limitation of ChatGPT, which is vulnerability to cyberattacks and the spread of information.^28,29 If ChatGPT is to be used in clinical practice, it should guarantee the main ethical principles such as beneficence, justice, nonmaleficence, privacy, and responsibility. Maintaining these principles, especially privacy, is challenging, and the protection of patient data is essential to prevent misuse of information.³⁰

The tasks that ChatGPT can be engaged in are diverse. The participants used ChatGPT in many clinical aspects, but the extent of use varies from one domain to another. Only one-third thought the ChatGPT utilized it to identify more sophisticated tasks such as wrongly prescribed medications, unnecessary medications, and dosage forms. Variation in the performance of this AI model was established, it was inferior in questions related to differential diagnosis and clinical management compared to general medical knowledge.³¹

Most participants agreed on the usefulness of ChatGPT for non-pharmacological care plans. This is probably due to the comprehensive information that the ChatGPT can provide on different aspects, especially if given sufficient parameters. Galido et al investigated the applications of ChatGPT in the management of Schizophrenia. The findings of the study showed that the AI model provided comprehensive nonpharmacologic treatment approaches, vocational and social skills training, and supportive services.³²

The most positively received functionality of the ChatGPT by the participants who used it, compared to other sources typically utilized for assessing drug interactions, was drug-drug interactions. However, Moratth et al demonstrated that in real-world scenarios, ChatGPT answered most of the 50 drug-related questions wrong or partly wrong.³³ Another study was conducted by Hsu et al that evaluated ChatGPT performance in Drug-Herb interactions for traditional Chinese medicine. They concluded that the responses were evaluated as inappropriate and lacked information that is needed to provide patients with clear recommendations.³⁴

The positive attitude toward using ChatGPT by the students implies that it will play a major role, with its pros and cons, in their academic activities. This necessitates that universities and institutions should design strategies, policies, and specialized centers to regulate the use of this source of information and familiarize students with its limitations and ethical dilemmas that might be encountered. On the other hand, AI sources should be developed, continuously improved, and tested for their use in the clinical field. We should bear in mind that different AI Chatbots may differ in the type of response and information provided. Consequently, large language models should be improved to provide better accuracy and relevancy. This can be achieved by iterative refinement, using examples to mimic human-like responses, and expanding the training data to suit medical purposes and scenarios.

This study is the first in Jordan to evaluate the perceptions and attitudes of PharmD students towards the incorporation of ChatGPT in their clinical practice. However, one limitation of the study is that it did not examine the effect of ChatGPT utilization by students on the accuracy of their clinical decisions or patient outcomes. Also, data collection relied on convenience and snowball sampling techniques, which may introduce selection bias. Moreover, utilizing social media for participant recruitment prevents the calculation of a survey response rate.

Conclusion

Our study examining Pharm-D students’ perceptions, concerns, and practices regarding ChatGPT integration in pharmacy practice training revealed diverse perspectives. Participants acknowledged potential benefits in expediting treatment-related problem identification and drug interactions but expressed skepticism about its accuracy in clinical information provision. Significant concerns included over-reliance, digital divide implications, compromised personal interaction, and ethical considerations. While nearly half had not used ChatGPT, many expressed future intent. Recommendations varied, with prior users indicating higher benefits and lower concerns, suggesting potential links to adoption and recommendations. The study emphasizes nuanced perspectives and highlights the need for careful consideration in effectively integrating ChatGPT into pharmacy practice.