Continuing Medical Education in Vietnam: A Weighted Analysis from Healthcare Professionals’ Perception and Evaluation

Introduction

In every country, continuing medical education (CME) is a compulsory requirement for health professionals, particularly important among those working in clinical settings.^1,2 This requirement is due to the development of science and technology in clinical fields as well as the emergence of new and complicated diseases.³ Therefore, participating in CME courses helps health professionals not only to reinforce and systematize knowledge and skills but also to update with new knowledge and skills to meet the healthcare needs in this changing world. Vandergrift, Gray, Weng⁴ analyzed a sample of 19,563 general internists and highlighted the association between CME requirements and improvement of physician knowledge, practices as well as confidence in clinical performance. Despite the low-quality evidence synthesized, a systematic review of 33 published studies has revealed both short-term and long-term effectiveness of CME courses.⁵ Because of this important role of CME, health professionals must have a certain number of CME credits to maintain their licenses. In Vietnam, health professionals in clinical settings must have at least 48 CME credits in two consecutive years.

For many health professionals, joining the CME courses has become a great opportunity to meet and discuss with experts and colleagues in the same or different disciplines and specialties to improve their perceptions and skills. However, many health professionals in resource-limited settings do not fully understand the role of CME⁶ and thus participating in CME courses is their way to get enough CME credits to maintain their licenses regardless of their improvement in clinical knowledge and performance. Therefore, to improve the effectiveness of CME courses, organizers have tried different approaches to enhance the course quality and to increase participants’ satisfaction,² such as inviting good instructors, and designing a practical curriculum to meet the attendee’s objectives. Several studies have revealed that CME courses with the focus on improving knowledge alone are not as effective and attractive as those with the focus on improving both knowledge and practice.^2,7,8 However, to date, there is a paucity of scientific research to evaluate the effectiveness of CME courses, and studies that have been conducted so far have focused on certain specialties, for example in family physicians or primary care physicians.^9,10 The effectiveness of CME courses in other specialties is still controversial or unclear.

In Vietnam, policies regarding implementation and requirements for CME in health professionals have been issued by the Ministry of Health since 2013. Technically, CME courses are short courses organized to provide health professionals with updated knowledge, attitudes, and skills in their specialties, or to re-train health professionals so that they can do their routine health care with minimum medical errors. Attendees at CME courses are not only physicians but also nurses, pharmacists and medical technicians such as rehabilitation therapists. However, to date, there has been no study on the effectiveness of these CME courses in Vietnam. As suggested by the Kirkpatrick model, the evaluation of CME programs should start with level one, which is how participants react to the training. Therefore, this study aimed to assess participants’ reaction through their perception and evaluation. Findings from this study can be helpful for healthcare policymakers to have an overall understanding of the effectiveness of CME and to have adjustment in implementing this important activity if needed.

Materials and Methods

Settings and Study Design

During a five-month period from September 2019 to January 2020, a cross-sectional study was conducted in all 62 CME courses at a university hospital in Vietnam. Each year, the hospital organizes over 100 CME programs to provide updated knowledge and experience sharing for healthcare professionals across the country. These courses are mostly on-site, in-person but vary in the study duration (from one day to several days, one full day or one session per day for several days), the specialty (i.e., for nurses only, for general physicians, for surgeons only or for all health-care workers), the qualification of attendees (mostly undergraduate) and the number of attendees (ranging from about 20 up to 200).

During the study period, attendees of all CME courses regardless of variation mentioned above were eligible to join the study. The formula used to calculate sample size was based on estimating a mean score of the Program Evaluation Instrument scale.¹¹ A type one error rate of 0.05, a standard deviation of the PEI score of 1.3 and its corresponding marginal estimating error of 0.1 were based on a previous study.² To consider the cluster effects where attendees from the same CME course might have similar evaluation toward that course and thus intra-class correlation was high, a design effect of 2 was chosen.¹¹ From this calculation, a minimum sample size was 1300. With an anticipation of refusal rate of 10%, at least 1445 participants were needed.

Study Procedures

During the CME courses, we randomly selected 30 participants from a list of participants in each course. For courses with fewer than 30 participants, all participants of that course were selected. Participation was through verbal consent and on a voluntary basis. A self-report, anonymous questionnaire was distributed to the participants during the course. Completing and returning the complete questionnaire indicated participants’ confirmation to join the study. At the end of the course, this questionnaire was collected. All procedures performed in this study including verbal informed consent were approved by the Human Ethics Committee at University Medical Center Ho Chi Minh City, Vietnam (Approval number: 24/GCN-HDDD) and were in accordance with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Measurement

The self-report questionnaire used in this study had three parts. The first part contained questions about demographic characteristics including sex, age, education level, institution level and profession. The second part had questions about experiences during the current CME course, including information resources about the current CME, receiving financial, practical support for the current CME course (i.e., being encouraged to attend, being allocated with time to attend), attendance rate and preferred learning style. Seven items about participants’ perceptions toward CME in general were adapted from a previous study and were also included in this part.¹² Participants rated each statement on a 6-point Likert scale from one (strongly agree) to six (strongly disagree).

The main outcome in this study was participant’s evaluation toward CME programs assessed through the Program Evaluation Instrument (PEI).¹³ This scale contains 19 items using a 6-point Likert scale from one (strongly disagree) to six (strongly agree) and covers four aspects of a training course, including program objectives (item 1, 3, 4, 13), learner’s objectives (item 6, 8, 12, 15), teacher’s behavior (item 2, 7, 9, 10, 14) and program satisfaction (item 5, 11, 16, 17, 18, 19). To calculate total scores of the PEI and four subscales, item 8, 11, 17 are reversed. The total score of PEI ranges from 19 to 114, a higher score indicates a higher level of positive evaluation toward CME programs. This scale has good psychometric properties with Cronbach’s alpha from 0.70 to 0.82 for the four subscales and 0.92 for the total scale, indicating a high level of internal consistency.^2,13 In this study, the scale was translated into Vietnamese by two researchers independently. Differences in the two translations were discussed and resolved to have the final version. Almost no difference was found in the two translations, except some minor wording which was then discussed and agreed between the two translators. A pilot study in a convenient sample of five participants was conducted to evaluate the wording, which resulted in no further changes.

Data Analysis

Conventional descriptive statistics were used to describe the data, including mean and standard deviation for quantitative variables, frequency and percentage for qualitative variables. To facilitate interpreting the results, Likert-type items were dichotomized. To adjust for the cluster effect and to consider the sampling weight, weighted estimate for these statistics was calculated. In this study, the sampling weight was the inverse probability of being chosen in this study among all attendees of a CME course. To compare the main outcome (i.e., PEI score) among participants with different characteristics, t-tests and analysis of variance (ANOVA) were used with and without the adjustment for the cluster effect and sampling weight. Type one error was set at 0.05 and thus a p-value of less than 0.05 was considered statistically significant. All data analyses were conducted using Stata version 16.

Results

From 62 CME courses, a total of 1419 attendees agreed to participate in this study. Among these, 157 (10.7%) attendees returned incomplete questionnaires and were excluded from the analysis. Among 1312 participants in this analysis, the majority were females (58.1%) with a mean age of 34.5 (SD 10.6) years. Most physicians (59.5%) had an undergraduate degree (48.0%), and worked at national or provincial institutions (49.9%) (Table 1).

Table 1 Demographics Characteristics and Experiences of Attendees of CME Events (n = 1312)

In terms of CME experiences, approximately 80% of participants reported ever having participated in a CME course. The most common information resources about the current CME courses were from colleagues and invitation letters. About three-quarters reported having practical support from home institutions but only 37% received financial support. Most participants reported an attendance rate of 80% or greater. More than 75% expressed the preference for offline, face-to-face CME courses and only 15.8% preferred online CME programs. Almost all attendees had good perceptions toward CME in general where CME was reported to be helpful in improving knowledge, skills and daily practice. However, about 5% of participants reported CME a waste of time.

Table 2 presents the participants’ evaluation toward the current CME program assessed through the PEI scale. Participants agreed on almost all the questions measured and thus indicated a high level of positive reaction toward the CME course they were attending. However, about 52% of participants reported that the program objectives demonstrated during the CME course were different from their expected objectives and nearly one-third thought that program materials were too difficult. Interestingly, nearly 15% reported not much was gained from the CME course. The pooled scores of these questions to reflect four dimensions of the PEI scale were analyzed in detail (Figure 1). The results indicated a high level of positive evaluation in all four dimensions measured including program objectives (20.7 ± 2.2), learner’s objectives (18.8 ± 2.3), teacher’s behavior (25.7 ± 2.7) and program satisfaction (29.7 ± 3.4). The total score of the PEI was high at 95.0 ± 8.9.

Table 2 Participant’s Evaluation of CME Assessed Through the Program Evaluation Instrument

Figure 1 Four dimensions of evaluation toward CME assessed through the Program Evaluation Instrument.

The levels of participants’ reactions as measured through the total score of the PEI among different characteristics of participants are shown in Table 3. While there was no association between demographic characteristics and the total score of PEI, attendees’ experiences were associated with the PEI score, particularly during the current CME course. Those who attended 90–100% of the CME sessions and chose an offline learning approach reported a higher level of positive evaluation (p = 0.019 and p = 0.021). Participants who had positive perceptions toward CME reported significantly higher level of positive evaluation compared with those with negative perceptions (p <0.001 to p = 0.035).

Table 3 Factors Associated with Participant’s Evaluation Toward CME Measured by the Total Score of the Program Evaluation Instrument

Discussion

In our study, attendees from 62 CME courses during a five-month period reported positive perceptions and high level of positive reaction. Four aspects of evaluation measured through the PEI scale including program objectives, learner’s objective, teacher’s behavior and program satisfaction were also high. Based on the same measurement, the level of positive perceptions in our study was higher than that reported in 601 attendees from 30 courses in Saudi Arabia.¹⁴ The positive reaction toward CME courses as rated by the attendees in our study was also higher compared with a previous study among 45 physicians in Iran,² where almost all the items in the PEI had lower mean values than ours. Although the absolute results might be different in various settings, it is difficult to compare the real participant’s reaction toward CME in alternative settings due to several reasons. Findings from a systematic review indicated that apart from the tools used to measure participant’s reaction, it is also important to compare the content, learning style, and administration-related procedures.¹ However, information about these aspects is often limited from published studies.

It is important to identify whether the participants’ reaction was related to their characteristics so that future interventions and modifications in CME courses can be made. In our study, none of the demographic characteristics was associated with the PEI total score. This indicates that the positive reaction toward these CME courses is consistently high among participants with diverse backgrounds. In previous studies, while older female CME attendees reported a higher level of attitude toward CME,¹⁴ the association between age and sex and CME perception was inconsistent. For example, the effectiveness of CME was positively associated with age in Saudi Arabia,¹⁴ and in females in the US¹⁵ but was found to have no association with age and sex in other studies.¹⁶ Interestingly, we found that the evaluation scores were not different between attendees who reported receiving practical and financial support and those who did not receive such support. Our findings and others demonstrate that we should not focus on background characteristics of attendees to increase the participant’s reaction toward CME programs.

Findings from our study indicate that experiences and perceptions toward CME are important factors associated with the overall CME evaluation. Those who had high attendance rate and preferred offline CME reported higher levels of positive reaction. This finding is as expected since study styles and attendees’ performance during the course have been proven to affect the overall effectiveness of medical education.¹⁷ Moreover, those with positive perceptions toward CME had higher reaction scores. The role of perceptions and attitudes toward CME and its importance as a lifelong learning has been shown to be a core component contributing to the success and effectiveness of CME courses.^8,14 However, it is important from our study to note that about 5% of participants reported that CME is a waste of time. Such perception toward CME has also been reported in previous studies, particularly in resource-limited countries where the role of CME is not fully understood.⁶ It is therefore essential to enhance the role of continuing medical education in clinical practice for healthcare personnel.

Several implications can be learned from our study findings. First, most CME attendees were young, undergraduate physicians working at national and provincial institutions. While this is a positive signal that CME has gained good attention from this population, a large proportion of health professionals such as those over 40 years old with postgraduate education and those who work at district health services or in private sector might have not realized the importance of CME. In Vietnam, older health-care professionals with high education level are often those who have high position at their institution with most of their time spent on management and administration issues rather than their specialty. Since CME is a compulsory requirement for every health-care professional in Vietnam, enhancement of the critical role of CME and lifelong learning in this population is needed. Second, in Vietnam, CME courses have not yet been well categorized to suit various levels of education, working experiences and each participant’s specific needs. It might be good to follow recommendations from previous studies to organize CME programs specifically for each audience group to optimize the effectiveness.

Our study still has several limitations. First, this study was conducted at only one hospital and thus limited in the generalization of the study findings. Although the study site is among the biggest hospitals in the south of Vietnam, CME courses at other hospitals and institutions in various areas of Vietnam might be different. Second, each participant’s reaction toward a training program depends significantly on the organizers, instructors, training curriculum and materials and equipment used, particularly for practical sessions. In this study, due to the inclusion of various specialties and a large number of CME courses with different learning and teaching styles, we were unable to collect and synthesize these factors. It is difficult to compare the participants’ reaction of CME courses among disciplines, for example CME programs for ophthalmologists, for pediatricians and for surgeons. Further studies are needed, possibly focusing on certain specialties to confirm findings in this study. Moreover, as recommended by Moore et al.¹⁸ evaluations of CME should be based not only on the perceptions and satisfaction of attendees, but also on other aspects such as competence, performance, patient’s health and community health. The effectiveness of CME should be confirmed through these evaluations in future studies.

Conclusion

From attendees’ perceptions and evaluation in a large number of CME courses, CME seems to be effective in providing and updating knowledge, attitudes and practice for health-care professionals in Vietnam. Attendance rate of participants during the courses and their perception toward CME are essential factors associated with their reaction toward such CME. However, further studies are needed in other settings and specialties to fully understand the effectiveness of continuing medical education in Vietnam.