Competency-Based Evaluation Methodology Enhances Learning During Internship: A Pilot Study

Introduction

The concept of “competency” was first proposed by David McClelland in 1973.¹ In 2002, Epstein and Hundert further defined professional competence, which involving communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection.² The competency includes not only explicit parts such as professional knowledge and professional skills, but also deep parts such as personality traits, motive, and self-concept.³ With the development of many years, various models of clinician competency have gradually emerged and developed.^4–6 Under the competency-based medical education (CBME) training and evaluation model, competencies are divided into distinct, independent dimensions. This compartmentalization makes it challenging for CBME to conduct holistic assessments of medical students within the clinical practice environment. To address this limitation, entrustable professional activities (EPAs) offer a promising solution.^7,8 EPAs are grounded in workplace-based teaching and assessment, aligning with the practical demands of medical education and clinical care by integrating multiple competencies into unified, real-world clinical tasks.^9,10 While competency frameworks outline the overall professional identity of physicians, EPAs specifically define the concrete clinical tasks that learners must be able to perform independently and effectively. In 2014, the Association of American Medical Colleges (AAMC) established a set of core entrustable professional activities for graduating medical students.¹¹ These EPAs have since been widely adopted in undergraduate medical curricula and are also used extensively in the evaluation and training of residents during postgraduate medical education. In 2018, China’s first Consensus on Core Competency Framework for Residency Education was published, and six core competencies for residency were identified, which including professionalism, medical knowledge and skill, patient care, communication and collaboration, teaching, and life long learning.¹² Peking Union Medical College (PUMC) has established a “competency-based” medical evaluation system of eight-year program, which runs through the whole process of medical students from enrollment to graduation, mainly including multiple mini-interview, objective structured clinical examination (OSCE), etc.³

Internship represents a crucial phase in medical education. Traditional surgical internship assessment in the Department of Urology faces several limitations, including singular evaluation dimensions, absence of quantitative standards, and inadequate process evaluation. The current emphasis on theoretical examinations and case analyses diverges from contemporary competency-based educational principles. The traditional written examinations have disadvantages for assessing knowledge, clinical reasoning, and self-confidence, while traditional viva of case analysis also has many disadvantages, including subjectivity, low validity, and low reliability.¹³ These methods inadequately assess students’ clinical reasoning, practical skills, and patient communication abilities, failing to comprehensively evaluate internship teaching quality. While OSCE evaluates medical students’ general capabilities during their education, research indicates its limitations in assessing surgical and communication skills.¹⁴ Additionally, the rapid advancement of minimally invasive urological surgery techniques and increasing disease complexity demand enhanced clinical learning assessment. Traditional evaluation methods demonstrate significant deficiencies in surgical learning assessment and fail to accurately reflect the comprehensive teaching process.

Currently, there are relatively few reports on CBME-related clinical internship evaluations for Chinese medical students. In light of the limitations of traditional internship evaluation methods, we aim to explore the application of a cpmpetency-oriented approach to internship assessment. This study will use urology internship as a starting point to develop and evaluate a new method for assessing internship outcomes grounded in CBME principles.

Materials and Methods

Establishing a Competency-Oriented Internship Learning Assessment Method

The competency-oriented internship learning assessment method was initially drafted by WD Wang and Y Zhao, and subsequently revised and finalized through discussions among WD Wang, Y Zhao, GY Zheng, and X Zhao. The Department of Urology at Peking Union Medical College Hospital (PUMCH) traditionally assesses internships through case analysis, wherein students gather patient medical histories, conduct physical examinations, formulate diagnoses with differential diagnoses, and respond to examiner questions. Building upon this foundation, a competency-oriented assessment utilizing standardized patients (SP) has been developed, comprising: (1) A refined assessment methodology. The updated SP-based evaluation simulates the complete hospital diagnosis and treatment process. This assessment encompasses medical history collection, physical examination, diagnosis and differential diagnosis, clinical decision-making, patient communication and informed consent, surgical procedures, and postoperative management. The “medical history collection” and “physical examination” are based on the communication with SP; “diagnosis and differential diagnosis” component includes evaluations based on auxiliary examinations; “decision-making” involves treatment plan development and rationale explanation; “communication and informed consent” requires interaction with SP regarding the patient’s condition and treatment plan to obtain consent; the “surgical detail” section presents surgical videos where students must explain procedural steps and identify critical anatomical structures (Figure 1); “postoperative management” involves explaining the patient’s post-surgical care. (2) Development of an SP-based examination case database covering common urological conditions, including renal cell carcinoma, renal cyst, prostate cancer, benign prostatic hyperplasia, bladder cancer, upper urinary tract urothelial cell carcinoma, urinary calculi, adrenal adenoma, and adrenal pheochromocytoma. The SP scripts were developed based on clinical cases. The SPs were selected from the PUMCH SP library, all of whom have extensive experience in SP-related work. The scripts were piloted and refined prior to the study.

Figure 1 Identification of important structures in the video of “laparoscopic-assisted radical nephrectomy”. (A) Renal artery (arrow); (B) ureter (arrow); (C) inferior vena cava (arrow); (D) adrenal gland (arrow).

Subjects and Grouping

The study was designed as a prospective study. The seventh-year interns of eight-year program at PUMC were selected as the research subjects. The Inclusion criteria: those who participated in the 1-month internship rotation in Department of Urology and participated in the after-department examination. The exclusion criteria: those who asked for leave for various reasons, failed to participate in the entire internship, or failed to participate in the after-department examination. The study lasted from March 1^st, 2024 to October 31^st, 2024 (10 months), and 41 interns were included in total.

The subjects were divided into two groups based on their internship period from March 1^st, 2024 to October 31^st, 2024. Students rotating during the time from March 1^st, 2024 to June 30^th, 2024, who were unaware of the updated assessment methods prior to evaluation, comprised the control group (20 interns). Students rotating during the time from July 1^st, 2024 to October 31^st, 2024, who received information about the updated assessment methods at internship commencement (awareness of the evaluation framework without scoring rubric), formed the observation group (21 interns) (Figure 2).

Figure 2 The flow diagram of the study.

The Implementation of the Evaluation

1. After-department examination scores encompassed medical history collection (0–10 points), physical examination (0–10 points), diagnosis and differential diagnosis (0–10 points), decision-making (0–10 points), communication and informed consent (0–10 points), surgical detail (0–10 points), and postoperative management (0–10 points). Each item had specific scoring criteria. Scores were assigned based on the accuracy and completeness of the response, ranging from a minimum of 0 to a maximum of 10. Assessment cases were randomly assigned to students. The evaluations were conducted by attending physicians with ≥ 3 years of clinical teaching experience. All attending physicians underwent evaluation training prior to the study. Scores were recorded immediately upon completion of each after-department examination.
2. Work engagement scores comprised engagement in ward work (0–10 points, based on the level of participation in ward rounds, a score of 0 indicates no participation whatsoever, while a score of 10 indicates full participation in all ward rounds), engagement in communication (0–10 points, based on the performance of participation in the working group regarding the informed consent process, a score of 0 indicates no participation whatsoever, while a score of 10 signifies full participation in all aspects of the informed consent procedures), and engagement in surgery (0–10 points, based on the performance of participation in the working group regarding surgery, a score of 0 indicates no participation at all, while a score of 10 indicates participation in all surgeries). Clinical teachers, who were attending physicians with ≥ 3 years of clinical teaching experience, evaluated these daily performances following the students’ after-department examination. All clinical teachers received evaluation training before the research.
3. Self-evaluation of ability improvement included clinical knowledge (0–10 points), decision-making capacity (0–10 points), surgical skill (0–10 points), colleague cooperation ability (0–10 points), communication ability (0–10 points), and continuous learning capacity (0–10 points). For the self-assessment of all project improvements, 0 indicates complete disagreement and 10 indicates complete agreement. Students completed their self-evaluation after the after-department examination.

Statistical Analysis

All statistical procedures were performed using IBM SPSS Statistics for Windows, version 27.0 (IBM Corp., Armonk, NY, USA). The measurement data were expressed as mean ± standard deviation, and the independent sample t test was used for comparison between groups; the count data were expressed as numbers (n) and percentage (%), and the Chi-square test was used for comparison between groups. P-values of <0.05 were considered to indicate statistical significance.

Results

Comparison of General Data Between Two Groups

The study included 41 interns: 21 students in the observation group (8 males, 13 females) and 20 students in the control group (9 males, 11 females). No significant statistical difference existed in gender between the groups (P= 0.654). The mean age was 24.62 ± 0.74 years in the observation group and 24.45 ± 0.69 years in the control group, showing no significant statistical difference (P= 0.453). The flow diagram was shown in Figure 2.

Comparison of After-Department Examination Performance Between Two Groups

Table 1 presents the SP-based assessment results for both groups. No significant statistical differences were observed in scores for “medical history collection”, “physical examination”, and “diagnosis and differential diagnosis”. However, the observation group demonstrated significantly higher scores in “decision-making”, “communication and informed consent”, “surgical detail”, “postoperative management” and “total score” compared to the control group. The analysis results for different genders are presented in Table 2.

Table 1 After-Department Examination Performance of Two Groups

Table 2 After-Department Examination Performance of Two Groups Considering Genders

Comparison of Work Engagement Between Two Groups

The work engagement scores of the observation group and the control group are presented in Table 3. The observation group demonstrated higher scores in “engagement in ward work”, “engagement in communication”, “engagement in surgery” and “total score” compared to the control group, with statistically significant differences observed between the two groups. The analysis results for different genders are presented in Table 4.

Table 3 Work Engagement of Two Groups

Table 4 Work Engagement of Two Groups Considering Genders

Comparison of Self-Evaluation of Ability Improvement Between Two Groups

The self-evaluation scores for capacity improvement in the observation group and control group are presented in Table 5. The analysis revealed no statistically significant differences between the groups regarding “clinical knowledge”, “decision-making capacity” and “continuous learning capacity”. However, the observation group demonstrated significantly higher scores in “surgical skill”, “colleague cooperation ability”, “communication capacity” and “total score” compared to the control group, with statistically significant differences observed between the groups. The analysis results for different genders are presented in Table 6.

Table 5 Self-Evaluation of Ability Improvement of Two Groups

Table 6 Self-Evaluation of Ability Improvement of Tow Groups Considering Genders

Discussion

This study aimed to further enhance the implementation of the internship process and improve outcomes by revising the internship assessment method based on CMBE. The results indicated that the competency-based evaluation approach not only improved performance in after-departmental examination but also promoted greater work engagement and enhanced self-evaluation of ability enhancement. This positive trend was consistent across both genders.

Scholars have defined competency as “possessing the required abilities in all domains in a certain context at a defined stage of medical education or practice”; thus, competency is multidimensional and dynamic, evolving with time, experience, and environment.^15,16 As competency requirements shift with temporal and environmental factors, competency-based medical education (CBME) may differ before and after graduation and across regions. Given diverse cultural backgrounds, educational systems, and social development, competency requirements in the Chinese medical student training system should incorporate both existing foreign frameworks and the unique characteristics of Chinese medical education.¹⁷ Matsuyama et al’s systematic review revealed that while CBME in Canada, Singapore, the Netherlands, the United Kingdom, and the United States shared structural and content similarities, differences emerged in terminology and expression due to national systems, cultures, and customs.¹⁸ A retrospective study by Urushibara-Miyachi et al demonstrates how CBME’s definition and expression evolved during its localization in Japan to align with Japanese medical education training requirements.¹⁹ Additionally, undergraduate medical education aims not only to enhance clinical ability but also to facilitate students’ professional identity formation (PIF).²⁰ This necessitates differentiation in CBME details between undergraduate and graduate medical education. In summary, whether it is undergraduate education or postgraduate medical education, the current clinical teaching based on CBME is the mainstream model for all stages of medical education, and its advantages are fully demonstrated. Nevertheless, in our hospital or in many regions of China, the assessment methods for medical interns are not in line with this concept. We have adjusted the internship evaluation model in an attempt to better align with the current mainstream educational philosophy and further enhance the learning process and learning outcomes of interns.

The “Standards for Basic Medical Education in China” defines medical education as a continuous process encompassing medical school education, postgraduate medical education, and continuing professional development. Undergraduate students must develop the capacity for continuing medical education post-graduation. During undergraduate education, graduates should acquire lifelong learning abilities in “science and scholarship”, “clinical practice”, “health and society” and “professionalism”²¹ The undergraduate internship phase represents a crucial bridge between “medical student” and “clinician” roles. Traditional evaluation methods in surgical departments typically emphasize theoretical knowledge and case analysis, inadequately reflecting the surgical learning process. This approach neither aligns with current clinical CBME nor facilitates improvement in clinical teaching processes. Consequently, we implemented changes to the traditional assessment method, developing an enhanced format aligned with CBME. The reformed after-department examination based on SP expanded from traditional evaluation components to include “medical history collection”, “physical examination”, “diagnosis and differential diagnosis”, “decision-making”, “communication and informed consent”, “surgical detail” and “postoperative management”, corresponding to competency-based training requirements.

To evaluate whether the competency-oriented SP-based assessment influenced internship processes and outcomes, the observation group received information about the assessment method, while the control group did not. During the after-department assessment, the observation group demonstrated significantly superior performance in “decision-making”, “communication and informed consent”, “surgical detail” and “postoperative management”. This indicates that competency-oriented SP-based assessment enhances internship learning outcomes. Throughout the internship, the observation group exhibited higher work engagement levels, suggesting that the competency-oriented assessment method stimulates learning process improvements. While surgical learning constitutes essential content in surgical department internships, previous assessment methods inadequately addressed this aspect. The competency-oriented assessment’s inclusion of “surgical detail” heightens student attention to surgical learning processes. The updated assessment method incorporates surgical video analysis for evaluating step identification and structural recognition, enhancing surgical teaching effectiveness while creating a “closed loop” with preclinical “topographic anatomy” coursework.

Students conducted self-evaluations to assess the competency-oriented assessment methods’ impact on their capability development in the Department of Urology. The self-evaluation encompassed six aspects: “clinical knowledge”, “decision-making capacity”, “surgical skill”, “colleague cooperation ability”, “communication ability” and “continuous learning capacity”. The observation group achieved significantly higher scores in “surgical skill”, “colleague cooperation ability” and “communication capacity”, indirectly confirming the competency-oriented assessment method’s effectiveness in enhancing students’ comprehensive abilities. Feedback, a crucial component of medical education, facilitates professional skill acquisition in CBME. Effective feedback should be timely, specific and encouraging, supporting learning goal achievement.²² While this study incorporated both clinical teacher and intern evaluation feedback, intern feedback was limited to self-evaluation, lacking assessment of teaching processes and methods.

This study is only an initial attempt at undergraduate internship assessment reform, without established systems, and has certain limitations. First of all, this study is non-randomized and sequential designed, and absent of sample size or power calculation. Secondly, the observation group interns were informed of the new assessment, although without much details, possibly leading to Hawthorne effect. Thirdly, there is subjectivity in scoring and needs for inter-rater reliability. In addition, this is only a preliminary research in Department of Urology in our hospital, and the generalizability in other departments or institutions is limited. Subsequently, further attempts can be made in other departments, and a multi-center randomized controlled study can be designed to further evaluate its feasibility. And future research should integrate theory and practice to enhance the CBME assessment framework, ensuring the updated assessment becomes integral to undergraduate CBME in China.

Conclusion

The competency-oriented SP-based assessment method implemented in the Department of Urology internship demonstrates potential promotion for undergraduate clinical education and internship learning outcomes. However, this conclusion is valid only for a single institution and department. Its application requires further evidence from higher-level studies involving multiple departments and centers.