Balancing standardized testing with personalized training in surgery

Standardized testing

For more than half a century in the US, standardized testing has been an integral part of the life of students, measuring their intellectual capacity and competency in a desired field of expertise. A standardized test is designed to assess the performance of students in settings in which the questions, conditions of test administration, scoring procedures, and interpretations of results are consistent.¹ The educational system in the US is replete with standardized testing to identify the most talented and intelligent students, with the SAT college admission examination being by far the most ubiquitous. Every year, more than 2 million students from more than 20,000 US high schools take this standardized test, which serves as a gauge for their acceptance into undergraduate colleges and universities across the nation.² Its impact is far-reaching; even fluctuations in real estate values have been reported to be linked with a county’s high school SAT scores.³

Despite its pervasive use, the validity of the SAT has been called into question because of alleged biases that are due to social inequities in American culture and not because of an inherent property of the test. Over time, however, multiple studies have shown that the test has an internal consistency coefficient of 0.91, and has high reliability, with the true score within 30 points of the measured score.⁴ In terms of validity, the SAT not only measures the construct that corresponds with other measures of academic performance such as grade point average, but also predicts future success in college.⁴

Standardized testing in medicine

The next big standardized test for someone pursuing a career in medicine in the US is the Medical College Admission Test (MCAT). Not only has MCAT been proved to be a reliable and valid predictor of unimpeded academic success in medical school, but the scores are also correlated with United States Medical Licensing Examination (USMLE) scores.^5,6 Similarly, USMLE is a predictor of success and competency during residency training and passage of in-training and board certification exams.^7–10 In the same way, the annual in-service exams, board certification exams, and increasingly, maintenance of certification examinations continue to be part of a physician’s medical career in the US.

Although the purpose of such continuous and rigorous testing is to develop medical personnel as lifelong self-directed learners, whether scoring high on each exam actually serves this purpose needs further consideration. These standardized exams function as a threshold to be crossed before medical students can progress to the next level in their academic life, ensuring that they are competent enough to fulfill the requirements of their desired profession. However, achieving that competency is shaped by the behavior and learning patterns of an individual student. The results of these exams not only represent a student’s performance but also reflect the standards of the school, program, and teachers (faculty) involved. All schools employ different teaching styles to prepare students to achieve high scores on their exams. It is worth mentioning that during high school, undergraduate, and medical school, the institutions, teachers, and students are equally responsible for learners’ education. However, once a medical graduate enters into residency training, the burden is solely the individual’s, and this partly explains the pervasive rigor in the process of resident selection by the residency programs.

The intent of this review is to shed light on teaching and learning patterns during general surgery residency training that not only guarantee trainees’ success on standardized tests and competency, but also help them become lifelong and self-directed learners.

Competency-based residency training

Due to the high cost and limited time and resources available for training activities in health care, especially during general surgery residency training, learners must be trained as efficiently as possible. Traditionally, in the US medical system, the dominant mode of teaching and learning during general surgery residency training has been the apprenticeship model, wherein residents work with faculty members under the “see one, do one, teach one” paradigm.¹¹ However, this model faces threats from new training standards imposed by Accreditation Council for Graduate Medical Education (ACGME) and the changing landscape of modern medicine that has brought limited work hours and a focus on reducing costs in health care.¹² With such a restricted environment and increasing demands for patient safety, both time for training and opportunities to observe a mentor are limited. In addition, a resident might be assigned to a specific procedure repeatedly, in which he or she has already achieved competency, while missing the opportunity to be trained in other difficult or rare procedures. Alternately, mentors may be caught up in teaching a task that they think is important, while a resident being instructed has already learned that particular task.

This situation gave birth to the competency-based model in which residents’ competency is routinely measured to make sure that their progress is satisfactory. Competency-based residency education is a process that focuses on acquisition of proven knowledge and skills and is not simply dependent on the length of training.¹³ The competency-based model overlaps cognitive, technical, and nontechnical domains.¹⁴ Timely achievement of milestones is measured not only by skills but also by the outcomes of performance. A true competency-based model de-emphasizes time-based training and promises greater accountability, flexibility, and learner-centeredness.¹² However, we believe that while objective and routine assessment systems have been put in practice, the concept of learner-centeredness has not yet been fully implemented in our medical education system. Attending physicians still regulate teaching, with less than optimal consideration for residents’ personal characteristics and attitudes toward learning. Moreover, a time-based residency curriculum is still in use rather than one based on achievement of milestones.

Learner-centered education and personalized training

Although the concept of personalized and learner-centered education has been successfully applied and practiced in various domains outside of medicine,^15,16 this practice is relatively new in medicine, and few studies are available to emphasize the utility of this approach.^17,18 These newer theories of education focus more on learners and their needs and personal factors that play an important part in the learning process. This allows for customization of the curriculum according to the needs and learning styles of the learners, in an attempt to increase the efficiency of the curriculum. This is very similar to the popular concept of personalized medicine, an example of which is individualized treatment plans for breast cancer that displays variable clinical, morphological, and biological features based on the molecular heterogeneity it exhibits.¹⁹ This gene expression profiling, along with biomarkers and prognostic multigene classifiers in a specific patient, then guides the diagnosis, prognostic evaluation, treatment, and follow-up.¹⁹ The purpose is to have a customized management plan according to the specificities of a disease in a given patient.

A similar concept is used in personalized education by identifying the learning patterns and styles of the students. Kolb and Kolb²⁰ described these learning styles as a part of their famous experiential learning theory. They explain that hereditary makeup, unique life experiences, and the demands of our present environment all contribute to developing a preferred learning mode. This has important implications for learners from different ethnicities and national backgrounds. Kolb and Kolb²⁰ argue that experience shapes the way learners grasp knowledge, which then affects their cognitive development.

One study measuring learning styles of otolaryngology residents showed that three quarters of the residents preferred converging and accommodating learning styles in which individuals process newly acquired knowledge by active experimentation and learn most effectively by being actively involved in simulations, laboratory assignments, presentations, and demonstrations.²¹ Studies such as these not only help to identify the preferred learning styles, but can also prove beneficial in implementation of a curriculum to fulfill the requirements of the trainees. It will allow them to achieve the learning milestones quicker, while continuing learning beyond those thresholds. Therefore, it is important to alter the learning process to accommodate residents’ styles and preferences to develop them as lifelong learners.

Principles of learner-centered training

Such a personalized approach to training in surgery can be based on the five key learner-centered principles described by Weimer²² (Table 1).

Table 1 Five key principles of learner-centered, personalized medical education

1. Need-based content: the first of these is that the need-based learning should be tailored for the trainees. To serve that purpose, an initial and then continuous assessment of knowledge and skills of the trainees is necessary. Based on the results of these assessments, a focused and targeted learning process should be initiated to overcome the deficient areas for both technical and nontechnical skills. This will not only improve the learning process and prepare the residents to demonstrate competence in standardized testing, but also help in reducing the time and cost associated with the process.
2. Facilitative role of the instructor: the second principle of a learner-centered approach is to change the role of teachers from instructive to facilitative. The trainees should be allowed to do more discovering on their own, while teachers redirect them to the right path whenever they get off course. This allows opportunity for new experiences and creativity, which ultimately leads to increased motivation.
3. Involvement of the learners: the third principle of a learner-centered approach focuses on getting learners involved in designing and implementing their curriculum, and generating a collaborative environment in which they choose their preferred content, while their teachers serve as coaches and mentors. This can be explained using the example of preparing for standardized testing. Among the hundreds of books and notes on the market to prepare students for the SAT, MCAT, or USMLE, their selection of what is best for them is still often based on the recommendations of their teachers or seniors.
4. Assessment and feedback: the fourth principle is assessment and feedback. Ongoing and immediate feedback from mentors helps to identify the specific deficiencies and allows for targeted remediation and learning in less time and at lower cost.
5. Learner’s autonomy: the fifth and last principle is the learner’s autonomy. Humans tend to lean toward self-directed and autonomous regulation of behavior. In medicine, however, evidence of internal motivation and demonstration of competence and self-confidence (that will allow the faculty to entrust learners with autonomy) are lacking in the majority of residents.²³ Because patient safety issues cannot be ignored, a gradual increase in responsibilities is wise. Such a scaffolding model, in which the role of teachers is to foster the learner’s development and to provide support structures to help the learner get to the next stage of entrustment and competence, should be considered for implementation.²⁴

Combining these learner-centered principles with individuals’ characteristics and attributes such as learning styles, personality traits, emotional intelligence, gender, ethnic backgrounds, etc, would give rise to a customizable and personalized model of education. We believe this approach can be as successful as any personalized item in real-life settings such as the answer to, “how do you like your eggs?”, at a restaurant, or getting an individualized plan for treatment of breast cancer (as in the example of personalized medicine described earlier). We do not propose that an individual classroom be set for each trainee; instead, we recommend that a menu of options be provided from which students can choose what suits them best.

Use of technology

Patient safety and administrative issues present challenges to total incorporation of such learner-centered, personalized medical education. However, with the development of simulation and virtual reality environments, the likelihood of success is increasing. Case-based and scenario-based simulations along with surgical simulators (low and high fidelity) are gaining popularity in different specialties to teach not only technical skills but also behavioral skills such as professionalism and interpersonal communication. A simulation environment may incorporate several elements that include providing feedback, opportunity for deliberate and repetitive practice, increasing levels of difficulty, ability to adapt to different learning strategies, and establishing clearly stated benchmarks and outcome measurements.²⁵ Having such defined goals allows trainees to indulge in deliberate practice at their convenience, thereby improving the learning curve and quickly reaching a plateau.

By increasing the difficulty level and applying real-life settings in simulation, rare and difficult situations may be reproduced. In addition, such simulators can measure even tiny errors and near misses, which are not easy to identify in real-life settings.²⁶ Because a simulator can accurately calculate the path of the virtual instrument and how close to an anatomical hazard an instrument is placed, it can measure near misses that may or may not result in a postoperative complication, thus training residents to avoid them in real life.

Virtual worlds are a new addition to the technology used in medical training. “Second Life” is one of the best known of these learning environments in which participants navigate as avatars through three-dimensional, computer-generated, realistic-looking environments.²⁷ Avatars can be personalized according to learners’ preferences. The virtual world offers opportunities for student interaction, intense engagement, scripted immersive experiences, simulations, role-playing, and constructivist learning. The anonymity afforded by the avatar appears to lead to less inhibition and greater interaction.²⁷ In addition, increased engagement and collaboration is possible with experts from different parts of the world. The Imperial College of London has created a game-based simulation in Second Life for undergraduate medical students whereby they can interact with patients in order to build their skills and confidence.²⁸ Similarly, another “Second Life” model has been shown to be effective for continuing medical education of family physicians.²⁷ Such real-life examples support virtual environments for residency education in which they can practice with virtual equipment, procedures, lab results, and patients.

Conclusion

Having passed through multiple stages of standardized testing and achieving milestones, residents are mature enough that they should be allowed to choose their learning process from a flexible menu of learning modalities. This will help them improve their self-evaluation skills and develop them as lifelong and self-regulated learners, which are basic requirements of the medical profession. This approach does not by any means make the faculty redundant, rather, they serve as ideals/benchmarks for the residents in their pursuit of expertise. Learner-centered personalized medical education has already shown significant promise in an otolaryngology residency program described by Reh et al¹⁸ and should be expanded in different forms such as simulation and virtual environments for accurate assessments and feedback. Learning styles and other psychometric traits, such as emotional intelligence, that have an effect on the learning process should be measured to tailor the learning process accordingly. ACGME has increasingly focused on achieving milestones, but has provided no guidelines on how to get there. Therefore, we should try to benefit from this flexibility, and employ methods of learning that are fruitful to both learners and the system. Residency training should not be time-based; rather, trainees should be allowed to sit for their board certification as soon as they achieve the desired milestones.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Popham WJ. Why standardized tests don’t measure educational quality. Educational Leadership. 1999;56(6):8–15.
2.		Long Jr L. SAT: The Validity and Reliability and it’s Effect on Cultural and Racial Minorities. Available from: https://psychsocialissues.com/2011/07/20/sat-the-validity-and-reliability-and-its-effect-on-cultural-and-racial-minorities/. Accessed September 14, 2016.
3.		Hanushek EA, Machin S, Woessmann L, editors. Handbook of the Economics of Education. Volume 3. Amsterdam: North Holland; 2011.
4.		Stickler L. A critical review of the SAT: Menace or mild-mannered measure? TCNJ Journal of Student Scholarship. 2007;9:1–8.
5.		Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80(10):910–917.
6.		Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA, Zhao X. The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants. Acad Med. 2013;88(5):666–671.
7.		Kenny S, McInnes M, Singh V. Associations between residency selection strategies and doctor performance: a meta-analysis. Med Educ. 2013;47(8):790–800.
8.		Sutton E, Richardson JD, Ziegler C, Bond J, Burke-Poole M, McMasters KM. Is USMLE Step 1 score a valid predictor of success in surgical residency? Am J Surg. 2014;208(6):1029–1034.
9.		Miller BJ, Sexson S, Shevitz S, Peeples D, Van Sant S, McCall WV. US Medical Licensing Exam scores and performance on the Psychiatry Resident In-Training Examination. Acad Psychiatry. 2014;38(5):627–631.
10.		Hu Y, Martindale JR, LeGallo RD, White CB, McGahren ED, Schroen AT. Relationships between preclinical course grades and standardized exam performance. Adv Health Sci Educ Theory Pract. 2016;21(2):389–399.
11.		Franzese CB, Stringer SP. The evolution of surgical training: perspectives on educational models from the past to the future. Otolaryngol Clin North Am. 2007;40(6):1227–1235.
12.		Bhatti NI, Ahmed A. Improving skills development in residency using a deliberate-practice and learner-centered model. Laryngoscope. 2015;125(Suppl 8):S1–14.
13.		Long DM. Competency-based residency training: the next advance in graduate medical education. Acad Med. 2000;75(12):1178–1183.
14.		Ahmed A, Reh DD, Bhatti NI. Technical skills and outcomes assessment in endoscopic sinus surgery: Surgical competency matters. Otorinolaringol. 2014;64(3):153–156.
15.		McCreay CL, Golde MF, Koeske R. Peer instruction in the general chemistry laboratory: Assessment of student learning. J Chem Edu. 2006;83(5):804–810.
16.		Freeman S, Eddy SL, McDonough M, et al. Active learning increases student performance in science, engineering and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410–8415.
17.		Lockspeiser TM, Kaul P. Using Individualized Learning Plans to Facilitate Learner-Centered Teaching. J Pediatr Adolesc Gynecol. 2016;29(3):214–217.
18.		Reh DD, Ahmed A, Li R, Laeeq K, Bhatti NI. A learner-centered educational curriculum improves resident performance on the otolaryngology training examination. Laryngoscope. 2014;124(10):2262–2267.
19.		Sabatier R, Gonçalves A, Bertucci F. Personalized medicine: present and future of breast cancer management. Crit Rev Oncol Hematol. 2014;91(3):223–233.
20.		Kolb AY, Kolb DA. Learning styles and learning spaces: a review of multidisciplinary application of experiential learning theory in higher education. Acad Manage Learn Educ. 2005;4:193–212.
21.		Laeeq K, Weatherly RA, Carrott A, Pandian V, Cummings CW, Bhatti NI. Learning styles in two otolaryngology residency programs. Laryngoscope. 2009;119(12):2360–2365.
22.		Weimer M. Learner-Centered Teaching: Five Key Changes to Practice. 2nd ed. San Francisco, CA: Jossey-Bass; 2013.
23.		Fonseca AL, Reddy V, Longo WE, Gusberg RJ. Graduating general surgery resident operative confidence: perspective from a national survey. J Surg Res. 2014;190(2):419–428.
24.		Hoffman BD. Using self-determination theory to improve residency training: learning to make omelets without breaking eggs. Acad Med. 2015;90(4):408–410.
25.		Wiet GJ, Stredney D, Kerwin T, et al. Virtual temporal bone dissection system: OSU virtual temporal bone system: development and testing. Laryngoscope. 2012;122 Suppl 1:S1–12.
26.		Fried MP, Satava R, Weghorst S, et al. The Use of Surgical Simulators to Reduce Errors. In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in Patient Safety: From Research to Implementation (Volume 4: Programs, Tools, and Products). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005.
27.		Wiecha J, Heyden R, Sternthal E, Merialdi M. Learning in a virtual world: experience with using second life for medical education. J Med Internet Res. 2010;12(1):e1.
28.		Toro-Troconis M, Mellström U, Partridge M, Meeran K, Barrett M, Higham J. Designing game-based learning activities for virtual patients in Second Life. J Cyber Rehabil. 2008;1(3):225–238.