Multiple mini-interview as a predictor of performance in the objective structured clinical examination among Physician Associates in the United Kingdom: a cohort study

Introduction

The process of admission in medical schools is quite demanding for both the admission committee and applicants. The wide gap between the number of applicants and the availability of places in the medical schools makes the correct selection procedures vital. The critical step of delivering a competent health care professional starts with selecting a suitable candidate using a rigorous selection process and further refining this selection with a panel interview. However, the use of multiple mini-interviews (MMIs) in their stead is growing consistently.¹ Evidence suggests that MMI can predict students’ future performance in the clinical clerkship and objective structured clinical examination (OSCE).^2–5 In 2015, Anglia Ruskin University, Chelmsford, UK, recruited the first cohort of students for MSc in the Physician Associate program using MMI.

Candidates’ future performance is of interest to authors, and many articles in the medical education literature have explored the possible future performance. According to one study, candidates’ academic merit is the best reflection of their predictive academic performance.⁶ There is a perception that cognitive skills are important, however, there is also a strongly held perception in the medical education research fraternity that the right attitude, as evaluated by an MMI assessment, significantly contributes to a productive workforce. The importance of a panel interview is highlighted by a study which concluded that students selected through the interview outperformed in communication skills during OSCE as compared to those selected by academic merit alone.⁷ The necessity of a structured admission interview to facilitate the selection of a candidate with the right attitude to work in the health care sector, is proven.⁸

In 1976, the term “noncognitive skills” was first developed to describe the various aspects of an individual’s personality other than cognitive traits which can contribute toward their behavior and success.⁹ Health care professionals need to have both cognitive and noncognitive skills. It is difficult to objectively test the noncognitive skills through traditional interviews.¹⁰

Communication skills should be an integral part of health care education programs. Experience alone cannot be attributed to a reliable factor in improving the communication skills of the students.^11,12 Effective communication is paramount to high-quality patient care and management.¹³ The majority of complaints received from dissatisfied patients at the National Health Services (NHS; UK) were attributed to poor communication.¹⁴ Therefore, it is suggested to assess both cognitive and noncognitive attributes of aspiring candidates wanting to join the health care profession.¹⁵

The MMI is conceptually derived from OSCE, but differs in being nonclinical.¹⁶ Evidence suggests that MMI may be a more impartial, reliable and valid assessment tool when compared to traditional interviews in testing noncognitive skills.¹⁷ The MMI has been positively received by the candidates, assessors and other stakeholders.^18,19

In paramedicine, MMI has been proven as a reliable and valid tool for the assessment of nonclinical attributes.²⁰ The ability of an individual to perceive, regulate and understand emotion, construct thoughts and then respond accordingly is defined as emotional intelligence.²¹ Emotional intelligence is a desirable trait in medical professionals, which may contribute to the development of professionalism and communication skills if integrated wisely in the curriculum.²² Assessing the emotional intelligence of the candidate through MMI has been proposed in the literature and is considered the most valid tool to measure noncognitive capabilities in an educational setting.²³

A comparative study of MMI versus panel interviews for medical school applicants showed that both were valid instruments. However, MMIs appeared more successful in assessing noncognitive skills and may objectively better reflect the overall persona of the applicants.²⁴ MMIs are appreciated for being innovative in finding the ways to test the compliance with the core values of the NHS in UK. It helps to avoid a rehearsed response enabling the candidate to better portray overall personality, and allows interviewers to discriminate between good and excellent.²⁵

This study aims to identify the correlation between the performance of the candidates’ communication skills during MMI and the subsequent OSCEs during their training and the effectiveness of MMI as a selection tool with the communication skills reflecting the ethos of the NHS among Physician Associates.

Materials and methods

A retrospective cohort (n=30) study was designed to examine the correlation between the average scores of communication skills of Physician Associate trainees recruited through an MMI-based selection process and subsequent two OSCEs in their first year of study (trimesters 2 and 3). The anonymous data of the average scores of communication skills attained in MMI and OSCE were collected from the Physician Associate program team. The study was approved by the ethics committee of the Faculty of Medical Sciences, Anglia Ruskin University, Chelmsford, UK.

MMI

In December 2012, “Compassion in Practice” was launched, which comprises a set of values: care, compassion, competence, communication, courage and commitment (6Cs), which are embedded in the daily care provided by the health care professionals within the NHS.²⁶

As part of the Physician Associate program, the Department of Medicine and Healthcare Sciences constructed an interview process that tested the potential candidates on their attitude toward the 6Cs, knowledge of general medical ethics and situational judgment.

It is essential to define the core communication skill for the item development, and it was based on the Calgary-Cambridge guide.^27–29 As part of the process, 10 simulated patient-based scenarios were constructed by clinicians and academics. The purpose was to evaluate the noncognitive skills, moral and ethical judgment of the candidate. The other skills that were assessed included problem solving, interpersonal communication, teamwork, flexibility and the reliability of a candidate.

Candidates were given 2 minutes to read and prepare a scenario or a question and then 8 minutes to present to an interviewer at each of the mini-interview stations. The assessors were clinical or academic members of the faculty and received MMI training prior to their assignment as an interviewer. An example of scenario is given in Box 1.

Box 1 Example of an MMI station Abbreviation: MMI, multiple mini-interview.

OSCE

OSCEs have become a favorite tool used in the assessment of students’ competence in both undergraduate and postgraduate medical education. This assessment method has been shown to be highly reliable when the following criteria are met: proper organization, blueprinting (the assessment should cover the learning objectives of the course), examiner training and consistent simulated patient training. It provided the opportunity to assess the communication skills and has been proven to measure the noncognitive ability of a student along with clinical competencies. During the first academic year, the students were formatively assessed for their communication skills by blueprinting and constructing OSCE scenario stations. These clinical and communication skill stations were based on the Faculty of Physician Associates’ (FPA) Curriculum Framework. Clinical consultants and academic lecturers designed the stations to meet the core standards set by the FPA and ensured a safe and robust scenario was created to test the competency of the students. These stations varied from focused clinical skills to different communication skill scenarios (eg, history taking, explanation of procedures and breaking the bad news).

Psychometric scale for both MMI and OSCE

To be consistent with the measurement of communication skills for both the MMI and OSCE, we used Explanation and Planning Scale (EPSCALE), a validated 4-point response scale designed to measure communication skills.³⁰

Statistical analyses

The statistical analyses were performed using IBM SPSS Statistics for Windows, Version 24.0 (IBM Corporation, Armonk, NY, USA). Pearson’s correlation coefficient was calculated to assess the relationship between the average scores of communication skills attained in MMI and OSCE during trimester 2 and trimester 3.

Results

There was a positive correlation between the average scores of communication skills attained in MMI and OSCE during trimester 2 (r=0.956, n=30, p<0.001) and trimester 3 (r=0.966, n=30, p<0.001; Tables 1 and 2). The results are summarized as scatter plots in Figures 1 and 2.

Table 1 Correlation between the average scores of communication skills attained in MMI and OSCE (trimester 2) Note: **Correlation is significant at the 0.01 level (two-tailed). Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Table 2 Correlation between the average scores of communication skills attained in MMI and OSCE (trimester 3) Note: **Correlation is significant at the 0.01 level (two-tailed). Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Figure 1 Scatter plot of the average scores for communication attained in MMI and OSCE (trimester 2). Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Figure 2 Scatter plot of the average scores for communication skills attained in MMI and OSCE (trimester 3). Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Discussion

Health care outcome is positively correlated with active physician–patient communication. Trained in a medical model, Physician Associates are a relatively new addition to the existing health care force with communication being an essential part throughout the curriculum. Communication is a crucial bridge between patients and health care professionals which can significantly enhance the recovery of the patients.³¹ It is one of the aspects of noncognitive traits and can be learned through practice and experience. A number of medical councils all over the world including the Accreditation Council for Graduate Medical Education (ACGME) and General Medical Council (GMC), UK, have endorsed the introduction of training in communication skills in the medical curriculum.^32,33

MMI has been proven as a successful technique for selecting candidates in various healthcare domain. Studies support the adoption of MMI as an admission protocol in conjunction with other academic selection criteria. The MMI score can be used as a predictive measure for future performance in examinations as well as clinical clerkship.³⁴ The predictive validity of MMI was confirmed and attributed to the similarity in the communication aspects of both examination types and the ability of an individual to perform well under pressure in a real-life clinical setting.³ The numerical score provided by MMI allows wider dispersion during the evaluation of the strength of the applicants’ communication skills.

MMIs have been shown to be more effective than typical interviews as the number of raters involved in the assessment of each candidate allows for a more reliable objective assessment of these individuals. The way in which the MMIs are carried out can provide insight into different aspects of candidates’ eligibility for the position. Communication and listening skills are often assessed, as well as their time management when put under pressure to understand the contents of the station, formulate their thoughts regarding the matter and effectively communicate them back to the assessor or, in some cases, the actor at the station.

Specific factors need to be considered to efficiently use MMIs as a tool for value-based recruitment (VBR). The critical criteria for each station need to be defined efficiently; the scoring system should identify both positive and negative behaviors to allow for more comprehensive assessment of the candidates. Having a defined set of criteria that should be met at each station facilitates fair judgment and comparison of candidates’ performance in different aspects of the interview.

Although MMI has established its capability to assess communication skills during the admission process,³² its validity as an admission tool for the Physician Associate programs in UK is still in its infancy. In the literature, evidence for the reliability, validity and acceptability of the MMI is increasing for the postgraduate admissions. However, data on the predictive validity of the MMI in medical education are sparse.³⁵

To the best of our knowledge, this is the first study describing the use of an MMI in a Physician Associate admission process. Every stratum of health care professionals requires communication skills, and the assessment of these skills has been adopted as a crucial part of the admission process and curriculum. Medical schools are continuously working toward developing innovative solutions for teaching and assessing communication skills at all levels of medical education.^36,37 The MMI as a selection tool has long been practiced to enhance the assessment of communication skills during the admission interviews. Conversely, OSCE has been proven as a useful tool for the assessment of communication skills in medical education.^38,39 For the successful practice of medicine, expertise in both verbal and nonverbal communication is essential, and both MMI and OSCE can assess this effectively.

Limitations

This study has few limitations. First, trainee students had varied experience such as some were fresh biomedical science graduates and many with 5–7 years of work experience in health care settings. This imposes a perspective bias in their MMI performance. Second, this study was conducted at a single center, with relatively small sample size limiting the generalizability. Future studies should focus on repeating and encompassing the findings with multiple Physician Associate programs, as well as establishing the correlation between MMI and national examinations for the Physician Associates conducted by the FPA which in turn will contribute to setting a benchmark criterion for the assessment of OSCE across the Physician Associate schools, thereby contributing to the patients’ safety and care.

Conclusion

This is the first study demonstrating the positive relationship between the performance of communication skills during the recruitment process using MMI and OSCEs in the Physician Associate program in UK. The MMI scores are an excellent indicator to predict the performance of candidates’ communication skills during OSCEs.

Acknowledgment

We would like to express our earnest thanks to Dr Greg Goodman, Associate Professor, Monash University, Melbourne, VIC, Australia, for comments that significantly improved the manuscript.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Rees E, Hawarden A, Dent G, Hays R, Bates J, Hassell A. Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: a BEME systematic review: BEME Guide No. 37. Med Teach. 2016;38(5):443.
2.		Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009;43(8):767–775.
3.		Husbands A, Dowell J. Predictive validity of the Dundee multiple mini-interview. Med Educ. 2013;47(7):717–725.
4.		Eva KW, Reiter HI, Rosenfeld J, Norman GR. The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med. 2004;79(10 suppl):S40–S42.
5.		Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007;41(4):378–384.
6.		Wilkinson D, Zhang J, Byrne GJ, et al. Medical school selection criteria and the prediction of academic performance. Evidence leading to change in policy and practice at the University of Queensland. Med J Aust. 2008;188(6):349–354.
7.		De Lusignan S. Comparison of communication skills between medical students admitted after interviews or on academic merits. Inform Prim Care. 2012;20(2):77–79.
8.		Powis D. How to do it: select medical students. BMJ Br Med J. 1998;317(7166):1149–1150.
9.		Samuel Bowles; Herbert Gintis. Schooling in Capitalist America: Educational Reform and the Contradictions of Economic Life. Am J Sociol. 1978;83(4):999–1006.
10.		Kreiter CD, Yin P, Solow C, Brennan RL. Investigating the reliability of the medical school admissions interview. Adv Heal Sci Educ Theory Pract. 2004;9(2):147–159.
11.		Pololi L, Clay MC, Lipkin M Jr, Hewson M, Kaplan C, Frankel RM. Reflections on integrating theories of adult education into a medical school faculty development course. Med Teach. 2001;23(3):276–283.
12.		Cantwell BM, Ramirez AJ. Doctor-patient communication: a study of junior house officers. Med Educ. 1997;31(1):17–21.
13.		Baig LA, Violato C, Crutcher RA. Assessing clinical communication skills in physicians: are the skills context specific or generalizable. BMC Med Educ. 2009;9(1):22.
14.		Workforce and Facility [webpage on the Internet]. Data on Written Complaints in the NHS 2016-17; 2017. NHS Digital. Available from: https://digital.nhs.uk/catalogue/PUB30080. Accessed November 1, 2017.
15.		Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM. Assessing personal qualities in medical school admissions. Acad Med. 2003;78(3):313–321.
16.		Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ. 2004;38(3):314–326.
17.		Lemay J-F, Lockyer JM, Collin VT, Brownell AK. Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ. 2007;41(6):573.
18.		Brownell K, Lockyer J, Collin T, Lemay JF. Introduction of the multiple mini interview into the admissions process at the University of Calgary: acceptability and feasibility. Med Teach. 2007;29(4):394–397.
19.		Razack S, Faremo S, Drolet F, Snell L, Wiseman J, Pickering J. Multiple mini-interviews versus traditional interviews: stakeholder acceptability comparison. Med Educ. 2009;43(10):993–1000.
20.		Tavares W, Mausz J. Assessment of non-clinical attributes in paramedicine using multiple mini-interviews. Emerg Med J. 2015;32(1):70.
21.		Mayer JD, Salovey P. What is Emotional Intelligence? In: Jhon.D. Mayer; Peter Salovey, ed. Emotional Development and Emotional iItelligence: Educational Implications. New York: Harper Collins.; 1997:3–31.
22.		Cherry MG, Fletcher I, O’ Sullivan H, Dornan T. Emotional intelligence in medical education: a critical review. Med Educ. 2014;48(5):468–478.
23.		Yen W, Hovey R, Hodwitz K, Zhang S. An exploration of the relationship between emotional intelligence (EI) and the multiple mini-interview (MMI). Adv Heal Sci Educ Theory Pract. 2011;16(1):59–67.
24.		Pau A, Chen Y, De Alwis R. What does the multiple mini interview have to offer over the panel interview? Med Educ Online. 2016;21(1):29874.
25.		Barnett R, Winterton A, Firth P, Davison J, Willis J, O’Brien A. Multiple mini interviews for undergraduate physiotherapy entry in the United Kingdom (UK). Physiotherapy. 2015;101:e1650.
26.		Health Education England. Values Based Recruitment Framework. London; 2016. Availbale from: https://www.hee.nhs.uk/sites/default/files/documents/VBR_Framework March 2016.pdf. Accessed February 20, 2018.
27.		Kurtz SM, Silverman JD. The Calgary-Cambridge Referenced Observation Guides: an aid to defining the curriculum and organizing the teaching in communication training programmes. Med Educ. 1996;30(2):83–89.
28.		Kurtz S, Draper J, Silverman J. Teaching and Learning Communication Skills in Medicine. 2nd ed. Oxford, UK: CRC Press; 2004.
29.		Silverman J, Kurtz S, Draper J. Skills for Communicating with Patients. 3rd ed. Boca Raton, FL: CRC Press; 2013.
30.		Silverman J, Archer J, Gillard S, Howells R, Benson J. Initial evaluation of EPSCALE, a rating scale that assesses the process of explanation and planning in the medical interview. Patient Educ Couns. 2011;82(1):89–93.
31.		Stewart MA. Effective physician-patient communication and health outcomes: a review. CMAJ. 1995;152(9):1423–1433.
32.		Oluwasanjo A, Wasser T, Alweis R. Correlation between MMI performance and OSCE performance – a pilot study. J Community Hosp Intern Med Perspect. 2015;5(3):27808.
33.		General Medical Council. Tomorrow’s Doctors. London: General Medical Council; 2009. Available from: http://www.gmc-uk.org/Tomorrow_s_Doctors_1214.pdf_48905759.pdf. Available from December 5, 2017.
34.		Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, Norman GR. Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. JAMA. 2012;308(21):2233–2240.
35.		Pau A, Jeevaratnam K, Chen YS, Fall AA, Khoo C, Nadarajah VD. The Multiple Mini-Interview (MMI) for student selection in health professions training – A systematic review. Med Teach. 2013;35(12):1027–1041.
36.		Kessler CS, Chan T, Loeb JM, Malka ST. I’m clear, you’re clear, we’re all clear. Acad Med. 2013;88(6):753–758.
37.		Levinson W, Lesser CS, Epstein RM. Developing physician communication skills for patient-centered care. Health Aff (Millwood). 2010;29(7):1310–1318.
38.		Duffy FD, Gordon GH, Whelan G, Cole-Kelly K, Frankel R. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Acad Med. 2004;79(6):495–507.
39.		Middlemas S, Haftel H, Ross PT, Lypson ML. Assessing interpersonal communications skills: the use of standardized patients in graduate medical education. J Grad Med Educ. 2013;5(3):515–516.