Multiple Mini-Interviews: Current Perspectives on Utility and Limitations

Introduction

The selection of a suitable candidate for health professional schools is one of the most important initial steps that pave the way for developing a competent and caring health professional. In recent years, schools have selected candidates who not only excel in academics but possess key personal attributes essential for competent health professionals. The importance of selecting the right candidate increases once we observe that the selected candidate for health profession education ultimately graduates, whether within the designated time or beyond.^1,2 To address these requirements, universities across the globe strive to develop robust assessment tools assessing multifaceted attributes.

In the UK, the Medical School Council Consensus Statement on attributes of a good doctor guides medical schools on admission criteria.³ This encompasses attributes and qualities required by doctors, such as good communication skills, the ability to work as part of a team, non-judgmental behavior, empathy and integrity. Thus, attaining a high level of performance only in the academics is inadequate for a career in medicine.

Similarly, the Liaison Committee for Medical Education – the accrediting body for the MD program in USA and Canada – has specified lists of core attributes for accepted applicants that cover intelligence, integrity, personal and emotional characteristics, for effective performance in future.⁴

Many institutes globally assess both cognitive and non-cognitive abilities during their admission process. Cognitive abilities are judged by academic records and written tests such as the MCAT (Medical College Admission Test in North America), PCAT (Pharmacy College Admission Test), the GAMSAT (Graduate Australian Medical Schools Admission test in Australia), UKCAT (the United Kingdom Clinical Aptitude Test), and the Undergraduate Medical School Admission Test (UMAT). A positive predictive ability of academic achievement is evident in the literature,^5,6 whereas the predictive ability of these tests is varied.

To quantify the challenging task of assessing individual aptitude, health sciences schools use a wide range of selection tools including reference letters, co-curricular and other accomplishment records and written personal statements, but have concluded with low reliability.^7–9

Going face-to-face with a single interviewer or a panel with varying degrees of structured interview is a common part of the selection process. A structured interview is intended to measure professional aptitudes such as communication skills, decision-making, teamwork skills and insight about self. Although the validity and the reliability of interview ratings are enhanced through this,¹⁰ the proven high degree of variability of its predictive ability makes it a weak measure for future performance assessment.

In an attempt to overcome the limitations of panel interview, Eva et al.,¹¹ in 2002 at McMaster, introduced a “multiple sample-based” approach to assess the non-academic attributes and named it the Multiple Mini Interview (MMI). This is an OSCE-style exercise, comprising multiple focused encounters. It intended to assess a range of personal attributes such as critical thinking, ethical decision-making, communication skills and knowledge of the healthcare system, which were previously inadequately assessed by personal interview. Initially, the MMI was used for admission in medical schools only; however, increasing research and proven robust psychometrics have led to its application in entrance to residency programs, dental, pharmacy, nursing and physiotherapy schools.

Utility of MMI

MMI-based testing needs to be verified on the basis of principles of assessment, which contributes greatly as to who should be admitted to the program. For this very reason, multiple studies have been done to establish the utility of MMI. The concept of utility of assessment is defined by Van der Vleuten as multiplicative function of following elements:¹²

1. reliability,
2. validity,
3. educational impact,
4. acceptability, and
5. cost/feasibility

However, depending on the context and purpose of assessment, the weightage of these elements could be varied. For example, in a high-stake examination reliability has more weightage, while on the other hand, educational impact would have higher weightage as compared to reliability. Being a high-stake examination, the focus of this study would be more on validity, reliability, acceptability and feasibility than educational impact.

Literature Search Strategy

In order to gain an overview of relevant and updated research on the topic of interest, PubMed, ERIC and Google Scholar databases were searched which included articles published from 2008 to 2018. A preliminary search yielded most useful results with the following key words:

“Multiple mini-interviews, Utility, Reliability, Validity, Acceptability, Feasibility, Limitations” in “all fields”

The research is further refined using limits – English language and Health Professions Education with full text available.

This literature search yielded 74 publications. To assess the relevance of articles, abstracts or if necessary full texts were reviewed. After removal of duplicates, systematic reviews, commentaries and editorials, 50 articles were selected for review (see Table 1). However, an article in the category of special communication is included because of the presence of original findings in our context.

Table 1 Literature Search Strategy

Reliability of MMI

Targeting various aspects, multiple studies have been done to establish the reliability of MMI scores.

Intraclass correlation, evidence of internal consistency of the station, is calculated by Cronbach's alpha. Indication of a well-constructed station is established by assessing the required constructs turns high among all.^13––15 Along with high internal consistency, item-total correlation for each station was from 0.53 to 0.96, assessing the desired attribute adequately.¹⁶ High composite reliability was also mentioned in the admission process of Malaysia.¹⁷

Identifying the appropriate number of stations has been the most discussed parameter in establishing the reliability evidence of the MMI. The G-coefficient along with D-studies (decision studies) has continually showed that increasing the number of stations results in greater reliability. An initial D-study by Eva et al.¹¹ claimed that using 12 stations could lead to a G-coefficient of 0.85, but later studies decreased the desired number of stations to 10, 9, 7 and even 5.^18–25

The duration of the station has also been discussed in context to establish reliability evidence. Dodson et al. compared the G-coefficient for 5- and 8-min stations and found minimal difference in reliability between the two.²⁶ Similarly, other studies also reflect adequate reliability with 5 or 6, or 7 min.^19,20,25 This is possible because a decrease in duration can give a margin of increased number of stations that in turn could positively enhance the reliability. However, fear of losing authenticity and acceptability of the process requires keeping the length of station to an optimal duration.²⁰ This is further proven by an Internal Medicine residency program induction, in which only 5 stations' MMI gave reliability of 0.9 in 8 min working time,²⁴ similar to Eva et al. with 12 stations.¹¹

The format and scoring system also effect the reliability. Sebastian et al.²⁷ showed that the reliability of MMI scores increased after changing the scoring system from Likert to a normative scoring rubric. Another interesting debate about reliability is based on the format of the stations. Using past behavioral and/or situational questions gave good reliability after minimizing the order bias.²⁸ Applicant–station interaction has proven to be the major source of variance in scores. This variance declined from behavioral interview to situational judgment question and hence internal consistency also varied in this sequence.²⁹

Raters have been identified as important facets for improving reliability. Inter-rater reliability, rater type and rater training is being discussed. Whether calculated by interclass correlation or G -coefficient, inter-rater reliability turns out high,^17,22,24,29 indicating less importance of number of raters per stations. To maintain the consistency in scoring, raters' training is highly recommended, as this helps in improving subjectivity reflecting in scores given by raters.^31–34 Griffin et al. highlighted the importance of rater training by comparing the variance in MMI scores after transforming information-based training to skills-based training. The resultant variance was reduced after improving strategies of training sessions.³¹

In addition, the effect of rater type (lenient vs stringent, faculty vs students or community members) on reliability has been studied. Contrasting evidence has been found between those who claim and those who do not claim a prominent difference.^11,18,19

Validity

Validity is a unitary construct that refers to different types of validity testing.³⁵ Studies assessing the validity of MMI have focused on the following aspects.

Content Validity

Blue-printing before construction of MMIs is the first step to ensure content validity. Themes are selected as per institutional guidelines and expected entry-level competencies defined by relevant professional colleges, especially by its originator at McMaster.^{16,17,34,36,37} Further validation of the content was achieved by calculating item difficulty and discrimination indices.¹⁷ Emergence of a unidimensional construct on factor analysis provided support to consider and use mean score as overall performance.¹⁷

A longitudinal study (2006–2012) claimed that the effect of item reuse, familiarity with the items, deliberate availability of material on the internet and candidate coaching showed no effect on MMI scores.³⁹

Convergent and Divergent Correlation

Studies suggest that MMI showed positive correlation with the assessment methods measuring the same construct, and vice versa. The MMI scores of the applicants for the Australian Medical School were correlated with scores on NEO PI-R Domains scale for assessing the Big Five personality variables. There was positive correlation among MMI scores with extraversion and conscientiousness. However, neuroticism was unrelated and self-consciousness was negatively correlated with MMI scores.⁴⁰

In another study for admissions to a Doctor of Veterinary program, the MMI scores were correlated with extraversion and emotionality, using HEXACO-PI-R-60. Total MMI scores had significant correlation with extraversion but were not related to emotionality.¹⁴

A multi-institutional study under the Australian Specialty Training program evaluated correlation of 6 station MMIs with 50-item Situational Judgment tests and demonstrated evidence for convergent validity.³⁶

Another significant correlation of the MMI with part 2 of UMAT (in the format of a situational judgment test) provides the evidence of its convergent validity. For instance, part 2 of UMAT in the format of a Situational Judgment test assessing interpersonal understanding was positively correlated with the MMI.⁴⁰

MMIs showed discriminant validity in many studies comparing no correlations with past academic performances like grade point average (GPA)³³ or logical reasoning ability or non-verbal reasoning.⁴⁰

Predictive Validity

For the predictive validity of MMI, evaluated through communication interviews administered 8 months later, the two-factors model was supported with high correlations for oral communication and problem evaluation constructs. MMI problem evaluation scores were significantly correlated with scores on building a relationship and explaining and planning portions of the interview.¹⁴

In the recruitment process for Psychology students, comparison was done between two admission processes, one being high GPA (75% of admitting students) and the other cognitive plus non-cognitive (MMI), selecting 25% of the students. The results showed fewer drop-outs after two years of study, better academic performances and better academic learning self-efficacy and critical thinking with the latter admission process.⁴¹

The predictive validity of MMI and OSCE was evaluated for an Internal Medicine Residency program, showing significant positive correlation between average MMI interpersonal score and communication score in the OSCE, and negligible relation between average MMI overall score and communication score on the OSCE.⁴²

The University of Hamburg evaluated the predictive validity of MMI conducted for medical school applicants by conducting follow-up studies over the course of two years. The result showed that MMI measures the competencies related to practical contexts like general practice evaluations of psychosocial competencies, suitability for the medical profession as well as in OSCE.⁴³ Similarly, in the Physician Associate Program and the Midwifery Program in the UK, MMI scores positively predict communication during OSCE.^13,37

In pharmacy admission, the MMI has significantly predicted the performance in the Pharmacy Examining Board of Canada (PEBC) and licensing examination.³⁸

Important evidence for the validity of the MMIs is provided by demonstrating that MMI scores are related to peer assessment conducted four years later. The scores of two MMIs (MOR and MIRKAM) were moderately correlated with the peer assessment done during clinical studies in medical school.⁴⁴ Empathy, medical knowledge, teamwork, ability to cope with pressure and communications skills were among the domains identified for peer evaluation. The most significant correlation was for communication skills.

Acceptability

Acceptability of the MMI accounts for the acceptance and the perception of the stakeholders, which includes evaluators/examiners and the candidates/examinees.

The combined analysis of both candidates and interviewers reflected that the MMI was a fair,^17,28,45 transparent, reliable⁴⁶ and authentic⁴⁷ assessment tool and were satisfied with the level and quality of the process.⁴⁸ The overall experience of the MMI was positive.^23,45,49

This was further verified by a qualitative analysis which indicated that a one-to-one interview improves the quality of the interaction and gives interviewers a chance to express diverse opinions. Multiple assessment opportunities in the MMI make the interviewer relaxed and facilitate multiple viewpoints of a candidate. The standardized scenario-based assessment in the MMI makes the process authentic and creates a more purposeful assessment process. However, in one study, candidates felt stressed as the time was short and they had to change their thought process after every other station.⁵⁰

Candidates

Candidates reflected that the MMI is an innovative idea for a selection process with no cultural,⁴⁹ age or gender³⁰ bias. They enjoyed the MMI,^24,30,51,52 especially the role plays⁴⁸ and found the experience relaxing and positive.^45,53

When comparing it with the traditional interview, it was found that the MMI was fair as opposed to the traditional interview and non-cognitive skills were evaluated more effectively through the MMI.^24,30 Another study claimed that the participants were in favor of mixed of MMI and unstructured interaction when compared to MMI alone (p < 0.01). One of the significant findings in the study was that a lower MMI performance was associated with higher preference of the traditional interview, a higher MMI performance was associated with higher MMI preferences and the mix of interview methods were not correlated with MMI performance.⁵⁴

MMI provides candidates with better opportunities to portray themselves.^17,30 They have sufficient time and require no specialized knowledge to express at stations.^23,45,49,55 However, in another study there was a mix of opinions regarding the time limitation, which varied from a shortage of time to more than enough time for each station.⁴⁹

Comparison between the past behavioral questions and situational questions showed that past behavioral questions were more acceptable than the situational questions.²⁸

The MMI was conducted through Skype in 2011 at The University of Sydney to assess international applicants for their medical and dental graduate programs. Candidates gave positive feedback regarding its format and experience.⁵⁶

Interviewer

Interviewers specified that MMI tested a valid range of competencies. It provided the opportunity to appreciate the complete and authentic picture with a larger range of skills from the candidates,³⁰ especially in professionalism and communication skills.⁴⁷ In their opinion, the MMI was a reasonable method for accurate portrayal of abilities of the candidates^23,46 by pushing them to think deeply.

The interviewers appreciated the organization of the process including the number of stations, the diversity of case scenarios and their relevancy to the program.⁴⁹ The interviewers stated that there was sufficient time to evaluate the candidates.^46,49,55 They valued the importance of orientation sessions for candidates,^30,45,49,55 but also stated that candidates required some specialized knowledge to perform at the stations.⁴⁶ They agreed that the time allotted was sufficient to evaluate the candidates.^46,49,55 For them, the overall experience and atmosphere was satisfying,⁴⁹ better than the traditional interview, and thus they were in favor of continuing with the MMI.²⁴

Feasibility

Most of the studies regarding the feasibility issue claim that the MMI is feasible and could be implemented successfully.^19,57,58 Few schools had feasibility problems in offering over 2 hrs of interview time per candidate.²⁰ Time and commitment were a key requirement from the station developers and the examiners in formulating the stations, attending training sessions and conducting the MMI.⁵⁹ However, due to interest in the process, the time commitment was manageable.⁵⁸

When it comes to cost, the MMI is more challenging compared to the traditional interviews.^11,60 The involvement of staff for planning, implementation and data entry as well as salaries for overtime⁵⁹ are required. Moreover, miscellaneous expenses for refreshment, actors and availability of the infrastructure are also made.

Successful implementation of the MMI requires sequential efforts in terms of planning and resource provision like OSCE. It is therefore suggested to direct efforts the same way to ease its development and implementation.¹¹

Limitations of the MMI

Candidates argued that the MMI restricts the opportunity to discuss the past achievements and experiences and, in turn, limits the opportunity to reflect the motivation and willingness to be a medical graduate. Furthermore, they found the MMI forced students to “perform” rather than respond naturally.⁵⁰

Cultural diversity between the interviewer and the candidates affected the assessment process and made the evaluator feel unreliable to gauge a response correctly. A language barrier was reported as another limitation. Those candidates who lack proficiency in English were reported as more nervous and having greater difficulty in expressing themselves.⁴⁷

Feasibility issues also pose limitations to its effectiveness. Being resource-intensive, the MMI requires more efforts in its planning, development, implementation and evaluation. Along with station structure, context and raters' training, the venue also has an important impact on examiners' rating. The availability of a comfortable and sound-proof venue is thus recommended for effective processing of MMIs.¹⁹

The limitation is also confined to inadequate generalizability as the data in studies are usually derived from a single cohort and a single institution.⁵⁵ Similarly, contrasting evidence regarding the effect of rater's type,^11,18,19 station structure and its format^28,29 have been found. This is apparently considered as a limitation, but in fact, it is urged to have variation in study results due to the need to adapt MMIs for the specific requirements of schools, and this in turn could lead to variable context and then content.⁶¹

Conclusion

Evidence regarding the admission criteria of health professions establishes the utility of the MMI considerably well. A well-established internal consistency, based on optimal number of stations and duration, and a well-constructed scoring system with a single interviewer has been recognized. Strong evidence regarding its content validity and predictive ability is available. MMI scores show positive correlation with assessment tools measuring the same attribute while at the same time showing negative correlation with tests measuring different attributes. Although its acceptability is established by an abundance of literature, evidence regarding its feasibility is less. Resources need to be allocated both financially and in terms of manpower. The cost should be calculated by considering the following: generation of interview material, human resource (i.e., interviewer and support staff) use, training of interviewer, and the support staff and infrastructure requirements. Further research is desired to address the limitations in terms of cultural diversity, language barrier, and effect of raters' type.