Selection as a learning experience: an exploratory study

Introduction

Selection of applicants for medical school is an expensive and time-consuming process that has been studied extensively in the literature. In general, given the main purpose of the selection procedures, research on this topic focuses on the “summative” aspects of selection, that is, its formal consequences for students or applicants.¹ The findings are generally distinguished in academic and nonacademic performance. Selection is predominantly based on academic performance and is a predictor of performance in the early years of medical school.^2,3 Also, attention has grown for selection on mainly nonacademic aspects, for instance, through multiple mini-interviews⁴ and situational judgment tests (SJTs).⁵ Performance on these tests has proven to correlate to performance in clinical practice.⁶

Neither research on selection for medical school nor research on the learning effects of summative assessment in medical education, for example, by Cilliers,^7–10 explores the “formative” aspects of selection, that is, selection as a learning experience. The impact of assessment on student learning, however, has become unquestioned⁷ and attention is growing for “assessment of learning”.^11,12 The “educational effect” is also emphasized as one of the criteria for good assessment in the “Consensus statement and recommendations” from the Ottawa Conference¹³ and as one of the elements defining the utility of assessments according to Van der Vleuten.¹⁴ Moreover, Eva et al¹⁵ stated that “to not consider the use of assessment for performance improvement even in high stakes contexts is a considerable missed opportunity”. This formative role of assessment is about supporting learners and allows for transforming previous experience into behavior, attitudes, skills, knowledge, and new learning goals.

Prideaux et al¹⁶ state that selection would benefit from an integrative approach, instead of the more common role as an activity that is being conceived separate from other educational and assessment activities. They encourage the field to focus on the impact of selection, and it turns out that there is a lack of research on this topic so far. Additionally, if selection resembles the curriculum that is selected for,¹⁷ this might also extend, and contribute to, the impact of selection. Although applicant experiences have been studied in the literature,^18–21 studies do not focus on learning effects of medical school selection. This apparent paucity of research indicates that the process of selection is perceived to be separated from learning and assessment in medical school and that learning only starts at the beginning of the formal curriculum.

Considering the above-mentioned literature and propositions, it can be hypothesized that selection for medical school is a learning experience in itself and provokes learning effects in applicants, concerning skills and knowledge as well as learning about the curriculum and one’s fit to it. The first aim of this exploratory study, therefore, is to determine whether students have learning experiences in the selection process, and, if so, what experiences (relevant for medical school) they have. If it were clear as to how these can be employed best to stimulate learning, medical schools could choose deliberately to either 1) adjust the selection procedure to make sure learning effects will be enlarged or improved, or 2) adjust the start of the curriculum in a way that integrates the learning effects of the selection procedure into the subsequent formal learning activities. The second aim of our study, therefore, is to understand what students need in order to employ the learning effects of the selection process at the start of the formal curriculum. Answers to these questions allow us to value the potential learning effects of medical school selection, and to make a stronger connection between learning from the selection process and learning in the formal curriculum. Given the substantial resources utilized in selection, the efforts of applicants and medical school in the procedure itself, and the volume of information gained on the applicants, there is great value in finding a way to connect the processes of selection and curriculum for enhancement of education.

Based on the aims, the research questions of the current study are 1) What learning experiences do students have in the selection process? and 2) What do students need in order to utilize learning effects of the selection process at the start of the formal curriculum?

Materials and methods

Setting

This study was performed at the Radboud University Medical Center (RUMC) in Nijmegen, the Netherlands. In Dutch medical education, a 3-year (mainly theoretical) Bachelor’s program is followed by a 3-year Master’s program with mainly practical education. Each year, 330 new students are admitted.

RUMC admission

According to Dutch law, students had direct access to medical school if their pre-university Grade Point Average is equal to or higher than 8 on a scale of 1 (poor)–10 (excellent). All other applicants are admitted through a selection procedure, and each medical school can design its own selection procedure.

The RUMC selection procedure took place when students were preparing for their pre-university exams. The procedure comprised two consecutive parts that aimed to connect to the early medical school curriculum. The first part was an assignment to be done at home. Applicants had to write a personal summary of the RUMC curriculum, and ask three people from their network to compare this description to the applicants’ personal qualities to consider its “fit”. The applicants, at their turn, had to reflect on these considerations. Furthermore, applicants had to write a personal study plan, based on a patient case they were presented in a video. In this personal study plan, they had to describe their personal learning goals and learning activities they would choose to achieve those goals. All these elements had to be written down and sent in and were assessed altogether as sufficient or insufficient.

The second part consisted of an on-site exam. The first section was a multiple-choice test requiring the application of pre-university-level biology, chemistry, and physics to medical school issues. In the last section, applicants were presented with situations and four or five possible actions responding to each situation. They were asked to put the actions in the order of appropriateness to the given situation (SJT).⁵ The situations represented dilemmas that may occur in the daily practice of medical school or the medical profession and were about working together, giving and receiving feedback, integrity, and dealing with mistakes made.

For the applicants who passed the first part, a final score was calculated by assessing the second part, both sections being equal in weight. The final scores were ranked, and the 309 best performing applicants were admitted.

Study design

To explore what and how students learned in the selection procedure, a qualitative study was considered suitable, as a qualitative approach “can shed light on phenomena that are not accurately understood in practice”.²² Following a constructivist approach, we used focus groups²³ to achieve depth in the discussion of topics by the interaction with group members.²⁴

Participants

We invited all 309 students admitted to RUMC medical school through selection in 2016 to participate in a focus group interview. Thirty students agreed to participate (9 male, 21 female, which resembled the actual male/female ratio of the cohort). We composed four focus groups of no more than eight students each.

Data collection

Each of the four focus group interviews took about 75 minutes and was held in the first week of the study program in September 2016. An experienced interviewer (CF) guided the session, while the main researcher (MdV) took notes and asked for clarification when necessary. We used a semi-structured interview guide of six themes, each illustrated by several guiding questions (Supplementary material) to explore the students’ experiences concerning the research topic.²⁴ Questions emerging in one group were discussed in the following groups to allow for deeper understanding of the phenomenon under study.^25,26 All interviews were audiotaped. Except for the two researchers and the interviewees, no other people were present during the interviews.

Data analysis

All four interviews were transcribed verbatim and entered into qualitative data analysis software (ATLAS-ti). The transcripts were analyzed applying thematic analysis,²⁷ in accordance with the guidelines described by Braun and Clarke.²⁸ Through reading and rereading, coding relevant text fragments, identifying themes and patterns, and discussion in the research team (MdV, CF, RE, JCS, RFL), we aimed to identify repeated patterns of meaning or themes reflecting the content of the interviews, keeping the focus on the aim of the study.

First, all transcripts were anonymized and read repeatedly by the first author to ensure accuracy and to promote familiarity. Coding was accomplished as an iterative process, although three consecutive stages were distinguished: descriptive coding, interpretative coding, and defining overarching themes.²⁷ In the first stage, three researchers (MdV, CF, RE) independently coded all transcripts without a predefined codebook, highlighting what respondents said in relation to the aim of the study and staying close to the data (descriptive coding). Following every interview, the researchers collaboratively carried out a comprehensive analysis of the coded transcripts, creating consensus on a shared codebook. Disagreements were mostly about codes that were closely related in meaning and were resolved through discussion, enhancing the definition of the codes.

In the second stage, we identified connections between the codes and interpreted the meaning of what had been said in the interviews (interpretative coding). In this stage, the codebook was adapted again to the experience gained so far. When the final codebook had been defined, MdV reread and analyzed all interviews once more and applied the final codebook to all data. After all data had been coded, MdV, CF, RE, and RFL discussed the phrases that had been coded and their relations in order to construct and confirm themes and find patterns in meaning in a recursive process (defining overarching themes). Themes were identified at a semantic level, taking into account what participants said not only literally, but also beyond the explicit level. This involved interpretation and was enriched by using the researchers’ own perspectives as well, which can be considered a constructivist approach.²⁹ This constructivist influence was also applicable as the interviews were held in focus groups, during which the participants interacted, influenced each other, and added to other participants’ answers.²⁴ By regularly discussing the analytic process, the perspectives, and the findings in the research team, reflexivity was practiced.^22,25,30

The diversity of the team contributed to a thorough process. The principal investigator (MdV) is an educationalist and medical education policy advisor. LF is an experienced educationalist, a researcher specializing in qualitative research and learning in transitions, and a medical doctor. RE is also an educationalist and policy advisor. JCS is a psychologist and a professor of medical education specializing in assessment. The supervising researcher (RFL) is a professor of medical education, the director of the RUMC Health Academy, and also a medical doctor.

Ethics

At the end of a regular introduction session at the start of the curriculum, students were invited by the researcher (MdV), who had no involvement or responsibility in the curriculum. Students were presented with an oral summary of the study design and given a written description of the research design and its purpose. Participation was entirely voluntary, and participants were given a €25 gift card afterwards. They all signed a written informed consent form at the start of the interview. Data were treated strictly confidentially and were available to the researchers only. All analyses were conducted anonymously. The study received ethical approval of the Ethical Review Board of the Netherlands Association for Medical Education (NERB dossier number 683).

Results

An overarching finding in the interviews was that students considered selection an obstacle and felt, therefore, that they could not have learned from it. They had not focused on learning either, as, besides mastering what was required to be successful in the procedure itself, they felt it was not important to learn in the selection process. When asked to describe what they had learned in the selection procedure, consequently, students thought that they had not learned much.

On an implicit level, however, they described a wide range of learning experiences. Questions such as “What insights did the selection give you?” or “Are there things you have been doing differently since the selection?” revealed a variety of previously implicit learning experiences. Analysis revealed that there were four themes, which were perceived as learning outcomes of the selection process: 1) Self-regulation; 2) (Fit with) the medical school curriculum; 3) Professional fit; and 4) Interpersonal effects. Students also reflected on the connection of the selection and the curriculum, related to the second aim of the current study.

Self-regulation

The theme of self-regulation has many aspects, one of which is personal leadership. In selection, students learned to manage themselves, to prioritize, and to have the courage to choose what activities to do or not to do [Quotation 1, see Table 1]. They had to set goals and take responsibility for reaching those goals. Each of them developed a personal strategy to get things done and reach their goals [Q2].

Table 1 Overview of themes, quotation examples, and codes

A second aspect is self-assessment. Through the steps of the selection procedure, students had implicitly been forced to ask themselves to what extent they mastered the subject material needed for the selection [Q3]. In this process, they experienced unfamiliar feelings of uncertainty and learned how to manage these. They were familiar with studying and being assessed in pre-university education and knew how to prepare themselves. The pressure felt in the selection process was described as being much higher as there was so much to lose, and students felt that the way they would be assessed was unclear, despite the information given. They had learned how to manage these uncertainties and had become aware of what had been “normal” or evident to them so far and reframed their personal approaches.

Students experienced that the selection had contributed to their self-knowledge and that the references’ comments had been a valuable source of information for this process. Some information about strengths and weaknesses brought forward by the references was new to the students, while other information was just a confirmation of what they already knew [Q4]. Students also mentioned that they were not used to explicitly asking for feedback and that this assignment showed the additional value of doing so [Q5]. Some students observed that they had adapted their behavior as a consequence of the feedback given, while others did not really relate the feedback to their daily life. They valued the feedback as just a part of the selection procedure, without consequences.

Furthermore, students discovered personal strengths and weaknesses through the selection process in general [Q6]. During the interviews, the students discussed various views on asking for help in preparing for the selection procedure. Some considered it professional to know what they needed from other people to be successful, but others considered it fraud-like to ask other people for help [Q7].

Students learned about different aspects of the medical school study program. Through the selection procedure, they developed an idea of being a university student in general and of being a medical student in particular [Q8]. They learned about the curriculum and its educational principles and appreciated this aspect of the selection procedure [Q9]. Furthermore, they gained an insight into the logistics of the study program and what would be expected of them as students. This relates to self-selection: students described the process of constant reflection on how they related to the elements of the selection procedure as samples of the study program. This encouraged them and boosted their motivation: their ambitions were confirmed, and they felt like they were already in medical school [Q10]. On the other hand, what drove students in doing the assignments was the reward of admission to medical school, and so they wanted all the materials they sent in to be assessed thoroughly. Some assignments were intentionally designed to stimulate self-selection as well, but this aspect was less valued by the students [Q11].

Professional fit

The selection process encouraged students to refine their ideas about what being a doctor meant in practice and the dilemmas that might pertain to the profession and how they related to these. For instance, they experienced that, in some situations, there is no clear best solution or approach for a patient and that it is very important for doctors to focus on each patient’s personal wishes in delivering health care [Q12].

At the same time, the interviews revealed that students conjectured whether the focus had implicitly been on getting an idea of the study program or of the profession. Although this may be less a learning experience per se, another angle on professional fit is that students worried whether the selection assignments were a good way of selecting the best students or future doctors. What became explicit here, in other words, was their personal frame of reference of who was a suitable student or doctor and how these traits should be measured [Q13].

Interpersonal effects

Students also described their learning experiences in relation to others, involving peer comparison. They had to face up to friends and peers being rejected in the selection procedure while they themselves had been admitted and found this hard to deal with [Q14]. Earlier in their educational careers, differences in performance had had fewer consequences than in this process. They experienced this in the examination hall as well at the moment of the on-site exam. Being gathered there with so many other applicants made them visually aware of what was at stake in terms of numbers, while they were all in this together, on the other hand, which instantly appeared to increase their feelings of relatedness. One student puts it as in Q15.

Furthermore, they learned that applicants had prepared for the selection in different ways. Despite the instructions given for the second part of the procedure (the on-site exam), some students had prepared extensively for this test. As applicants were connected with their competitors (classmates, friends, for instance), in online selection forums and in person at the on-site exam, they became aware of different strategies [Q16]. Some were very confident of their approach and stuck to their natural strategy, while others were not and tried to adapt it.

Students also experienced a changing relationship with others after they had been selected. They obtained a position that is coveted by many others, and their friends and relatives felt involved and proud. This in itself caused pressure on the students: they indicated that being selected meant they had to like medical school and had to be successful. Hesitation was unacceptable after being selected, if only for their peers who had been rejected [Q17]. Furthermore, family and friends presented articles or news items on medical education or health care to the students once they had been selected [Q18]. Or, the other way around, students also started to read magazines that had been available before but got their attention only after being admitted [Q19].

Connecting selection and the curriculum

To be able to employ their selection procedure learning experiences at the start of the formal curriculum, the students needed more specific feedback [Q20]. Except for a ranking number, the students did not receive any feedback on their selection performance. Although their admission felt as a confirmation of eligibility for medical school, the students emphasized that they needed feedback, at least the correct answers, to be able to learn from this experience, as it would otherwise remain unclear what they could improve. They suggested discussing the answers in small-group coaching sessions (which is a regular feature in the curriculum) to learn from each other’s perspectives and to discuss the answers with professionals. This would give them in-depth information and the backgrounds of considerations made [Q21].

Students had different perspectives on whether or not it would make sense to add the references’ reports to their medical school portfolio as a start for their personal development plan. Some thought that this would help them as a starting point for reflection and development, whereas others thought that this was information from a different context and, therefore, not useful for development in medical school. They wanted their coaches to start with an open mind, without information about previous performance.

Students were not really keen on using early medical school content for selection. As described earlier, they focused on selection rather than on learning. They thought that it was particularly attractive that they had not needed to master a lot of new content for the RUMC selection procedure, as was the case for some other medical schools. Finally, students felt disappointed if what they had learned about the curriculum during selection would be taught again rather than be built upon in the first week of the formal program.

Discussion

Although the students were under the impression that they had not really learned from selection, after the concept of learning had been reframed, the students described a wide range of learning experiences. We found that four learning experience themes can be distinguished. These themes are 1) Self-regulation; 2) (Fit with) the medical school curriculum; 3) Professional fit; and 4) Interpersonal effects. Furthermore, as a result of the second research question, our study produced leads for optimizing the selection–curriculum connection.

These findings indicate that medical school selection can have educational impact,^13–15 and can be approached as a process of development rather than a single moment of high-stakes assessment. In the selection process from start to finish, applicants learn and work in two different contexts simultaneously: secondary school and medical school. In both contexts, the stakes are high: final exams and admission to medical school, respectively. Physically and mentally, therefore, they face two contexts in which they must perform at the same time, going back and forth. In our discussions about the results of the current study, the theoretical concept of learning through boundary crossing³¹ appeared to be helpful; the process of medical school selection can be examined from this angle to gain more insight into the applicants’ learning. Boundary-crossing research suggests that a boundary crosser is a person who is exposed to and active in two related but potentially discontinuous work spaces. Boundaries are known to hold learning potential through the processes of “identification, coordination, reflection, and transformation”.³¹

The process of “identification” applies to what students learned about (their fit with) the medical school study program and their professional fit. The process of identification starts when they familiarize themselves with it by doing the assignments. Obviously, applicants are not blank slates when they enter the selection process; however, the selection helps them to validate, adjust, and enrich their perspectives. One aspect of the “coordination” process is that it “entails efforts of translation between the different worlds”. The self-regulation learning in the current study connects to this coordination learning, which can be illustrated by applicants becoming aware that their secondary school strategies are not appropriate or entirely helpful in the “other” context and they need to adapt and extend their repertoire. The translation aspect may require more attention from the medical school perspective as well. As an example, in the interviews, students discussed various views on asking for help in preparing for the selection procedure, from it being professional to it being fraud-like to ask other people for help. In secondary school final exams and in the selection process, individual performance is what is being rewarded, whereas medical school and the medical profession require teamwork, a clear view of the limits of one’s expertise, and asking for help when applicable. Discussing these contradictions could enrich students’ identification and coordination processes.

The “reflection” process of the boundary-crossing theory emphasizes how people become aware of and explain differences between practices and thus learn something new about their own and other people’s practices. This applies to the theme of the fit with the medical school curriculum. The questions that came up in students when doing the selection procedure assignments, such as “Do I fit into this programme? What about my motivation and my abilities to be successful in this curriculum?”, can be understood to be manifestations of reflection. Reflection is also linked to self-regulation, as this theme is about adapting strategies as a result of new experiences.

Some of our findings, finally, connect to the “transformation” aspect of boundary crossing, although less evidently so. Akkerman and Bakker³¹ observe that “Transformation leads to profound changes in practices, potentially even the creation of a new, in-between practice, sometimes called a boundary practice”. In the current study, students describe how they judge their context differently after being selected and before starting medical school (during the school holidays, roughly) and how they are perceived differently by their social context. Transformation also applies from a “negative” perspective, in the way that students describe that there is no clear link between the selection procedure and medical school: both are separated in time, and the start of the formal curriculum refers explicitly to neither selection experiences nor to selection content. Strengthening this link would be efficient, however, if all the effort made by the selected applicants would be rewarded in medical school, self-selection would be encouraged and this would provide the opportunity to add new content to the medical school program as some learning goals would already have been achieved beforehand.

In addition to reflections on the results through the lens of boundary-crossing theory, the issues introduced by Gibbs and Simpson about learning from assessment³² are a relevant perspective in optimizing the learning effects of selection. They state that 1) assessment can influence the quantity and distribution of student effort; 2) assessment can influence the quality and the level of the students’ effort; and 3) assessment can be accompanied by timely and sufficient feedback. The first two already appear to be on a satisfactory level, given the very high-stakes situation the students are in during medical school selection, although the curriculum and the integration of its assessments is probably a key factor to enhance student effort in the long run. The third issue of Gibbs and Simpson (feedback), however, is lacking. Besides a formal decision (selected or rejected), applicants do not receive feedback. We suppose that a “hidden curriculum”³³ is at stake here. As students have not been provided with any feedback as a starting point for learning besides their final ranking number, neither directly after selection nor at the start of the curriculum, we might implicitly have strengthened the impression that selection is just relevant for admission and nothing more. By providing feedback and explaining why, the “hidden” message that selection is not worth learning from or reflected on could be prevented.

Strengths and limitations

The current study is the first aiming to understand students’ learning experiences in medical school selection. From the perspective of efficiency and effectiveness, it is important to have a clear picture of how all teaching and assessment activities relate to each other and support students, in a coherent way, in becoming competent professionals.^34,35 This study adds to this by taking into account the additional value of selection as an important first step in the learning continuum of the medical professional.

A limitation of our study is that it might be prone to selection bias.³⁶ As participation in the interviews was voluntary, it may have been particularly the most active and committed students who shared their experiences. Our study, moreover, involved only one medical school in the Netherlands. Therefore, we do not claim that our results represent the experiences of all students. We are also aware that learning effects may depend on the nature of the selection procedure. Although studies like the current one are often criticized for low generalizability because of their specific context and the small number of respondents, they have a high exploratory capability, and develop general statements and hypotheses.^37,38 Furthermore, it is argued that transferability (also called “reader generalizability”), rather than theoretical generalization (to theoretical propositions) and statistical generalization (to populations), should be applied to these studies.³⁹ To enlarge transferability, we have provided a rich description of our study methods, reflexivity and results, and reflect on it in this discussion section.⁴⁰ This supports readers to evaluate how the results and principles found can be applied to their own settings, which is an important criterion for scientific rigor of studies like the current one.³⁹ In summary, based on our exploratory findings, we aim to encourage others to apply the formative perspective to their medical school selection procedures, to study and discuss its additional value, and to act upon the findings.

Finally, it is unclear what rejected applicants learn in the selection procedure. As our focus was on learning to improve the connection between selection and the formal curriculum, the perspective of rejected applicants was less relevant for our study. We suppose that, for them as well, the selection procedure enhances self-selection, which can be seen as reflection learning in the boundary-crossing theory. Notwithstanding the rejection, this can be either “self-selecting out”, or, like for those who were admitted, a confirmation of one’s wish to study medicine. Nevertheless, the positive experience of being selected may have influenced how the admitted students perceive the selection procedure in hindsight.

Implications for practice

First of all, to benefit from learning in the selection procedures, it is important that selection characteristics match curriculum characteristics. This is also supported by the findings of Burgess et al, focusing on applicant experiences in general.¹⁹ As assessment drives learning,^1,8 the content and study approach required in the selection process will stimulate curricular learning needs. Curriculum sample selection has shown predictive value for medical school performance¹⁷ and the current study provides one more important argument to align selection and curriculum.¹⁶

As mentioned earlier in the “Discussion” section, another important practical implication of the current study is that students should be provided with feedback, which encourages learning through reflection on what they have learned. They do not really experience the selection process as a learning opportunity because they focus strongly on admission. Medical schools have a responsibility to help them consider the procedure in more than just the high-stakes aspect of it and to connect selection to the curriculum. In RUMC medical school practice, for example, we now encourage students to use the reference letters as a starting point in meetings with their coaches. Doing so, we aim to make students aware of the additional value of the selection procedure for their personal development and to illustrate that previous experiences can be used in medical school. However, as long as medical schools do not consciously use selection as a first step to learning, students will not do so either. Another practical suggestion from the context of our medical school is to keep in touch during summer, for instance, by sending students information to foster the process that has started during selection. This could serve as a “boundary-crossing object” in the boundary-crossing theory.³¹

Further research

As they relate to the local selection procedure and curriculum, the results of our study are applicable in the RUMC context. It would be interesting to explore how the results apply to different contexts and, for instance, what specific elements of the selection procedure enhance learning and how the curriculum picks up on these elements or not. We hypothesize that the curriculum sample approach in the current study has enhanced learning about “the fit”. It would also be instructive, furthermore, to ask teachers who contribute to early medical school to share their perspective: how do they perceive what students learned from the selection procedure and what would they do to stimulate using selection as a starting point for further development? Finally, we do not really know how self-selection (which qualifies as learning) applies to this procedure, as adequate self-selection would result in applicants quitting during the procedure, and we have not asked these applicants to share their experiences. We hypothesize that self-selection (two-dimensional: either the experience of a fit or the lack of it) also happens to the applicants who were rejected, but the current study did not include that aspect.

Conclusion

Students can learn a variety of things while participating in a medical school selection procedure and selection could be considered an inclusive part of students’ learning processes. In the context of this study, students learn about the curriculum, themselves, their relation to others, and the profession they have been selected to enter. From the start of the selection onwards, they appear to be engaging in a boundary-crossing process, although this process remains implicit and is not experienced or labeled as learning. This exploration revealed new opportunities for taking advantage of the learning that takes place during selection and to value the selection procedure as the start of the future doctors’ development.

Data sharing statement

Transcripts of the interviews are available to qualified researchers upon request through the first author.

Acknowledgments

The authors thank the students for their participation in the focus group interviews and their willingness to share and reflect on experiences. The authors also thank Rikkert Stuve, The Text Consultant, for editing the manuscript.

Author contributions

All authors contributed substantially to the conception and the implementation of the study, as well as data analysis and interpretation. MdV and CF have collected all data together, MdV has written the first draft of the article, and CF, RE, JCS, and RFL have critically revised it and approved the final version. All authors agree to be accountable for the accuracy and integrity of the work.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Heeneman S, Oudkerk Pool A, Schuwirth LW, van der Vleuten CP, Driessen EW. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49(5):487–498.
2.		Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. BMJ. 2002;324(7343):952–957.
3.		Siu E, Reiter HI. Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development. Adv Health Sci Educ Theory Pract. 2009;14(5):759–775.
4.		Dore KL, Reiter HI, Eva KW, et al. Extending the interview to all medical school candidates-computer-based multiple sample evaluation of noncognitive skills (CMSENS). Acad Med. 2009;84(10 Suppl): S9–S12.
5.		Patterson F, Zibarras L, Ashworth V. Situational judgement tests in medical education and training: research, theory and practice: AMEE Guide No. 100. Med Teach. 2016;38(1):3–17.
6.		Patterson F, Knight A, Dowell J, Nicholson S, Cousans F, Cleland J. How effective are selection methods in medical education? A systematic review. Med Educ. 2016;50(1):36–60.
7.		Cilliers FJ, Schuwirth LW, Adendorff HJ, Herman N, van der Vleuten CP. The mechanism of impact of summative assessment on medical students’ learning. Adv Health Sci Educ Theory Pract. 2010;15(5):695–715.
8.		Cilliers FJ, Schuwirth LW, Herman N, Adendorff HJ, van der Vleuten CP. A model of the pre-assessment learning effects of summative assessment in medical education. Adv Health Sci Educ Theory Pract. 2012;17(1):39–53.
9.		Cilliers FJ, Schuwirth LW, van der Vleuten CP. Modelling the pre-assessment learning effects of assessment: evidence in the validity chain. Med Educ. 2012;46(11):1087–1098.
10.		Cilliers FJ. Is assessment good for learning or learning good for assessment? A. Both? B. Neither? C. It depends? Perspect Med Educ. 2015;4(6):280–281.
11.		Dannefer EF. Beyond assessment of learning toward assessment for learning: educating tomorrow’s physicians. Med Teach. 2013;35(7):560–563.
12.		Schuwirth LW, Van der Vleuten CP. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33(6):478–485.
13.		Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–214.
14.		Van Der Vleuten CP. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1(1):41–67.
15.		Eva KW, Bordage G, Campbell C, et al. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 2016;21(4):897–913.
16.		Prideaux D, Roberts C, Eva K, et al. Assessment for selection for the health care professions and specialty training: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):215–223.
17.		de Visser M, Fluit C, Fransen J, Latijnhouwers M, Cohen-Schotanus J, Laan R. The effect of curriculum sample selection for medical school. Adv Health Sci Educ Theory Pract. 2017;22(1):43–56.
18.		Kumar K, Roberts C, Rothnie I, du Fresne C, Walton M. Experiences of the multiple mini-interview: a qualitative analysis. Med Educ. 2009;43(4):360–367.
19.		Burgess A, Roberts C, Clark T, Mossman K. The social validity of a national assessment centre for selection into general practice training. BMC Med Educ. 2014;14:261.
20.		Patterson F, Zibarras L, Carr V, Irish B, Gregory S. Evaluating candidate reactions to selection practices using organisational justice theory. Med Educ. 2011;45(3):289–297.
21.		Niessen ASM, Meijer RR, Tendeiro JN. Applying organizational justice theory to admission into higher education: admission from a student perspective. Int J Sel Assess. 2017;25(1):72–84.
22.		Tavakol M, Sandars J. Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part I. Med Teach. 2014;36(9):746–756.
23.		Liamputtong P. Qualitative Research Methods. 4th ed. Melbourne, VIC: Oxford University Press; 2013.
24.		Stalmeijer RE, McNaughton N, Van Mook WN. Using focus groups in medical education research: AMEE Guide No. 91. Med Teach. 2014;36(11):923–939.
25.		Frambach JM, van der Vleuten CP, Durning SJ. AM last page. Quality criteria in qualitative and quantitative research. Acad Med. 2013;88(4):552.
26.		Tavakol M, Sandars J. Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Med Teach. 2014;36(10):838–848.
27.		King N, Horrocks C. Interviews in Qualitative Research. Thousand Oaks, CA: Sage; 2010.
28.		Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
29.		Bergman E, de Feijter J, Frambach J, et al. AM last page: a guide to research paradigms relevant to medical education. Acad Med. 2012;87(4):545.
30.		Barry CA, Britten N, Barber N, Bradley C, Stevenson F. Using reflexivity to optimize teamwork in qualitative research. Qual Health Res. 1999;9(1):26–44.
31.		Akkerman SF, Bakker A. Boundary crossing and boundary objects. Rev Educ Res. 2011;81(2):132–169.
32.		Gibbs G, Simpson C. Conditions under which assessment supports students’ learning. Learn Teach High Educ. 2004;1:3–31.
33.		Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69(11):861–871.
34.		Walsh K. Costs and assessment in medical education: a strategic view. Perspect Med Educ. 2016;5(5):265–267.
35.		van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. Twelve Tips for programmatic assessment. Med Teach. 2015;37(7):641–646.
36.		Henderson M, Page L. Appraising the evidence: what is selection bias? Evid Based Ment Health. 2007;10(3):67.
37.		Gable GG. Integrating case study and survey research methods: an example in information systems. Eur J Inform Syst. 1994;3(2):112–126.
38.		Mayring P. On generalization in qualitatively oriented research. 2007;8(3).
39.		Polit DF, Beck CT. Generalization in quantitative and qualitative research: myths and strategies. Int J Nurs Stud. 2010;47(11):1451–1458.
40.		Kuper A, Lingard L, Levinson W. Critically appraising qualitative research. BMJ. 2008;337:a1035.

Supplementary material

Guide for semi-structured interview

Categories with examples of probing questions

Starter: attitude

• How do you perceive selection as a tool for admission to medical school?

Preparations

• How did you prepare for the selection assignments?
• What was your strategy?
• Who did you consult?
• How did your participation influence your school performance?
• What was the key factor in your performance in the selection?

Selection content

• What have you learnt through participation in the selection procedure?

Steps in the procedure

• What helped you to learn in the selection procedure?
• What elements have hampered or promoted your learning?
• How do you value selection as an event you can learn from?

After selection

• How did you feel when you heard you had been admitted?
• What were your thoughts about yourself?
• To what extent do you perceive things differently after being selected?

In the study program

• To what extent do you think selection and the study program form one whole?
• To what extent have you experienced selection as the start of the study program? Why? How do you assess this?
• How would you want the selection procedure to connect to the study program?
• What do you need to use the selection results in the study program?
• How should selection results be integrated or not in the study program to be able to learn from them?