Serious games as an educational strategy for management and leadership development in postgraduate medical education – an exploratory inquiry

Introduction

The practice of medicine has witnessed many notable changes in the organization and delivery of care. The scope of these changes has been broad, ranging from the discovery of novel therapeutic modalities to shifts in the health care delivery process; for example, from the paternalistic approach to a more consumer-oriented focus. In the former approach, doctors solely decided what needed to be done to solve a patient’s problem. Nowadays, however, patients are increasingly engaged and involved in sharing the responsibility of care decision-making within a fast-growing consumerist health care context.¹ As a result, there are growing expectations from physicians to ensure the judicial administration of health care services and prescribe affordable and efficient therapies to patients. There is also an increased focus on the reallocation of expensive and potentially risky medication in exceptional situations to patients.² These changes, and more, have resulted in the need for doctors who can communicate well with their patients and effectively negotiate with other stakeholders in the delivery of health care services. To adequately address all of these expectations, both the consumers and providers of health care need a clear understanding of the real value of health care. Several institutions have developed and continue to develop initiatives to improve the quality of their services.³ Some of these interventions include the (re)organization of the delivery of care to adequately meet the needs of patients, investing in more scientific research to improve the process of care, and reappraising the training of health care providers to ensure that they are better prepared for new challenges from a fast-changing health care environment.⁴ In medical education, an outcome of this reappraisal process has been the CanMEDS competency framework by the Royal College of Physicians and Surgeons of Canada.⁵

Management and leadership training for physicians

In line with reappraisals of the ways physicians were being trained, we conducted a study to investigate medical residents’ (MRs) level of preparedness in the seven CanMEDS professional domains in the Netherlands.⁶ The findings from that study revealed that Dutch MRs perceived their leadership skills to be average, with very low ratings in their negotiation skills. The study also showed a clear need for training in practice management and leadership (PM&L), with extra focus on developing their negotiation skills. There was a need to know more about how to run clinical practices effectively, about the organization of health systems, and how to gain more proficiency in developing their career aspirations.⁷ In a separate multinational survey among MRs from the Netherlands, Canada, Australia, and Denmark, the data showed similar findings in the perceived knowledge and skills in essential areas of PM&L as well as the need for additional training in those areas.⁸ Many of the respondents preferred that the training takes place early in their professional training as doctors.^9–11

In response to most of these findings, many postgraduate training programs have reviewed their curricula and are now implementing PM&L development during residency. Examples include the mandatory leadership training program for doctors in postgraduate medical training in Denmark and the introduction of a clinical leadership project in the United Kingdom (a joint initiative of the NHS Institute for Innovation and Improvement and the Academy of Medical Royal Colleges). Other examples include the Harvard Business School, which has been working with local hospitals in Boston, offering electives in management development as well as designing a management skills training program for residents during their training.⁹

In addition to developing formal leadership training programs, the execution of management decisions in clinical practice requires novel educational strategies to integrate the different types of knowledge, critical judgment skills, and the execution of management decisions in clinical practice.¹² The integration of all of these components means that effective educational strategies must be sought to achieve this objective. In a previous study we published, we described the phenomenon of “generational” expectations otherwise known as the “generational gap.” We explained how this phenomenon had impacted the educational landscape, and the demands it had placed on curricular design, which subsequently has resulted in attuning the choice of instructional methods to the needs of the “Millennial resident” (ie, collaborative, technologically adaptive, feedback-driven learners).^13,14 This has resulted in the emergence and adoption of educational strategies such as e-learning, the flipped classroom, and serious games (SGs) into medical education as novel teaching methods.

Serious games as a leadership development strategy

In 2014, we developed a curriculum for PM&L development for postgraduate medical trainees in our institution (Busari JO. Longitudinal Curriculum for Leadership Development; unpublished data; 2014). The development of the PM&L curriculum evolved as a result of the extensive needs assessment survey we conducted among MRs and faculty in the Netherlands.^7,11 The content of the PM&L curriculum included 10 topics, ranging from knowledge of the health care system to effective negotiation skills, and the latter was a theme that the residents felt for additional training. The outcome of another needs assessment conducted amongst Dutch consultants (ie, attending physicians) also showed similar findings.¹⁰ Therefore, we set out to design a leadership training module in effective negotiation and chose to use a SG as a novel teaching modality. SGs are simulation contexts where specific “situations or activities” are used to replicate various “real life” events in the form of a game for prediction, analysis, or training purposes.^9,15,16 Furthermore, a SG is an educational context that we felt many residents would be familiar with and, like the flipped classroom, would appeal to the learning needs of residents.^14,17 Although we had reasons to assume that SGs would be a useful educational strategy, we lacked a clear understanding of whether it was an ideal intervention for teaching negotiation, a subset of our PM&L curriculum, and also how the MRs would perceive its utility and potential benefit for their learning. To define the appropriate research approach to investigate this phenomenon, we reconstructed the purpose of designing the leadership training module as well as the choice of the educational intervention using the typology of research purpose.¹⁸ There are nine categories of research purpose that have been identified in behavioral and social science: 1) prediction; 2) adding to the knowledge base; 3) having a social, educational, institutional, or organizational impact; 4) measuring change; 5) understanding complex phenomena; 6) testing new ideas; 7) generating new ideas; 8) informing constituencies; and 9) examining the past.

The purpose of our study fell into the fifth category of trying to understand complex phenomena, cultures, people, or change since we were interested in investigating the suitability of SGs to teach effective negotiation within our PM&L curriculum. We chose to conduct an exploratory inquiry, using in-depth interviews to examine the perceptions of residents on the utility of SGs as a teaching strategy. We were also interested in knowing whether SGs were effective teaching methods within a PM&L curriculum and, in this case, for negotiation skills. For our inquiry, we formulated the following research questions: 1) what are the MRs’ perceptions of their negotiation skills as a subset of the CanMEDS leader role; 2) what are MRs’ personal views about negotiation; 3) what are the MRs’ views about using SG to learn negotiation; and 4) what do the MRs expect in the construct/design of a SG to teach negotiation? Finally, we were interested in exploring examples of potential scenarios for effective negotiation in a SG.

Method

The participants of this study were MRs at the Zuyderland Medical Centrum in Heerlen, the Netherlands. We approached 10 residents for this study and six finally participated in the study. Five of the respondents were female, and all were from different specialties, namely pediatrics (N=3), gynecology (N=2) and from the Department of Emergency Medicine (N=1). The number of years in training varied among the participants from 1 to 5 years. The information about this study was sent to all of the respondents by e-mail after their consent and support from their program directors was obtained. The Ethical Review Board of the Zuyderland Medical Center approved the study.

We invited the six residents who gave their written informed consent to participate in separate interviews during the first few weeks of January 2015. A semistructured questionnaire was designed and used during the 60-min interview of each participant. The questionnaire comprised of two sections. The first section comprised of nine questions to investigate residents’ views about management skills. An example of one of the items in this section was “What has been the biggest challenge for you within the management functions you have to perform?” The second section of the questionnaire focused on a subset of the leader competency (ie, negotiation) and how to design a SG to develop this competency. This section consisted of eight questions. An example of one of the items in this section was “How would you train management and leadership skills in a game if you were the developer?”

The sessions were conducted after the MR’s working shift or during an afternoon break at work. Each interview was recorded, transcribed verbatim, and sent to the participants for approval of content before further analysis. Following approval, thematic analysis was used to analyze the data inductively. HY, DV, and JOB reviewed the transcripts for excerpts revealing participants’ perceptions of the SG. These were coded using theoretical coding methods (ie, open, axial, and selective coding). The coded excerpts were grouped into relevant themes that were named (ie, Theme 1. CanMEDS leader competency, Theme 2. MRs’ personal views about negotiation, Theme 3. Views about SGs, and Theme 4. Educational needs in a SG) in collaboration with JOB.

Results

The data analysis resulted in four themes that provided insight into how MRs perceived the utility of SGs as an educational strategy for PM&L development. A fifth theme illustrated examples of what residents thought the scenarios should look like in the SG.

CanMEDS leader competency

The opinions of the MRs differed on which of the subsets of CanMEDS leader role they felt well prepared in and those in which they needed additional training within the curriculum. Time management was one of the subsets of worrisome leader roles that was highlighted by three of the MRs. Two of them mentioned that they had difficulties with time management and would like additional training in it, while one MR said that time management was well covered in the current training program. The other three MRs did not mention time management.

Another subset of the leader role with contradicting feedback was communication. Two MRs explicitly indicated that they needed additional training in communication, while another one said communication was trained well enough in medical school and during residency. Public health and ethics was a subset of the leadership role that did not need additional training (n=2). Leadership, on the other hand, was described as a task that was difficult to manage. The two MRs who mentioned this competency also suggested they would like additional training in this within a SG.

The MRs were explicit about their views on their negotiation skills. Two MRs said they had difficulties with effective negotiation and were in need of additional training. Another MR said that negotiation and communication were part of time management, citing that “If you cannot negotiate or communicate effectively, you cannot manage your time properly.” This resident wanted additional training in time management but felt it was not possible without training in communication and negotiation. The other three MRs said they used negotiation skills every day.

Personal views about negotiation

As mentioned above, half of the respondents wanted additional training in negotiation. However, the individual views of the MRs on the concept of negotiation differed. One MR defined negotiation as “having to compromise viewpoints/decisions,” while another explained it as two parties having differing opinions on what is important and having to convince the other of what should be done. Another MR said: “Negotiation is not about getting it your way. It is about discussing the possibilities with another person and get to the best solution for both. The most satisfying negotiation is the one in which both individuals are critical because then you will get the best solution. Negotiation stimulates active thinking about a problem.” Asked how negotiations take place in practice, the respondents mentioned that this happened with several stakeholders. These included physicians (peers and superiors), clinical supervisors, patients, nurses, other MRs, and medical interns. We observed that it differed per MR as to which of the abovementioned stakeholders was experienced as hardest to negotiate with. Two MRs said that nurses were the most difficult to negotiate with, while another was of the opinion that because doctors and peers had more or identical professional authority, they were harder to negotiate with. Another MR thought that who was the most difficult to negotiate with differed per person and not necessarily per professional group. One of the respondents also noted that patients who visited hospitals regularly were often harder to negotiate with than those who almost never come to hospitals.

The MRs also had particular thoughts about negotiation with patients. One of the MRs said that negotiation with patients was needed when the patient had other expectancies than theirs, while another buttressed the importance of elaboration when negotiating with patients, to ensure that patients correctly understood the rationale of their treatment decisions or procedures.

We also observed that the context of an MR’s work (ie, the nature of the work shift) influenced the quality and type of negotiations they had. One MR felt that negotiating during night shifts was harder, because of reduced staffing and operating facilities during those hours of the day. Another felt, though, that that the difficulty in negotiating itself did not differ, instead the context within which it occurs during day shifts is what differs. For example, having to compete with other doctors to squeeze one’s patient into a surgery program. This MR concluded with “every shift has its difficulties when it comes down to negotiation.”

Views about serious games as a teaching strategy

Three of the MRs had a positive attitude toward learning management skills using a SG. Two of them preferred management training in a face-to-face setting while the third was indifferent about it. They argued that the attitude and motivation toward computer-based learning could be influenced positively by familiarity with games and also affect the educational outcome. Interestingly, the three MRs who were confident with SGs were also familiar with medical applications for mobile phones, which they regularly used for training skills or medical calculations. One of the MRs mentioned that she was unaware of what was possible with such a game. Admitting that she might be more enthusiastic when the tool was ready, the limited knowledge she had about gaming at the moment made her prefer the classical, traditional didactic method, arguing that the traditional, face-to-face approach of management training would be more realistic. Furthermore, the lack of nonverbal interaction, like body language and posture, would make the game less realistic and vivid. A SG would, therefore, require feedback according to one of the MRs who preferred the traditional method.

Educational needs in the construct of a serious game

The biggest concern the MRs had about the leadership SG was the ability of such a game to simulate real-life situations efficiently. Most of the MRs (n=4/6) wanted a game that would be realistic. However, the precise definition of realistic differed among them. In their terms, “realistic” was mostly referred to as a proper reflection of daily practice. Therefore, for a game to qualify as a real representation of everyday practice, it should include, amongst other things, time and finance-related restrictions (eg, not being able to perform particular treatments due to high costs or lack of time). One of the MRs even said that management skills could not be taught using a SG, because of the lack of body language, posture, and feedback. On the other hand, it would be of interest when combined in a module with real-life training.

The MRs who were enthusiastic about gaming complained that current medical applications that they regularly used are not realistic and offered options that they are not likely to encounter in real-life settings. The assumption that all generic brands of medication or specialist disciplines are available in all hospitals and that all specialists would be readily accessible to negotiate with were some of the examples given to describe this. Furthermore, half of all the respondents felt that, in such a game, management skills should be taught in combination with medically related problems and not in isolation. They argued that they would not enjoy a game if it focused purely on management skills since they had to deal with both medical and management-related problems in their day-to-day jobs. Although they did not explicitly mention that they would like to see management and medical training combined, the scenarios they described as examples implicitly implied this. Moreover, half of the interviewed MRs would prefer a game with a limited set of possible options or answers. The reason they cited most was that MRs would have too many choices to choose from when using open-answer options, resulting in dilemmas of which choices to choose or where to start. Another reason was that open-answer options were demotivating because it would cost MRs unnecessary time to search and find the needed information to answer the questions and type them out.

Two MRs mentioned they would like a game for each specialty. They would not be interested in a generic game for all specialties. Two of the MRs mentioned that they would like cross-training. These MRs wanted to play a character from another specialty because in this way they would learn more about their point of view. One of them thought that MRs would get better in negotiation when they gain more insight into the thinking process of the other person. The other MR’s thought was that understanding the other person’s ideas would lead to an increase in mutual understanding. Furthermore, three MRs agreed that the game did not need a particular order of actions. “It should not matter which one you pick first when you have a couple of options in the game. However, one’s action should determine the rest of the game, as in, when a wrong option is chosen the complexity of the game automatically increases.”

Lastly, there were a couple of remarks mentioned separately by individual MRs The remarks included: 1) combining a SG with the training of management skills in real life; 2) assessment of competency in the SG should not involve a summative scoring system; and 3) SGs should preferably be linked to applications that could be used to take notes, like “Evernote.” Another MR said that SGs needed variability in content, to maintain the interest and engagement of the user and prevent predictability of the outcome in the SG. One MR mentioned that the SG should be available without Wi-Fi while another highlighted the need for an affordable price to acquire the SG.

Educational scenarios

The MRs were asked to give examples of educational situations that exemplify “contexts of negotiation” which can be used to develop a SG. The examples of such potential scenarios included negotiations with patients, doctors (supervisors), and nurses. Most of the examples were during day shifts and just a couple during night shifts. We have listed a few illustrative scenarios with detailed descriptions in Box 1 that reflect different interactions involving MRs and a patient, a physician, and a nurse (see Box 1).

Box 1 Examples of situations where medical residents (MRs) engage in negotiation

Interaction with a patient

The example scenarios with patients were mostly about those who had read something on the internet and wanted additional tests or unneeded treatment from physicians. In some cases, patients demand being admitted to hospital even when it was not necessary. Thus, the focus of the negotiations with patients was mostly on reassuring them (see Box 2).

Box 2 Scenario 1: example of an interaction between a medical resident (MR) and a patient

Interaction with a doctor

The examples of scenarios with doctors were mostly about physicians from other specialties who do not have time for the MR’s patient. Another scenario was of physicians who are at home during night shifts and sometimes unwilling to come to the hospital to help the MR’s patient, because the attending considers the condition not to be as severe as the MR considers it to be. The example scenario with a supervisor is about an MR who was unsure about a patient’s treatment plan and wanted an attending physician from a different specialty to come by and check the patient (see Box 3).

Box 3 Scenario 2: example of an interaction between a medical resident (MR) and a doctor Abbreviations: Obs & Gyn, obstetrics and gynecology; PCA, patient controlled analgesia.

Interaction with a nurse

Most of the example scenarios with nurses included nurses who refuse to admit a patient because of a shortage of admission beds. Another example was nurses who declined to help a new patient because they were too busy taking care of other patients. There was also an example scenario about a nurse who wanted treatment for a patient other than what the MR had requested (see Box 4).

Box 4 Scenario 3: example of an interaction between a medical resident (MR) and a nurse

Discussion

The purpose of this study was to investigate the views of MRs with regards to the development of a SG for training management and leadership skills. Five questions that represented the theme of this paper were used to investigate the respondents’ preferred methods of using SGs as an educational strategy for leadership development. Regarding the subquestion on how PM&L skills should be trained using a SG, our findings revealed contradictory responses on communication and time management skills. Some respondents felt that their current curricula equipped them sufficiently with these skills, while others felt that they needed additional training.

The residents felt that leadership and negotiation were two domains they had difficulties with and for which they desired further training. It was interesting to observe that besides negotiation skills the findings from this study did not corroborate the findings in a previous study where MRs expressed their desire for training in negotiation skills, departments and group practices, knowledge of health systems, and career development.⁸ The results of the current study also differed from the study by Daugird and Spencer¹⁹ where they demonstrated that most of the medical students needed additional knowledge about the organization of health care, diagnosis-related groups, and practice management and an understanding of tax-related expenditures. We attribute these observations to differences in the size and composition of the samples as well as to the time lapse of more than a decade between the current study and that of Daugird and Spencer.¹⁹

Our analysis of the respondents” opinions of negotiation as a subset of the CanMEDS leader roles suggest that MRs needed to (and regularly) negotiate(d) with different stakeholders, namely doctors, supervisor, patients, nurses, other MRs, and medical interns. They described the process of negotiation as attempting to “convince” the other person on a subject or viewpoint. They also had different ideas about negotiation when it involved patients. For example, one of the respondents said that negotiation with patients was needed in those situations when patients have other expectations than theirs, usually after patients have obtained information about disease and treatment options from other sources (eg, the internet). In general, the respondents’ opinions differed on which group of stakeholders was the most difficult to negotiate with. There was also no agreement on the influence of work shift (ie, day or night) as well as on the frequency and intensity of negotiation conversations.

Further exploration of the residents’ views about using SGs to learn negotiation skills showed that those who were familiar with medical applications and had them installed on their smartphones for use had a more positive attitude toward gaming than those who were not. Also, the MRs who preferred traditional training methods were unaware of the vast array of possibilities for SGs and particularly concerned about their authenticity (ie, whether it would be realistic or not). We assume that these “novice” MRs might develop a more positive attitude to gaming if and when they encounter fully developed games that pay sufficient attention to the authenticity of context. The use of animations that show nonverbal communication, as well as direct feedback on actions in the SG, are examples of achieving this.

The fourth subquestion we intended to answer was what MRs would want to see in a SG should such an educational intervention be developed. Our findings showed that the MRs were mostly interested in how realistic the content of the game would be and that it should also be able to combine PM&L skills with medical knowledge and clinical expertise. While just a few of the respondents mentioned that they would like to see their PM&L and medical training combined, all of them used SG scenarios that combined both of these educational contexts. The model of van Merriënboer and Kirschner²⁰ also supports the need for MRs to combine medical knowledge with the development of management skills during their training. This model argues that complex skills such as management and medical practical (and cognitive) skills need to be taught holistically. Therefore, the complex skills of a PM&L curriculum should not be trained in isolation. This is because a “paradox of transfer” could occur if these two skills are taught separately. The paradox of transfer or (transfer paradox) refers to the high contextual interference that has 1) adverse effects on performance during practice, 2) no effect on the number of retention problems solved after training, but 3) positive effects on the number of new problems solved after an exercise or learning activity (transfer). Also, the methods that work best for reaching isolated goals are often not the best ones to help achieve integrated goals. By avoiding compartmentalization and fragmentation, however, the isolated goals can easily be prevented.²⁰ Hence, it makes sense to offer the training of management skills within (or in close alignment with) medical cases.

Another preference of the MRs was that SGs should be domain specific and not generic. In the examples of the scenarios they provided, all of the respondents used examples within their specialties. When asked to give a more generic context, they all said that they could not find one. Furthermore, some MRs said the SG needed variability to keep it interesting and to prevent the game from being predictable. Bearing the theory of complex learning in mind,²⁰ it is evident that variability of practice is of utmost importance to achieve the transfer of knowledge. Our last inquiry was about the possible scenarios that could be used to develop negotiation skills in a SG. The proposed scenarios were those involving negotiations between residents and doctors, direct supervisors, nurses, or patients and could be either during the day or night shifts. See the overview of examples of these scenarios in Box 1.

Although we set out to interview 10 residents for this study, the logistics and realities of roster coverage forced us to contend with interviewing six residents who were available. Fortunately, the depth of the information we derived from the interviews conducted compensates for the small number of residents interviewed. We are cautious of the limitations of the generalizability of our findings, bearing in mind that the purpose of this study was to investigate the suitability of SG as an educational strategy for PM&L. Concerning the latter, we believe this study informs us about the potential use of SGs to teach certain aspects of PM&L. It shows residents’ readiness to use this educational modality for this purpose and, if well designed, could be an effective strategy to teach specific aspects of PM&L.

Conclusion

To investigate whether the identified needs are generalizable for all MRs, a survey consisting of the findings from this explorative inquiry can be administered in a more significant sample of MRs in different hospitals. It is recommended for future research to investigate the view of educational experts on this subject, as well as the possibilities of collaboration between gaming companies, MRs, and experts in medical education to design SGs for medical education. This study provides a basis for the use of SGs for PM&L education in postgraduate medical education.

Summary points

• MRs want additional training in the following CanMEDS competencies: leadership, negotiation, communication, and time management.
• MRs negotiate with different stakeholders in the clinical workplace, namely doctors, supervisor, patients, nurses, other MRs, and medical interns.
• MRs who are familiar with medical applications for mobile phones have a more positive attitude toward learning through serious gaming than those who are unfamiliar with these apps.
• The biggest concern of the MRs is that the game should be realistic, should not be generic, and should be capable of offering enough variability.
• MRs would prefer a game in which the management skills are combined with medical knowledge and expertise.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Jung HP, Wensing M, Grol R. [Difference between paternalism and consumerism the dilemma of the GP] Tussen paternalisme en consumentisme Het dilemma van de huisarts. Huisarts Wet. 2001;44(13):19–25.
2.		Beleidsnotie. [A more efficient use of medicines] Een doelmatiger gebruik van geneesmiddelen (April 2005). Available from: https://zoek.officielebekendmakingen.nl/kst-29477-17-b1.pdf. Accessed April 18, 2018. Dutch.
3.		Porter ME, Teisberg EO. Redefining Health Care: Creating Value-Based Competition on Results. Boston: Harvard Business School Press; 2006:71–95.
4.		Bohmer RMJ. Designing Care: Aligning the Nature and Management of Health Care. Boston: Harvard Business School Press; 2009:98–102.
5.		Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29(7):642–647.
6.		Berkenbosch L, Busari JO. [The management competences of Dutch physician assistants] De managementcompetenties van Nederlandse arts-assistenten. Tijdschrift voor Medisch Onderwijs. 2011;30(6):302–314. Dutch.
7.		Berkenbosch L, Bax M, Scherpbier A, Heyligers I, Muijtjens AM, Busari JO. How Dutch medical specialists perceive the competencies and training needs of medical residents in healthcare management. Med Teach. 2013;35(4):e1090–e1102.
8.		Berkenbosch L, Schoenmaker SG, Ahern S, et al. Medical residents’ perceptions of their competencies and training needs in health care management: an international comparison. BMC Med Educ. 2013;13:25.
9.		Stephenson J. Doctors in management. BMJ. 2009;339:4595.
10.		Berkenbosch L, Brouns JW, Heyligers I, Busari JO. How Dutch medical residents perceive their competency as manager in the revised postgraduate medical curriculum. Postgrad Med J. 2011;87(1032):680–687.
11.		Brouns JW, Berkenbosch L, Ploemen-Suijker FD, Heyligers I, Busari JO. Medical residents perceptions of the need for management education in the postgraduate curriculum: a preliminary study. Int J Med Educ. 2010;1:76–82.
12.		Busari JO, Berkenbosch L, Brouns JW. Physicians as managers of health care delivery and the implications for postgraduate medical training: a literature review. Teach Learn Med. 2011;23(2):186–196.
13.		Sharma N, Lau CS, Doherty I, Harbutt D. How we flipped the medical classroom. Med Teach. 2015;37(4):327–330.
14.		Lucardie A, Busari J. The flipped classroom as a pedagogical tool for leadership development in postgraduate medical education. Educ Sci. 2017;7(2):63.
15.		Banks J, Carson JS, Nelson BL, Nicol D. Discrete-Event System Simulation. 5th ed. Upper Saddle River, NJ: Prentice Hall; 2010.
16.		Schroedl CJ, Corbridge TC, Cohen ER, et al. Use of simulation-based education to improve resident learning and patient care. J Crit Care. 2012;27(2):219.e7–219.e13.
17.		Lucardie AT, Berkenbosch L, van den Berg J, Busari JO. Flipping the classroom to teach Millennial residents medical leadership: a proof of concept. Adv Med Educ Pract. 2017;8:57–61.
18.		Newman I, Ridenour CS, Newman C, DeMarco GMP. A typology of research purposes and its relationship to mixed methods. In: Tashakkori A Teddlie C, editors. Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: Sage; 2003:167–188.
19.		Daugird AJ, Spencer DC. The perceived need for physician management training. J Fam Pract. 1990;30(3):51–52.
20.		van Merriënboer JJG, Kirschner PA. Ten Steps to Complex Learning. London: Taylor & Francis; 2007.