Is there a way for clinical teachers to assist struggling learners? A synthetic review of the literature

Introduction

Clinical teachers are faced with the daily challenge of managing patients while simultaneously supervising medical trainees from all levels of training and with varying degrees of autonomy.¹ Steinert has defined struggling learners as medical trainees who do “not meet the expectations of a training program”.² Such trainees, who experience more difficulties, often pose an even greater challenge to clinical teachers, since these learners warrant closer supervision,³ which may be time-consuming and compete with time spent on patient care, but may also require specific teaching skills.⁴ As a result, the supervision of struggling learners has been associated with reactions from clinical teachers ranging from helplessness to frustration and avoidance,^2,5 as clinical teachers often perceive that they are ill equipped to supervise these learners efficiently.⁶

The discomfort that clinical teachers may feel when supervising struggling learners may have concrete consequences for learners and patients alike. Clinical teachers in the field are often best positioned to detect difficulties early on, due to their privileged access to learners’ actual performances with real patients, and to the fact that struggling learners very rarely step forward themselves to ask for help.⁷ When difficulties are not addressed as soon as they are detected, undue delays might occur before remediation and closer follow-up can take place.^8,9 In certain cases, learners’ difficulties may in fact never be addressed, which may eventually allow less than fully competent trainees to be promoted to independent practice.^10–12

Although clearly delineated guidelines for supervising struggling learners could offer a useful solution to these issues, no such guidelines exist, due to the paucity of evidence to guide best practices in remediation.^9,13 Intervention studies regarding remediation have often been limited to small single-institution endeavors, and have focused on the remediation of cognitive difficulties.^9,14 Although recommendations for remediation at an institutional level have recently been published,¹⁵ the best approach for clinical teachers in the field remains to be established.

Therefore, we aimed to present here a synthetic review of the empirical evidence and theory that may guide clinical teachers in their daily task of supervising struggling learners. This article reviews our current knowledge on the challenges and solutions that have been identified and explored.

Materials and methods

Our methodology was based on Grant and Booth’s¹⁶ typology of reviews, which defines a literature review as the “examination of recent or current literature”, whose “analysis may be conceptual or thematic” and is “typically narrative”. A search of the relevant literature was performed in both medical (Medline, Embase) and educational (ERIC, Education Source) databases, using the following search terms: underperformance, fail, at risk, difficult or problem, combined with clinical or medical student or resident. The search covered the 1995-2015 period, and was limited to the English and French languages. This search strategy generated 440 titles. After an initial title-based screening, 70 abstracts were read, of which 37 were rejected based on relevance. A total of 33 articles were thus included in this review (Figure 1).

Figure 1 Flow diagram for article inclusion.

Articles were considered for inclusion if they addressed the issue of struggling learners from the perspective of medical teachers in the field, or if they presented empirical evidence or theory that could inform their perspective. Because this review focused on clinical teachers’ daily supervision tasks, articles that addressed the issues of learner difficulties solely from an institutional or systematic perspective were excluded. A meta-analysis was not deemed feasible, due to the wide variety of research contexts and methodologies encompassed, nor was an exhaustive review the purpose of this article. Rather, it aimed to synthesize the most relevant concepts to guide clinical teachers when working with struggling learners.

Results

Theoretical framework

A theoretical framework for addressing struggling medical trainees was provided by Langlois and Thach,¹⁷ who first suggested that the SOAP (subjective, objective, assessment, plan) clinical model could be used to structure the approach to learners with difficulties. This model has since been used repeatedly in medical education.^18–20 It takes advantage of the parallel often drawn, since Irby²¹ pointed it out, between the clinical and pedagogical diagnostic approaches.^1,22–24 When applied to struggling learners, the SOAP model divides the task into four steps that are well known to clinicians: 1) detecting problems, based on a subjective impression; 2) gathering and documenting objective data, according to diagnostic hypotheses; 3) making a pedagogical diagnosis based on this assessment; and 4) planning a targeted remediation. These steps can guide clinical teachers to approach learners’ difficulties in the same order that they assess patient symptoms. In this review, the relevant literature and recommendations (Table 1) are also organized as per these four main steps.

Table 1 Suggested approach to struggling learners Abbreviations: EPAs, “entrustable” professional activities.

Subjective: Detecting a problem based on a subjective impression

Between 10% and 15% of learners will experience significant difficulties during their medical training.^25–27 The various steps leading to their formal identification are most often initiated based on the subjective impressions of clinical teachers.⁷ These subjective impressions, or intuitions, can be either shaped by direct observation of the learner in action or by interacting formally or informally with the student.²⁸ Clinical teachers’ perceptions are indeed considered to be a reliable predictor of learners’ difficulties. Weller et al,³ for instance, found that when clinical teachers were asked to identify subjectively which learners required closer supervision, a reliability coefficient of 0.7 was reached with nine observers, whereas 50 observers were required to reach this same coefficient when more traditional measures were used.

In theory, clinical teachers’ recognition of difficulties in a learner should prompt the next steps of documentation, assessment, and remediation rapidly, since early identification is considered the gold standard to attain.²³ In practice, however, the process is very often halted at this stage.^2,9,18,29,30 One of the most frequently cited reasons for this, according to many exploratory studies done with clinical teachers, is the discomfort of not knowing which steps should be followed and how to achieve them.^6,31–33

The consequence of not acting on clinical teachers’ early impressions is the frequent delay in identifying and addressing learners’ difficulties. Late identification of difficulties was defined by Lacasse³⁴ as identification that takes place after the first quarter of a rotation. When this occurs, practical implications are that less time is left for efficient remediation, but also that critical incidents may take place, with concrete consequences to patients, before a red flag is raised.^8,28 A US survey of program directors in internal medicine in 2000 drove the point home by revealing that up to 59% of their residents in difficulty were indeed identified after a critical incident.³⁵ In order to avoid consequences associated with late identification of difficulties, and because their subjective impressions can be considered reliable, clinical teachers should be encouraged to act on their perceptions of difficulties in learners as soon as a doubt arises,³⁴ beginning with the gathering and documenting of objective data.

Objective: Gathering and documenting objective data

According to Bearman et al,³⁰ establishing a diagnosis in medical education consists in “identifying discrepancy between the expected performance standard and the demonstrated performance, and then trying to establish the reason for underperformance”. In many clinical programs, expected abilities at various stages of training are defined through milestones,³⁶ while key tasks in a discipline that learners should eventually be trusted to perform are defined as “entrustable” professional activities (EPAs).³⁷ Milestones and EPAs can both be very useful in identifying and documenting discrepancy. Once it is identified, however, underperformance should be seen as a symptom, not as a diagnosis in itself, which once made could justify ending the observation phase.^17,34,38 The first step in addressing difficulties should consist, on the contrary, in further observations to define the issue better²⁸ and to clarify its cause better, in order to determine the most appropriate response.^18,19 Further verification of the initial impression formed also helps determine whether the learner is in fact experiencing difficulties that require follow-up or whether the issue was contextual or even an isolated incident.³⁹

Through this collection of data, tangible examples must be gathered. This should be done as much for clinical teachers, who will thus verify their impressions, as for the learner, who is more likely to see credibility in the feedback received,³⁵ and for the institution, who will be able to justify its decisions better.^18,34 Objective data are indeed an integral part of a fair approach.²⁸ According to many authors, this initial documentation is the responsibility of clinical teachers in the field, because only they have access to learners’ performances in a real clinical context.^30,40,41 The documentation of clinical performance by clinical teachers is essential, since only 2%–6% of struggling learners will self-identify.^7,35 Various hypotheses have been put forward to explain why such low rates of learners will themselves ask for help. These hypotheses range from fear of academic and social consequences³⁵ to self-appraisals, which are significantly different from actual performance.⁴² Kruger and Dunning⁴³ have indeed shown that those who are “unskilled” in a certain discipline are often unaware of it, because the same knowledge is often necessary to be both competent and have an accurate perception of one’s competence in a given field. For instance, participants in the Kruger and Dunning study whose test scores were in the 12th percentile estimated themselves to be in the 62nd percentile. These participants only recognized their errors once their skills had improved.

How to collect pedagogical data

Data collection can be achieved either through indirect or direct observation of learners’ performances. The latter occurs relatively rarely unless procedures or techniques are involved,^9,44 but will often be more effective. In a national US survey of the Association of Program Directors in Internal Medicine, a clear majority (82%) reported that problem-learner difficulties had been identified in their institution through direct observation.³⁵ Difficulties, however, can also be inferred from the manner in which cases are presented or from indirect observation,⁴⁵ but where indirect observation is the only type of observation, certain difficulties could be masked.²⁹ Data can also be collected by reviewing notes written by the learner in patient files or through their formal and informal interactions with staff and faculty. No matter which approach is preferred or available, data should always be based on more than one context and on as many observations as possible.^9,46

Many observations are indeed necessary, due to the subjective nature of observations made in clinical settings, but also due to case specificity. As Nendaz et al⁴⁷ put it, the competence to diagnose varies from one case to another for the same clinician. It is thus more reliable to question a learner more briefly on many cases. For an observed behavior to lead to a diagnosis, it ought to have been observed repeatedly⁹ and in various types of situations. Indeed, each performance depends as much on the context as it depends on the content,⁴⁸ and hence Jouquan’s statement⁴⁹ that the nonmanifestation of a certain knowledge does not systematically equate with its absence [translation].

Hinson et al⁵⁰ stressed the importance of an additional source of data, which they called “diagnostic conversations”: these conversations consist of “discussing the problem and getting a fuller picture of everything that is going on for that student”. This conversation allows a more accurate pedagogical diagnosis, but may also bring to light potential gaps in perceptions, giving an opportunity for both parties to share the same understanding of a given situation.³⁴ The diagnostic conversation can henceforth be used as a common starting point, which will facilitate a shared appreciation of whether or not progress has been made later on.³⁰ When relevant, it will also allow the taking into account of the learner’s context and reality with regard to the remediation plan, allowing it to be more realistic and increasing the chance that it will be followed.⁵⁰ In fact, the reasons for the observed difficulties should be determined jointly with the learner, since only the learners hold the whole picture of their personal situation.^28,30,38 Faculty can only gain from adding this insider’s perspective to the outside perception of the learner’s professional performance. Diagnostic conversations can take the form of a brief exchange at the end of a shift or of a more substantial discussion in light of all the assembled information to date, but Lacasse³⁴ recommends that at the very least, an informal discussion be held with the learners themselves before any further step is undertaken.

Assessment: Making a pedagogical diagnosis based on assessment of the collected data

Underperformance may be due to one of three main categories of difficulties – related to cognition, mental health, or attitude – among which cognitive difficulties are the most frequent.^27,35,51 When considering which category is most likely involved, it must be borne in mind that causes of underperformance are often interrelated.³⁴ A study by Guerrasio and Aagaard,¹⁴ for instance, showed that among 53 learners followed for clinical reasoning difficulties, three-quarters had “comorbidities”, ranging from insufficient knowledge (17% of students) to mental health issues (12% of residents). In such cases, the best approach is to address difficulties one by one, beginning with the predominant difficulty for that learner or the one that interferes most with their work.

One prominent finding when reviewing the literature on the causes of underperformance to consider when a learner is in difficulty is the absence of consensus among authors on what the differential diagnosis should include and on how to classify it. Indeed, up to 17 different classification systems of underperformance have been identified in this review. They have been synthesized in one table for the purpose of this article, and are presented in Table 2. This striking disparity of classifications seems to be related to the disparity of sources used by each author to classify learner difficulties. Cited sources vary from regulating institutions, whose classification systems are based on expert consensus^7,35,52 to systematic reviews^34,53 to compilations of clinical teachers’ narrative descriptions.^51,54,55 In a few cases, no source was cited.^18,22,56,57 This multitude of classification systems also stems from the fact that none of these systems was reused in its entirety by authors other than the original author. Rather, new classification systems were created each time by adapting or summarizing one or more preexisting classifications.^17,19,20,58

Table 2 A synthesis of classification systems of underperformance in medical education

A quick look at the synthesis of these classification systems in Table 2 reveals that among the main causes of underperformance that are cognitive in origin, a lack of knowledge is cited significantly more often than clinical reasoning difficulties. However, while insufficient knowledge is indeed much more prevalent among medical students in difficulty, clinical reasoning difficulties have been shown to be at least as frequent during residency.⁷ The fact that its consideration as a cause of underperformance is so markedly less frequent (four authors of 17 cited it explicitly) could also suggest a lack of knowledge transfer from one research field to the other: although clinical reasoning has been the subject of 40 years of research, it may at times be confined to a debate among experts,²⁹ which may be partly due to the hermetic terminology used in cognitive psychology, as suggested by Kempainen et al.⁵⁹ Moreover, there seems to exist a misconception that clinical reasoning errors are driven by a lack of medical knowledge, rather than an inability to apply that knowledge in clinical practice.⁶⁰

Plan: Planning a targeted remediation

Remediation was defined by Guerrasio et al⁷ as “additional teaching above and beyond the standard curriculum, individualized to the learner who without the additional teaching would not achieve the necessary skills for the profession”. As pointed out by Hauer et al’s⁹ thematic review of the literature on remediation, no standardized and universally accepted remediation processes have yet been developed. Remediation initiatives will thus often resort to ad hoc and uncoordinated processes^27,61 or to the sterile repetition of inconclusive strategies.^30,62–64 One possible cause for the absence of a standardized remediation process may very well be the aforementioned absence of consensus on the specific causes of underperformance. It is indeed much more difficult to solve a problem that has not been well defined, and the demonstrated disparity among classification systems of underperformance means that there exists at present no consensual basis on which to build common guidelines for managing difficulties.

In practice, obstacles to efficient remediation also arise when the pedagogical diagnosis has not been well established at the outset. Remediation may in fact be fruitless if the previous step, clarifying the diagnosis, has been overlooked.²⁸ Therefore, the first step in planning a targeted and efficient remediation is to have pinpointed the underlying issue as precisely as possible. This point cannot be overstressed, knowing that Bearman et al³⁰ have shown, consistent with our anecdotal experience, that the very idea of matching a remediation intervention with the underlying difficulty is rarely a consideration for clinical teachers in the field. This approach may lead, for instance, to simply adding a supplemental clinical rotation as a remediation strategy for issues related to professionalism. The lack of well-defined remediation strategies for clinical teachers in the field can only (understandably) deter them from naming difficulties that they would not know how to address. Therefore, there is a strong need for further research with regard to remediation that is evidence-based, individualized, and targeted according to pedagogical diagnosis.^{2,18,35,65,66} In the absence of formal guidelines, however, some guiding principles may be drawn from previous studies on remediation.

In situ remediation

Whereas learners with noncognitive difficulties should be referred, it is generally recommended that cognitive difficulties first be addressed within the learners’ current rotations,^9,18,67 an approach that Bearman et al³⁰ termed “in situ learning”. Only if this approach fails should the issue be brought to the next institutional level. Remediation that is integrated with the learners’ usual clinical activities is generally regarded as the most efficient,^18,30 because in situ remediation is based on the cognitive psychology principle of experiential learning. First, it allows the learner to anchor new knowledge to skills that are already mastered and to a context that is already familiar.⁶⁵ Second, it also allows this new knowledge to be contextualized and transferred immediately to an authentic practice setting.^68,69 Finally, in situ remediation takes advantage of experiential learning by focusing the learning process on real-life issues that the learner truly has to solve.³⁴ Unfortunately, in practice, a survey conducted by Saxena et al⁶⁵ among 71 US medical schools showed this integrated form of remediation to be the least utilized, possibly (according to the same study) because of the lack of confidence clinical teachers often express regarding their own capacity for remediating learners’ difficulties.

Time-efficient remediation

The management of struggling learners is notorious – and sometimes dreaded – for being time-consuming.^{7,14,20,34,61} In one study based on 151 medical learners with difficulties, Guerrasio et al⁷ observed that the remediation of clinical reasoning difficulties was among the most time-consuming remediation processes, with an average of 20 hours spent on individual interventions per student. There is an obvious need for time-efficient remediation strategies, because for remediation activities to be realistically integrated with a regular rotation, it is crucial that these activities be time-efficient, as they will otherwise not be acceptable to clinical teachers.⁷⁰

Discussion

Our review has shown that many principles are well established regarding the approach to struggling medical learners. As presented in Table 1, an effective approach to the clinical supervision of struggling learners should aim for early and precise identification of difficulties, based on repeated observations and on an initial informal discussion with the learner. A pedagogical diagnosis should be made, supported by various sources of data, acknowledging that difficulties can be interrelated. Only once this pedagogical diagnosis is clear should a targeted remediation strategy be developed and ideally integrated with the learner’s regular clinical activities. This review has also shown, however, that there remain both theoretical and practical challenges to address, in order to operationalize these principles further. Three-pronged recommendations are discussed in response to these challenges.

Faculty development

Certain barriers to applying best practices in the field could be addressed by faculty-development endeavors. Firstly, a lack of proper documentation by clinical teachers has been identified as one of the main causes of failure. It has been suggested that this step may be omitted, because clinical teachers do not know what to observe and document specifically. Knowledge of key elements to document proof of competence in a learner, or lack thereof, should be made explicit to clinical teachers, in accordance with the expected development path at respective institutions. Secondly, the identification of precise pedagogical diagnoses to explain underperformance has repeatedly been identified by clinical teachers themselves as a pressing faculty-development need. Since the next practical steps will derive from these diagnoses, faculty support in identifying the underlying issues would be instrumental in helping clinical teachers carry through an effective approach to struggling learners.

Institutional procedures

This review has also pointed out some steps in the management of struggling learners, which if implemented locally as preset procedures could then serve as common references among faculty. We hypothesize that having a simple and preestablished procedure to which learners could be referred should foster earlier identification of struggling learners. In this perspective, the first process to formalize should be a comprehensive description of steps to follow when referring a learner for remediation. Such processes should be determined locally, based on best evidence in remediation and organized according to available resources. For instance, clinical teachers, as well as rotation supervisors, should know ahead who to notify, formally and informally, if an issue arises. It should be clear who is responsible for overseeing the remediation process, and who serves as faculty advisor for this purpose. A second procedure, which could be preestablished locally, is a list of relevant descriptors for clinical teachers to observe and document, in order to situate the learner effectively with regard to the expected level of competence. Such descriptors will generally be drawn from the milestones relevant to each level of training.

Further research

The absence of a common framework to organize learner difficulties and from which to build standardized remediation strategies has been highlighted in this review. The establishment of such a framework should be given priority, in order for future research to build around a common structure. This objective also appears as a preliminary step to address a second pressing need: that of developing and evaluating efficient remediation strategies on a wider scale. Such endeavors should focus on targeting remediation strategies according to diagnosis, hence the need for a common framework to build from, and should aim for them to be time-efficient, so as not to impede their implementation in practice.

Conclusion

This article aimed to review the theory and empirical evidence available regarding the global approach to struggling learners during clinical supervision. Our review has shown that the perception that a learner is experiencing difficulty is assessed intuitively and reliably by clinical teachers, but that their unfamiliarity with the following steps, from pedagogical diagnoses to remediation strategies, leads to delays before the formal identification of these learners. Because early identification of struggling learners should be considered a gold standard, the subsequent steps for clinical teachers have been presented here, organized around the clinical model SOAP. Many obstacles have been identified to their practical application, among which the lack of standardized remediation procedures is not a lesser deterrent. Is there a recipe for supervising struggling learners? The answer remains to be completed, and future research and faculty-development endeavors should pursue the common goal of further operationalizing remediation strategies in response to specific diagnoses and making these processes more accessible to clinical teachers in the field.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Audétat MC, Laurin S. Clinicien et superviseur, même combat! [Clinician and supervisor, same battle!]. Med Que. 2010;45(5):53–57. French.
2.		Steinert Y. The “problem” junior: whose problem is it? BMJ. 2008;336(7636):150–153.
3.		Weller JM, Misur M, Nicolson S, et al. Can I leave the theatre? A key to more reliable workplace-based assessment. Br J Anaesth. 2014;112(6):1083–1091.
4.		Walsh A, Antao V, Bethune C, et al. Fundamental Teaching Activities in Family Medicine: A Framework for Faculty Development. Mississauga (ON): College of Family Physicians of Canada; 2015.
5.		Dory V, Audétat MC. “Ils sont carrément incurables”: comment les métaphores des cliniciens enseignants révèlent leur malaise dans la gestion des difficultés de raisonnement clinique de leurs internes. [“They’re downright incurable”: how the metaphors clinical supervisors use reveal their unease at dealing with the clinical reasoning difficulties of their residents]. Pedagog Med. 2013;14(2):83–97. French.
6.		Audétat MC, Faguy A, Jacques A, Blais JG, Charlin B. Étude exploratoire des perceptions et pratiques de médecins cliniciens enseignants engagés dans une démarche de diagnostic et de remédiation des lacunes du raisonnement clinique. Pedagog Med. [Exploratory study on the perceptions and practices of clinical teachers involved in a clinical-reasoning difficulty diagnosis and remediation process]. 2011;12(1):7–16. French.
7.		Guerrasio J, Garrity MJ, Aagaard EM. Learner deficits and academic outcomes of medical students, residents, fellows, and attending physicians referred to a remediation program, 2006-2012. Acad Med. 2014;89(2):352–358.
8.		Challis M, Flett A, Batstone G. An accident waiting to happen? A case for medical education. Med Teach. 1999;21(6):582–585.
9.		Hauer KE, Ciccone A, Henzel TR, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature. Acad Med. 2009;84(12):1822–1832.
10.		Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med. 2005;80(10 Suppl):S84–S87.
11.		Guerrasio J, Furfari KA, Rosenthal LD, Nogar CL, Wray KW, Aagaard EM. Failure to fail: the institutional perspective. Med Teach. 2014;36(9):799–803.
12.		Yepes-Rios M, Dudek N, Duboyce R, Curtis J, Allard RJ, Varpio L. The failure to fail underperforming trainees in health professions education: a BEME systematic review – BEME Guide no. 42. Med Teach. 2016;38(11):1092–1099.
13.		Ziring D, Danoff D, Grosseman S, et al. How do medical schools identify and remediate professionalism lapses in medical students? A study of U.S. and Canadian medical schools. Acad Med. 2015;90(7):913–920.
14.		Guerrasio J, Aagaard EM. Methods and outcomes for the remediation of clinical reasoning. J Gen Intern Med. 2014;29(12):1607–1614.
15.		Kalet A, Guerrasio J, Chou CL. Twelve tips for developing and maintaining a remediation program in medical education. Med Teach. 2016;38(8):787–792.
16.		Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26(2):91–108.
17.		Langlois JP, Thach S. Managing the difficult learning situation. Fam Med. 2000;32(5):307–309.
18.		Ronan-Bentle SE, Avegno J, Hegarty CB, Manthey DE. Dealing with the difficult student in emergency medicine. Int J Emerg Med. 2011;4:39.
19.		Vaughn LM, Baker RC, DeWitt TG. The problem learner. Teach Learn Med. 1998;10(4):217–222.
20.		Hicks PJ, Cox SM, Espey EL, et al. To the point – medical education reviews: dealing with student difficulties in the clinical setting. Am J Obstet Gynecol. 2005;193(6):1915–1922.
21.		Irby DM. What clinical teachers in medicine need to know. Acad Med. 1994;69(5):333–342.
22.		Steinert Y, Levitt C. Working with the “problem” resident: guidelines for definition and intervention. Fam Med. 1993;25(10):627–632.
23.		Evans, Alstead EM, Brown J. Applying your clinical skills to students and trainees in academic difficulty. Clin Teach. 2010;7(4):230–235.
24.		Irby DM. Excellence in clinical teaching: knowledge transformation and development required. Med Educ. 2014;48(8):776–784.
25.		Yates J, James D. Predicting the “strugglers”: a case-control study of students at Nottingham University Medical School. BMJ. 2006;332(7548):1009–1013.
26.		Faustinella F, Orlando PR, Colletti LA, Juneja HS, Perkowski LC. Remediation strategies and students’ clinical performance. Med Teach. 2004;26(7):664–665.
27.		Smith CS, Stevens NG, Servis M. A general framework for approaching residents in difficulty. Fam Med. 2007;39(5):331–336.
28.		Steinert Y. The “problem” learner: whose problem is it? AMEE Guide no. 76. Med Teach. 2013;35(4):e1035–e1045.
29.		Audétat MC. L’identification et la remédiation des difficultés de raisonnement clinique en médecine. [The identification and remediation of clinical reasoning difficulties in medicine]. (État des pratiques, recherche d’outils et processus pour soutenir les cliniciens enseignants). Montreal: Université de Montréal; 2010. French.
30.		Bearman M, Molloy E, Ajjawi R, Keating J. “Is there a plan B?”: clinical educators supporting underperforming students in practice settings. Teach High Educ. 2013;18(5):531–544.
31.		Rees CE, Knight LV, Cleland JA. Medical educators’ metaphoric talk about their assessment relationships with students: “You don’t want to sort of be the one who sticks the knife in them”. Assess Eval High Educ. 2009;34(4):455–467.
32.		Monrouxe LV, Rees CE, Lewis NJ, Cleland JA. Medical educators’ social acts of explaining passing underperformance in students: a qualitative study. Adv Health Sci Educ Theory Pract. 2011;16(2):239–252.
33.		Cleland, Knight LV, Rees CE, Tracey S, Bond CM. Is it me or is it them? Factors that influence the passing of underperforming students. Med Educ. 2008;42(8):800–809.
34.		Lacasse M. Diagnostic et prise en charge des situations d’apprentissage problématiques en éducation médicale. [Educational diagnosis and management of challenging learning situations in medical education]. Quebec: Université Laval; 2009. French.
35.		Yao DC, Wright SM. National survey of internal medicine residency program directors regarding problem residents. JAMA. 2000;284(9):1099–1104.
36.		Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system: rationale and benefits. N Engl J Med. 2012;366(11):1051–1056.
37.		ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39(12):1176–1177.
38.		Paice E. Identification and management of the underperforming surgical trainee. ANZ J Surg. 2009;79(3):180–185.
39.		Steinert Y, Nasmith L, Daigle N, Franco ED. Improving teachers’ skills in working with “problem” residents: a workshop description and evaluation. Med Teach. 2001;23(3):284–288.
40.		Arawi T, Rosoff PM. Competing duties: medical educators, underperforming students, and social accountability. J Bioeth Inq. 2012;9(2):135–147.
41.		Bernstein S, Atkinson AR, Martimianakis MA. Diagnosing the learner in difficulty. Pediatrics. 2013;132(2):210–212.
42.		Regehr G, Eva K. Self-assessment, self-direction, and the self-regulating professional. Clin Orthop Relat Res. 2006;449:34–38.
43.		Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77(6):1121–1134.
44.		Torre DM, Treat R, Durning S, Elnicki DM. Comparing PDA- and paper-based evaluation of the clinical skills of third-year students. WMJ. 2011;110(1):9–13.
45.		Pulito AR, Donnelly MB, Plymale M, Mentzer RM Jr. What do faculty observe of medical students’ clinical performance? Teach Learn Med. 2006;18(2):99–104.
46.		Lacasse M, Théorêt J, Skalenda P, Lee S. Challenging learning situations in medical education: innovative and structured tools for assessment, educational diagnosis, and intervention. Part 2: objective examination, assessment, and plan. Can Fam Physician. 2012;58(7):802–803, e418–e820.
47.		Nendaz M, Charlin B, Leblanc V, Bordage G. Le raisonnement clinique: données issues de la recherche et implications pour l’enseignement. [Clinical reasoning: from research findings to applications for teaching]. Pedagog Med. 2005;6(4):235–254. French.
48.		Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387–396.
49.		Jouquan J. L’évaluation des apprentissages des étudiants en formation médicale initiale. [The assessment of medical students’ learning]. Pedagog Med. 2002;3(1):38–52. French.
50.		Hinson JP, Griffin A, Raven PW. How to support medical students in difficulty: tips for GP tutors. Educ Prim Care. 2011;22(1):32–35.
51.		Paul G, Hinman G, Dottl S, Passon J. Academic development: a survey of academic difficulties experienced by medical students and support services provided. Teach Learn Med. 2009;21(3):254–260.
52.		Yates J. “Concerns” about medical students’ adverse behaviour and attitude: an audit of practice at Nottingham, with mapping to GMC guidance. BMC Med Educ. 2014;14:196.
53.		Mitchell M, Srinivasan M, West DC, et al. Factors affecting resident performance: development of a theoretical model and a focused literature review. Acad Med. 2005;80(4):376–389.
54.		Hendren RL. Predicting success and failure of medical students at risk for dismissal. J Med Educ. 1988;63(8):596–602.
55.		Boulé R, Girard G, Bernier C. Dépistage précoce des résidents en difficulté. [Early detection of residents in trouble]. Can Fam Physician. 1995;41:2130–2135. French.
56.		Tonesk X, Buchanan RG. An AAMC pilot study by 10 medical schools of clinical evaluation of students. J Med Educ. 1987;62(9):707–718.
57.		Kahn NB. Dealing with the problem learner. Fam Med. 2001;33(9):655–657.
58.		Reamy BV, Harman JH. Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency. Fam Med. 2006;38(4):252–257.
59.		Kempainen RR, Migeon MB, Wolf FM. Understanding our mistakes: a primer on errors in clinical reasoning. Med Teach. 2003;25(2):177–181.
60.		Scott IA. Errors in clinical reasoning: causes and remedial strategies. BMJ. 2009;338:b1860.
61.		Luhanga FL, Larocque S, MacEwan L, Gwekwerere YN, Danyluk P. Exploring the issue of failure to fail in professional education programs: a multidisciplinary study. J Univ Teach Learn Pract. 2014;11(2):1–24.
62.		Winston KA, Van Der Vleuten CP, Scherpbier AJ. The role of the teacher in remediating at-risk medical students. Med Teach. 2012;34(11):e732–e742.
63.		Audetat MC, Laurin S, Dory V. Remediation for struggling learners: putting an end to ‘more of the same’. Med Educ. 2013;47(3):230–231.
64.		Cleland, Leggett H, Sandars J, Costa MJ, Patel R, Moffat M. The remediation challenge: theoretical and methodological insights from a systematic review. Med Educ. 2013;47(3):242–251.
65.		Saxena V, O’Sullivan PS, Teherani A, Irby DM, Hauer KE. Remediation techniques for student performance problems after a comprehensive clinical skills assessment. Acad Med. 2009;84(5):669–676.
66.		Hauer KE, Teherani A, Kerr KM, O’Sullivan PS, Irby DM. Student performance problems in medical school clinical skills assessments. Acad Med. 2007;82(10 Suppl):S69–S72.
67.		Regehr G, Norman GR. Issues in cognitive psychology: implications for professional education. Acad Med. 1996;71(9):988–1001.
68.		Nendaz M, Perrier A. Diagnostic errors and flaws in clinical reasoning: mechanisms and prevention in practice. Swiss Med Wkly. 2012;142:w13706.
69.		Tardif J. Pour un enseignement stratégique : l’apport de la psychologie cognitive. [Strategic education: contributions from cognitive psychology]. Montreal: Editions Logiques; 1992. French.
70.		Setna Z, Jha V, Boursicot KA, Roberts TE. Evaluating the utility of workplace-based assessment tools for speciality training. Best Pract Res Clin Obstet Gynaecol. 2010;24(6):767–782.
71.		Audétat MC, Laurin S, Sanche G, Béïque C, Caire Fon N, Blais JG, Charlin B. Clinical reasoning difficulties: a taxonomy for clinical teachers. Med teach. 2013;35(3):e984–e989.