Back to Journals » Advances in Medical Education and Practice » Volume 8

Interns reflect: the effect of formative assessment with feedback during pre-internship

Authors McKenzie S, Burgess A , Mellis C

Received 7 June 2016

Accepted for publication 29 September 2016

Published 11 January 2017 Volume 2017:8 Pages 51—56

DOI https://doi.org/10.2147/AMEP.S114480

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 3

Editor who approved publication: Dr Md Anwarul Azim Majumder



Susan McKenzie,1 Annette Burgess,2 Craig Mellis1

1Central Clinical School, 2Education Office, Sydney Medical School, The University of Sydney, Sydney, NSW, Australia

Background: It is widely known that the opportunity for medical students to be observed and to receive feedback on their procedural skills performance is variable in the senior years. To address this problem, we provided our Pre-Intern (PrInt) students with “one-to-one” formative feedback on their ability to perform urethral catheterization (U/C) and hypothesized that their future practice of U/C as interns would benefit. This study sought to evaluate the performance and practice of interns in U/C 4–5 months after having received feedback on their performance of U/C as PrInt students.
Methods: Between 2013 and 2014, two cohorts of interns, (total n=66) who had received recent formative feedback on their U/C performance as PrInt students at Central Clinical School, were invited to complete an anonymous survey. The survey contained nine closed unvalidated questions and one open-ended question, designed to allow interns to report on their current practice of U/C.
Results: Forty-one out of 66 interns (62%) completed the survey. Thirty-five out of 41 respondents (85%) reported that the assessment with feedback during their PrInt term was beneficial to their practice. Thirty of 41 (73%) reported being confident to perform U/C independently. Eleven out of 41 respondents (27%) reported that they had received additional training at intern orientation. Nine of the 11 interns (82%) reported that they had a small, but a significant, increase in confidence to perform U/C when compared with the 30 of the 41 respondents (73%) who had not (p=0.03).
Conclusion: Our results substantiate our hypothesis that further education by assessment with feedback in U/C during PrInt was of benefit to interns’ performance. Additional educational reinforcement in U/C during intern orientation further improved intern confidence. Our results indicate that extra pre- and post-graduation procedural skills training, with feedback, should be universal.

Keywords: formative assessment with feedback, performance and practice of new interns, medical interns, urinary catheterization, pre-intern students

A Letter to the Editor has been received and published for this article.

Introduction

The opportunity for senior medical students to perform core procedural skills, such as urethral catheterization (U/C), and to be observed1 and receive feedback varies considerably.2 Effective feedback is critical in procedural skills training; it allows the students to have insight into their clinical performance and it is the main component of formative assessment.3 While many authors have provided definitions for feedback, we chose Ende’s (1983) description of feedback as, “information describing students’ or house officers’ performance in a given activity that is intended to guide their future performance”.4 Receiving accurate feedback can assist in narrowing the gap between the students’ actual performance and the performance which is desired.5 In order for feedback to occur, the clinical task requires direct observation and assessment according to the specific performance criteria.6,7

Senior medical students perform procedures in the hospital wards during clinical placements, and at the completion of their clinical attachment, they were generally given a “global assessment”.810 However, safe performance is complex, requiring the application of many skills, including psychomotor skills, clinical judgment, communication skills, decision making, and patient-focused interactions.11 Clearly, time constraints often prevent students from being directly observed7 or adequately supervised.6 Therefore, formal specific “one-to-one” feedback that enhances the learning process is often overlooked.12

We used a faculty-validated checklist for the formative assessment. This had previously been used by the students in this study for self-directed learning following urinary catheter training in their second year of medicine; it was also used as a reference point in their senior years. An emphasis was placed on Ericson’s model of “deliberate practice”.13 This involved the processing of information and the acquisition of skills with the type of feedback that reinforces positive aspects performance, which allows the learners’ insight into what they did well and what they could improve upon.

The socio-constructivist model of feedback is relative to our study as it contributes to the learning of the students by allowing conversation between the student and the assessor.14 To create good feedback, Vickery and Lake15 recommended that the learners need to be directly observed, the learners are provided with clear goals and outcomes, and that the assessors must be able to provide positive and negative feedbacks.15 Pendleton’s positive critique16 method of delivering feedback was used as a framework for the assessment component of this study to allow students to interact with the assessor and to construct meaning from their feedback information. This type of feedback allowed students to be immediately informed of their progress and allowed the students to reflect upon the aspects of their performance that needed to be improved.

It is generally assumed that interns are able to perform basic core skills, such as U/C under supervision when first entering the workplace.17 However, our previous study found that immediately prior to the commencement of internship, almost one-third (29%) of Pre-Intern (PrInt) students had never performed male U/C.18 Similarly, a recent UK study found that 20 out of 50 doctors (40%) had not performed male catheterization as medical students.19 Thomas et al reported that 6% (51/864) of urological consultations from July 2006 to June 2007 were related to U/C morbidities, and 74% (38/51) of these were intern related, with most (73%; 28/38) occurring during the first 6 months of internship.20 A follow-up study by Sullivan et al that involved intern education at the start of the intern year found that male U/C-related morbidities that were attributed to new interns during their first 6 months fell by 27%, with 63% reporting confidence in performing male U/C compared with 35% prior to the program.21 Another study involving medical student education in U/C found that knowledge could be retained from skills laboratory training (with feedback) to real-life clinical situations for up to 6 weeks.22 While both studies reported the positive effect of education, neither had reported on the effects of assessment with feedback on early intern performance in U/C.

Our study sought to evaluate participants’ practice and performance of U/C as new interns following formative assessment in U/C performance, using Pendleton’s method of positive feedback16 immediately prior to graduation.

To the best of our knowledge, there are no recent data that report on the effects of “one-to-one” feedback in U/C in PrInt and its subsequent impact on the performance capabilities in early internship.

Context

The participants in this study were new interns who had completed a 4-year graduate entry medical degree at Sydney Medical School. All were students at Central Clinical School (CCS), which is attached to Royal Prince Alfred Hospital (RPA). While some remained as interns at their original student hospital, RPA, most were dispersed to regional, interstate, and international hospitals. Procedural skills training in U/C had been provided to the CCS medical students in years 2 and 3, and in PrInt, which is the final stage of medical student training immediately prior to internship. The duration of PrInt term was 8 weeks during 2012–2013.

Methods

PrInt assessment with feedback

Two cohorts of final-year PrInt students (n=66) at CCS were assessed and given individual feedback on their performance of male U/C as part of their preparation for internship between 2012 and 2013.

Study instruments

Our study instruments for providing assessment with feedback were a part-task training model to assess students’ psychomotor skills and knowledge in U/C and faculty-validated content referenced performance criteria checklist.

Part-task trainer

We used a part-task trainer as neither was it appropriate to use a real patient, nor was it financially viable to use an integrated procedural performance instrument (IPPI).11 We felt that we could achieve a level of authenticity with the use of assessment protocols.23 We used a faculty content-based validated checklist18 to facilitate direct observation. The checklist was developed by clinicians at Sydney Medical School. Its purpose is to act as an educational guideline for students’ self-directed learning. The performance criteria ranged from 1 to 26 (which included all criteria that are involved in catheterization, such as obtaining consent for the use of aseptic technique).18 The assessment criteria were a binary “yes” or “no” score.18 Assessors were senior clinicians who had been trained in the use of the checklist and the theory of Pendleton’s method16 of providing feedback (Table 1). The assessment took place over 15 minutes per student, followed by 5 minutes of one-to-one feedback; which was modelled on Pendelton’s method.16

Table 1 Feedback model


Note: Use of the of feedback model was adapted from Pendleton’s positive critique method.16

Follow-up

The follow-up study took place between 2013 and 2014, when (n=66) interns who had received assessment with feedback as PrInt students at RPA were invited to complete an anonymous electronic survey questionnaire. This consisted of nine closed unvalidated questions and one open-ended question, which were compiled in consultation with senior clinicians. The questions included a number of factors that may have influenced interns’ current U/C practice including demographics, such as gender location, additional education, interns’ confidence with independent performance, frequency of performance, and the benefits associated with U/C feedback during PrInt.

Statistical analysis

All questionnaire results were recorded as categorical variables and presented as percentages. Chi-square tests were used to determine any differences between those who attended their parent hospital RPA, as well as U/C performance by gender. We considered p<0.05 to be statistically significant. Because of small numbers, and to perform chi-square testing, we combined responses namely: for “Intern success in U/C performance”, “Sometimes” and “Never” were combined. For “Intern confidence by hospital” and “Intern confidence to perform by additional training”, “Not very” and “Somewhat” were combined. Responses were missing for the following questions: one from question 1, three from question 2, one from question 4, and one from question 7. All statistical analyses were performed in Statistical Analysis System (SAS) Version 9.3.

Qualitative data analysis

The final question was open ended to elicit further comments from interns on their experience and performance with U/C. Respondents’ free text responses were collated and thematically analyzed by the first author.

Ethics approval

Ethics approval was obtained from The University of Sydney Human Research Ethics Committee (protocol number 2012/1764). A statement was included in the survey notifying the participants of the study intentions, and return of the survey was taken as consent to participate.

Results

Over a period of 2 years, a total of 41 out of 66 (62%) interns who had participated in assessment with feedback as PrInt students completed the survey 4–5 months into their internship.

Of the 41 respondents, 21 (51.2%) were males, 17 (41.4%) were females, and 3 (7.4%) did not report their gender. Fourteen (36%) respondents reported that they were allocated to their “parent hospital”, RPA, and 25 of 39 (64%) respondents were allocated to other hospitals (two frequencies were missing). Most (35/41, 86%) respondents reported that “assessment with feedback” in PrInt was beneficial to their U/C practice as interns. Virtually all (39/41, 95%) respondents reported that they were “somewhat or very confident” to perform U/C on a patient and 30 of 40 (75%) respondents stated that they had successfully performed U/C insertion.

Performance by hospital

Interns who were allocated to their original teaching hospital (RPA) reported a statistically higher frequency of independent performance of U/C, 13 of 14 (93%), when compared with 15 of 24 (63%) of their cohort peer interns who were students at RPA, but allocated as interns to other hospitals (p=0.04).

There was a non-significant trend for higher confidence in performing U/C among interns who were allocated to their original teaching hospital, RPA. Eight of 14 (57%) interns reported a higher level of confidence to perform U/C, when compared with 10 of 25 (40%) of interns who were attached to other hospitals (p=0.30). Interns at RPA also reported a higher, but not significant, success rate (12/14, 86%) when compared with their cohort peers who were allocated to other hospitals (17/24, 71%; p=0.30).

Performance by gender

Female interns reported a small, but not significantly higher, frequency of independent performance (13/16, 81%) when compared with male interns (15/21; 71%; p=0.49). However, 11 of 21 (52%) male interns reported to be more confident in performing U/C when compared with 7 of 17 (41%) female interns (p=0.79).

Male interns also reported a far higher frequency of performing U/C on male patients, with 7 of 21 (33%) having inserted more than 5 male U/C catheters, while 1 of 17 female (6%) interns performed U/C on a male patient.

Confidence by additional training in U/C

Eleven of 41 (27%) respondents reported that they had received additional training at intern orientation. Nine of these 11 interns (82%) reported that they were significantly more confident to perform U/C, when compared with the 30 of 41 (73%) interns who had minimal or no additional training (p=0.03).

Qualitative data

Analysis of open-ended responses resulted in three themes, which interns felt were helpful in their performance, namely, tips from experienced clinicians, use of checklists, and further education at intern orientation.

Tips from experienced clinicians

Empirical learning theory asserts that individual learning depends on building, drawing upon one’s existing knowledge.24 Interns reported that “tips from experienced practicing clinicians” added knowledge, as they speak from experience and offer insight on how to deal with challenges that were ahead of them: “tips from experienced clinicians were of benefit, as they offered insight into challenges that we may soon encounter.” [Intern: 16]

The use of a checklist

Interns found the checklist to be insightful into the procedural protocol: “The Supervised (assessment) session with checklist was useful, as it gave me insight as to what I should have been doing”. [Intern: 32]

A checklist allows for an objective step-by-step assessment of microcompetencies9 to which a student must recall and implement. This is useful as it can be used in both teaching and assessment.25

Further education at intern orientation

Interns found that having further education and training in U/C during intern orientation provided additional confidence and complemented the educational benefits of assessment with feedback during PrInt. They also found education as motivation to perform U/C on patients. “I had one additional session during my orientation week, which complements what I did last year” (i.e., it was on the pelvic models) “it inspired me… and gave me confidence to perform, I have since inserted two Indwelling catheters both in men”. [Intern:4]

Discussion

We sought to investigate the impact of assessment with feedback during PrInt on interns’ practice and performance of U/C in early internship. While many authors have explored the effect of assessment with feedback, we were also able to draw further perspectives from our results and were given some insight into U/C performance patterns of new interns. The key findings included demographic information, intern performance frequencies, and gender and performance of male U/C.

Intern performance patterns

Those who did their internship in their original student teaching hospital reported a higher frequency of independent performance of U/C, when compared with their peers from the same teaching hospital, but who were allocated to other hospitals as interns. Familiarity with their original teaching hospital procedures and staff may partially explain this. Alternatively, it may be that RPA students had more teaching and training on this procedure during orientation than other hospitals. Interns who were allocated to their original student hospital also reported significantly (p=0.03) higher confidence and success rate in U/C performance, when compared with interns who were allocated to other hospitals.

Gender specificity

Unsurprisingly, we found U/C performance patterns were “gender specific”,26 with male interns reporting a far greater frequency of performing male catheterization, compared with females. These results validate a previous report of gender specificity, in that male interns receive more experience in male-specific skills.26 This observation of gender specificity may also be explained by the policy of some hospitals that male U/C is done by interns, rather than nursing staff.

Education

Further training on U/C during internship orientation was inconsistent across different hospital sites. However, we found that additional training during orientation did improve interns’ confidence to perform U/C. Our qualitative findings indicated that interns who were provided with further education at orientation in the intern year reported that it provided both additional confidence and motivation to perform U/C. This supports a recent study by Sullivan et al, who reported that a new intern training program in U/C at the start of the intern year resulted in an increase in confidence in performing U/C, from 35% before training to 63% after the training program (p<0.05).27 Although mandatory intern orientation and education are now part of an Australian national training program,28 the content and consistency of U/C training in individual hospitals are unknown.

The use of checklists

Interns reported favorably on the use of a checklist, in that it gave insight into what they “should” have been doing. Checklists have been developed in some programs as an attempt for a standardized curriculum to teach and evaluate procedures.29 When checklists are combined with prompt feedback, students are made aware of what they “know and don’t know”, which focuses on learning.30 However, it must be noted that checklists also have drawbacks, in that the steps to perform a procedure are not differentiated between the important and the less important steps, and sometimes not all steps are essential for the successful completion of a procedure.31

Tips from experienced clinicians

Experienced clinicians who are experts in their field can provide valuable tips to students. The value of their expertise can be transmitted to the patient by the work of others in the form of training other clinicians.32 As stated by Irby, “experienced clinicians can also act as role models33 for students by helping them to establish their identity”, as students learn by imitations and observation of the clinicians that they respect.34

The significance of this study

The results from this study support the assessment with feedback in the theoretic framework of Pendleton’s method,16 by allowing PrInt students to reflect on their performance and allowing them to be aware of their performance strengths and limitations. If this was implemented in all clinical schools, the threat to the safety of the patients would be diminished.

Study limitations

The response rate to our survey was only 62%, despite one initial and two reminder e-mails; therefore, selection bias cannot be ruled out. We performed multiple subgroup analyses (gender, hospital, and additional training); despite the small sample size, this along with the absence of a control group must be taken into account when considering our conclusions.

Although all the initial participants attended the same clinical school as medical students (RPA), the main focus of this research relied on their experience as interns and the majority were placed at a variety of city, rural, and international hospitals. They reported an inconsistency of additional training during early internship, which led to our main outcome being confounded. Nevertheless, we consider this finding to be of significance as it indicated the need for assessment with feedback in pre-internship.

We utilized a part-task training model to assess students’ psychomotor skills and knowledge in U/C as it was not appropriate to use real patients, nor was it financially viable to employ actors to utilize an IPPI,11 which would have been useful for nontechnical skills. We also acknowledge that assessment and feedback on a part-task trainer cannot replace the complexities of a real-life situation in a busy hospital ward. However, our aim was to seek standardization and technical proficiency in U/C, thus enabling direct observation, assessment, and feedback prior to our students entering internship.

Conclusion

Our results highlight the value of direct observation, assessment, and feedback on performance of U/C for PrInt students. This additional training ensures that new interns enter the workplace with an updated assessment of their ability to perform this common procedure. Despite the benefit of additional training during intern orientation, this was inconsistent with many interns receiving little or no further education at orientation.

Definitions

Intern training program

A period of 47 weeks of mandatory, supervised, work-based clinical training that includes medicine, surgery, and emergency medical care terms to meet regulatory requirements. The program also includes orientation, formal and informal education sessions, and assessment with feedback, and it may be provided by one or more intern training providers. Also called postgraduate year 1 (PGY1) (Australian Medical Council).28

Global assessment

Assessment that requires the assessor to make an overall judgment on a procedure rather than a step by the assessment of microcompetencies.9

Acknowledgment

We would like to thank the interns who contributed to this study.

Disclosure

The authors report no conflicts of interest in this work.

References

1.

Boud D. Feedback: ensuring that it leads to enhanced learning. Clin Teach. 2015;12(1):3–7.

2.

Liddell MJ, Davidson SK, Taub H, Whitecross LE. Evaluation of procedural skills training in an undergraduate curriculum. Med Educ. 2002;36(11):1035–1041.

3.

Sadler DR. Formative assessment and the design of instructional systems. Instr Sci. 1989;18(2):119–144.

4.

Ende J. Feedback in clinical medical education. JAMA. 1983;250(6):777–781.

5.

Taras M. Assessment – summative and formative – some theoretical reflections. Br J Educ Stud. 2005;53(4):466–478.

6.

Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: a literature review. Med Educ. 2000;34(10):827–840.

7.

Irby DM. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med. 1995;70(10):898–931.

8.

Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387–396.

9.

McKinley RK, Strand J, Ward L, Gray T, Alun-Jones T, Miller H. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. Med Educ. 2008;42(4):338–349.

10.

McIlroy JH, Hodges B, McNaughton N, Regehr G. The effect of candidates’ perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination. Acad Med. 2002;77(7):725–728.

11.

Kneebone RL, Nestel D, Vincent C, Darzi A. Complexity, risk and simulation in learning procedural skills. Med Educ. 2007;41(8):808–814.

12.

Gordon J. One to one teaching and feedback. BMJ. 2003;326(7388):543–545.

13.

Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79 (Suppl 10):S70–S81.

14.

Nicol D. From monologue to dialogue: improving written feedback processes in mass higher education. Assess Eval High Educ. 2010;35(5):501–517.

15.

Vickery AW, Lake FR. Teaching on the run tips 10: giving feedback. Med J Aust. 2005;183(5):267–268.

16.

Pendleton D, Schofield T, Tate P, Havelock P. The Consultation: An Approach to Learning and Teaching. Oxford, UK: Oxford University Press; 1984.

17.

Gleason AJ, Daly JO, Blackham RE. Prevocational medical training and the Australian curriculum framework for junior doctors: a junior doctor perspective. Med J Aust. 2007;186(3):114–116.

18.

McKenzie S, Burgess A, Chapman R, Mellis C. Pre-interns: ready to perform? Clin Teach. 2015;12(2):109–114.

19.

Forsythe RO, Eylert MF. Medical students and foundation doctors need more exposure to basic urology. Bull R Coll Surg Engl. 2014;96(7):240–243.

20.

Thomas A, Giri S, Meagher D, Creagh T. Avoidable iatrogenic complications of urethral catheterization and inadequate intern training in a tertiary-care teaching hospital. BJU Int. 2009;104(8):1109–1112.

21.

Sullivan JF, Forde JC, Thomas AZ, Creagh TA. Avoidable iatrogenic complications of male urethral catheterisation and inadequate intern training: a 4-year follow-up post implementation of an intern training programme. Surgeon. 2015;13(1):15–18.

22.

Todsen T, Henriksen MV, Kromann CB, Konge L, Eldrup J, Ringsted C. Short- and long-term transfer of urethral catheterization skills from simulation training to performance on patients. BMC Med Educ. 2013;13(1):1–6.

23.

Eva KW, Bordage G, Campbell C, Ginbraith R, Holmboe E, Regehr G. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ. 2016;21(4):897–913.

24.

Dyrbye LN, Harris I, Rohren CH. Early clinical experiences from students’ perspectives: a qualitative study of narratives. Acad Med. 2007;82(10):979–988.

25.

Lammers RL, Davenport M, Korley F, et al. Teaching and assessing procedural skills using simulation: metrics and methodology. Acad Emerg Med. 2008;15(11):1079–1087.

26.

Levy BT, Merchant ML. Differences in clinical experiences based on gender of third-year medical students in a required family medicine preceptorship. Acad Med. 2002;77(12 Pt 1):1241–1246.

27.

Sullivan JF, Forde JC, Thomas AZ, Creagh TA. Avoidable iatrogenic complications of male urethral catheterisation and inadequate intern training: a 4-year follow-up post implementation of an intern training programme. Surg. 2015;13(1):15–18.

28.

Australian Medical Council Limited. Intern training –National standards for programs. Available from: http://www.amc.org.au/files/37fdfb7b6fd3a1acdcc7800c23b42217737c2aa5_original.pdf. Accessed November 24, 2016.

29.

Sullivan ME, Ortega A, Wasserberg N, Kaufman H, Nyquist J, Clark R. Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional teaching methods? Am J Surg. 2008;195(1):20–23.

30.

Chickering AW, Gamson ZF. Seven principles for good practice in undergraduate education. AAHE Bull. 1987;3:7.

31.

Lammers RL, Davenport M, Korley F, et al. Teaching and assessing procedural skills using simulation: metrics and methodology. Acad Emerg Med. 2008;15(11):1079–1087.

32.

Prideaux D, Alexander H, Bower A, et al. Clinical teaching: maintaining an educational role for doctors in the new health care environment. Med Educ. 2000;34(10):820–826.

33.

Irby DM. Clinical teaching and the clinical teacher. J Med Educ. 1986;61(9 Pt 2):35–45.

34.

Harden RM, Crosby J. AMEE guide no 20: the good teacher is more than a lecturer--the twelve roles of the teacher. Med Teach. 2000;22(4):334.

Creative Commons License © 2017 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.