What are Supervisor Perceptions of Clinical Placement Sign-off Forms for Medical Students?

Background

Progression through medical school curricula involves the completion of different clinical rotations, in line with situated learning theory, whereby learning is promoted through engagement in the working environment.¹ Clinical competence in these rotations is normally assessed by an end-of-placement supervisor evaluation,² which acts as an interim barometer of student progress prior to summative end-of-year examinations. The usual assessor is a consultant-level supervisor who is a highly experienced content expert, able to recognise unsafe levels of performance.³ On sign-off forms, overall competence is typically subdivided into constituent sub-domains such as professionalism, knowledge and clinical skill, which are then combined to arrive at an overall outcome.⁴

Despite widespread use, the supervisor ratings system has long been criticised for being outdated and suboptimal.⁵ Assessors are a major cause of measurement error,⁶ and multiple supervisors are required given the high numbers of students per year-group, which introduces significant measurement error from inter-rater variability. Assessors are seldom briefed on the student curriculum⁷ and are often not accustomed to the marking framework.⁸ Additionally, raters may concentrate on contrasting components of performance,⁹ may not appreciate expected student standards,¹⁰ and may even score the same performance differently on the same rating scale.¹¹ These issues can be somewhat mitigated through assessor training, which may reduce differences in marking stringency.¹¹ However, in practice the use of untrained assessors is common.¹²

Furthermore, the format of undergraduate clinical attachments in hospitals has markedly changed since the introduction of the supervisor sign-off system. Rather than the archaic method of one longitudinal placement,¹³ it is now routine to undergo a range of multiple different sub-speciality rotations. Consequently, there are shorter rotations and frequent changes of supervisor,¹⁴ with less supervisor-learner contact for individual supervisors in any one rotation.² This reduces assessment validity as supervisor ratings are based on a smaller number of interactions,¹⁵ leading some critics to argue that the advantages of rater-based clinical assessments are offset by psychometric drawbacks.¹⁶

Notably, in studies which investigated the correlation between clinical ratings from single assessors and end-point assessment scores, low or non-significant correlation coefficients have been found.^17,18 Additionally, ratings by resident doctors (non-consultants) demonstrate better correlation to other assessment scores than ratings by consultant-level assessors,^19,20 which calls into question the prevailing method of consultant-led end-of-placement sign-off assessments and suggests that changes should be made to the current consultant-led process.

In educational theory, any proposed changes to curricula or assessments should include input from key stakeholders.²¹ Current supervisors are well placed to elucidate existing problems within the sign-off process and their perceptions might inform possible solutions. Therefore, this study aims to ascertain clinical supervisor perceptions of the end-of-placement single supervisor sign-off process, to aid in determining what changes should be made. A systematic literature review was conducted (Figure 1), using the PubMed database and OVID Medline with the following search terms: “supervisor perceptions” OR “faculty perceptions” AND medical student OR undergraduate AND “sign-off” OR assessment OR evaluation OR rating.

Figure 1 shows the research search strategy flowchart, adapted from Page et al.22

Two single-centre quantitative studies were identified, indicating there is a dearth of literature on supervisor perceptions of the clinical evaluation of medical students. Both studies only used Likert scale questions to gather supervisor perception data. Study characteristics, evaluation methods and outcomes are summarised in Appendix 1.

In one study, supervisors perceived their assessments of students were fair and involved adequate observation time.²³ However, they also reported themselves to be fail-adverse, a finding which is widespread in assessment literature; assessors are less confident in making fail decisions than passing decisions,²⁴ and may award passing grades to avoid the consequences of expressing concerns over learner performance.²⁵

The second study compared supervisor perceptions of evaluating students over a longitudinal placement lasting a whole academic year, versus a shorter clinical rotation. It found supervisors considered longitudinal supervision better for assessment validity, quality of clinical skills evaluation, and the ability to provide feedback,²⁶ although no comparison with end-point assessments occurred.

Overall, the studies are limited in their exploration of supervisor perceptions. Neither collected qualitative data, and it is unclear what changes supervisors would advocate making to the sign-off process, which merits further exploration.

We therefore formulated the following research questions:

1. What are the perceived positives of the single supervisor sign-off process?
2. What are the perceived negatives of the single supervisor sign-off process?
3. What changes to the end-of-placement sign-off process could be considered based on supervisors’ opinions?

Methods

Institutional Context (Imperial College London, UK)

Supervisors of third-year medical students were chosen as this year has the longest clinical placements at our institution. In this year students rotate through three different clinical placements: “Medicine”, “Surgery”, and “General Practice (GP)”. Placement duration is eight weeks for GP, and either eight weeks or two four-week mini-placements in Medicine and Surgery. A named consultant supervisor provides an in-person sign-off rating for each student at the end of their attachment (on four-week attachments, students will have a different supervisor for each mini-placement and therefore will receive two sign-off ratings). Supervisors are not made aware of student performance on prior rotations.

The sign-off form (Appendix 2) uses a four-point ordinal scale with the categories “Below Expectations”, “Borderline”, “Meets Expectations” and “Above Expectations”. An overall grade is awarded, as well as a rating for each of the sub-domains (Clinical Skills; Practical Skills; Communication; Knowledge; Professionalism). Students must achieve either “Meets Expectations” or “Above Expectations” across all ratings to successfully pass the placement. If this is not achieved, completion of a remedial period is required before they are eligible to attempt the high-stakes end-of-year examinations.

Several techniques are used to safeguard assessment validity and reliability. All supervisors undergo assessor training on expected student standards and how to complete the assessment form, including frame-of-reference training. Additionally, students are pre-assigned assessors, removing the possibility of assessor selection bias.²⁵

Data Collection

An anonymous web-based mixed-methods questionnaire was used to collect data (Appendix 3). Supervisors (all consultants or fully qualified GPs) were recruited at training days and via email using pre-existing institutional mailing lists, with two reminder emails sent to boost response rates. All 161 Year 3 clinical placement supervisors in hospitals and GP practices affiliated with Imperial College School of Medicine were considered eligible.

Open-ended questions provide better depth of response but may lead to responder fatigue if too many are required,²⁷ therefore Likert scale questions were also included to complement findings. Seven-point scales were used to provide respondents with more scope to differentiate between values whilst maintaining validity.²⁷

It was explained to supervisors that participation was voluntary and no personal or identifiable data would be collected. They were provided with a Participant Information Sheet and informed that completion of the questionnaire was evidence of consent. Participants were made aware that once the questionnaire was completed it would not be possible for them to withdraw retrospectively as all data was recorded anonymously.

Ethical Approval

The study was conducted in line with the British Educational Research Association’s 2018 guidelines.²⁸ Ethical approval was obtained from Imperial College’s Education Ethics Review Process (reference EERP2223-060).

Data Analysis

Statistical Analysis

Mean and Standard Deviation (SD) were calculated for each Likert scale question. Ordered chi-squared testing was performed to determine statistical differences between responses by specialty (medicine-surgery; surgery-GP; GP-medicine), and by setting (hospital-GP) with surgery and medicine combined to form a hospital cohort.

Thematic Analysis of Qualitative Questionnaire Responses

NVivo12 software²⁹ was used to facilitate coding. Template codes were pre-identified (Appendix 4) based on the literature review. Qualitative analysis of free-text questionnaire responses started on receipt of the first questionnaire, with emergent coding of unforeseen data.

Repeat analysis of codes was performed to ensure consistency and promote refinement into overarching themes.³⁰ The questionnaire was closed when thematic saturation was reached; this was determined as the point at which no new additional codes were generated from three consecutive questionnaire responses.³¹ Saturation was reached within subgroups after analysing 32 questionnaires overall (11 medicine, 10 surgery, 11 GP).

Results

Responses were received from 59 of the 161 Year 3 clinical placement supervisors in hospitals and GP practices affiliated with Imperial College School of Medicine (24/61 medicine; 19/49 surgery; 16/51 GP). The overall response rate of 36% is considered adequate within the literature on online questionnaires.³²

Statistical Analysis

Table 1 shows summary statistics (mean and SD) for each Likert scale questionnaire item on a seven-point scale where 1 = strongly disagree and 7 = strongly agree, arranged by specialty. It also shows the results for Chi-squared testing of responses between specialties (medicine-surgery; surgery-GP; GP-medicine).

Table 1 Likert-Scale Responses to Questionnaire Statements by Specialty

The Holm multiple correction method³³ was used to correct for multiple statistical tests (12 tests: four Likert scale questions for each of the three specialty-specialty comparisons). The adjusted significant p-value was calculated at <0.008 for each sub-test.

Responses from supervisors in surgery and medicine were not statistically significantly different. However, a statistically significantly higher proportion of GP supervisors perceived they had adequate time to assess students’ knowledge and skills. Additionally, statistically significantly higher proportions of GP supervisors perceived their knowledge and skills ratings would correlate with end-of-year assessment scores than medicine supervisors. This same trend was evident between GP and surgery supervisors, but was not statistically significant.

Figure 2 illustrates the reported supervisor-student contact hours by specialty. GP supervisors had significantly increased contact time with learners, with 68.8% of GPs reporting over 20 contact hours with each student across their placement compared to 9.3% of hospital-based supervisors (GP-medicine: p<0.001; GP-surgery: p<0.001).

Figure 2 shows clinical supervisors’ self-reported contact time with students by specialty.

Additionally, on Chi-squared testing there was a statistically significant disparity (p=0.041) between supervisors’ desire to make changes to the single supervisor sign-off system. Those based in hospital settings were more in favour of making changes to the process (medicine 62% in favour; surgery 64% in favour) than GP-based supervisors (27% in favour).

Thematic Analysis of Qualitative Questionnaire Responses

The full list of codes and themes is shown in Appendix 5. The key themes identified during coding were divided into Positives and Negatives of the single supervisor sign-off model:

1. Positives of the single supervisor sign-off model:

1. Accountability
2. Continuity
3. Feasibility

2. Negatives of the single supervisor sign-off model:

1. Effect on learning
2. Inflexibility
3. Limited contact time
4. Psychometric concerns
5. Sign-off form shortcomings

Positives of the Single Supervisor Sign-off Model

Accountability

Having a single named supervisor was perceived to improve robustness of assessment with “accountability” and “oversight and scrutiny of student performance” compared to having multiple assessors. A named supervisor was seen to improve students’ “motivation”, “engagement” and attendance at the supervisor’s teaching sessions, whilst pre-assigning supervisors was considered beneficial to prevent students from only seeking sign-offs from individuals perceived to be lenient.

Continuity

The enhanced supervisor-student contact time and “consistency” of observation was felt to aid the identification of gaps in student knowledge, which could facilitate tailored teaching and individualised feedback. Likewise, supervisors articulated that the greater direct contact time with learners provided a more accurate view of competency.

Feasibility

Supervisors believed a single supervisor model made it more feasible to provide all supervisors with training, which in turn would safeguard ratings against assessor variance. Furthermore, a single assessor could complete the sign-off form efficiently without needing to review multiple mini-assessments from other parties. Finally, the single sign-off model was considered more achievable and less stressful for students than seeking out multiple assessors.

Negatives of the Single Supervisor Sign-off Model

Effect on Learning

Interestingly, supervisors thought that having a named supervisor had a negative impact on student learning behaviours with other staff members. Students were viewed as “sign-off focused”, and willing to forgo learning opportunities with doctors who were not their allocated supervisor. When the named supervisor was absent, it was perceived that students often displayed “little effort to attend or impress other members of the team”.

Additionally, respondents articulated that feedback content and style differed between supervisors, which could impact learners more in a single supervisor model as feedback is limited to one voice. Having multiple students simultaneously assigned to each supervisor was also seen as a potential disadvantage, with dominant students masking the competency of others.

Inflexibility

Several logistical difficulties with the single sign-off model were highlighted by supervisors, including scheduling issues and a lack of flexibility when supervisors were absent due to annual leave, sickness, or the need to provide emergency clinical cover. Timetabling clashes between university and hospital teaching agendas were also reported, with centralised university teaching often precluding student attendance on clinical placement.

The in-person sign-off requirement was criticised for reducing depth of feedback due to time-pressure. Multiple supervisors desired the ability to complete sign-off forms remotely to overcome this.

Limited Contact Time

Short rotations and mixed student attendance levels were perceived to lead to “brief” or “variable contact with individual students”, to the extent that “feedback can be subjective” or “prone to observation bias”. Supervisors acknowledged that variation in case presentations, informal teaching opportunities and student motivations were further drawbacks in making accurate rating judgements within a limited contact time.

Psychometric Concerns

Variation in rating stringency, differing perceptions of acceptable standards, and assessor bias were seen as threats to sign-off rating reliability. A lack of triangulation via multiple assessor ratings was felt to compound this. One supervisor commented that the single supervisor model was no longer fit for purpose due to “modern teamworking practices in hospitals where inpatient care is shared between a number of consultants…so no one person has oversight across the placement”.

Validity concerns were also widespread with the absence of formalised quantitative assessments of knowledge or skills during placements, and supervisors’ reluctance to fail students due to repercussions. Numerous hospital supervisors acknowledged that grades awarded were often based on student attendance, punctuality, and enthusiasm, rather than knowledge or skill.

Suggested changes included incorporating supervised learning events such as Case-Based Discussions into sign-off criteria, or introducing a formalised in-placement assessment with a standardised OSCE or written examination.

Sign-off Form Shortcomings

Numerous supervisors articulated that practical skills should be removed from the sign-off form, with a separate assessment performed by the clinical skills team/teaching fellows who were acknowledged to spend more time with learners. Feedback from others in supervisory roles was also recognised as important, and multiple respondents desired the creation of a multi-source feedback component to increase reliability of assessment.

Finally, there were mixed opinions regarding whether clinical placement performance could be expected to predict examination performance. Some viewed clinical ratings as a better indicator of future performance as a resident doctor than a predictor of end-of-year assessment scores.

Discussion

This study adds rich qualitative detail to the existing literature on supervisor perceptions. Variation in clinical assessment scores should be due to differences between those being assessed, but rater error is known to exert a considerable influence.² This psychometric quality was a source of concern for some supervisors, with large numbers of hospital-based supervisors referencing the possibility of subjectivity and bias in a single supervisor assessment model.

The sign-off model was perceived more favourably by GP supervisors, which tallies with advantages cited in the existing literature on longitudinal assessment;³⁴ community-based placements are more longitudinal in nature than hospital rotations and thus provide increased learner observation time¹⁴ and this improves inter-rater reliability³⁵ and accuracy of performance ratings.²⁶

In hospitals there are numerous barriers to making competency judgements, which exert less effect in GP settings; the more varied environment with frequent interruptions,³⁶ multiple individuals acting in supplemental supervisory roles when the allocated supervisor is not present,³⁷ and multiple grades of students in the environment with different learning goals.³⁶

Perceptions that student behaviour differed in the absence of their named assessor suggests ratings awarded by supervisors are under-sampling true performance. This is alluded to in the wider literature in a deception study where statistically different competency ratings were awarded to students between expected and unexpected raters, with higher scores awarded when students knew they were being assessed.¹⁹

Despite its perceived advantages, there was mixed support for the single supervisor sign-off method overall, with calls for more formalised, defensible, and validated mechanisms to aid in the sign-off process. The statistically significant split between opinions of hospital- and GP-based supervisor suggests the sign-off process need not be uniform between community and hospital-based placements.

Possible Changes to the Sign-off Process

Use of Multi-Source Feedback in Hospital Settings

The use of multiple assessors is beneficial for assessment reliability² and reduces the effect of leniency-bias/generosity error that is hypothesised to lead assessors to rate positively across clinical ratings.³⁸ Multi-source feedback may also help to alleviate some of the assessment focused student behaviours noted by supervisors, where students were perceived as only attending teaching from the clinician tasked with their sign-off, at the expense of other learning opportunities.

As part of promoting multi-source feedback, hospital-based supervisors suggested including the opinions of junior team members within the sign-off process. This could be advantageous as students spend more time with resident doctors than consultants,²⁰ and residents may more accurately evaluate student performance than consultant assessors.¹⁹ This surprising phenomenon may be rooted in Schmidt and Moust’s concept of social and cognitive congruence (1995). With closer proximity in age and position in the learning hierarchy, junior assessors are likely more aware of expected levels of student competence than consultants, equipping them to make better performance judgements. The effect could be accentuated by student behaviours; junior team members are more approachable from a student’s perspective,³⁹ and students may therefore be more willing to ask them questions which display a lack of knowledge.

Nevertheless, there are downsides to the use of multi-source feedback. Accountability and oversight of student performance could be lost with multiple raters, so having a primary named supervisor may still be required. A study in postgraduates indicates four to 11 raters are required for 80% inter-rater reliability,⁴⁰ but this reduces feasibility and necessitates a sizeable increase in institutional administration load. Furthermore, student workload/stress levels might increase when tasked with undertaking multiple sign-offs, and larger volumes of required assessments may act as a barrier to learning rather than an aid.⁴¹ To improve feasibility students could be asked to self-select raters, however assessor selection bias would reduce validity.³⁸

Increase Supervisor-Learner Contact Time on Hospital Placements

Increasing assessor-learner contact time would improve both reliability and validity of assessment, as the brief contact time with learners in hospitals was perceived to lead to the normalisation of “snapshot” competency ratings. This fits with the dominant discourse within the literature, with concerns regarding student supervision time being more pronounced in hospital settings.⁴² Longer contact time leads to more accurate competency ratings; in a cohort study where four-week supervisory periods were compared with two-week periods, correlation between clinical knowledge ratings awarded by supervisors and students’ end-point assessment scores was superior for longer supervision duration.⁴³

Opinions on how to safeguard increased supervisor-learner contact time were united; supervisors desired a contractual change so that supervision time was built into their clinical schedules. However, this may be difficult to achieve due to competing clinical pressures.

Using Different Assessors for the Practical Skills Sign-off

The assessor being present is one of the cornerstones of assessment validity,⁴⁴ yet questionnaire responses suggest that practical skills observation is rarely performed by supervisors in their day-to-day roles. Supervisors were mindful that this lack of observation affected assessment validity, and desired removal of the practical skills component from the sign-off form, suggesting a separate sign-off should be carried out by the clinical skills team.

Embedded in-Placement Assessments

Proposed embedded in-placement assessments options included either a standardised quantitative assessment of knowledge (eg Single Best Answer Questions, SBAQs), a practical examination (eg an Objective Structured Clinical Examination, OSCE), or the completion of workplace-based assessments (WPBAs). This would mean the use of multiple sampling methods, which would have the benefit of increasing reliability through triangulation.⁴⁵

Advantages of these assessments include the objectivity of SBAQs and a reduction in assessor variance with increased assessor numbers in OSCEs versus single supervisor assessments.⁴⁴ Post-hoc calculations of cut-scores would also safeguard against assessors’ reluctance to fail.⁴⁶ Furthermore, assessment stimulates learning behaviours,⁴⁷ and targeted feedback from assessments is valuable for learners,⁴⁸ so embedded in-placement assessment could help drive student development.

Unfortunately, additional assessments pose logistical challenges. A question bank would be required where written questions would need to be continually written, whilst recruitment and training of examiners plus the creation of stations and marking schemes are needed for practical examinations. However, embedded in-placement assessment are commonly used in undergraduate medicine in other countries,^17,19,49 indicating feasibility concerns can be overcome.

Remote Sign-Offs

Reflection on feedback is an important driver of deep learning,¹ however supervisors reported that students typically present in groups for the sign-off, and the need to complete forms for multiple learners in a timely manner restricted the value of feedback given. Additionally, the provision of detailed feedback for learners was sometimes limited by competing clinical demands. Therefore, some supervisors called for the ability to conduct remote sign-offs to improve feasibility of providing personalised feedback.

The drawback would be the lack of student involvement, with no possibility of dynamically addressing learner queries or gauging student self-awareness of performance. Students could also be overwhelmed by feedback which has not been previously mentioned in a face-to-face environment, and may be less willing to accept it as valid.⁴⁷ Thus, a remote sign-off would perhaps not be appropriate when wishing to give significant feedback to enact meaningful change. A hybrid model could present a possible sign-off method to explore; either with remote sign-offs plus the option of face-to-face feedback, or where remote feedback is permissible alone only if both supervisor and student are content not to meet for face-to-face feedback. This may reduce administrative burden whilst safeguarding the pedagogical benefits of face-to-face feedback if deemed preferable by either assessor or learner.

Rating Scale Changes

Assessment reliability is threatened where rating scale anchors are not meaningful to the assessor.⁵⁰ The institution’s anchors (“Above Expectations”; “Meets Expectations”; “Borderline”; “Fail”) were criticised by some supervisors as being “vague” and unhelpful, and supervisors reported that expected levels of performance could be easily confused or forgotten, especially where staff were also supervising students from different years of the curriculum.

Descriptor training is carried out at the institution, but the descriptor framework may need improvement. Some supervisors articulated that they would prefer a pass-fail grading scale for each domain, rather than the discriminatory Expectations scale with multiple passing categories. The use of pass-fail scales does not affect summative assessment outcomes but has been shown to improve student satisfaction, by engendering a more collaborative learning environment with reduced competitiveness between students.⁵¹ Alternatively, clinically anchored rating scales with entrustment terminology (rather than Expectations scales) have been shown to improve both inter-rater reliability,⁴⁴ and discrimination between levels of learner performance.⁵²

Limitations

There are a few limitations to this study. Generalisability is limited as only supervisor perceptions from one medical school were sampled, although this included several different affiliated hospitals and GP practices. Other institutions may have different sign-off requirements, placement durations, rating scales, and assessor training methods. Triangulation through collaboration with different institutions would be beneficial, with sharing of best practice.

A limitation in study design using questionnaire data is that there is a lack of adaptability in data collection,⁵³ although this is somewhat mitigated by also collecting qualitative data. We also note the potential for volunteer bias in questionnaire responses.

Contribution to Knowledge in This Area

We could not locate any studies in the existing literature which explore clinical supervisor perceptions of the undergraduate end-of-placement supervisor sign-off process from a qualitative perspective. This study adds rich data on stakeholder perceptions to explore why there might be limitations with the predictive validity of supervisors’ end-of-placement clinical ratings of medical students.

Conclusion

Hospital-based supervisors were less positive towards the sign-off process compared to GP supervisors, likely due to modern working practices of shared patient loads, shorter rotations, and reduced contact time with students.

This study can contribute to broad and meaningful change in long-term medical education policy beyond the local context, with rich qualitative data informing possible changes to the widespread but outdated clinical placement sign-off process for medical students. Differing sign-off processes could be considered in GP and hospital-based settings. In hospital settings, a multiple assessor model with embedded in-placement assessments could be considered. The assessment of practical skills should be removed from the end-of-placement clinical supervisor sign-off, and instead should be performed by clinical skills tutors, teaching fellows or resident doctors, who more frequently supervise students’ performance of practical skills.