A mixed methods study of student perceptions of using standardized patients for learning and evaluation

Background

Health care educators use a variety of pedagogical methods to help students develop their interpersonal, clinical assessment and treatment intervention skills.¹ Educators have emphasized teaching and evaluating skill performance, in addition to cognitive knowledge, as written tests and multiple-choice questions can demonstrate low external validity in relation to clinical aptitude.^2,3 Clinical skills can be taught and evaluated in a variety of ways including clinical education and use of a standardized patient (SP), also referred to as a simulated patient. An SP refers to “a healthy person who has been carefully trained to realistically portray a patient in a reproducible fashion”.⁴ These actors are provided with comprehensive training and reproduce appropriate physical symptoms, relevant history, and emotional responses consistent with a structured clinical scenario.⁵ SPs were first used in medical education by Howard S Barrows⁶ in 1965 and have gradually been introduced into the field of nursing⁷ and allied health professions such as social work, speech and language pathology, occupational therapy (OT), physical therapy (PT), and dietetics.^4,5,8–10

SPs have proven valuable in both the teaching and evaluation of students. Barrows⁶ outlined several advantages of using the SP method, including the ease of access at any time and in any location. The SP method offers students a transition from classroom/didactic learning to patient-based clinical learning without unnecessary concern or risk related to their clinical abilities.⁶ While evaluation in a clinical setting provides a valid assessment of student ability, SP use has the advantage of demonstrating robust reliability in addition to high external validity.^2,11 As healthy individuals, SPs may lack some degree of authenticity in clinical presentation, but are able to “present an illness or scenario in a standardized, unvarying manner”.⁷

Use of SPs across health professions

Historically, SPs have been primarily employed in the education of medical students, and the majority of the literature examining SP use for education and evaluation comes from the perspective of medical students and their educators. Despite a growing interest in SP use among the allied health professions,¹² a gap in this literature is apparent.^4,5 SPs play a large role in the curriculum for PT and OT students at the University of Manitoba. At the time of this study, the OT program was a 24-month master’s degree, and the PT program a 3-year bachelor’s degree. While there were no shared courses between the programs, there was considerable collaboration between faculty around teaching and evaluation strategies. SPs were used for teaching interviewing/interpersonal skills, clinical assessment skills, and provision of interventions, and to examine student knowledge and skill through objective structured clinical examinations (OSCE) in PT and practical examinations in OT. With increasing application of SP use in our professional programs, the authors felt it important to explore the effectiveness of this approach to ensure students receive a high quality, evidence-based education. SP-based programs are expensive to implement and maintain, and require considerable investment in training;^13,14 evaluating effectiveness is critical to justify the expense, and the knowledge gained could enhance and further improve existing programs.

Rationale

Several factors contributed to the need for this study. Despite a growing trend in SP use among OT and PT professional education programs, there is little discipline-specific evidence reported in the literature. There is a great deal to be learned from differences between professional programs as well as discerning where and when SPs are most optimally used. Furthermore, much of the literature tends to focus on either the learning process or the evaluation of outcomes; a broader understanding of the SP experience in these areas collectively could provide considerable insights into sequencing and targeting SP use. In fact, to our knowledge, there is no literature exploring the programmatic use of SPs in professional education, nor how students globally perceive the use of SPs as a teaching and evaluation tool. Several discipline-specific issues are also of particular interest. First, over the past decade, nearly all Canadian OT and PT university programs have transitioned from an undergraduate to a graduate degree, and it is important to evaluate the comprehensiveness of these new programs. Second, as professions that advocate an adult learning approach, including self-directed and reflective learning strategies, it is appropriate that educators also reflect on teaching strategies and involve students in the evaluation of educational methodologies. Third, as health care disciplines that espouse a patient-centered approach to service delivery, educators should promote a student-centered perspective to learning. To fully capture student perspectives, we need to know how they experience aspects of learning and evaluation, including the use of SPs.

Purpose

The purpose of this study was to explore the following questions: what are past and current OT and PT students’ perceptions of SP use, and how does it influence their professional education?

Methods

A sequential-explanatory mixed-methods design¹⁵ was used to first capture a comprehensive perspective of students across programs through a quantitative survey (phase 1), then obtain an in-depth perspective of representative students through focus groups configured on the basis of initial quantitative analysis (phase 2), and finally interpret these findings through integration of the quantitative and qualitative data.

Phase 1: quantitative

An electronic (online) survey design was used to capture anonymous responses from current students and graduates between 2006 and 2009 from the University of Manitoba OT and PT programs. Of the total population within these parameters (N=524), we had contact information for, and sent personal invitations to, 432 potential participants. Current students (N=200) were invited via their university email account and public advertisement on campus. Students received an initial email as well as two follow-up emails, as multiple contacts have been demonstrated to improve response rates.¹⁶ Each email identified the purpose of the study and a link to the online survey. Previous graduates were approached through an alumni email list (N=242) as well as public posters and advertisements in provincial professional organization newsletters.

Survey

The study investigators developed the survey over several months, developing content based on a review of the literature and the pedagogical questions of the investigators. After pilot testing and revision of the survey, a total of 24 questions were configured in four content sections (see Supplementary material). Section one (“Standardized Patients/SP”) dealt with global issues of SP performance, student–SP interaction, and relevance to their professional education. Section two (“Teaching/Education”) related to SP effectiveness as a strategy for learning different types of skills. Section three (“Feedback”) explored the relevance, timing, and perceived benefit of feedback from SP interactions. Section four (“Examination”) covered perception of SP use as a method of evaluation. The questions were presented as statements, using a four-point Likert scale with response options of “strongly disagree” (1), “somewhat disagree” (2), “somewhat agree” (3), and “strongly agree” (4). Questions varied with respect to positive and negative wording (eg, “SPs remain in their role throughout the duration of the interaction” versus “SPs often try to trick students”). Three questions had a “not applicable” option; one related to the use of videotaping, which PT did not incorporate into their program, and two asked about multiple encounters with the same SP, which some students would not have experienced. Demographic data was also collected related to age, sex, program (OT/PT), graduation year, and level of education prior to entering the program. Respondents also indicated whether they would be willing to be contacted to participate in the subsequent qualitative phase.

Data analysis

Demographic data were summarized using mean and standard deviation (for continuous variables) and frequencies and proportions (for nominal variables). Individual survey question responses were weighted from 1 to 4; negatively worded questions were adjusted for reverse weighting. Since each of the four content sections was composed of 5–7 items, composite scores could be calculated for each of the four content sections (total weighted score/[number of questions – not applicable responses]).¹⁷ The primary analysis involved descriptive statistics related to overall student ratings. Mean scores and standard deviation were calculated for each content section.^17,18 Secondary analysis explored potential differences between programs, sex, and graduation year, and any potential association between age and respondent rating. Sex was a potential variable of interest as rehabilitation programs in Canada have historically demonstrated an imbalance related to enrollment and employment, with OT being roughly 92% female¹⁹ and PT 78%.²⁰ Graduation year was considered because we postulated that after graduation and a period of clinical practice, students might change their perspectives of SPs, compared with being in the midst of their education. We collapsed the data in 2-year increments by graduation year (current students =2010 and 2011; recent graduates =2008 and 2009; past graduates =2006 and 2007). To create consistency, and because PT students have limited exposure to SPs in their first year, only PT students in their second and third year of study were enrolled.

Student t-tests were used to compare mean scores between groups, using the independent variables of study program (OT/PT) and sex. Analysis of variance was used to compare scores based on graduation year. Regression using the Pearson statistic was used to identify any relationship between age and respondent ratings.¹⁸

Phase 2: qualitative

A series of focus groups were conducted to investigate the student perspectives in greater detail. Results of the quantitative data analysis informed the selection of participants. Correlation between survey scores and participant age was apparent, as well as significant differences in scores between study program (OT versus PT) and participant sex. Males account for only 10% of OT and 22% of PT students, precluding sex as a primary factor for group differentiation. Therefore, four focus groups were configured to contrast both program and age: older OT, young OT, older PT, and young PT. This stratification enabled us to consider responses for each program and age category across two focus groups. Only respondents who, on the quantitative survey, had provided consent to follow-up in the qualitative phase (N=94) were eligible. To optimize contrast, these respondents were ranked by age and, beginning with the oldest, email invitations were extended until a group of five was achieved. A comparable process was conducted, beginning with the youngest respondent in each program. A research assistant (RA) administered all invitations to ensure confidentiality of participants from the authors, who were faculty members. The focus groups were conducted at the university, but with the authors off site. Each group was co-facilitated by two RAs; one a current student and the second a recent OT program graduate, and both receiving training from the study authors. An interview guide was developed based on study-predicated issues and results of the quantitative phase. Each focus group interview was audio-recorded and transcribed verbatim. One RA transcribed all interviews, while the second RA audited correspondence between audio recordings and transcriptions.

The initial data analysis focused on transcripts from the young OT and PT focus groups. All authors reviewed these transcripts several times and independently open coded. Next, one author (EMG) transferred these open codes from each reviewer into a matrix. All authors met to examine and discuss similarities and differences until consensus was reached and redundancies removed. A second author (PFW) collapsed initial codes to create a preliminary framework with 12 categories. All authors again met to discuss the coding framework and, after reaching consensus, independently recoded the transcripts based on the framework, remaining open to new/emerging concepts. The authors reconvened and reviewed the transcripts, achieving consensus on categories. Finally, all authors used the coding framework to analyze the remaining two transcripts, with three additional categories emerging.

Results

Phase 1

Participants

Email invitations were sent to a total of 432 individuals from the total complement of current students and recent graduates (N=524). We had no way of knowing how many of these invitations were viewed or how many of the email accounts were dormant. A total of 179 individuals initiated the online survey; ten respondents did not answer any questions, and two respondents answered all the questions but did not provide demographic data, precluding their use in the analyses (see Figure 1).

Figure 1 Participant enrollment.

The 167 usable datasets represent a response rate of 32% of all potential participants and 39% of targeted individuals. Response rates were higher among OT (43%) than PT (21%) students (see Tables 1 and 2). The mean difference in age between programs was 1.24 years (95% confidence interval =[−0.09, 2.57]) and was not significant. The proportion of female respondents in each program was not significantly different (χ²=1.50, df=1, P=0.221) nor was the distribution of respondents by graduation year (χ²=0.525, df=2, P=0.769). The female response rate was slightly higher in OT than PT, although not significantly, and both were comparable to Canadian employment ratios for each profession.

Table 1 Participant demographics reporting mean ± SD or frequency (percentage of responses) Abbreviations: OT, occupational therapy; PT, physical therapy; SD, standard deviation.

Table 2 Participant response rates by number (percentage of potential respondents) Abbreviations: OT, occupational therapy; PT, physical therapy.

Primary analysis: overall perceptions of SP use

Participant ratings for each of the four content areas are summarized in the box plot in Figure 2, and mean scores are reported in Table 3. Median scores were slightly higher than means in all four areas, with a negative skewing due to the longer whiskers in the negative direction and several low outliers. For all respondents (N=167), there was a significant difference in mean rating between content areas (F=48.9, df=3, P=0.000); post hoc analysis identified differences between all categories except Feedback and Examination.

Figure 2 Box plot of participant survey ratings.

Table 3 Mean Scores ± SD for survey content areas by response group Notes: Bold font = statistically significant difference (P<0.05); for all respondents, mean scores sharing the same letter superscript demonstrated a significant different from one another (Tukey’s HSD). Abbreviations: HSD, honest significant difference; OT, occupational therapy; PT, physical therapy; SD, standard deviation; SP, standardized patients.

In the Examination section of the survey, 86% of respondents indicated they somewhat, or strongly agreed, SP-based examination was an effective way to evaluate their knowledge. In comparison with other evaluation formats, 69% agreed it was more effective than written exams, and 72% felt it was more effective than written assignments. However, 89% somewhat, or strongly agreed, that interacting with an SP in an exam was anxiety provoking, and 84% disagreed that anxiety improved their exam performance.

Secondary analysis: ratings by subgroup

A breakdown of survey rating by subgroups is also provided in Table 3. Graduation year was not related to any differences in rating. Females had significantly higher ratings than males in the Teaching (t=3.49; P=0.001) and Feedback (t=3.77; P=0.000) content areas. Respondents in the OT program had higher mean ratings for all four categories; this difference was significant for the SP (t=2.88; P=0.004) and Examination (t=2.78; P=0.006) sections. Student age was positively associated with ratings in the SP category (r_p=0.284, adjusted r²=0.075, P=0.000).

Phase 2

While five individuals agreed to participate in each focus group, the number who ultimately attended was variable. When attendance conflicts were identified in advance, additional recruitment was pursued using the ranked list of survey respondents. Recruitment was constrained by the number of survey respondents consenting to follow-up and the point at which differentiating age categories approached convergence. By stratifying groups, we were able to analyze responses by program (OT N=7; PT N=5) and age (older N=7; young N=5). A summary of focus group participant characteristics is provided in Table 4.

Table 4 Focus group participant characteristics Abbreviations: Grad, graduation; OT, occupational therapy; PT, physical therapy.

Emergent themes

In exploring the qualitative data, a variety of themes were identified in relation to the four components of the initial survey and the quantitative findings.

The SP experience

Focus group participants were overwhelmingly positive about their educational experience with SPs, including overall quality and performance. They reported the acting to be of excellent quality, and a number of students reflected on how they became “lost in the moment:” “… we had one lab which was on Parkinson’s … we totally thought that they actually had Parkinson’s until afterwards … they just showed up [and] were normal” (young PT).

One variable aspect of the SP experience that became evident was the students’ willingness to “suspend disbelief” – that they were prepared to approach the encounter as if it were a genuine clinical interaction. For example, some participants reflected on very authentic experiences, such as a young OT student who said “it was really neat how that person can just play that role over and over again and just provide that same information but in different ways … they play the role really well … they open up differently for different people.” Conversely, others alluded to their reluctance or inability to move beyond the mechanics of the encounter. A young PT provides the following insight:

I think they’re a valuable assessment tool … but other than that, I think the whole method and for me personally, I learn a lot more through interacting with actual patients … more than with standardized patients … you kinda just go through the motions.

This perspective was more apparent among PT participants, who commented that they perceived the SP as a “means to an end” or tool to demonstrate competence to their instructor, rather than investing in the interactive experience.

Appreciation for SP interaction appeared to have a temporal aspect for some. There were instances in which participants indicated that as they moved through the latter part of their program, they lost interest in the SP encounters and no longer approached them with a mindset of simulating a clinical experience.

Use of SP for teaching and education

Participants identified value in using SPs to learn and practice clinical skills. In particular, interviewing skills were highlighted as being conducive to this methodology. Broadly speaking, participants agreed that learning from and practicing with SPs was preferable to didactic strategies and was far superior to practicing with peers/colleagues and instructors. In particular, when practicing with peers or instructors, students felt their “patient” already knew what to anticipate and had comparable clinical knowledge, making it difficult to interact in a genuine manner. Furthermore, peers were inclined to facilitate the success of the encounter or intervention as they had a preexisting collegial relationship: “you get awfully comfortable doing it with just your lab partner” (older PT). Instructors provided a more desirable portrayal, as a young OT articulates: “I thought it was … better in that they gave you direct feedback … but I felt more self-conscious of myself when I was doing it with a faculty member than whenever it was with a [SP].” Participants also noted that, despite appreciating the value of SPs in these clinical learning encounters, they preferred “real” patients and saw their clinical placements as a superior venue for learning.

Students noted that practicing with SPs was particularly valuable early in their professional education. Several benefits were identified, including the sense of safety knowing they would not hurt their patient if they made an error, and having the opportunity to stop an interaction for feedback/correction or have a “redo”, which were not generally options when working with real patients in a clinical context. A number of participants reflected back on how these SP learning encounters provided a boost to their confidence, specifically in preparation for the clinic. An older OT articulates: “I forgot it was a simulated client. I was just doing my thing and [thought] oh yeah this is just like being on fieldwork. I think that it actually builds confidence in those interactions as time goes by.”

Feedback related to SP interactions

Some participants felt that feedback provided by SPs reflected the experience of a “true” patient and placed high value on these responses, whereas instructors provided more accurate technical feedback on skill performance. Those in the OT program were more inclined to see the patient’s subjective experience as relevant to them: “… as much as the instructor is giving us instruction, but it doesn’t necessarily mean it’s the right one. What’s really valuable is what the patient feels in the end of it” (older OT). This type of feedback was perceived as contributing to greater confidence, particularly as a prelude to fieldwork. A young OT describes her reaction to feedback following interview training in this way:

[SP feedback] was more important than [the instructor’s] … like the teacher’s perspectives were more like … their own experiences but from the patient’s view that is still really important because I mean you want patients that keep coming back to you in real practice so you want to know how they’re feeling too.

Participants from the PT program reported few opportunities to obtain feedback from either instructors or SPs directly, whether in learning or examination situations, and were more equivocal in their desire for SP input. In situations where feedback was provided, participants indicated SP responses were more benign and students tended to focus on their instructor’s remarks, as illustrated by this exchange among young PTs:

Facilitator: Was the [SP] feedback helpful … did you reflect on it after?

Participant 1: For me no.

Participant 2: That’s an OT thing [group laughter]

Facilitator: So was instructor feedback helpful … during your interactions with SPs?

Participant 2: Yeah, I think that’s where we got most feedback from. The lab instructors coming around to observe, they obviously prepped the patients so they know what’s supposed to happen and they’re watching you.

Use of SP for evaluation purposes

The benefits to using SPs instead of actual patients were noted in all four focus groups, including issues of student security, patient safety, consistent portrayal, and pragmatics:

So even in an exam situation, you know going in that this person is acting, that you are not really going to hurt them if you mess up and that’s why I think it’s … that safe zone (older OT).

       We couldn’t do this on a real patient because they would fatigue or it would hurt (young PT).

Despite the generally positive view of overall SP performance and belief that SP-related evaluation was preferable and more reflective of student proficiency than written evaluations, participants raised the issue of fairness with respect to evaluation situations. The quality of SP portrayal took on particular importance, where inconsistent performance might benefit or handicap a student. There was concern that an error on the part of an SP could impact a student’s performance and, ultimately, their examination grade. This created a particular sense of anxiety for students, as they perceived this to be outside of their control. One young PT expressed the viewpoint of classmates in this way: “… some others felt that it wasn’t consistent … like the way the actor portrays themselves, what you have to do in your OSCE station … but because of their acting it kinda makes it harder for you to get what your task is supposed to be.”

Student anxiety related not only to SP performance specifically but also the SP evaluation format generally. Participants identified a number of factors that increased stress, including location and configuration of the examination room, familiarity with equipment and supplies, previous encounters with the same SP, and having the examiner in the room. A number of participants suggested students would feel less anxious and better able to engage with the SP without their examiner present, even if they knew they were being watched from outside the room (eg, through a one-way mirror).

Another expressed concern was deciphering whether a questionable clinical manifestation during an exam was part of the rehearsed scenario (eg, a contrived anomaly the student was expected to pick up and address appropriately), the consequence of an SP error (eg, not complying with the prescribed scenario), or the SP’s inability to accurately portray a health condition (eg, hypertonicity). There was an underlying question of laying blame for the resulting student performance deficiency, whether fault lies with the student, SP, or even the evaluator. One young PT reflected on this dilemma when a perceived SP misstep occurred during an exam: “you’re not sure if this is part of it or whether you are supposed to continue doing what you are asked to do or you have to try and change your approach because of the way the actor presented.”

One issue many participants identified, was seeing the same SP in different contexts. For example, an SP might portray a patient in a learning situation in one course and then subsequently portray a different patient and scenario in an examination in another course. This created dissonance for some students; even if they were prepared to enter the situation authentically, seeing a familiar individual in a new role created a disconnect, and participants felt this inhibited optimal performance. These multiple encounters with the same SP in different contexts also seemed to compromise verisimilitude for students in both learning and exam situations.

Discussion

Students in these rehabilitation programs were generally quite satisfied with the use of SPs in their professional education. The ratings for all four content areas were encouraging, with an overall mean around 3. Median scores were higher in all categories, indicating that a large number of responses congregated between “agree” and “strongly agree”, while a small number of respondents provided ratings at the very low end of the scale, drawing the mean downwards. There may have been a ceiling effect due to the four-point range in the survey design, as the upper limits in two content areas extended to a score of 4. No corresponding floor effect was apparent; however, a few outliers were noted with scores between 1 and 2 and likely reflect a small number of students who had a very negative perception of their SP experience. During the focus groups, it became apparent that even a singular event could impact a student’s overall perception of SP use or performance. It is possible that the outlier scores were a response to a frustrating event (eg, a poor exam) rather than the student’s global experience.

There were some significant differences in ratings between content areas. Using SPs for learning and practicing clinical and communication skills appears to be the aspect most appreciated by students. Some participants were willing and able to “temporarily suspend disbelief” – that is, to act as if they were a therapist really seeing a client. Velde et al²¹ reported similar findings among OT students, who believed they were consulting therapists while working with SPs. Students in many disciplines have identified a lack of believability or authenticity when practicing with peers and educators, and view SP training as an effective learning strategy, particularly with communication skills.^22,23 Between the two programs in our study, there appeared to be a stronger willingness among the OT students to invest personally in the SP experience, whereas PT students found it more contrived and approached SP encounters mechanically as a means to demonstrate competence to their instructors. Baptiste and Solomon¹² also reported PT students, particularly early in their program, were quite mechanistic when approaching their SP encounters. Our program uses both actors and individuals with a disability, and the survey questions were not sufficiently granular to determine whether students had a stronger preference for one or the other.

In exploring the qualitative data, a continuum of student preferences emerged. Student colleagues are often used to role-play during initial exposure to clinical and communication skills because it is pragmatic and cost-effective; however, study respondents clearly identify this as a largely undesirable option because it lacks authenticity in clinical presentation and interaction, and a successful outcome is typically enabled by the “patient”. Similar issues of unrealistic portrayal and challenges taking peer-learning seriously have been identified in OT,²⁴ nursing,²⁵ and medical education.²⁶ Having educators role-play was preferable because they took the role more seriously and were more demanding and critical of student performance, but students found it challenging to entirely disengage from the preexisting relationship. SPs were identified as more authentic, and students felt the SP reacted appropriately in response to their intervention, albeit within the confines of a prescriptive scenario. However, students still preferred practicing with an individual who genuinely had a health condition versus a “healthy” SP, and ultimately felt that practice in a clinical environment with “real” patients (ie, fieldwork) was the optimal method of learning (Figure 3). There is little in the literature exploring this spectrum of portrayal preferences, although a mixed-methods study did find students rated SP cases higher than role-play as a teaching method in an OT curriculum.²¹ Gallimore et al²⁷ also reported pharmacy students found SPs more useful in their learning than peers, followed by instructors and staff members, although peers were identified as least believable and authentic.

Figure 3 Learning continuum: preferred scenarios for learning and practicing clinical skills. Abbreviation: SP, standardized patient.

There was recognition among students that, from both a pragmatic and readiness perspective, they needed to progress through these learning strategies, suggesting a second learning continuum. Participants confirmed they found role-playing and SP practice as most beneficial early in their professional education, particularly prior to their intermediate and advanced fieldwork placements. The perceived value and benefit of SP interactions, particularly for learning and practicing skills, diminished as their exposure to, and experience with, actual patients in clinical placements increased. Velde et al²¹ identified a desire among OT students for inclusion of SPs earlier in their curriculum. Other authors have noted that practicing with an SP early on provided a safe and supportive context to build both skills and confidence, which students felt was good preparation for the clinical setting.^28,29 In particular, these early “success experiences” provided a boost of confidence going into their basic and intermediate fieldwork.

In contrast to learning scenarios, rating of SP use in Examinations was the lowest among all categories. On the one hand, students appeared to appreciate and even prefer this style of examination as better reflecting their capacity to demonstrate acquired knowledge and skills. Stehle et al³⁰ have also found a strong positive correlation between students’ evaluation of learning and use of standardized practical examinations. However, participants in our study reported the experience to be highly stressful – a factor that did not necessarily elicit their best performance. For most students in rehabilitation, the OSCE/practical exam is a very new and intimidating experience. It is understandable that students would be more apprehensive and uncomfortable with this style of evaluation. Admission to the OT/PT programs is restricted, and applicants must achieve high academic performance to successfully gain entrance. It is typical for many students to be “high achievers” who associate substantial self-worth with their academic grades; it may be that the unfamiliarity and anxiety of practical examination (and perhaps resulting lower grades) may be factors in students’ lower ratings with this component.

Related to examination anxiety and performance, the issues of blame and responsibility emerged in the qualitative analysis. Respondents expressed uncertainty about whether a mistake on an exam (and consequently a low score) was due to student error, SP error or poor portrayal, confounding contextual factors, or examiner issues. The high premium placed on evaluation grades may exacerbate this tendency to assign external responsibility for discrepancies, which in turn may mar the SP experience. It may be worthwhile exploring whether students perceive exams differently when administered as pass/fail rather than a graded format, as was the case in our program.

Students in the OT program rated all categories higher than those in PT, particularly in the SP Experience and Examination areas. Hale et al³¹ have reported fairly neutral ratings of SP usefulness in a diabetes teaching module among PT students. The OT discipline has a slightly higher ratio of females, who rated all categories higher than did males, but the proportion of male to female respondents was not significantly different among our survey respondents. Our OT and PT programs use SPs in similar ways and draw from the same pool of actors; however, at the time of this study there were some differences between the programs. The OT program used SPs more frequently for teaching purposes, including a communication skills block where they interview several SPs over a period of weeks. The sessions were video recorded, and students debriefed and reviewed their evaluation with their instructor using the video. Another variation between programs was the PT use of “bell-ringer” exams with multiple SP stations in an OSCE format, while the OT program employed an extended practical exam with the same SP, where students performed several interventions in succession. It may be that differences in use and format of SP interactions between the two programs accounted for some of the difference in students’ ratings.

SP feedback was perceived to be of substantial importance to students, particularly in the OT program. Students in our OT program had more opportunities than PT students for direct SP feedback (verbal and written) because of increased SP applications in teaching labs and structured written feedback on practical exams. More study is required to determine whether difference in student perception is a result of the manner in which the SPs are used and provide feedback, or whether this is a reflection of differences between students in the two programs. Interestingly, students in the OT program placed a higher value on the SP feedback than they did on instructor feedback. They appeared to derive a better sense of success from affirming SP comments and linked a positive SP experience with their own potential for success in a clinical setting, whereas instructor feedback was perceived as more discretionary and related to performance mechanics. Conversely, PT students tended to place higher value on their instructor’s input, and suggested even the limited SP feedback they received was a consequence of how their instructor had prepared the SP. Both disciplines highlighted the importance of mastering skills of clinical reasoning through SP interactions, but OT students tended to focus on developing these skills through clinician–patient relationships, whereas PT students focused on eliciting the appropriate information required to inform decision-making. This finding is not entirely surprising given the OT conceptualization of patients as equal partners,³² whose input and perception of outcomes³³ is critical to the clinical decision-making process, and the emphasis placed on students developing patient–centeredness³⁴ in the OT program.

Student age was associated with higher ratings in the SP experience section, suggesting that older students were more likely to take these interactions seriously and less inclined to be distracted by issues such as seeing the same SP in more than one role or suspicion that the SP was trying to “trick” them during the interaction. We might speculate that with increased life and perhaps vocational experience, older students are better able to see the value of simulated practice and the relevance to educational goals. While age was not associated with any of the other content area scores, post hoc analysis did reveal an association between increasing age and lower anxiety when interacting with an SP in exams. It may be that older students have less exam anxiety in general (whether it be in practical or written format) and their rating of SP use in evaluation is merely an artifact of a different outlook on examinations generally. Alternately, as individuals move further away from their education, they may look back retrospectively on SP use in evaluation, adjusting their ratings accordingly. However, the lack of any significant relationship between graduation year and respondent rating suggests this was not the case.

Lessons learned

Several issues pertinent to our professional programs came to light through the course of this study. First, there is variability in performance quality among actors in any SP pool. Given the high premium students place on accurate and error-free SP portrayal, educators are inclined to select and use a smaller subset of actors. While this tends to increase student (and instructor) satisfaction, it also increases the risk of recurrent student encounters with the same actor in different scenarios – a situation which students identify as problematic. This produces a “catch-22” where instructors have a limited pool of SPs and must choose between diversity (with the risk of performance issues) and consistency (with the risk of repeated exposure). One approach we used at a course level was to split classes into groups and rotate the SPs through the different groups in each successive exam. However, this approach requires considerable coordination and is particularly challenging to implement across an entire program of courses.

A second area of insight was the selective use of SPs. It became clear that SPs were particularly useful early on in a professional program, providing a safe and efficient method of learning and enhancing student self-confidence prior to clinical fieldwork. As students progress and develop greater competency, SP use had to be adapted accordingly. Later in the program, SP scenarios needed to be more complex or the SP portrayal needed to be more challenging for students to continue investing in and gaining from the experience. SP activities focus more on integrative skills and clinical reasoning as students advance through their program. Furthermore, not all learning scenarios lend themselves to SP use, and instructors need to be selective about when and where they employ SP strategies.

Finally, consistency, structure, and preparation were critical to the successful use of SPs for both teaching labs and examinations. Specific guidelines have been adopted to enhance the SP interaction for students. A trained backup SP is now standard practice for exams in case a primary SP is ill or absent, rather than back-filling with an instructor or faculty member. Students are provided with access to the evaluation venue, as well as consistent training equipment, prior to SP evaluations to reduce contextually related exam anxiety. SPs receive a structured training session and comprehensive script materials prior to training for all exams. Qualitative exam feedback to students is differentiated between evaluators (ie, student performance) and SPs (ie, patient experience) and integrated into the evaluation process.

Limitations

The findings from this study are reflective of respondents from the OT and PT programs at our university and cannot be generalized beyond this cohort. While we had a reasonable response rate to our survey and focus group invitations, there is always a risk of response bias. We found considerable synergy between the quantitative and qualitative findings, particularly in regard to the OT and PT programs; however, while we were able to consider responses across focus groups (ie, program/age), some individual focus groups were small in number, which may have limited the breadth of data.

Conclusion

OT and PT students reported strong satisfaction with the use of SPs in their professional education. Students in the two programs approached SP interactions somewhat differently, with OT students perceiving the interaction as more authentic and the SP feedback having greater value; this may reflect the increased application of SPs as a learning tool and intentional use of SP feedback in the OT program. Students identified a continuum of preferred options for learning and practicing skills: peers, instructors, SPs, and actual patients. The use of SPs is most optimal early on in the professional education program and can serve to bolster self-confidence and prepare students for clinical fieldwork. SP scenarios and portrayal requires increased complexity and challenge later in the program and should focus on integration and clinical reasoning skills. Educators need to be aware of pragmatic and contextual issues when using SPs for examination purposes, including repeated exposure to the same actor.

Authors’ contributions

EMG was responsible for administration of the grant and oversight of the study, contributed to the development and implementation of the study design, provided oversight of the quantitative analysis, and contributed to the qualitative analysis and interpretation. PFW contributed to development and implementation of the study design, provided oversight of the qualitative analysis, and contributed to the quantitative analysis and interpretation. GMP participated in development and implementation of the study design and provided contributions to qualitative and quantitative data analysis and interpretation. EMG wrote the first draft of the manuscript, and all authors reviewed and contributed to the final version.

Acknowledgments

Funding for this project was provided through a grant from the Centre for Higher Education Research and Development (CHERD), University of Manitoba. The authors would like to express their appreciation to statistician James Douglas Staley for assistance with the quantitative analysis; Lisa Salter and Jeannette Logan for their work as RAs, including focus group facilitation; and all of the study participants for their willingness to share their experiences openly and honestly.

Disclosure

The authors declare that they have no competing interests.

---

References

1.	Ainsworth MA, Rogers LP, Markus JF, Dorsey NK, Blackwell TA, Petrusa ER. Standardized patient encounters: a method for teaching and evaluation. J Am Med Assoc. 1991;266(10):1390–1396.
2.	Ladyshewsky R, Baker R, Jones M, Nelson L. Evaluating clinical performance in physical therapy with simulated patients. J Phys Ther Educ. 2000;14(1):31–37.
3.	Zraick RI, Allen RM, Johnson SB. The use of standardized patients to teach and test interpersonal and communication skills with students in speech–language pathology. Adv Health Sci Educ. 2003;8(3):237–248.
4.	Liu L, Schneider P, Miyazaki M. The effectiveness of using simulated patients versus videotapes of simulated patients to teach clinical skills to occupational and physical therapy students. Occup Ther J Res. 1997;17(3):159–170.
5.	Black B, Marcoux BC. Feasibility of using standardized patients in a physical therapist educational program: a pilot study. J Phys Ther Educ. 2002;16(2):49–56.
6.	Barrows H. An overview of the uses of standardized patients for teaching and evaluating clinical skills. Acad Med. 1993;68(6):443–453.
7.	Becker KL, Rose LE, Berg JB, Park H, Shatzer JH. The teaching effectiveness of standardized patients. J Nurs Educ. 2006;45(4):103–111.
8.	Badger L, MacNeil G. Standardized clients in the classroom: a novel instructional technique for social work educators. Res Soc Work Pract. 2002;12(3):364–374.
9.	Hampl J, Herbold N, Schneider M, Sheeley A. Using standardized patients to train and evaluate dietetics students. J Am Diet Assoc. 1999;99(9):1094–1097.
10.	Syder D. The use of simulated clients to develop the clinical skills of speech and language therapy students. Eur J Disord Commun. 1996;31(2):181–192.
11.	Nicol M, Freeth D. Assessment of clinical skills: a new approach to an old problem. Nurse Educ Today. 1998;18(8):601–609.
12.	Baptiste S, Soloman P. Developing communication skills. In: Baptiste S, Soloman P, editors. Innovations in Rehabilitation Sciences Education: Preparing Leaders for the Future. Berlin: Springer Verlag; 2005:136–145.
13.	Vessey JA, Huss K. Using standardized patients in advanced practice nursing education. J Prof Nurs. 2002;18(1):29–35.
14.	Yudkowsky R. Should we use standardized patients instead of real patients for high-stakes exams in psychiatry? Acad Psychiatry. 2002;26(3):187–192.
15.	Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 2nd ed. Thousand Oaks: SAGE publications Inc.; 2003.
16.	Schaefer DR, Dillman DA. Development of a standard e-mail methodology. Public Opin Q. 1998;62(3):378–397.
17.	Boone HN, Boone DA. Analyzing Likert data. J Extension. 2012;50(2):2TOT2. Available from: http://www.joe.org/joe/2012april/tt2.php. Accessed July 5, 2014.
18.	Norman G. Likert scales, levels of measurement and the “laws” of statistics. Adv Health Sci Educ. 2010;15(5):625–632.
19.	Canadian Institute for Health Information. Occupational Therapists in Canada, 2008. Ottawa: Canadian Institute for Health Information; 2009. Available from: http://www.caot.ca/pdfs/OTreporteng.pdf. Accessed July 5, 2014.
20.	Canadian Institute for Health Information. Physiotherapists in Canada, 2010 – National and Jurisdictional Highlights and Profiles. Ottawa: Canadian Institute for Health Information; 2011. Available from: http://www.cihi.ca/CIHI-ext-portal/pdf/internet/PTDB2010_PROVINCIAL_PROF_EN. Accessed July 5, 2014.
21.	Velde B, Lane H, Clay M. Hands on learning: the use of simulated clients in intervention cases. J Allied Health. 2009;38(1):17–21.
22.	Sanson-Fisher RW, Poole AD. Simulated patients and the assessment of medical students’ interpersonal skills. Med Educ. 1980;14(4):249–253.
23.	Schultz K, Marks A, Dunn B. Community-based collaboration with high school theatre students as standardized patient. Am J Pharm Educ. 2007;71(2):1–10.
24.	Lindstrom-Hazel D, West-Frasier J. Preparing Students to hit the ground running with problem-based learning standardized simulations. Am J Occup Ther. 2004;58(2):236–239.
25.	Hilton P, Barrett D. An investigation into students’ performance of invasive and non-invasive procedures on each other in classroom settings. Nurse Educ Pract. 2008;9(1):45–52.
26.	Hendry GJ. Barriers to undergraduate peer-physical examination of the lower limb in the health sciences and strategies to improve inclusion: a review. Adv Health Sci Educ. 2013;18(4):807–815.
27.	Gallimore C, George AK, Brown MC. Pharmacy students’ preferences for various types of simulated patients. Am J Pharm Educ. 2008;72(1):1–6.
28.	Vallevand A, Paskevich D, Sutter B. Use of simulations to assess the injury evaluation and management skills of advanced student athletic therapists at a Canadian university. J Allied Health. 2007;36(3):244–256.
29.	Kay CJ, Johnson JA, Kopp KC. Standardized patients for teaching geriatric dentistry. Special Care Dent. 1994;14(6):229–232.
30.	Stehle S, Spinath B, Kadmon M. Measuring teaching effectiveness: correspondence between students’ evaluations of teaching and different measures of student learning. Res Higher Educ. 2012;53(8):888–904.
31.	Hale LS, Lewis K, Eckert RM, Wilson CM, Smith BS. Standardized patients and multidisciplinary classroom instruction for physical therapy students to improve interviewing skills and attitudes about diabetes. J Phys Ther Educ. 2006;20(1):22–27.
32.	Law M, Baptiste S, Mills J. Client-centred practice: what is it and does it make a difference? Can J Occup Ther. 1995;62(5):250–257.
33.	Law M, Baptiste S, McColl MA, Polatajko H, Carswell A, Pollock N. Canadian Occupational Performance Measure. 4th ed. Ottawa: Canadian Association of Occupational Therapists; 2005.
34.	Ripat J, Wener P, Dobinson K. The development of client-centredness in student occupational therapists. Brit J Occup Ther. 2013;76(5):217–224.

---

Supplementary material

Simulated Clients

At the School of Medical Rehabilitation, Simulated Clients are used for teaching students in both the occupational therapy and physical therapy programs.

The following statements relate to your experience with Simulated Clients in the OT or PT program.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement. Use N/A only if the statement does not apply to your SMR experience.

Teaching/Education

Simulated Clients are used in various aspects of the education process. For example, occupational and physical therapy students practice skills related to interviewing, communication, clinical assessment and interventions.

The following statements will relate to your experience with Simulated Clients as a learning tool.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement. Use N/A only if the statement does not apply to your SMR experience.

Feedback

Feedback is an important component of the learning process. Receiving feedback allows students the opportunity to gain insight into their performance, learn from their mistakes and gain confidence in their abilities.

The following statements relate to your perceptions/experiences with receiving feedback about your Simulated Client interactions.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Examinations

Simulated Clients are also used for practical examination and Objective Structured Clinical Exam (OSCE) purposes. Many different components are examined, including communication and physical assessment skills.

The following statements will relate to your perceptions/experiences with Simulated Clients in exam situations.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Participant demographics