Objective Structured Practical Examination in Experimental Physiology Increased Satisfaction of Medical Students

Background

The assessment encompasses the learning and two of its main indicators are the interpretation and use of the information that is categorized for the intended purpose.¹ In addition, assessment can affect students learning, their motivation to learn, and the way instructors teach which has an impressive influence on students’ performance.² Instructors can help students to be more focused and motivated if their ability of integration, usage, and linkage of knowledge are thoroughly assessed and especially if students’ skills are observed and graded.^2,3 Therefore, it seems the selection of assessment methods by professors is an important part of the medical students’ education.⁴

Although many options are available to achieve this purpose, the Objective Structured Practical Examination (OSPE) is a highly recommended method.^2,5-9 This assessment tool is derived from Objective Structured Clinical Examination (OSCE) by Harden and Gleeson and used for evaluating fundamental sciences, preclinical and paraclinical students.^6,10 Many medical schools all over the world have used OSPE for assessing students’ performance in laboratory exercises.^11,12

OSPE is a specified set of tasks which assesses what students can do in a structured pattern objectively under direct observation and is able to assign the above-mentioned capabilities.¹³ Miller’s framework of developing clinical qualifications concentrates on four levels of assessment: “knows, knows how, shows how and does”. It was reported OSPE evaluates the third level “shows how” of this framework by focusing on the assessment of the performance of specific skills in a controlled setting.¹⁴ It is believed that OSPE can reduce the number of examiners in marking the students and it is a standardized tool, which its advantages outweigh Traditional Practical Examination’s (TPE).^12,15

In the physiology department of Ahvaz Judishapur University of Medical Sciences (AJUMS) which works under the supervision of the Medical Council of Iran, the course of human physiology is considered as a fundamental sciences subject, which can be taken in the first 4 semesters of medical student’s courses. In this department, students evaluate experimental physiology using TPE. This method involves performing a particular experiment randomly and the assessment is based on the global performance rather than individual skills.¹² According to Miller’s framework, TPE mainly focuses on the base of this pyramid, ie the “knows” and “knows how” aspects.⁴ Therefore, it seems that application of a method which is more objective and structured is necessary so as to provide feedback to the students for gaining a better understanding of their weakness, eventually resulting in improving their skills.¹²

It was reported that OSPE in most cases improves students’ performance, especially in experimental physiology.¹⁶ In addition, in some studies feedback from students revealed that using OSPE increased the satisfaction in medical students,¹⁷ and the validity and reliability of this method is more than TPE.¹⁸

Department of physiology of AJUMS is trying to implement a series of reforms in teaching methods and assessment strategies to improve the quality of medical education. Therefore, considering the OSPE advantages comparison to TPE, this study was undertaken to determine student satisfaction regarding the OSPE as a method of assessment of experimental physiology before import it in the forthcoming university examination.

Methods

After obtaining the permission from the Ahvaz Jundishapur University of Medical Sciences Ethics Committee for research on human subjects (IR.AJUMS.REC.1398.234), 120 sophomore students (3rd semester) who took part in this project were studying at the department of physiology of AJUMS, Ahvaz, Iran.

The lecturers were well trained to achieve an optimal organization of the TPE and OSPE. In addition, these two assessment methods were introduced to the participants by a short lecture. It is worth mentioning that, at the beginning of semester, the students were permitted to choose which group they are interested to participate and in this research and we confirm that verbal consent process was informed verbal consent and this study complied with the declaration of Helsinki and Ahvaz Jundishapur University of Medical Sciences Ethics Committee for research on human subjects approved the informed verbal consent process.

TPE Session

In the physiology department of AJUMS, it is common for students who practice TPE to select randomly an experiment between all tests performed during a semester. Students should completely know the whole procedure of that experiment, what tools they need to use, presenting the final result and its interpretation is necessary. It is worth mentioning that the method of evaluation is thoroughly taught to the student in advance and through the execution of the experiment, students’ performance was evaluated.

OSPE Session

OSPE sessions were held according to a blueprint of the syllabus and structured checklist which was prepared as Bloom’s taxonomy. In addition, a senior faculty member validated all responses and skills stations. The OSPE consisted of three-minute stations, including the following items: 1. Response station; questions depend on the theoretical aspects of topics such as blood, muscle, nerve physiology, the respiratory and cardiovascular system. In this regard, we used different tools: a) Flashcards; including some pictures related to a disorder and asking the student to diagnose the disease. b) Videos; showing a correct or incorrect procedure related to one experiment and the examiner expect the student to realize what is the experiment? What is its goal? Moreover, how can we use this experiment in a clinic? c) Microscope; especially in the case of white blood cell differentiation. 2. Skill stations; questions regarding different topics, mainly blood cell experiments, for example, white and red blood cells (RBC) counting, fragility test for RBC and hematocrit measurement. In these stations, examiners directly graded student’s performance as; 1 (weak), 2 (moderate), 3 (great). It is imperative to know that examiners frequently discussed the required arrangements for conducting the OSPE and preparation checklist/key to reaching a consensus without any ambiguity.

Experimental Design

Students were assigned to three groups: 1. TPE group (n=40), the assessment method was just based on the gained score in the TPE session; 2. TPE+OSPE group (n=41), TPE was used for half of the topics which were taught during the semester and OSPE was utilized for another half and 3. OSPE group (n=39), students were evaluated according to the total earned scores within eight stations. Therefore, in this group, the total score was the sum of the obtained results from TPE and OSPE. It should be noted that the total grade in Iran is evaluated out of 20.

Feedback Questionnaire

In this study, immediately after TPE and at the 8th station of OSPE a reliable, validated and structured questionnaire was used.¹⁹ In the TPE group, the students were asked to express their attitude about the TPE and in TPE+OSPE and OSPE groups the students were surveyed regarding the OSPE. The questionnaire consisted of 10 items and each item had five options. It is worth mentioning that anonymity and confidentiality were taken into account by asking students not to write their names. In addition, 10 items were based on Likert’s 5-point scale to assess the impact of TPE and OSPE on student’s satisfaction and their efficiency in experimental physiology.

Likert Scale Items

Likert scale items analyzed and compared the effect of the TPE and OSPE on various aspects of student’s satisfaction and efficiency: 1) The questions asked were relevant, 2) Sufficient time was given to students, 3) TPE/OSPE is more fair, 4) TPE/OSPE is easier to pass, 5) TPE/OSPE is a better method of assessment, 6) TPE/OSPE improves learning physiology, 7) TPE/OSPE provides a chance to score better, 8) TPE/OSPE is less stressful, 9) TPE/OSPE makes students think more, 10) TPE/OSPE eliminates bias. These 10 items had the following options: strongly agree (option A; score 5), agree (option B; score 4), neutral (option C; score 3), disagree (option D; score 2) and strongly disagree (option E; score 1). A heightened score 5 and a score 1 was given for the options with, respectively, strong and having completely no impact of TPE and OSPE on student’s satisfaction and efficiency.

Statistical Analysis

The student’s total grades and total score for each item in questionnaires were averaged and expressed as mean ± SD using GraphPad Prism version 6.01 for Windows (GraphPad Software, Ca, USA). To evaluate the effect of OSPE and TPE on average of the total grade and average scores of each item in the questionnaire one way ANOVA was followed by posthoc Tukey’s test. P-value of less than 0.05 was considered to represent a significant difference. In addition, the results of different options in the Likert questionnaire were presented as frequency percentages.

Results

The total grade of experimental physiology for each item in questionnaires were averaged and expressed as mean ± SD. In TPE group (18.09±0.22) it was significantly higher compared to TPE+OSPE (16.68±0.26, P<0.01) and OSPE (17.15±0.32, P<0.05) groups (Figure 1). Students’ perspectives regarding the TPE and OSPE for each option of 10 questions were presented as frequency percentages (Table 1). It is worth mentioning that there was no significant difference in age average (TPE: 20.65±0.43, TPE+OSPE: 20.77±0.52, OSPE: 21.02±0.38) and gender distribution (TPE: 18 Male, 22 Female; TPE+OSPE: 22 Male, 19 Female; OSPE: 20 Male, 19 Female) of students in three different experimental groups. Therefore, the results of this study did not affect by the age average and gender distribution in different experimental groups.

Table 1 Students’ Perspective Regarding TPE and OSPE

Figure 1 The average of total grade in different experimental groups. One way ANOVA showed that, there is a significant increase in the average of total grade in TPE group (n=40) compared to TPE+OSPE (n=41, p<0.01) and OSPE (n=39, p<0.05) groups. Data are shown as mean±SD. *p<0.05 and **p<0.01 compared to TPE group.

One hundred percent of the TPE+OSPE group and 85% of the OSPE group felt that the OSPE questions were relevant, comprehensive and covered a wider area of knowledge in comparison to the TPE group (72.5%). In addition, the average score of TPE+OSPE (4.63±0.07) and OSPE (4.46±0.17) groups for the 1st question showed a significant increase compared with TPE group (3.62±0.2) (P<0.001) (Figure 2A).

Figure 2 The average scores for all of items in feedback questionnaire. (A–J) The results of one way ANOVA showed that, students’ attitude in all of items was in favor of OSPE. The average scores significantly were increased in all of items in TPE+OPSE (n=41, p<0.001) and OSPE (n=39, p<0.001) groups compared to TPE group (n=40). Data are shown as mean±SD. *** p<0.001 compared to TPE group.

Our results for the second item of the questionnaire showed that the majority of students in TPE+OSPE (88%) and OSPE (77%) groups expressed that sufficient time was given to them during OSPE, while 52.5% of TPE group agreed with this statement about TPE. Analyzing the second item in questionnaires revealed a significant increase of average score in TPE+OSPE (4.63±0.13) and OSPE (4.07±0.23) groups compared with TPE group (2.97±0.24) (P<0.001) (Figure 2B).

Although the average of total grade in the TPE group was significantly higher than in two other groups, a minority of students in response to items three and four believed that TPE is a fair method (12.5%) and easier to pass (37.5%). By contrast, most of the students in TPE+OSPE (respectively, 95% and 63%) and OSPE (respectively, 74% and 70%) groups declared this opinion regarding the OSPE. Furthermore, the average score for items three and four showed a significant increment in TPE+OSPE group (respectively, 4.51±0.09 and 3.87±0.15) as well as OSPE group (respectively, 4.07±0.22 and 3.94±0.22) compared with TPE group (respectively, 2.25±0.17 and 2.62±0.21) (P<0.001) (Figure 2C and D).

In response to item five, unlike the TPE group (27.5%), the majority of students in TPE+OSPE (88%) and OSPE (74%) groups expressed OSPE is a better method of assessment. One way ANOVA analysis for item five revealed a significant increase of average score in TPE+OSPE (4.29±0.12) and OSPE (4.02±0.23) groups compared with TPE group (2.55±0.21) (P<0.001) (Figure 2E).

The results showed, just 22.5% of students in the TPE group believed that TPE improved learning in physiology, while most of the students in TPE+OSPE (76%) and OSPE (77%) groups express this issue regarding OSPE. Analysis of the average score for this item indicated a significant increase in TPE+OSPE (4.09±0.14) and OSPE (4.07±0.22) groups in comparison to the TPE group (3.32±0.22) (P<0.001) (Figure 2F).

Interestingly, data analysis for items 7 and 8 revealed that less than 50% of students in TPE group (respectively, 37.5% and 27.5%) agree with the statement that TPE provides a chance to score better and is less stressful, while most of the participants in TPE+OSPE (respectively, 66% and 71%) and OSPE (respectively, 58% and 69%) verify this issue regarding OSPE. One way ANOVA analysis for an average score of items 7 and 8 showed a significant increase in TPE+OSPE (3.85±0.12 and 3.95±0.15) and OSPE (respectively, 3.71±0.23 and 3.84±0.23) groups compared to TPE group (respectively, 2.67±0.21 and 2.30±0.2)(P<0.001) (Figure 2G and H).

Considering item 9 in three questionnaires indicate that, just 15% of students in the TPE group declared that TPE makes students think more, while 83% of students in both TPE+OSPE and OSPE groups stated this matter about OSPE. In addition, average score analysis revealed a significant increase in TPE+OSPE (4.19±0.12) and OSPE (4±0.20) in comparison to the TPE group (2.05±0.20) (P<0.001) (Figure 2I).

Finally, analysis of different options of item 10 showed a few students in the TPE group (12.5%) expressed TPE eliminates bias, while more than 50% of students in TPE+OSPE (61%) and OSPE (64%) groups declared this issue regarding OSPE. Moreover, one way ANOVA analysis indicates a significant increment of the average score in TPE+OSPE (3.78±0.14) and OSPE (3.74±0.22) groups compared with TPE group (1.95±0.17)(P<0.001) (Figure 2J).

Discussion

This study was aimed to receive the acceptability of the OSPE for the first time between students as a relatively new assessment method for experimental physiology in AJUMS. The obtained results showed that the average total grade in the TPE group was significantly higher in comparison to TPE+OSPE and OSPE groups. In spite of better grades in the TPE group, students’ satisfaction and learning in experimental physiology increased in TPE+OSPE and OSPE groups compared with the TPE group.

Although in our study, the average of total grade in the TPE group was significantly higher than two other groups, Dissanayake et al reported a marked improvement in the mean scores for the experimental physiology at King Faisal University Medical School.¹¹ We assume that our finding is presumably the results of students’ first experience with OSPE in TPE+OSPE and OSPE groups. Moreover, they were not familiar enough with this assessment method. On the other hand, it was reported that getting feedback has a profound effect on students’ examination performance,²⁰ as in TPE+OSPE and OSPE groups, students did not receive any feedback about OSPE before exam day, while students in TPE group were completely familiar with this assessment method.

In the present study, feedback from students immediately after the exam indicated that they were in favor of the OSPE compared with the TPE (Table 1). In line with our results, in many studies, OSPE was rated by the students as a credible, effective, useful, interesting and challenging assessment method.^2,5,9,21

Unlike TPE group, the majority of students in TPE+OSPE and OSPE groups, similar to other studies,^5,7,16,22 felt that OSPE questions were more relevant, sufficient time was given to them, this assessment tool is fairer, helps to score better, less stressful and eliminates bias. These items were satisfactions criteria in the feedback questionnaire. As students’ declaration, exposure to similar types of questions with the same difficulty was an important advantage for validation of OSPE by students. In line with our findings, it was reported that OSPE is accepted between students because it includes a large number of questions and skills with a wide variety of physiological concepts, in which students can test much more abilities compared to TPE^2,9 and they will be more satisfied if their talents are considered for assessment.^3,23 In addition, Lakshmipathy pointed out that students’ concentration increased when their skills were demonstrated during OSPE.² Considering OSPE advantages, few students in this study expressed the opposite attitude regarding OSPE and this could be attributed to habituation to the TPE. In addition, the presence of examiners in some of the stations with the checklist in their hands may make OSPE more stressful and anxiety driven for some of the students; thus, they were not happy with this assessment method. Moreover, Pramod Kumar et al pointed that, although most of the participants believed that OSPE is a better method of examination and covered a wide range of knowledge than TPE, a large number of students in their study (63.5%) expressed that the OSPE may be exhausting and stressful if numbers of stations are increased.²⁴ Therefore, the number of stations and difficulty of questions are important points in the acceptance of OSPE among students, and in this study, this issue was considered, therefore nobody complained about it.

An interesting achievement in this study was, despite a significant reduction in average of total grade in TPE+OSPE and OSPE groups compared to the TPE group, most of the students declared that OSPE improved their learning in physiology and made them think more. Although the feedback questionnaire was presented to students before the score announcement, this result illustrated that students agreed with an assessment method which increased the level of learning. It is worth mentioning that, it was the first counter of some students with OSPE as a final exam, while all of the medical students are completely familiar with TPE procedure, because TPE is using for other practical lessons like Biochemistry, Histology, Anatomy, Immunology and Bacteriology as an assessment tool. Therefore, they feel more comfortable during TPE compared to OSPE. In addition, in order to understand the deficits of the present study, beside the feedback questionnaire students could write their suggestions and felling about this kind of examination. Most of students, especially the TPE+OSPE group, expressed that although OSPE is more useful they felt stressed during the exam and it had some negative feedback on their function.

It is suggested that assessment as learning is where it helps instructors gain information about what students understand and how they can use their knowledge.² OSPE skill stations assess the “shows how” level of Miller’ framework, while response stations demonstrate “knows” (recall of facts, principles, and theories) and “knows how” (problem-solving, application and interpretation).² Therefore, it seems OSPE can be useful to improve medical education and prepare medical students to develop better clinical competencies.

There were few limitations in the present study, such as the students’ first encounter with OSPE, random distribution of students in groups, the chosen test model and the dependence of the results on the expertise of the applicators which can be the basis of the future development to reform and refine the OSPE as an assessment tool. On the other hand, item analysis which incorporated the difficulty index and discrimination index was not used to compare the questions for the TPE and OPSE and it may be another limitation. In addition, this study emphasized the need for continuous faculty development in the field of medical education for its betterment.

Conclusion

The present study showed that the OSPE was well accepted by the medical students of AJUMS compared with the TPE. Moreover, it provides an atmosphere for refining the method before OSPE is implemented in the forthcoming university examination.