Psychometric Properties of Visual Indicators of Teaching and Learning Success “VITALS” Instrument for Evaluation of Clinical Teachers

Introduction

Clinical teaching is one of the important components of medical students’ education. The quality of clinical teaching delivered by clinical faculty is an essential component of medical students’ learning experience. The assessment of clinical teachers is often based on questionnaires completed by students. However, it is important that these questionnaires have good psychometric properties to reflect the quality of teaching by clinical faculty.¹

Students describe the ideal clinical teacher as being a positive role model. The positive role model has been defined as being competent clinically and personally, while exhibiting quality teaching abilities. These role models were further described as being compassionate, supportive, just assessors, that provide constructive feedback and guidance, while providing opportunities for students to get involved in patients’ care.^2,3 In addition, students valued active research, clear instruction, organization, and enthusiasm as core traits possessed by the ideal instructor.⁴

Clinical teachers also play an important role in teaching values and professionalism.⁴ Indeed, one of the unique qualities of clinical teachers is scaffolding, which is an instructor’s awareness of the level of their students, providing additional guidance according to their deficiencies.⁵ Moreover, clinical teachers should promote articulation, which is when students are encouraged to be explicit about their knowledge and skills.⁵

Student evaluations are an important component of the evaluation of clinical teachers in medical education. It was shown that student evaluation of clinical teachers is reliable, and correlates well with student learning, peer evaluation and tutor self-rating.⁶ Indeed, clinical tutors can get feedback directly from their students on their performances in multiple settings, which can be used to enhance the quality of teaching. Moreover, such feedback provides useful data for administrators to use. The feedback can be utilized to aid in tenure selection, promotions, faculty development programs and for allocating clinical teaching responsibilities.⁷

There are many published papers that study instruments developed for evaluating clinical tutors.^{5,7–9,17,18,21–23} Examples include: Maastricht Clinical Teaching Questionnaire (MCTQ),⁵ Cleveland Clinic Teaching Effectiveness Instrument (CCTEI),⁷ and System for the Evaluation of Teaching Qualities (SETQ).⁸ We have previously introduced the Visual Indicators for Teaching and Learning Success (VITALS) instrument which uniquely has a smaller number of items compared to the previously published methods for clinical teacher evaluation.¹⁰ VITALS is based on the theoretical framework of cognitive apprenticeship for clinical teaching. In addition, VITALS can visually display the evaluations of clinical teachers graphically, which simplifies the interpretation of the data output.

Psychometric properties of evaluation methods include the assessment of both reliability and validity of score interpretations from these instruments. Validity is often divided into distinct subcategories such as face, content, and criterion validity. Emerging conceptualizations consider construct validity as a term that encompasses all the validity subcategories.¹¹ In this paradigm, construct validity represents the degree to which score interpretations from the psychometric instrument are used to provide evidence to support or refute the underlying construct. The sources of evidence include the content (do the instrument items represent the intended construct), relations to other variables (correlations with the scores from other instruments), internal structure (acceptable reliability and factor structure), response process (relations between the underlying construct and the thought process of subjects), and consequences (whether assessment scores make a difference or not).¹¹ Face validity represents subjective judgement of how well the items represent appropriate operationalization of the measured construct based on its face value.¹² On the other hand, reliability or consistency of the scores from one test to another is considered a requisite for, but not sufficient evidence of validity. Generalizability theory provides an integrated framework for measuring the various forms of reliability. Although we previously reported some evidence of validity for the VITALS instrument,¹⁰ the other sources of evidence for validity including generalizability of the scores were not tested. After the widespread use of this instrument, especially in many other medical colleges in the Gulf region, we thought to provide further evidence of the psychometric properties of a study instrument. The findings from the current study could establish the evidence of its utility and stimulate further testing of the instrument in other contexts.

This study is conducted to answer the following research questions:

1. What is the reliability of student evaluation scores of clinical tutors using VITALS across raters and items?
2. What is the minimum number of student raters of clinical tutors that yields an acceptable reliability of the VITALS instrument?
3. What is the evidence of construct validity of the students’ evaluation scores of clinical tutors using the VITALS instrument?

Materials and Methods

Study Context

This is a retrospective study measuring the validity and reliability of students’ evaluation of clinical tutors using the VITALS instrument. The study was conducted at the College of Medicine and Medical Sciences, Arabian Gulf University (AGU) in Bahrain. The College of Medicine program of AGU is six years long. It is composed of three phases: Phase I (year 1), Phase 2 or pre-clerkship (years 2, 3 and 4), and Phase 3 or clerkship (years 5 and 6). This study was conducted in 2015 during the clerkship phase of the program, which is composed of hospital-based clinical rotations of the following major disciplines: Medicine, Surgery, Pediatrics, Obstetrics and Gynecology. It was approved by the “Research and Ethics Committee” of Arabian Gulf University. A total of 25 students evaluated each clinical tutor. Inclusion criteria for the reliability study are: clinical tutors teaching in the clerkship phase with complete evaluations and no missing values. Out of 100 clinical tutors teaching at the clerkship phase, 73 were eligible for the reliability study. This produced a total of 1825 complete evaluation forms. For the face validity study, 72 tutors responded from the 100 tutors questioned.

Description of the VITALS Instrument

We have previously demonstrated the development of the VITALS instrument.¹⁰ The evaluation form used by medical students to evaluate clinical tutors is composed of 10 items, with a 4-point Likert scale for each item. The maximum score is 4 and the minimum score is 1. All Likert scale item responses of “strongly disagree” or “disagree”, were combined as “disagree”, while all “strongly agree” or “agree” responses were combined as “agree”. Clinical tutors are informed of their respective collective ratings in the form of a horizontal bar graph showing the teacher’s strong points and weak points displayed at the end of the academic year.¹⁰

Face Validity of the VITALS Instrument

A questionnaire was distributed to tutors measuring their general acceptance of VITALS. The questionnaire was composed of 8 items. Each item was rated on a 1 to 4 response scale, where 1 = strongly disagree, 2 = disagree, 3 = agree and 4 = strongly agree. The items in the questionnaire were: 1) Tutors to be evaluated by medical students using VITALS. 2) Tutors should receive results of this evaluation. 3) Such evaluations are important to improve teaching. 4) Evaluations should be shown to academic administrators 5). Academic administrators should discuss this evaluation with clinical tutors. 6) This evaluation should inform decisions such as promotion. 7) It can be considered for tutors’ contract renewal. 8) Medical students can judge tutors.

Reliability of the VITALS Instrument

Reliability was measured using statistical models based on generalizability theory (G-theory). We have previously described in detail the assessment of reliability using G-theory analysis.^11,12 The object of measurement in the current study was the tutor’s evaluation scores by students. The design of the G-theory analysis included two facets (raters and items) with raters nested within tutors, and both crossed with items. We selected a design where items were fixed at 10, while the raters were considered random, as we were interested in generalizing the findings beyond the context used in this study. We used the generalizability coefficient (Φ), as we were interested in the relative inferences for inter-individual comparison of tutors’ performance. A G-study was conducted to estimate sources of error in the tutors’ ratings and to determine reliability of students with a different set of raters. A D-study was used to determine the reproducibility of these ratings, and to identify the minimum number of raters and items required to achieve reliability of ≥0.80. We have also reported the standard error of measurement (SEM) which represents the standard deviation of all the errors of measurement in the study.

The statistical software package GENOVA was used for the G-theory analysis. Data from the questionnaires from tutors and students analyzed using SPSS- version 16. Descriptive statistics were calculated based on the percentages of those who agreed and those who disagreed with each item in the form and in the questionnaire. Strongly agree and agree anchors were considered as “agree”, and strongly disagree and disagree were considered as “disagree”

Results

Generalizability Study (G Study)

The overall G-coefficient of the VITALS instrument scores across the two facets of the study (25 student raters and 10 items) was 0.89. In addition, the percent variance attributed to the study subjects (object of study) was 15.8%. Because of the nested design of the study, we were not able to determine the percent variance due to the rater’s facet. However, a large percent of variance (43.5%) was found due to the interaction between raters (students) and tutors. On the other hand, the percent variance due to items (0.2%) was negligible, and the variance due interaction between tutors and items was zero. This indicates that the relative ordering of tutors’ scores did not differ when tested on different items. Finally, the interactions between tutors, raters and items represented 40.5% of total variance. This component represents both the variance attributable to the three-way interaction, and the variance ascribed to source of error (facets) that were unmeasured in the study. Table 1.

Table 1 Generalizability Study Showing the Variance Components and Their Percentages for Tutors’ Evaluation by Students Using the VITALS

Decision Study (D Study)

Figure 1 demonstrates the predicted changes in the G-coefficient by the combined effects of increasing the number of items and raters. As shown in the figure, changing the number of items used in the evaluation form, does not appear to influence the G-coefficient. However, G-coefficient increased as number of raters (students) increased. With 5 raters, the G-coefficient was 0.631, at 10 raters it was 0.774, at 15 raters it increased to 0.837 and became 0.872 at 20 raters. Generalizability coefficient was increased to 0.91, if the number of raters is increased to 30, and reaching to as high as 0.923 with 35 raters.

Figure 1 An example of the application of visual indicators for teaching and learning success (VITALS) of a clinical tutor evaluated by students (Reproduced with modifications from Hamdy et al, 2001).

Evidence of Face Validity of VITALS

The response rate for the clinical tutors to their perception about VITALS was 72% (72 out of 100 tutors). Results indicated that 94% of tutors agreed with students’ evaluation of tutors using the VITALS; 97% of the responding tutors agreed that clinical tutors should receive their evaluation by medical students; 93% agreed that such evaluation is important to improve teaching; 90% agreed that an evaluation should be discussed with tutors; 89% agreed that the evaluation should be shown to administrators; 78% agreed that such evaluation can be considered for tutors’ promotion; 67% agreed that students are capable of judging their clinical tutors; and 65% agreed that the evaluation should be taken into consideration for tutors’ contract renewal. Figure 2.

Figure 2 Decision study (D study) results for the evaluation scores of clinical tutors (n = 73) using VITALS instrument. The D-study estimated the projected G-coefficient using different numbers of student raters (from 5 to 35 raters) and number of items in the VITALS instrument (from 5 to 20 items).

Discussion

This study is an extension to our previously published paper on developing and applying VITALS in reporting evaluations of clinical teachers.¹⁰ We have demonstrated in that study an evidence of content and face validity of the instrument by using a descriptive qualitative approach. In the current study, we evaluated the reliability and face validity of the instrument using a quantitative approach. We have demonstrated in this study that the VITALS instrument can be used for students’ evaluation of clinical tutors with a good reliability. The decision study indicated that the main determinant for the changes in reliability was the number of student raters, while the items of the instrument did not contribute to the variance. In addition, we showed that at least 12 students are required to achieve a dependable estimate of the evaluation of clinical tutors. Furthermore, the study revealed that a large percent of variance is attributable to unmeasured sources of error, which opens the venue to explore other facets in future studies. The majority of clinical teachers were positive and agreed with the importance of VITALS as an instrument for the evaluation of clinical teaching, demonstrating evidence of face validity. This could be explained as a subjective judgment about the appropriate operationalization of the measured construct.¹²

The variance ascribed to the subject of measurement (clinical tutors) indicates that averaging over raters and items, clinical tutors differed somewhat systematically in their evaluation scores. These findings indicate an acceptable degree of variability in students’ ratings of clinical tutors due to unsystematic sources of error. This relatively small percent of variance indicates that a larger percent of variance is attributed to different sources of error. Results of the present study demonstrated that students nested in tutors contributed 43.5% to variance components. Obviously, the nested model design of the study did not allow determining the percentage of variance imputed to differences between raters from the variance imputed to the interaction between raters and items independently.

The finding of the zero variance for the facet of item indicates that the score obtained on a particular item is representative of scores obtained on all similar items of the construct. In another way, it is an evidence of the internal consistency of the VITALS instrument. In addition, the interaction between items and raters was also zero, suggesting that the rank order of tutors did not change significantly across items, and there were small changes in rating behavior across items.

Gillmore et al, who used generalizability theory to study reliability of scores provided by students to tutors, found that increasing the number of students had a much greater impact on G-coefficient than increasing the number of items.¹⁵ Mazor et al used decision study to measure the effect of changing the number of items on generalizability coefficient. They also demonstrated that increasing the number of items from five to ten did not have significant impact on the generalizability coefficient.¹⁷ Interestingly, Mazor et al, who used the application of generalizability theory to students’ ratings of tutors, also found that the largest percentage of variance (45%) was associated with students (raters) nested within tutors. Furthermore, they have also found that relatively little variability (2%) was associated with the item facet, ie, students tended to rate a given tutor the same on all or most items.¹⁷

There are several published instruments for evaluation of clinical teachers with different types of evidence for their validity.^{5,7–9,17–25} The G-theory analysis indicated good reliability of the scores from Student Evaluation of Clinical Teaching Questionnaire (SETQ),²² the Maastricht Clinical Teaching Questionnaire (MCTQ),⁵ and The Cleveland Clinic’s Clinical Teaching Effectiveness Instrument.⁷ However, the advantage of the VITALS instrument is the relatively smaller number of items compared with previously published instruments, and the visual appeal of the provided information enhances its impact, and increases the likelihood that the information will be transformed into action.¹⁰

The results of the D-study indicated that at least 12 student raters are required to achieve an acceptable reliability of the VITALS instrument, irrespective of the number of items. Increasing the number of student raters from 5 to 12 students resulted in significant improvement to achieve an acceptable dependability coefficient of 0.8. However, any increase above 12 students did not result in striking changes in the dependability coefficient. This message is important for academic administrators when taking this evaluation into consideration for promotion or contract renewal. Previous studies have reported inconsistent findings regarding the minimum number of raters for achieving an acceptable reliability.^5,7,15,21 The number of raters of these studies ranged from a minimum of two or three in some studies,^7,21 seven raters in a different study,⁵ and fifteen raters in another study.²¹ Differences between these studies and the current one could be related to the structure of the study instrument, and the context of the study.

This study has some limitations that are worth reporting. Firstly, applying the VITALS instrument has been restricted to one medical school in the clerkship rotations. Further studies will be required to test generalizing the results of the current study to other medical schools. Secondly, although we have provided an evidence of reliability and face validity of the instrument, other sources of validity evidence such as predictive validity and criterion-related validity need to be determined. Future studies need to examine the relationship between other measures of performance for clinical tutors, and their evaluations using the VITALS instrument. Finally, the large percent of variance due to the unmeasured error raises the flag for future studies using larger sample sizes to examine other relevant facets in the G-theory analysis. One important facet could be the occasion where temporality of the evaluation scores using the VITALS could be tested.

Conclusion

Results of this study provide an additional evidence to the construct validity of using VITALS for clinical tutors’ evaluation by medical students in the clerkship phase. We demonstrated an evidence of reliability of the VITALS instrument taking into consideration the errors due to raters and items. In addition, there should be at least 12 students rating each clinical tutor in order to have an acceptable level of reliability for the study instrument. Finally, we have provided a quantitative evidence of face validity for the study instrument by the high level of agreement for using the instrument by clinical tutors. Results of this study may guide administrators when using evaluation results in supporting certain decisions such as promotion, retention, identification of tutors deserving recognition for teaching excellence, and general documentation of teaching quality. Additional studies are required before generalizing the use of this instrument in other medical schools.