Use of cross-institutional progress test as a predictor of performance in a new medical college

Introduction

Progress testing is a form of longitudinal examination that evaluates the complete domain of knowledge considered a requirement for medical students upon graduation during the medical school years.¹ There has been increasing use of progress testing among medical schools.² Progress testing is an approach to evaluate knowledge acquisition and learning outcomes.³ It is used throughout the curriculum to test student performance and learning. It is believed that what is asked in the examination drives what students learn.⁴ In 1991, progress testing was introduced at McMaster University.⁵ The test is administered to all classes in the medical school three times a year, contains questions about the population’s perspectives, behavior, and biology, and is 3 hours in duration.

All students of the medical college are enrolled in the progress test. First-year students are expected to answer a small percentage of questions compared to senior medical students. Progress testing allows for monitoring of the rate of acquisition of knowledge among students at the same college. It also evaluates the performance of students at a medical college with students from other medical colleges.^6,7 There has been a move to use progress tests for cross-institutional comparisons.^7,8 One of the advantages of cross-institutional progress testing is that it creates opportunities for comparing curricular effectiveness and identifying problem areas in medical schools.⁶ Despite the pitfalls of interinstitutional progress testing, the expected benefits are quality improvement, comparing curricular effectiveness, and identifying problem areas in schools.

In 2010, Qassim College of Medicine in Saudi Arabia introduced progress testing as a tool for enhancing and monitoring educational process in medical colleges in Saudi Arabia. Qassim University College of Medicine invites medical colleges in Saudi Arabia to contribute in progress testing. All medical colleges sits the same test together at the same time. The individual progress of each student is analyzed throughout the curriculum, and each student is given written feedback on their progress. This feedback enables students, as well as staff, to monitor their progress.¹

The College of Medicine at Princess Nourah Bint Abdul Rahman University (PNU) was founded in 2012. The college implemented the same innovative reformed medical curriculum of King Saud University (KSU). The development of any new medical school is a challenge in many aspects, from running the curriculum to the establishment of an assessment system and evaluation of the students’ performance. It is important for any new school to provide evidence to the students and faculty members that the new curriculum is producing sufficient learning outcomes. To meet this challenge, the idea of using progress testing as an indicator for learning acquisition was important. To date, students at PNU have taken the progress test every year for three years: 2012–2014. In 2010, the College of Medicine at KSU revised the curriculum to meet the international standards of medical education. The reformed curriculum is a system-oriented, student-centered, integrated, community-oriented curriculum. The medical program is a 5-year curriculum, with the first 2 years of basic sciences focusing on nine blocks, based on systems and divided into four semesters. The third year is a transition between basic sciences and the 2 years of clinical clerkship. During the first 2 years, the courses share full vertical and horizontal integration of basic science subjects: anatomy, physiology, and biochemistry with pathology, microbiology, pharmacology, and a few relevant topics of community medicine according to the themes of weeks.

The aim of the present study was to use the progress test to evaluate the rate of knowledge acquisition among a new medical school compared to other long-established medical schools in Saudi Arabia. KSU and PNU medical colleges are implementing the same innovative reformed curriculum.

Materials and methods

The study was approved by the ethical committee at the College of Medicine in PNU. The progress test is composed of 200 single best-answer multiple-choice questions targeting core knowledge in basic and clinical sciences that medical graduates are expected to have. The exam blueprint considered various body systems, medical disciplines, and processes.

The progress test was conducted in February 2015 as a multicentric exam at all participating medical colleges in Saudi Arabia. All medical students needed to participate in the test. The students were encouraged to participate by receiving a 5-mark bonus to their grades. The exam was prepared by Qassim College of Medicine. It was a paper- and -pencil test lasting 4 hours.

Thirteen medical colleges in Saudi Arabia participated, with a total of 4,252 students. A total of 177 students from PNU College of Medicine participated in the progress test: 64 from the first year, 62 from the second year, and 51 from the third year (Table 1).

Table 1 Student participation in the progress test in 2015 Abbreviation: PNU, Princess Nourah Bint Abdul Rahman University.

Results

Of the 4,252 students from 13 medical colleges in Saudi Arabia participating in the progress test in 2014, 177 (4.2%) students from PNU participated. Figure 1 shows the average yearly progress in knowledge of medical students throughout their education. The results showed that there was a yearly increase in the level of knowledge among medical students in Saudi Arabia. Table 2 demonstrates the average percentage total score per year for each medical college that participated in the progress test in Saudi Arabia.

Figure 1 Average yearly progress among medical education years for all students participating in the progress test in 2014 in Saudi Arabia.

Table 2 Average percentage total score per year for each medical college that participated in the progress test in Saudi Arabia Abbreviation: PNU, Princess Nourah Bint Abdul Rahman University.

Figure 2 shows the progress test results for PNU students with the other 12 medical colleges in Saudi Arabia in 2014. The figure shows the mean scores of students in the 5 years of medical school and the results of PNU students of years 1, 2, and 3. We compared the mean scores of students at PNU with students from other medical colleges. It is apparent that the acquisition of knowledge of PNU students was similar to peer students at years 1, 2, and 3. The results showed that the level of knowledge acquisition was similar for PNU students compared to other medical schools in Saudi Arabia.

Figure 2 Mean scores of the progress test of students from medical colleges participating in the progress test in Saudi Arabia. The PNU arrow is pointing to the yellow line. Abbreviation: PNU, Princess Nourah Bint Abdul Rahman University.

Discussion

Progress-test experience in Saudi Arabia shows that it is possible to deliver a progress test across multiple medical institutes. Previous studies have demonstrated that it is possible to deliver the progress test to multiple medical colleges.^8,9 The analysis we have presented provides reassurance that students at PNU are performing as expected compared to students at other medical colleges in Saudi Arabia.

Enrolment of PNU students in the progress test has benefited PNU in several aspects. At the student level, it has reassured students about their knowledge-acquisition level in comparison with peers across Saudi Arabia, according to results of the feedback collected from students regarding the progress test. At the PNU level, it has provided faculty members with a measure of curriculum effectiveness. It has shown up weak students that need support. Student Affairs consults students for support and mentoring based on low progress-test scores. It has also provided accrediting bodies with a tool to compare students’ performance and quality of curriculum implemented across medical colleges. There is evidence that the progress test was accepted by students as a means of measuring growth of knowledge without any apparent adverse effects.⁵

Limitations

Differences in student and teacher experience can create differences in the results. In addition, the student populations of the medical schools in Saudi Arabia are almost homogeneous, because the Ministry of Education determines the admission criteria to Saudi medical colleges.

Another major limitation of the study was the small number of students in PNU and probably some of the other comparative medical colleges. Cooperation on progress testing was variable. Some medical schools encouraged students to participate with a reward of bonus marks. The progress test is a test for which students do not study, leading to lower-than-anticipated scores.

Conclusion

The present study demonstrated that the progress test provides a measure of reassurance to students, faculty, and accredited bodies regarding the level of knowledge acquisition among different medical schools. The study showed that the interinstitutional progress test demonstrated that the level of acquisition of knowledge and performance of students in a new medical school was similar to other medical colleges in Saudi Arabia.

Acknowledgment

The authors acknowledge the faculty of Qassim University Medical College who assessed in the development and implementation of the progress test.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	McHarg J, Bradley P, Chamberlain S, Ricketts C, Searle J, McLachlan JC. Assessment of progress tests. Med Educ. 2005;39(2):221–227.
2.	Wrigley W, van der Vleuten CP, Freeman A, Muijtjens A. A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide no 71. Med Teach. 2012;34(9):683–697.
3.	Verhoeven BH, Verwijnen GM, Scherpbier AJ, van der Vleuten CP. Growth of medical knowledge. Med Educ. 2002;36(8):711–717.
4.	Newble DI, Jaeger K. The effect of assessments and examinations on the learning of medical students. Med Educ. 1983;17(3):165–171.
5.	Blake JM, Norman GR, Keane DR, Mueller CB, Cunnington J, Didyk N. Introducing progress testing in McMaster University’s problem-based medical curriculum: psychometric properties and effect on learning. Acad Med. 1996;71(9):1002–1007.
6.	van der Vleuten CP, Schuwirth LW, Muijtjens AM, Thoben AJ, Cohen-Schotanus J, van Boven CP. Cross institutional collaboration in assessment: a case on progress testing. Med Teach. 2004;26(8):719–725.
7.	Muijtjens AM, Schuwirth LW, Cohen-Schotanus J, Thoben AJ, van der Vleuten CP. Benchmarking by cross-institutional comparison of student achievement in a progress test. Med Educ. 2008;42(1):82–88.
8.	Finucane P, Flannery D, Keane D, Norman G. Cross-institutional progress testing: feasibility and value to a new medical school. Med Educ. 2010;44(2):184–186.
9.	Kerfoot BP, Shaffer K, McMahon GT, et al. Online “spaced education progress-testing” of students to confront two upcoming challenges to medical schools. Acad Med. 2011;86(3):300–306.