Prediction of Admission Tests for Medical Students’ Academic Performance

Introduction

There is a plethora of information on the admission process in medical schools. Finding a suitable student candidate has become more scrupulous than ever before. The competition is high due to the increasing number of outstanding applicants and the continuous struggle by students to maintain their standards high and competitive. The admission criteria have been critiqued, and there is a need for re-defining the pre-admission factors that are thought to be obsolete and weak in assessment.^1–3

Before admission, there is a worldwide discussion over academic and non-academic assessment tools. Academic assessment tools focus on the student’s perception, comprehension, and, most importantly, their depth of scientific knowledge.⁴ This category of assessment is considered the most reliable tool in predicting student’s academic performance in medical schools.^5–10

When it comes to selecting medical students, there is a general understanding of the importance of non-academic assessment tools.^10–13 These tools cover a broad scope of capabilities vital for success in medicine, especially during the clinical years, such as problem-solving, critical thinking, and quantitative prowess.¹⁰ The examples of non-academic skill assessment tools include the United Kingdom Clinical Aptitude Test (UKCAT) and the Undergraduate Medicine and Health Sciences Admission Test (UMAT).

In Saudi Arabia, different factors are considered in the student selection process, including scores from the cumulative high school average, the Standard Achievement Admission Test (SAAT), the General Aptitude Test (GAT), and an English language proficiency exam. The goal is to consider the academic and non-academic skills of the candidate as a predictive tool for assessing their suitability as medical students.

The GAT is composed of verbal and quantitative sections; it aims to assess the reading and comprehension abilities of the candidates.¹⁴ It further helps assess the candidate’s understanding of logical relationships, problem-solving skills, and cultural competence. However, there is uncertainty over the validity of such predictive tools for assessing non-academic skills.^1–3 SAAT is utilized as an assessment tool to apprehend the cumulative knowledge obtained throughout the last three years of high school in five subjects: physics, chemistry, mathematics, biology, and English.^15,16

The Cumulative Grade Point Average (cGPA) is globally considered a college performance indicator. It is the most valid and reliable tool for assessing a candidate’s college performance in university settings.^17–19 However, this has not been confirmed as a standard practice for medical colleges as relying on cGPA scores may have limitations because of grade inflation and institutional differences in grading.¹⁸ Be that as it may, by now, there has not been any better alternative for assessing student performance than cGPA. Therefore, it is now the most popular tool in academic research to correlate the pre-admission variable with the candidate’s college performance.¹⁸

This research article aims to test the predictive validity of GAT and SAAT for medical students’ performance in their first two years at a Saudi medical school. This study is significant since, unlike other Saudi studies, it focuses on comparing the prediction of both tests between the two years and examining the difference in students’ performance in basic and advanced science courses.

Research Questions and Objectives

The study looks at the following questions:

Do both GAT and SAAT predict medical students’ Grade Point Average (GPA) in the first and second years? If so, Is there any difference in terms of prediction strength for both years?

Do both GAT and SAAT predict students’ medical students’ academic performance in basic and advanced science courses? If so, is there any difference in terms of prediction strength for both science categories?

Materials and Methods

The study participants were 650 students at a Saudi medical school from 2018 - to 2019. Students take intermediate English courses and basic science courses in biology, chemistry, and physics during their first year. They take advanced science courses in the second year that include medical terminology, behavioral science, biochemistry, biological statistics, and medical computer science.

Data of GAT and SAAT grades and their GPA in the first and second years were obtained from students and validated with their college ID numbers. The data were obtained directly from the college records. There is no selection here, as we take the whole population of both years at this university.

As illustrated in the next section, advanced statistical tools in R were used to analyze the data. In addition, an explanation will be provided in detail on the statistical method we used in this study, how we test the prediction of GAT and SAAT for the first and second-year GPA, and test if there is a significant marginal comparable trend. Also, we would talk about the prediction of GAT and SAAT for students’ performance in basic and advanced science courses and if there is any difference between both.

Results

GAT and SAAT Prediction of Students’ Performance in First and Second Year

A multivariate multisensory regression analysis was applied to examine the relationship between first-year GPA and second-year GPA performance scores and GAT and SAAT test scores. Both GPA scores were simultaneously entered as dependent variables (DV), and GAT and SAAT score was entered as simple predictors with no interaction term. Both dependent variables were continuous and were significantly skewed: the residual analyses of their regression models indicated non-normality, which was visually assessed using QQ-plots and histograms. Therefore, these variables were log-transformed. Multicollinearity was assessed using the vif.lm function in R (Heiberger, 2018), and all VIF values were below 2 implying no serious multicollinearity. For the regression models, a Breusch-Pagan test for heteroscedasticity was conducted using the function bptest (lmtest package; Zeileis & Hothorn, 2002). This test was significant (BP(2) = 215.66, p < 0.001), that is, signaling significant heteroscedasticity. Consequently, we estimated corrected standard errors for the model using the heteroscedasticity-consistent covariance matrix estimate via vcovHC (sandwich package; Zeileis, 2004; Zeileis, 2006). Note also that in cases where heteroscedasticity was detected, the models with and without the heteroscedasticity consistent covariance estimate yielded similar p-values.

As a result, we found that the model was overall significant for first year GPA (R2 = 27.41; F(2, 657) = 124.1, p < 0.001). Specifically, as in Table 1, the SAAT was a significant predictor of first year GPA (ß = 0.053, SE = 0.0055, t (657) = 9.63, p < 0.001). Similarly, the GAT was a significant predictor of first year GPA (ß = 0.034, SE = 0.0056, t(657) = 5.98, p < 0.001).

Table 1 The Prediction Extent of GAT and SAAT of the First Year GPA

For the second year GPA, the model was also overall significant (R2 = 28.9; F(2, 657) = 133.5, p < 0.001). The SAAT, as in Table 2, was a significant predictor of second year GPA (ß = 0.059, SE = 0.0054, t(657) = 10.91, p < 0.001). Similarly, the GAT was a significant predictor of second year GPA (ß = 0.029, SE = 0.0056, t(657) = 5.15, p < 0.001).

Table 2 The Prediction Extent of GAT and SAAT of the Second Year GPA

To further examine whether the strength of the relations of the two predictors (GAT and SAAT) were comparable between the two dependent variables (first-year GPA, second-year GPA), we have first-run seemingly unrelated regressions (to account for the correlated errors across the models) using the “systemfit” library in R. In the subsequent steps, we have used the “multcomp” library in R to test the coefficient contrasts directly. As a result, we found that there was a significant difference in first-year GPA vs second year GPA predictions by SAAT (ß = −0.0054, SE = 0.00261, z = −2.073, p = 0.038). On the other hand, there was only a non-significant trend indifference in first-year GPA vs second year GPA prediction strength for GAT (ß = 0.0050, SE = 0.0028, z = 1.799, p = 0.072). In short, while SAAT was a slightly better predictor of GPA in the second year, the GAT was a better predictor in the first year, albeit this effect showed only a marginal trend. Note also that both SAAT and GAT were overall significant and reliable predictors of GPA’s in both years 1 and 2.

GAT and SAAT Prediction of Students’ Performance in Basic and Advanced Science Courses

In different analyses, we examined whether SAAT and GAT performance scores could reliably predict students’ grades in basic science courses (biology, chemistry, and physics) and advanced science courses (medical terminology, behavioral science, biochemistry, biological statistics, and medical computer science). Note that the grades are provided in a letter format (A, B, C, and D). Therefore, to be able to average the grades across different courses in each DV category (basic science and advanced science), we have first converted the letter notations into numbers (correspondingly, 4, 3, 2, and 1) and then created a mean of the corresponding grades per DV category.

Since the average scores were now an average of multiple values, we treated both DVs as continuous. We found that they were both significantly skewed as assessed via the residual analyses of their regression models. Therefore, these variables were also log-transformed. The Breusch-Pagan test for heteroscedasticity was again significant (BP (2) = 232.2, p < 0.001). Therefore, we estimated corrected standard errors for the model using the heteroscedasticity-consistent covariance matrix estimate via vcovHC (sandwich package; Zeileis, 2004; Zeileis, 2006).

As a result, we found that the model was overall significant for the basic science (R2 = 27.4; F (3, 657) = 124, p < 0.001). Specifically, as in Table 3, the SAAT was a significant predictor of basic science (ß = 0.087, SE = 0.0074, t (657) = 11.69, p < 0.001). Similarly, the GAT was a significant predictor of basic science (ß = 0.037, SE = 0.0084, t (657) = 4.37, p < 0.001).

Table 3 The Prediction Extent of GAT and SAAT of Basic Science

For the Advanced Science, the model was also overall significant (R2 = 22.4; F (3, 657) = 94.9, p < 0.001). The SAAT, as in Table 4, was a significant predictor (ß = 0.063, SE = 0.0062, t (657) = 10.02, p < 0.001). Similarly, the GAT was a significant predictor (ß = 0.020, SE = 0.0068, t (657) = 3.00, p < 0.01).

Table 4 The Prediction Extent of GAT and SAAT of Advanced Science

Finally, and similarly to the analysis of GPA scores, we tested whether the strength of the relations of the two predictors (GAT and SAAT) were comparable between the two dependent variables (basic science, advanced science) by means of, first, running seemingly unrelated regressions and subsequently testing the coefficient contrasts. As a result, we found that there was a significant difference in basic science vs advanced science predictions by SAAT (ß = 0.024, SE = 0.0054, z = 4.47, p < 0.001). Additionally, there was a significant difference in basic science vs advanced science prediction strength for GAT (ß = 0.0163, SE = 0.0058, z = 2.79, p < 0.01). These analyses revealed that both SAAT and GAT were stronger predictors of Basic Science course performance relative to Advance Science course performance. Nevertheless, both SAAT and GAT were significant overall predictors of course performance.

Discussion

The results showed that both GAT and SAAT significantly predict medical students’ GPAs with sufficient strength (ie, R2 =27% and 28% for the first and second years). It also revealed that if students’ SAAT increases, 5–6% of students’ GPAs’ will increase in the first and second year correspondingly, while 3% of the GPA will increase if students’ GATs increase in both years. On the one hand, these results might probably be explained as the GAT examination’s content aims to assess the students’ reading and comprehension abilities, which could be less related to the students’ performance in the first and second years at the Saudi medical school. On the other hand, the SAAT tool is used to apprehend the cumulative knowledge in English subjects and scientific subjects such as physics, chemistry, and biology, which are very close to medical students’ learning in their first and second years of Saudi medical school.

This part of our results showed an agreement with the general literature consensus that the National Achievement test scores, such as the GAT and SAAT, are a significant predictor of students’ performance at college in Saudi medical schools, i, e., like many studies, those test scores was even considered the best predictor of the pre-admission variables.^15–24

Furthermore, both GAT and SAAT are significant predictors of students’ academic performance in their basic and advanced science courses with an acceptable strength (ie, R2= 27% for the basic science course, and R2= 22% for advanced science course). This finding suggests that 8% of the student’s GPA is expected to increase if they have high SAAT scores, and only 6% of the student’s GPA is expected to increase if they have high SAAT scores in the advanced science course. The results could be attributable to the basic science course that students learned during the first year of medical school, similar to what they were preparing for in the SAAT exam. This finding is not a surprise as it is in line with other studies conducted in Saudi Arabia and UAE 15–20 that indicated the students’ performance at this level in the medical school is more advanced than the basic science course.

Generally, this study’s results indicate that SAAT is a better predictor of GPA in both the first and second years of medical school and the basic and advanced science performance. Nevertheless, results suggest that GAT and SAAT may significantly predict students’ GPA during the first two years of medical school study.

Conclusion

A composite of pre-admission variables (ie, GAT and SAAT) and the English language proficiency exam score are prerequisites for entering medical schools in Saudi Arabia. This study aimed to test GAT’s and SAAT’s predictive validity for medical students’ performance in their first two years at a Saudi medical school. This study is significant since, unlike other Saudi studies, it also examines the difference in students’ performance in basic and advanced science courses. The study showed that 14% of the students’ GPA would increase if the SAAT and GATs increased in both years. Similarly, it is expected that 14% of the student’s GPA in both basic and advanced science courses was expected to increase if the students’ had high SAAT scores. Also, 5% of the student’s GPA will increase in science courses if the students’ GATs increase. So we can say that only 33% of the students’ GPAs and performance in basic and advanced sciences could be predicted by the students’ GATs and SAAT.

So we can say that the student’s GPA would increase by 33% in the first two years in medical school and both science courses if the SAAT and GATs were high. These results raise the flag that other factors that could affect the medical students’ GPAs and performance should be investigated and discovered to improve the student’s performance. Thus, further studies could be conducted to shed light on this area and explore the best possible pre-admission variables that positively affect the performance of medical students.