University Admission Test Associates with Academic Performance at the End of Medical Course in a PBL Medical Hybrid Curriculum

Introduction

The selection process of students to gain access to a medical course is complex not only for students, but also for the faculty, society and health system as a whole. For students, the entrance in a medical course might represent the fulfilment of a dream. For the faculty, the challenge is to select the best students to practice medicine in terms of knowledge, skills, and professional attitudes to ultimately become a good doctor. For the society, there is expectance to be assisted by a doctor professionally competent and with sympathy for people. For the health system, the future doctors should be committed with the principles and guidelines established for care management.

Multiple-choice questions (MCQ) have been used to assess students’ declarative knowledge in the university admissions test (UAT) to medical schools, which appears to reliably predict academic achievement of medical students.^1–5 The pre-university grade point average analysis (GPA), a useful tool within a general domain of cognitive measures, is also used in the selection process of students to gain access to the medical course.⁶ Another type of admission test in the cognitive domain is a written sample examination in addition to MCQ; this essay is associated with academic performance in the clerkship at the end of the medical course.⁷

An admission test may also include an interview in workstations, similar to that seen in the Organized Structured Clinical Examination (OSCE), which assesses interpersonal skills, ability to communicate, and performance in traditional interview questions, the so-called Multiple Mini-Interview (MMI).⁸ This type of preadmission assessment seems to predict performance in either the clerkship in a medical course and/or national license examination following medical graduation.⁹ Other forms of tools used in the selection process of medical students rely on an autobiographical letter, a personal interview, a letter of support, and a personal statement.⁶

It should be pointed out that the majority of studies assessing the value of the admission test to predict academic performance at the end of medical course were carried out on lecture-based medical courses. It remains, therefore, controversial whether UAT could predict academic achievement in medical education, particularly in a medical school using a Problem-based learning (PBL) hybrid curriculum approach, as is the case in our medical course.

Accordingly, the aim of this study was to correlate marks in the UAT focusing on declarative learning with those obtained in the OSCE, in the progress testing (PT), and in the curricular units of the clerkship to evaluate the association of performance in admission tests with academic achievement at the end of a medical course in a PBL medical hybrid curriculum.

Materials and Methods

The Selection Process

The UAT for students gaining access to the medical course at our institution consists of 70 MCQ, as well as an essay, and it lasts 4 hours. MCQs have been used because they would appear to discriminate accurately high achievers from under achievers.¹⁰ Each question has five alternatives, since MCQs with three options have a low reliability and are associated with a low difficulty index.¹¹ The items include Biology, Portuguese Language, Mathematics, Geography, General and Brazilian History, Chemistry, and Physics. To entry medical school, a student has to score at least 40% of the total in the essay. A formula developed by our university is then used to compose the final mark by joining together the MCQ marks with the marks obtained in a National Highschool Exam. The final marks are organized in descending classificatory order, which determines whether students are appropriate for the course. In 2009, the applicants who scored at least 48% in the UAT joined our medical school.

We used data from all 48 students who were regularly matriculated last semester (12th stage) of the medical course in 2015. The only inclusion criteria was to be regularly matriculated in the 12th stage after having passed the 11th stage. The study was performed according to the principles outlined in the Declaration of Helsinki. Every precaution was taken to protect privacy of research subjects and the confidentiality of their personal information. Accordingly, the data from students’ marks was not identifiable, since the material was supplied by the Division of Academic Registers, whose members did not take part in the investigation. The authors did not have access to the students’ names and their marks were analyzed anonymously. Consequently, the Research Ethical Committee of our university waived the need for written informed consent, and approved the development of the study.

The Medical Course

Details of our medical course have been reported elsewhere.^12,13 Briefly, the pre-clinical stage of the medical course at our institution runs from the 1st to the 8th stage (the first four years of the medical course), whereas the clerkship runs from the 9th to 12th stage (the last two years of the medical course). In the pre-clinical stage, the curricular units are Tutorial, Medical Skills, and Primary Care.¹² The PBL method is used to run Tutorials. The Medical Skills are run in the Skills laboratories or at hospitals or specialized ambulatories seeing real patients. Primary care discusses cases obtained from real patients. Therefore, the PBL method is used during the preclinical stage.

In the clerkship, students rotate in Internal Medicine, Gynecology and Obstetrics, Surgery, Pediatrics, Family Medicine, and Emergency Medicine. At least 80% of the medical school workload is spent attending patients under the supervision of a preceptor/teacher; about 20% of the time is devoted to seminars, and a few lectures on the most prevalent subjects of each curricular unit are given. Since we have to follow the national guidelines for medical courses,¹² the PBL teaching approach cannot be used in the clerkship. The final mark in the clerkship (FMC) is made by averaging the means of each curricular unit, as outlined above. In each subject, the final mark is obtained by the mean of the written test at the end of each curricular unit, the marks obtained in the OSCE, the marks obtained in the PT, and the summative assessment provided by preceptors.

The Assessment Process in the Medical Course

The students’ assessment focuses on declarative and procedural knowledge, skills, and attitudes during the entire medical course. In Tutorial sessions, learning is assessed by MCQ before and after the reporting phase of each session.¹³ Furthermore, attitudes are evaluated by a formative assessment, given by the tutor after the end of each session based on a pre-specified list of attitudes.¹⁴ Formative assessment is also provided in Medical Skills and Primary Care based on a standardized instrument developed by our group.

The Objective Structured Clinical Examination (OSCE)

The OSCE allows assessment of medical student clinical skills and also a fair and particular evaluation of individual abilities. The students are evaluated twice a year by the OSCE since the end of the second year (the 4th stage) of the medical course. Each OSCE test has six questions (clinical stations) performed sequentially about important medical topics. Specialists on the subject at hand elaborate all the OSCE stations, which are revised by an experienced committee with expertise on the subject. Each OSCE station score is 70% objective (“item evaluation”) and 30% subjective (“general evaluation”).

The Progress Testing

The PT consists of a longitudinal learning assessment of all medical students of our medical course and adapted from the one originally described at Maastricht University.¹⁵ The PT consists of 120 MCQ, usually Key-Feature items, encompassing all subjects studied during the medical course. The same cluster of items must be answered by all students at the same time; this means that a student from the first stage answers the same questions as a student from the 12th semester. Therefore, the PT is important for students monitoring their own pace during the construction of the learning process, stage by stage, throughout the medical course. In addition, it is also valuable for the faculty to monitor the adequateness of curriculum implementation.

Statistical Analysis

A single dependency study was performed using Pearson test to assess bivariate correlation between UAT and PT, UAT and OSCE, and UAT and FMC. Additionally, multiple linear regression was used to determine multiple correlation among UAT x FMC x PT, UAT x OSCE x PT and UAT x OSCE x PT. Finally, a multiple dependency study was performed to assess multiple correlation between all variables, in which UAT marks were regarded as the dependent variable and OSCE, PT, and FMC marks as the independent variables. The regression analysis tool performs a linear regression analysis using the least squares method to fit a line into a set of observations. Thus, we can analyze how a single dependent variable is affected by the values of one or more independent variables. ANOVA test was used for multiple linear regression and p-value < 0.05 was considered significant in all circumstances.

Results

Table 1 summarizes the findings of the single dependency study. No correlation was found between marks in the UAT and PT (r=0.10; p=0.47), and in the UAT and OSCE (r=−0.20; p=0.17). However, a positive correlation was observed between marks in the UAT and FMC (r=0.34; p=0.02).

Table 1 Bivariate Correlation Between UAT and PT, UAT and OSCE, and UAT and FMC

In the multiple linear regression analysis between three variables, a positive correlation was observed among UAT, FMC, and PT marks (R=0.36; p=0.04, Figure 1). The same occurred with regard UAT, OSCE and FMC marks (R=0.43; p=0.01, Figure 2). However, no correlation was found between UAT, OSCE, and PT marks (R=0.22; p=0.32, Figure 3).

Figure 1 Projection of the multiple linear regression among UAT, FMC, and PT marks. Abbreviations: UAT, university admission test; FMC, average final marks in the clerkship; PT, progress testing.

Figure 2 Projection of the multiple linear regression among UAT, OSCE and FMC marks. Abbreviations: UAT, university admission test; FMC, average final marks in the clerkship; OSCE, Organized Structured Clinical Examination.

Figure 3 Projection of the multiple linear regression among UAT, OSCE, and PT marks. Abbreviations: UAT, university admission test; OSCE, Organized Structured Clinical Examination; PT, progress testing.

The results of the multiple dependency study are shown in Table 2. In this analysis, when all variables were used in the multiple linear regression, in which UAT marks were regarded as the dependent variable and OSCE, PT, and FMC marks were regarded as the independent variable, a higher R-value was obtained (R=0.46; p=0.01). This indicates that greater correlation is achieved when FMC, OSCE and PT are all taken into account, suggesting that UAT correlates with students’ global performance at the end of medical course.

Table 2 Results from ANOVA Test for Multiple Linear Regression, Showing the Relationship Between Marks in the UAT and the Independent Variables PT, OSCE and FMC

A possible relation between the essay scores and the independent variables mentioned above was also evaluated in this study. No correlation was found among the essay scores and PT, FMC, and OSCE scores (R=0.30; p=0.24).

Discussion

This study shows that performance in UAT correlates with academic achievement at the end of a medical course in a PBL medical hybrid curriculum. Particularly, UAT had positive correlation with FMC in the bivariate study. However, in the multivariate analysis, all final assessments performed in our PBL curriculum (FMC, PT, OSCE) had a contribution in the correlation with UAT.

The role of an admission test based on items of natural science knowledge to predict educational attainment during a medical course has produced conflicting results. Meyer et al¹⁶ did not find any association between marks on admissions test and academic achievement of students during the first three semesters of the medical course. Similar results were observed by Bodge et al¹⁷ who failed to show any prediction of admissions test on both cognitive and non-cognitive domains. In addition, admissions test on cognitive domain did not predict academic performance in second-year medical students.¹⁸

On the other hand, marks in the UAT (Medical College Admissions Test, for example) have been associated with medical school performance during the first 2 years of the medical course.¹⁹ A recent study by Yang et al²⁰ showed that admission test, taking into account cognitive ability, predicted academic achievement of first-year medical students. Furthermore, Patterson et al²¹ observed that when biology, physics and chemistry are included in the items of the selection procedure, aptitude tests appear to be effective. Our findings, therefore, reinforce the value of an admissions test, based on natural science items in predicting academic performance at the end of a medical course. Furthermore, our results expand this view by showing that marks in the UAT are associated with final academic performance, mainly in curricular units of clerkship at the end of a medical course.

Studies evaluating the association between the UAT and in-course OSCE scores in the earliest years of medical school have shown controversial results. While some studies have reported a relationship between the UAT and the OSCE second-year medical students scores,^6,22 others have failed to find the same result.^17,23 Also, in the pharmacy course, the correlation between the UAT and OSCE performance was weak. Indeed, it occurs in the non-cognitive domain (interviewing skills) of the OSCE.²⁴ Although our study also shows that there is no relationship between UAT and OSCE scores in the bivariate analysis, OSCE marks had a small contribution in the multivariate correlation with other variables.

The PT is a reliable method to measure the students’ knowledge acquisition during the medical course, not only in terms of students’ progress compared with their colleagues, but also regarding the longitudinal knowledge gain expected by the faculty.¹⁵ Little is known about the association between marks in admissions tests and in the PT at the end of a medical course. Dabaliz et al²⁵ showed that performance in the PT in the preclinical stages of a medical course was not predicted by five variables of the admissions test.²⁵ In our single dependency study, no correlation between marks in the UAT and in the PT performed at the end of the medical course was observed. However, similarly to OSCE, PT scores helped to increase the correlation between the variables when multivariate analysis was performed. This indicates that pre-university students with higher declarative learning at medical admission would also have a higher academic achievement at the end of the medical course.

The striking feature of this investigation was the association between variables of the UAT with those of the clerkship at the end of our medical course (FMC). Such association is scanty and controversial in current literature. Students who scored higher in the UAT do not perform better in the clerkship compared with those who scored lower.²⁶ Silver and Hodson²⁷ analyzed the relationship between declarative learning in the admissions test and students performance in the clerkship and did not find any association between these variables. By contrast, a study performed on 59-minority student’s ie, Black/Hispanic with parents with low educational level, found a positive correlation between marks in the admissions test and total marks in the clerkship.²⁸

Our study also showed a significant correlation between marks in the UAT and FMC. This is interesting because the FMC in our institution depends on a learning test that represents only 40% of the final mark; the other 60% is accounted for by the marks obtained in the OSCE, in the PT, and in the formative assessment given by preceptors. Therefore, our data suggest that the student who scores higher in the UAT is also able to better acquire competence in problem-solving, clinical reasoning, and communication skills, which are the mainstay of clinical competence in the clerkship.

It should be pointed out that in this investigation the essay, as part of the admission test, did not predict academic performance at the end of the medical course. In physical therapy, the scores obtained in the admissions essay were able to independently predict students’ academic achievement at the end of a two-year professional program.²⁹ A study involving more than one thousand medical students demonstrated that those who scored higher in the admissions essay performed better in the clerkship.⁷ However, another study enrolling 1162 medical students failed to show a correlation between the admissions essay and clinical performance in the clerkship.³⁰

It is difficult to explain why a written sample of the admissions test did not predict clinical competence in the clerkship at the end of the medical course. The essay assesses the student’s ability to organize thoughts, to present ideas in a cohesive way, and to give evidence regarding analytic thinking.⁷ Such characteristics are of utmost importance for students to acquire problem-solving skills, which are essential to master clinical competence mainly in the clerkship. In general, it is believed that the essay measures factors that are not related to scientific achievement.⁷

The main practical implication of the results of our study is that a selection process made of items on natural science, mainly biology, physics, and chemistry correlate with academic performance at the end of a medical course in terms of abilities, attitudes, and knowledge. This is important because such a selection process fits our Pre-University educational level, in which PBL is not used, and it is cost-effective. Whether the MMI, for example, will perform better than our admissions test remains to be determined.

Another important practical implication of this investigation is about the need of the essay in admissions test of a medical course in our educational environment. Our results have shown that the essay did not associate with academic performance at the end of medical course. Therefore, the usefulness of an essay as part of an admissions test to a medical school, at least in our educational reality, seems to be low. Further studies will be necessary to fully determine the necessity of an essay in the student’s selection process to a medical course.

This study had some limitations. We used data related to skills (OSCE) and knowledge domain (PT), but we did not use data regarding other non-cognitive domains, which might have a positive impact on the applicants selection,²¹ particularly that related to narcissism, which has been associated with poor academic performance in terms of knowledge and skills at medical school.³¹ Also, because of logistical reasons and cost, we could not use contextualized instruments of assessment, such as the MMI and Situational Judgement Tests, which applicants judge to be appropriate for the selection process of an admissions test,³² and have been considered a promising tool in the selection process of medical schools.²¹

Moreover, since the study was undertaken in a single center, it remains to be determined whether this applies to other schools as well. Also, the retrospective nature of the investigation indicates that the data should be received with caution in view of the potential selection bias. Finally, the relatively small sample size may interfere with the accuracy of data. However, we included the data of all students in the last semester (12th stage) of our medical course in 2015 who underwent the same academic process. In this way, it is important to emphasize that the higher the F-alpha value, the lower the sample size. This mitigates the error produced by other variables. Furthermore, the correlation between variables were moderate to high. Therefore, we are confident that our data has not been obtained by chance alone.

Conclusion

Our study shows that there is a correlation between UAT marks and academic achievement at the end of our medical course, with greater relation with FMC. Therefore, marks in the admissions test, but not essay scores, can predict academic performance at the end of a medical course in a PBL hybrid curriculum.