A Comparison of the Academic Achievement at the End of the Medicine Undergraduate Degree Program Between Students Who Only Used the University Admission Test and Those Who Used the University Admission Test Plus Marks from the High School National Exam (ENEM) at a Single Brazilian Center

Introduction

Selecting students to enter a Medical Undergraduate Degree program is of utmost importance for the medical profession. Students who passed a selection process have better academic performance and a lower rate of dropout in the Medicine undergraduate degree program during the preclinical stage in comparison with those who did not.¹ There are several types of selection methods for admission to an undergraduate degree program, including university admission tests (UAT), academic records, personal statements, letters of recommendation, situational judgment tests, personality and emotional intelligence, interviews and multiple interviews.²

The role of the UAT as a means of selecting students to enter a medical course is still debatable, as many studies have shown conflicting results.² Nonetheless, the UAT comprised of questions on biology, physics, and chemistry appears to have better psychometric properties in the medical selection process as compared with the ones that do not use such questions.³ This is consistent with our experience, which has shown an association between marks in the entrance examination test, composed of such questions, with academic achievement at the end of our medical course.⁴ In addition, students who were approved at UAT have higher motivation in comparison with those who did not.¹

Pre-university academic records seem to predict students’ outcomes not only during Medical School but also during their medical careers in many countries.^2,5 Furthermore, academic records are associated with a low rate of student dropout.^6,7 However, other studies suggest that such academic records do not predict performance during the medical course.² Finally, the association of UAT with Pre-University academic attainment is believed to predict students’ performance at the end of the first year of the medical undergraduate degree program.

In Brazil, there are several different types of UAT depending on the decision of each Medical School. A collection of pre-university marks has not been used in the admission process of a Medical School. However, marks of a pre-university national exam could be used in the selection process of medical students. A High School National Exam - ENEM (Exame Nacional do Ensino Médio) is a non-compulsory, standardized exam, managed by the Ministry of Education and launched in Brazil in 1998 to assess the quality of the Education provided by Brazilian High schools. Since 2009, the ENEM has been offered at the end of each year. It consists of 180 multiple-choice questions regarding Natural Sciences, Mathematics, Human Sciences, Portuguese and English or Spanish, and an essay.

The score obtained in the ENEM can be used by students to enter a Higher Education Institution. Some schools use only the total ENEM score to select their students, while others use the ENEM marks as part of the total score. About 57% of the students who undergo the ENEM use the score obtained in the exam to enter a Higher Education Institution; the remaining students appear not to use such a score.⁸ As far as we know, the results of the ENEM have not previously been correlated with the scores of UAT. In our institution, the ENEM marks may enter a formula to compose the final score of the UAT (see below).

This study aimed to determine whether students who use the ENEM score to enter our medical program have a better academic performance at the end of the medical undergraduate degree program in comparison with those who use only the marks obtained in the UAT. To our knowledge, such a comparison has not previously been undertaken.

Methods

The Medicine Undergraduate Degree Program

From the 1st to the 8th semester we run Problem-Based Learning (PBL) in our medical undergraduate degree program. By law, from the 9th to the 12th semester, all private and federal Medicine undergraduate degree programs in our country must cope with the National Guidelines for Medical Courses enacted in 2014. Details of this curriculum have been published elsewhere.⁹ Briefly, during the preclinical stage (from the 1st to the 8th semester) the curriculum is composed of three units: Tutorial sessions, Medical Skills, and Primary Care. In the clerkship, from the 9th semester to the 12th semester, students rotate in the main medical specialties (pediatrics, surgery, internal medicine, family medicine, urgency and emergency medicine, and gynecology and obstetrics).

Tutorial sessions are run with tutors who are Medical School teachers specially trained to act as tutors in tutorial sessions¹⁰ and are specialized in the subject.¹¹ As the curriculum is competency-oriented, the tutors from the 1st to the 4th semester are not necessarily physicians but are specialists on the topic. From the 5th to the 8th semester because students see real patients, all tutors are physicians specialized in the topic under discussion. The Tutorial sessions are adapted from the classical description by Schmidt,¹² with the inclusion of tests before the reporting phase.¹³ Medical skills are a discipline devoted to developing communication skills and medical skills necessary to the medical profession and run in parallel with the Tutorial sessions. Primary care uses a modification of the classical Tutorial session, and wherever possible it also runs in parallel with Tutorial Sessions and Medical Skills.¹⁴ During the clerkship, teaching is fundamentally ward-bedside, outpatient services, and emergency rooms. Students also take part, under direct medical supervision, in surgical procedures and childbirth.

The assessment process consists of formative assessment in each discipline during the preclinical stage as well as in the clerkship. Marks in the standardized formative assessment do correlate with marks from the Progress Test (PT), the Objective Structured Clinical Examination (OSCE), and the mean marks obtained in the disciplines of the clerkship.¹⁵ The summative assessment consists of multiple-choice questions in a Key-Feature format and open-ended questions applied at the end of each curricular unit, including those of the clerkship. The PT is applied each semester of the medical course. The OSCE is applied from the fourth semester onwards.

As far as the clerkship is concerned, the final mark is composed of the marks obtained from the OSCE (30%), the PT (18%), regarding the final written examination of each discipline (46%), and the formative assessment (9%). Therefore, the evaluation of the last semester of the Medicine undergraduate degree program is very comprehensive, approaching cognition, (PT and written examination of each discipline), skills (OSCE), and attitudes (formative assessment).

Data Collection

Fifty-one (51) students from the 12th semester (the last year) of our medical undergraduate degree program of the year 2015 were included in the study. These students had undergone an entrance test to be admitted to our medical undergraduate degree program. Briefly, all candidates underwent: 1) an essay test; those who scored < 40% are inapt; 2) a written examination in the Portuguese language, Brazilian literature, and the essay (maximum score: 20 points for Portuguese language and Brazilian literature, and 20 maximum points for the essay); 3) a multiple-choice test in chemistry, biology, mathematics, physics, and Brazilian or general history (the specific exam), whose maximum mark was 30 points. The final mark was obtained according to the formula: Total Marks = (2x NG) + (4 x NE), where NG=sum of the points obtained in the writing items on the Portuguese language, Brazilian literature, and essay. The NE is the sum of the points obtained in the specific examination (Biology, Mathematics, etc.).

The applicants who wanted to use the marks obtained from the ENEM were evaluated as follows: Total Marks = (1 x NG) + (3 x NE) + 0.7 x ENEM marks, where NE and NG are marked as mentioned earlier, and the only difference is the inclusion of ENEM marks in the final formula. The maximum ENEM score is 1000 points. Since scores on the ENEM are totaled to a maximum of 100 points, we first divided the student’s ENEM score by 10. Therefore, the equation for students who have not used the ENEM marks has a maximum score of 200. The equation for students who have not used the ENEM marks has a maximum score of 200 [(2 x 40) + (4 x 30)], which equals 200. The equation for students who have used the ENEM marks also has a maximum score of 200 [40 + (3 x 30) + (0.7 x 100)], which equals 200 as well.

We compared the marks obtained in the OSCE and in the PT between students who used the ENEM score and those who did not use such a score in the admission test. The mean marks obtained in the disciplines run in the clerkship (pediatrics, internal medicine, gynecology and obstetrics, urgency and emergency medicine, family medicine, and surgery) were also compared between ENEM x non-ENEM group.

The students’ marks were supplied by the Division of Academic Registers, whose members did not take part in the investigation. The research was conducted complying with the principles outlined in the Declaration of Helsinki; the privacy of research subjects was assured, the marks were not identifiable, and the analyses were performed anonymously. Hence, the informed consent of subjects was waived by the Research Ethics Committee of the University of Ribeirão Preto and the review body approved the aspects of the study.

Statistical Analysis

Continuous variables are given as mean ± standard deviation. The t-test for the unpaired sample was used to compare continuous variables between ENEM x non-ENEM group. The Spearman test was used to establish the correlation between marks obtained in the UAT and marks obtained in the OSCE, in the PT and those obtained in the clerkship in the ENEM and non-ENEM group. The software version string R Version 4. 2. 2 and the software R studio 2022.12. 0 were used for the statistical analysis. Differences at a p-value < 0.05 were considered statistically significant.

Results

Nine (9) students used the UAT marks plus those obtained in the ENEM, whereas the remaining 42 students only used the marks obtained in the UAT. The mean marks obtained in the disciplines of the clerkship were higher in the non-ENEM group in comparison with the marks obtained by the students in the ENEM group (7.32 ± 0.41 versus 6.98 ± 0.31; p=0.01). Figure 1 illustrates these findings.

Figure 1 Comparison of mean marks (±SD) obtained in the disciplines of clerkship in the ENEM and non-ENEM groups at the end of the medical undergraduate degree program (*p=0.01 unpaired t-test analysis). ENEM: High School National Exam.

Nevertheless, the scores obtained in the OSCE and the PT were not different between groups. The mean marks obtained in the OSCE were 7.90 ± 0.54 in the non-ENEM group, and 7.67 ± 0.79 in the ENEM group (p=0.42) and the mean marks in the PT were 4.87 ± 2.02 in the non-ENEM group, and 3.79 ± 2.02 in the ENEM group (p=0.09).

No correlation was found between the marks obtained in the UAT and in the OSCE (p=0.89; r=−0.06), in the PT (p=0.21; r=−0.46), and in the clerkship (p=0.99; r= 0.003) in the ENEM group. No correlation was found between the marks obtained by the non-ENEM group at the UAT and the marks in the OSCE (p=0.10; r=0.28) as well as in the PT (p=0.40; r=0.15). However, a moderate correlation was observed between marks in the clerkship with those of UAT in the non-ENEM group (p=0.00006; r=0.45).

Discussion

Our study shows that students who used the marks obtained in the ENEM had lower academic achievement in the disciplines of clerkship at the end of the medical undergraduate degree program than those who did not. Besides, a moderate correlation was found between marks in the entrance examination test and the mean marks in the disciplines of the clerkship in the non-ENEM group. However, the same has not been observed regarding skills and cognition, as no difference was observed between both groups regarding OSCE and PT. It must be emphasized that our results refer to the ENEM marks, which is a national examination and not a proper pre-academic record.

Although some studies have demonstrated the role of UAT in predicting students’ outcomes pursuing a medical degree,² others have not shown any benefit of UAT to predict academic performance at the end of the medical course.¹⁶ In addition, previous reports that have used the UAT along with academic records in the students’ selection process showed an increase in validity of the students’ selection process,^17,18 which contrasts with our data. On the other hand, our investigation concurs with some studies that showed no impact of pre-university academic records on final study achievement¹⁹ or in predicting students’ outcomes at the end of the Medical School.^20–22

One possible explanation for our finding is that students who passed the UAT and did not use the ENEM marks were more intrinsically motivated than those who used the ENEM marks. Because we run a PBL medical teaching approach, intrinsic motivation is paramount for students to have good academic achievement at the end of Medical School. Furthermore, we try to increase the extrinsic motivation of the students by applying an assessment test before the reporting phase of a Tutorial Session.¹³ Therefore, more motivated students have a higher probability of striving for the hard learning process inherent to the medical undergraduate degree program.

Another possibility to account for the discrepancy between our study and the few previously published ones is that the degree of difficulty between a UAT and the ENEM tests may be higher in the former than in the latter. However, we concede that further studies will be necessary to determine the importance of UAT marks over the ENEM test in the prediction of academic achievement in a medical course.

It has been shown that a UAT consisting of items related to physics, chemistry, and biology are independent predictors of academic achievement for students in the second year of a medical undergraduate degree program. Importantly, this finding was independent of the pre-university academic records.³ In the entrance test for our medical undergraduate degree program, the UAT consists, among other things, items related to those disciplines mentioned earlier. Our study, therefore, may expand the view that UAT comprised of items related to natural sciences may be associated with academic achievement at the end of the medical course as well. Furthermore, the addition of ENEM marks can mask a low performance in our entrance aptitude test. This might also be a hypothesis to explain the higher academic achievement of students who did not use the ENEM marks in comparison to those students who did.

This study has several limitations. First, the small sample size reduces the strength of our findings. However, the differences between data means are significant and the mean standard deviations are so narrowed that these turn the results unlikely to be the play of chance. Furthermore, a moderate correlation was found between marks obtained in the disciplines of clerkship with marks in the UAT in the non-ENEM group. Another limitation is that the study was single-centered; therefore, the generalizability of our findings is limited. Nonetheless, the difference between both groups was observed in the mean marks of the disciplines of the clerkship. Under that circumstance, the assessment of the learning process is concerned with high standards of the Bloom taxonomy, where the knowledge applicability and the creation of learning are of utmost importance for clinical practice. As the clerkship marks are also related to attitudes (formative assessment), it reassures that the UAT may be associated with an important characteristic of good doctors.

Conclusion

Medical students who use marks only from the UAT scored higher than those who used the UAT plus the ENEM marks in the clerkship at the end of a Medicine undergraduate degree program with a moderate correlation between these variables. This was observed in terms of cognition and knowledge applicability (a high standard in the Bloom taxonomy), but not in either the OSCE or the PT. Further studies are necessary (ideally a prospective, randomized trial) to confirm our generating hypothesis findings.