Factors associated with practicing evidence-based medicine: a study of family medicine residents

Introduction

Evidence-based medicine (EBM) is “the conscientious, explicit and judicious use of the current best evidence in making decisions about the care of individual patients”.¹ This approach integrates the latest research findings and clinical experience into individualized patient care.² In this sense, EBM is crucial for the implementation of beneficial and effective health care practices and the abandonment of harmful or ineffective ones.¹ Thus, EBM aims to help doctors make evidence-informed decisions in order to potentially reduce variation in medical decisions and to improve outcomes.^3,4

Although the concepts surrounding EBM have been highlighted in medical education for decades, many residents and physicians lack the skills and knowledge to practice EBM.⁵ Many barriers for implementing EBM in practice have been identified.² Negative attitudes toward EBM and a lack of familiarity with this approach are among the top barriers: doctors who viewed EBM as only “research-based decision[s]” that neglect “clinical expertise” were less likely to employ this perspective.² Other important barriers for implementation include lack of EBM competencies and knowledge and skills as well as non-conducive teaching styles.² The culture of practice, on the other hand, can counteract such barriers: a culture that cultivates respectful and reciprocal communication seems to encourage EBM.³

While the literature has identified key barriers, more research is needed about how residents develop and use their EBM skills. Several review studies of scores of EBM implementation studies have examined the impacts of different EBM instructional techniques.⁶ While these studies have found that multifaceted, clinically integrated approaches are generally effective for residents, they have also found that gains in EBM knowledge or skills diminish over time.⁶ Additionally, previous studies showed limited success in helping residents develop these EBM skills and knowledge, especially in contrast to medical students.^7,8 Finally, most studies do not consider residents’ previous experience when assessing their changes in EBM capacities, and thus, we have limited understanding on which experiences matter more in terms of long-term EBM development. Thus, we aim to provide insight into what experiences are associated with residents’ development of EBM attitudes, habits, and skills.

Various approaches for measuring EBM have been used in the literature. Evaluating EBM’s benefit to patients and learner’s behavior may require the use of patient-specific outcomes and active monitoring of learners.⁹ However, attitudes, self-efficacy, and skills are easier aspects for assessment and they can be evaluated using self-administered assessment tools.⁹ A systematic review identified a variety of instruments, many of which have limited validity.¹⁰ Among these tests, the Fresno Test has probably been used the most extensively, with over 300 citations and validations of variations of the test. We similarly chose the Fresno test to assess residents’ skills in EBM.¹¹ Among EBM instruments, the one by Taylor et al is valuable because it has been shown to reliably measure EBM habits and self-efficacy.¹² Using multiple measures increases confidence in effectively understanding complex variables such as EBM.¹³

This pilot exploratory study begins to examine variations in family medicine resident development of EBM by considering residents’ previous experience relative to their EBM mastery. For the purposes of this study, we focus on habits related to, self-efficacy in, and skills related to EBM. Using a survey based on two different validated assessments of EBM, this study analyzes responses from a cross section of family medicine residents and new interns to better understand what factors are associated with improved EBM outcomes. These factors include level of training, previous experience in research, previous training in EBM, and background variables such as gender and location of medical school (US and non-US medical schools) and type of residency program (such as university-based, community-based, and military-based programs).

Materials and methods

Design

We used a cross-sectional design for the purposes of this study. We administered a survey to a non-random voluntary sample of family medicine residents that asked some questions about their background as well as about EBM habits, self-efficacy, and skills.

To understand the existing measurements of EBM, we consulted key databases such as Medline, PubMed, and Google Scholar. We identified five instruments that assessed different aspects of EBM.^{9,11,12,14,15} We believe, like a previous EBM research, residents’ self-efficacy in practicing EBM as well as their EBM habits and skills to be particularly important.^1,2,5 Given this focus, we created an online survey that combined the two different validated instruments cited above: the Fresno test of EBM competence and Taylor et al EBM assessment.^11,12

We included the Fresno test to assess residents’ skills in EBM.¹¹ The Fresno test provides a brief vignette as context for asking residents how they might research the question and identify useful studies, followed by a few questions asking residents to estimate certain outcomes (e.g., predictive values, risk reductions, and so on). Fresno test scores can be as high as 212, although previous research suggests novices score on average around 95 and experts score around 148.⁹ The Fresno test has been validated using populations similar to that in our study and, thus, is expected to be a reliable measure of EBM skills.⁹

We used parts of the questionnaire formulated by Taylor et al¹² to assess residents’ sense of self-efficacy and EBM-associated habits. One part of the instrument has respondents assess their ability to conduct EBM skills (e.g., study design, bias, sample size adequacy, generalizability, and statistical tests). These six items are scored on a 6-point scale. For the measure of EBM-associated habits, a different part of the instrument asks respondents to estimate the amount of time spent each week reading medical literature and the number of articles read.

Finally, we used RedCap^® for data collection and management. The survey was conducted at the end of June and the beginning of July to capture residents from all 3 years of training, in addition to the newly recruited interns. This is a particular virtue of the study as it allows us to consider the influence of medical school in contrast to residency on EBM development. The Indiana University Institutional Review Board reviewed and approved the study. The survey included information page, which stated that completing the survey is considered as consent to participate and to publish the resulting data.

As part of the assessment of students’ EBM self-efficacy, habits, and skills, we explored several explanatory variables of interest. We considered the length of training. In the survey, residents indicated their year of residency, ranging from starting residents (hereafter called “resident recruits”) and former interns (“rising junior”) to those who had completed 2 or 3 years of residency training (“upper levels”). We expected that advancing in resident level of training would be associated with more favorable EBM outcome indicators (habits, self-efficacy, and skills). This is based on the assumption that residents grow in their EBM understanding and ability as they advance in training. We also took into account the role of previous EBM training. We anticipated that students with previous training (as measured in a series of questions in the Taylor et al questionnaire) would have greater understanding and engagement with EBM than those without previous training. Additionally, we contemplated that previous experience (as measured in the Taylor et al piece, “Have you personally been involved in conducting any kind of research?”) in conducting research lends itself to a sense of self-efficacy in EBM skills and provides knowledge on how to conduct EBM. Previous research experience could also affect EBM habits, as researchers would be more likely to stay abreast of research in their area of research. Finally, we included binary variables to account for the type of residency (university based compared to non-university based), gender, and place of medical training (US and non-US schools).

Participants

We sent an email invitation with a survey link to the program directors of family medicine residencies using the Association of Family Medicine Residency Director listserv. The program directors were asked to forward the invitation to their residents. Because we do not know which directors received the invitation and then forwarded it along, we are unable to estimate the response rate. Participants were sent a $10 gift card for completing the survey. Residents in the sample indicated their family residency program, and thus, the sample includes residents from 40 different family residency programs, representing different types of programs and different regions. This sample captures a small, but diverse slice of the family residency programs in the country. The sample likely draws on those particularly interested in EBM, which suggests results are likely biased upward.

Statistical analysis

We conducted reliability analyses of the particular subset of Taylor et al questionnaire used in our study. We are primarily interested in whether the items consistently measure the same underlying construct, EBM self-efficacy. Therefore, we used Cronbach’s alpha with the data we gathered to estimate this reliability and found it to be quite satisfactory, α = 0.89.

This pilot study sample was sufficiently large to run multivariate analyses on some outcomes of interest, including EBM habits and self-efficacy, but not for EBM skills due to partial completion of the Fresno test. Since EBM habits represented count data that were positively skewed toward zero, we used a negative binomial regression to determine the relationship between residents’ attributes and EBM habit outcomes. The count-dependent EBM habit variables (number of hours studying and number of articles read per week) suggested significant evidence of overdispersion (G2[1]=42.16 and 256.22, respectively, p<0.001), indicating that a negative binomial regression model was the preferred model for evaluating such a distribution. We estimated the impact of differences on key relationships by calculating the average marginal effects. We used a linear regression to evaluate the role of the same set of resident attributes on their self-assessed self-efficacy in EBM competencies.

The nature of the small, uneven sample for EBM knowledge and skills (N=42) indicated the use of nonparametric analyses.^16,17 We conducted Kolmogorov–Smirnov two-sample tests to compare outcomes in EBM knowledge relative to different binary variables (e.g., previous EBM training, previous research experience, and so on).¹⁶ Additionally, we conducted Kruskal–Wallis analyses of rank to compare residents at various stages of their training on the same outcomes of interest.¹² Data are available for future analyses.

Results

The survey was completed, at least partially, by 95 participants. Our sample is split fairly evenly between men (53%) and women (47%), with a majority having previously studied in the USA (66%) and a slight majority (58%) coming from community-based residencies. A majority also reported having had previous training in EBM (81%) and experience in conducting research (79%). Table 1 presents the descriptive statistics of participants and their EBM habits and EBM self-efficacy.

Table 1 Descriptive statistics of participants, EBM habits, and EBM self-efficacy Abbreviation: EBM, evidence-based medicine.

EBM habits

The results describing the relationships between resident attributes and EBM habits did not match our hypotheses. Contrary to expectations, the level of training does not suggest a constant upward growth in EBM habits (i.e., hours of studying and articles read in the literature per week). Rather, rising junior residents suggested significantly more hours of study per week compared to both resident recruits and upper level residents. While the difference in articles read per week was not statistically significant, it followed the same pattern. Figure 1 illustrates this pattern by presenting the average resident’s EBM habits from year to year.

Figure 1 Predicted EBM habits by residency level. Abbreviation: EBM, evidence-based medicine.

As shown in Table 2, different resident attributes were also related to the number of hours spent studying each week and the number of articles read each week. Previous EBM training was associated with a decrease in both EBM habits, although this relationship was only statistically significant for the number of hours spent studying per week, showing an average marginal effect of a decrease in 3 hours of study or two articles per week, compared to that of those who had not received EBM training. On the other hand, previous research experience had a positive and statistically significant relationship with EBM habits. Predictions based on the model suggest that residents with previous research experience read 2.5 hours longer and about 3.5 more articles than those without previous research experience. Other variables such as type of residency, gender, and place of training did not relate to EBM habits.

Table 2 The predicting variables for EBM habits and self-efficacy Notes: ap<0.001. bp<0.01. cp<0.10. dp<0.05. Abbreviation: EBM, evidence-based medicine.

EBM self-efficacy

The linear regression investigating EBM self-efficacy also found previous research experience to be a statistically significant predictor (Table 2). Holding all other variables constant, previous research experience was associated with a statistically significant increase in level of EBM self-efficacy of over 3 points on the 36-point EBM self-efficacy scale. Level of residency, or the comparison of upper level residents to the new recruits, indicated a nearly 1-point increase in self-efficacy when all other variables were held constant, although this was not statistically significant. Interestingly, previous EBM training had no relationship to self-efficacy. Similar to the other models, type of residency, gender, and place of training had no relationship with self-efficacy.

Each of the models described above also suggested that the measurements of EBM self-efficacy and EBM habits were not significantly related to each other. Both variables served as only statistically insignificant predictors in the other models.

EBM skills

The sample used to examine EBM skills via the Fresno test was much smaller in number (N=42) and, thus, limited our power to test for statistically significant differences among groups. However, the patterns in the data are worthy of consideration (Table 3). Upper level residents on average scored 10 points higher (on a 212-point scale) than resident recruits and 15 points higher than rising juniors. The value of formal EBM training appears to relate to EBM skills as suggested in the 15-point difference (97–82) between those who have and have not participated in such courses. Also, previous research experience appears to be associated with EBM skill mastery, as shown in the 19-point (98–79) higher average score between those who have and have not participated in research. Previous investigation of EBM found a 52-point average difference between novices and experts,⁷ which suggests that these average differences, while not at the same magnitude, are meaningful. For example, the difference between a resident who has and has not had previous research experience is over a third of the difference between an EBM novice and expert.

Table 3 EBM skills group comparisons Notes: aKruskal–Wallis analysis of ranks to test hypothesis of group difference. bKolmogorov–Smirnov two-sample test to test hypothesis of group difference. Abbreviation: EBM, evidence-based medicine.

Discussion

This initial pilot study reports on the differences in EBM learning habits, self-efficacy, and skills among family medicine residents at different stages of training. Our study indicated that rising juniors were engaged in EBM learning habits more than resident recruits and upper levels. This comes as somewhat of a surprise since the first year of residency is commonly perceived as heavily clinical, especially in comparison with the second and third years of training when residents are given more time to pursue their learning interests. It is also interesting to observe that upper level residents felt more confident in their EBM skills and scored higher on the Fresno test despite engaging less with EBM learning habits relative to their rising junior peers, although these patterns lacked statistical significance. Thus, further research should investigate the relationship between EBM habits and EBM self-efficacy and knowledge relative to training experience.

The average rates of EBM habits, self-efficacy, and skills give insight into how EBM habits and skills develop among trainees. Medical schools appear to provide resident recruits relatively undeveloped EBM habits, but a similar level of EBM knowledge as rising juniors. This appears to accord with an identified distinction in medical education, namely, teaching medical students EBM skills rather than the EBM habits, which is the true mechanism of life-long learning.¹⁸ While the small sample suggests EBM knowledge is higher among upper levels, the evidence is not definitive and should thus be examined in future studies.

The patterns related to EBM habits raise the question as to whether staying up to date with the literature is seen as a necessity only at certain points in training rather than as a career-long habit of professional development. The nature of EBM education may be such that students draw on the literature and EBM resources to respond to cases that are new to them, but as they broaden their experience, they draw on these skills and habits less frequently.

Residents who participated in formal EBM training exhibited a higher (not statistically significant) average EBM skills test score while engaging statistically significantly less in EBM learning habits. This finding follows previous research showing that EBM training interventions do improve EBM-related knowledge and skills.^19,20 Even though EBM training may be associated with higher skills, it may not instill the habits of continual learning. While it is difficult to perform an evaluative assessment of specific EBM training, there is also a need to assure that training interventions achieve their ultimate objectives: improving EBM knowledge and skills as well as the continual use of evidence in practice.

The effect of previous research experience, on the other hand, was associated with significant positive effects on EBM habits and self-efficacy and a positive trend in EBM skills. This finding supports the importance of including research as a critical part of medical education. This is consistent with the study by Oussalah et al which showed that enrollment in a PhD or a research Master is an independent predictor of PubMed searching skill and knowledge of main journals.²¹

This work is not without limitations. The use of voluntary sampling suggests that statistical relationships may be biased upward, as those who decided to participate were likely more familiar with and enthusiastic about EBM. While the sample draws on respondents from 40 different family residencies, a larger sample would also help to determine to what extent these data patterns can be further generalized. Because our analysis was insufficiently powered, there is still uncertainty among trends that appeared in the data, especially related to higher levels of EBM self-efficacy and skills among upper level residents relative to their peers, or our inability to detect differences among types of residency programs. Additionally, we cannot effectively determine whether there is any causality in these relationships or merely a correlation. We were unable to explore the confounding effect of varying EBM curricula among the various residency programs as it demands a far more intensive approach. This pilot study does generate interesting questions for future analysis. Further research into this topic and the role of various EBM curricula, especially using longitudinal study designs, is warranted.

Our work is relevant and has multiple strengths. First, it examined factors that contribute to EBM learning and should serve as an invitation for further research on the relationship between these factors and EBM habits, self-efficacy, and skills. It provided additional understanding of the relative impact of previous research experience and EBM training, as well as class standing in a different way from previous research. Furthermore, this study used validated tools to assess the learning outcomes, thus providing more evidence for the utility of standardized evaluation methods. We used a promising method of recruiting residents by engaging program directors to invite residents. This resulted in a number of programs represented. Recruiting residents is often a challenging task and using the method we did, we were able to achieve fairly wide representation across institutions, settings, and regions.

Conclusion

Our findings suggest the practice of EBM may benefit from medical education increasing research experiences and EBM training. Research experiences provide the practical training, while EBM training provides focused instruction necessary for EBM habits, self-efficacy, and skills. These EBM outcomes are not inherently gained through time in family medicine residency. Future research, particularly longitudinal designs, should continue to pursue this line of inquiry.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Rohwer A, Young T, van Schalkwyk S. Effective or just practical? An evaluation of an online postgraduate module on evidence-based medicine (EBM). BMC Med Educ. 2013;13:77.
2.		Swennen MH, van der Heijden GJ, Boeije HR, et al. Doctors’ perceptions and use of evidence-based medicine: a systematic review and thematic synthesis of qualitative studies. Acad Med. 2013;88(9):1384–1396.
3.		Wennberg JE. Practice variation: implications for our health care system. Manag Care. 2004;13(9 Suppl):3–7.
4.		Mardon R, Shih S, Mierzejewski R, Halim S, Gwet P, Bost JE. National Committee for Quality Assurance: The State of Health Care Quality. Washington, DC: National Committee for Quality Assurance. 2003.
5.		Maggio LA, Tannery NH, Chen HC, ten Cate O, O’Brien B. Evidence-based medicine training in undergraduate medical education: a review and critique of the literature published 2006–2011. Acad Med. 2013;88(7):1022–1028.
6.		Young T, Rohwer A, Volmjink J, Clarke M. What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLoS One. 2014;9(1):e86706.
7.		Ilic D, Diug B. The impact of clinical maturity on competency in evidence-based medicine: a mixed-methods study. Postgrad Med J. 2016;92(1091):506–509.
8.		Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. CMAJ. 1998;158(2):177–181.
9.		Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva E. Development and validation of the ACE tool: assessing medical trainees’ competency in evidence based medicine. BMC Med Educ. 2014;14:114.
10.		Shaneyfelt T, Baum KD, Bell D, et al. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006;296(9):1116–1127.
11.		Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003;326(7384):319–321.
12.		Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical appraisal skills training for health care professionals: a randomized control trial. BMC Med Educ. 2004;4(1):30.
13.		Malhotra MK, Grover V. An assessment of survey research in POM: from constructs to theory. J Oper Manag. 1998;16(4):407–425.
14.		Kunz R, Fritsche L, Neumayer HH. Entwicklung eines Gegenstandskatalog als Basis einer reproduzierbaren Ausbildungsqualität in evidenzbasierter Medizin [Development of quality assurance criteria for continuing education in evidence-based medicine]. Z Arztl Fortbil Qualitatssich. 2001;95(5):371–375. German.
15.		Holloway R, Nesbit K, Bordley D, Noyes K. Teaching and evaluating first and second year medical students’ practice of evidence-based medicine. Med Educ. 2004;38(8):868–878.
16.		Siegel S. Nonparametric Statistics for the Behavioral Sciences. New York, NY, USA: McGraw-Hill; 1956.
17.		Elliott AC, Woodward WA. Statistical Analysis Quick Reference Guidebook: With SPSS Examples. Thousand Oaks, CA: Sage; 2004.
18.		Vrdoljak D. Teaching evidence based medicine in family medicine. Acta Medica Academica. 2011;41(1):88–92.
19.		Al Achkar M, Davies MK. A small group learning model for evidence-based medicine. Adv Med Educ Pract. 2016;7:611–615.
20.		Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva E. A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Med Educ. 2015;15(1):39.
21.		Oussalah A, Fournier JP, Guéant JL, Braun M. Information-seeking behavior during residency is associated with quality of theoretical learning, academic career achievements, and evidence-based medical practice: a strobe-compliant article. Medicine (Baltimore). 2015;94(6):e535.