Back to Journals » Advances in Medical Education and Practice » Volume 7

A simple heuristic for Internet-based evidence search in primary care: a randomized controlled trial

Authors Eberbach A, Becker A, Rochon J, Finkemeler H, Wagner A, Donner-Banzhoff N

Received 30 November 2014

Accepted for publication 25 January 2016

Published 4 August 2016 Volume 2016:7 Pages 433—441

DOI https://doi.org/10.2147/AMEP.S78385

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 3

Editor who approved publication: Dr Md Anwarul Azim Majumder



Andreas Eberbach,1 Annette Becker,1 Justine Rochon,2 Holger Finkemeler,1Achim Wagner,3 Norbert Donner-Banzhoff1

1Department of Family and Community Medicine, Philipp University of Marburg, Marburg, Germany; 2Institute of Medical Biometry and Informatics, University of Heidelberg, Heidelberg, Germany; 3Department of Sport Medicine, Justus-Liebig-University of Giessen, Giessen, Germany

Background: General practitioners (GPs) are confronted with a wide variety of clinical questions, many of which remain unanswered.
Methods: In order to assist GPs in finding quick, evidence-based answers, we developed a learning program (LP) with a short interactive workshop based on a simple ­three-step-heuristic to improve their search and appraisal competence (SAC). We evaluated the LP ­effectiveness with a randomized controlled trial (RCT). Participants (intervention group [IG] n=20; ­control group [CG] n=31) rated acceptance and satisfaction and also answered 39 ­knowledge ­questions to assess their SAC. We controlled for previous knowledge in content areas covered by the test.
Results: Main outcome – SAC: within both groups, the pre–post test shows significant (P=0.00) improvements in correctness (IG 15% vs CG 11%) and confidence (32% vs 26%) to find evidence-based answers. However, the SAC difference was not significant in the RCT.
Other measures: Most workshop participants rated “learning atmosphere” (90%), “skills acquired” (90%), and “relevancy to my practice” (86%) as good or very good. The ­LP-recommendations were implemented by 67% of the IG, whereas 15% of the CG already conformed to LP recommendations spontaneously (odds ratio 9.6, P=0.00). After literature search, the IG showed a (not significantly) higher satisfaction regarding “time spent” (IG 80% vs CG 65%), “quality of information” (65% vs 54%), and “amount of information” (53% vs 47%).
Conclusion: Long-standing established GPs have a good SAC. Despite high acceptance, strong learning effects, positive search experience, and significant increase of SAC in the pre–post test, the RCT of our LP showed no significant difference in SAC between IG and CG. However, we suggest that our simple decision heuristic merits further investigation.

Keywords: decision making, medical informatics, evidence-based practice, continuing ­medical education

Introduction

Unanswered questions

In primary care, patients present a wide spectrum of clinical problems to their general practitioners (GPs). The questions that are brought up in these encounters touch upon virtually every clinical topic.13 According to a recent systematic review, ten patient visits left GPs with approximately six unanswered questions on average. Only ∼50% of GPs attempted to find an answer.4 The main reasons given by GPs for not pursuing their open questions further were lack of time and doubt that relevant information was available at all.

Available information

There is a discrepancy between the situation described earlier and the large amount of medical information available. Structured databases and other sources are accessible via the Internet. Online evidence has the potential to provide quick answers to physicians’ questions.5

Clinicians’ use of electronic resources has been reported to be increasing.6 Examinations of resources used in primary care have revealed that, after textbooks and discussions with other physicians, Internet use represents the third most frequently used resource.4,610

Answering questions

Evidence-based medicine (EbM) has provided a framework for accessing, appraising, and using the medical literature relevant to individual patients. One of its pioneers, David Sackett, coined the paradigm of “Five Steps”: question formulation, literature search, critical appraisal, implementation of results, and evaluation.11 However, time constraints and lack of skills prevent most physicians from following this model.1,10,1216

As an alternative to the classical appraising mode, the McMaster Group already suggested the searching mode, which primarily uses databases of processed information for quick answers to questions arising in everyday care situations. Even physicians familiar with EbM appraisal methods would resort to less sophisticated strategies for their everyday clinical questions. Slawson and Shaughnessy have formulated a need for simple heuristics not just to appraise but to manage medical information.17,18

Methods

Learning program (LP)

We developed an LP that is based on a simple three-step heuristic (Figure 1). The aim of the program is to help GPs use Internet sources for their clinical questions. This can be taught in interactive seminars. All information is also available in the form of a booklet for individual study. A CD-ROM with relevant URLs and background material helps GPs put into practice what they have learned. In order to improve the effectiveness of the program, we used multimedia (workshop [WS], booklet, CD, Internet) and multiple exposures (lectures, exercises, homework).19

Figure 1 The three-step heuristic of decision making.

Simple heuristic

The content of the LP (Figure 2) was structured around the three-step heuristic,20 shown in Figure 1. Its application ensures that GPs search and appraise sources in a methodological way and also helps them decide when to end their search. We regard this as especially important because GPs often feel overwhelmed by the amount of available information.2123 This set of three questions is not meant to be followed strictly in every literature search. It is also meant to encourage GPs to reflect on their own search and appraisal habits.

Figure 2 Workshop table of contents.


Abbreviations: ARR, absolute risk reduction; RRR, relative risk reduction; NNT, number needed to treat.

Example 1: a commercial manufacturer mentions in the drug-sheet of a lipid-lowering drug that no long-term studies with patient-relevant outcomes have been conducted so far. This information is not in line with the interest of the manufacturer (source); it should therefore be accepted without further search. The claim of a manufacturer that a herbal preparation alleviates tumor pain, however, would not lead to immediate acceptance but should motivate further research.

Example 2: if for a herbal preparation, no formal evaluation, such as a controlled study, can be found, its use should be avoided. Only if the claim is in line with the interest of the source and trials evaluating a particular treatment exist, a more detailed appraisal of study design features is required. Here the most valid and relevant criteria should be checked first, such as:

  • is this a randomized controlled trial (RCT)?
  • is the main outcome clinically relevant?
  • can the choice of the control treatment be justified?
  • which patients have been recruited into the study (external generalizability)?

Participants

GPs located in five different cities in Germany were invited to take part in the main study. Participants had to be motivated to attend two WS sessions and contribute to the evaluation.

Stratification

GPs were asked whether they had performed an Internet search for a clinical question during the previous year. The answer (yes/no) was used for stratification.

Randomization

The computer-assisted randomization was undertaken by the central clinical trials unit at the University of Marburg. Within the two stratification-groups, permuted blocks of variable length were formed, including one to four participants. According to the order of registration, the GPs were assigned to either the intervention group (IG) or the control group (CG). In order to maintain the allocation concealment, the participants were not given details about the study design or the allocation process. The control arm was a waiting group of GPs who were offered the WS after study completion and evaluation of the main outcome.

Intervention

GPs attended two interactive small group WS in a computer classroom. Figure 2 shows the contents of the WS. Sessions provided information on databases that are useful for GPs. Their scope, advantages, and disadvantages were discussed. There was an emphasis on processed evidence-based information as opposed to original papers, such as drug bulletins, clinical evidence (British Medical Journal) or national and international guidelines. Practical search strategies, bias of sources, and simple criteria to check the validity of information were covered. There was ample opportunity for hands-on exercises and discussion of participants’ experience with the sources provided. WS lasted at least 2 hours each with a 4-week interval in-between. More specific background information can be found in the “Supplementary materials” section (English) and on the Internet (German).24

Study outcomes

The primary outcome of the study was GPs’ search and appraisal competence (SAC). The secondary outcome was to determine the acceptance and satisfaction with the WS.

WS acceptance

We asked GPs to evaluate both WS using a semi-standardized questionnaire each time directly after the event. Fifteen questions in a 6-point Likert scale covered organization, educational process, and relevance of content (1= very good, 6= deficient). Finally, they estimated their own commitment for change regarding content covered by the WS in a standardized way.25

Search satisfaction

GPs were asked to rate the relevance of content and their satisfaction using the strategies learnt. Acceptance and satisfaction were evaluated not only by IG participants but also by control subjects after their participation in the WS.

SAC

We evaluated GPs’ SAC by additionally (after pre–post test) correct answered clinical questions.

Clinical questions

Thirty-nine clinical questions from seven clinical domains were developed by the study team. Questions concerned prevention, therapy, diagnosis, and prognosis. Most questions were in multiple-choice format, but GPs were also asked to elaborate on some open questions. The appropriateness of answers was checked by experts in EbM and clinical areas covered (see the “Acknowledgments” section).

Pre–post test

All participants had to answer each question twice: first, spontaneously in order to establish their baseline knowledge and again after a thorough search. We could thus control for previous knowledge.

If participants improved, ie, gave an incorrect answer at baseline but a correct answer after thorough search, the answer was scored “+1”. In the case of a correct answer at baseline but a wrong answer after the search, the score was “–1”; no change resulted in “0”. We arrived at an overall SAC score by adding up the single question scores for each participating GP. The result can theoretically vary between –39 (–100%) and +39 (+100%) additional correct answers. Besides the objective professional correctness, we checked the subjective confidence (0%–100%).

Pilot study

Before starting the study, we checked the comprehensibility and the level of difficulty of the 39 clinical questions with a group of 12 GPs. Subsequently, we held the workshop (WS) with another 14 GPs to improve methods and content including learning material. These GPs did not take part in the main study.

Statistics

The data analyses were performed with SPSS software (SPSS Inc., Chicago, IL, USA). For baseline comparisons, we report medians, interquartile ranges, and results of the Mann–Whitney U test (P-values) for continuous variables and percentages, and results of asymptotic χ2 tests for categorical variables. Differences in SAC improvement (before vs after; between study arms) were evaluated by the Mann–Whitney U test (P-value). We used the Hodges–Lehmann estimator to obtain the 95% confidence interval for the difference between medians.26

Results

Study participants

Recruitment, randomization, and flow of participants are shown in Figure 3. Twelve percent (n=110 of 905) of the GPs originally approached gave written consent to participate and were randomized thereafter. Seventy-six percent (n=84 of 110) of subscribers attended at least one WS. Finally, complete data were available from 46% (n=51 of 110) of the original group. Consequently, the dropout rate was 64% due to delayed information search (n=30 “did not search”), absence (n=10 “did not attend”), and incomplete WS attendance (n=15 “only one WS”) or incomplete data (n=4).

Figure 3 Flowchart of general practitioners’ participation, and dropout.


Abbreviation: WS, workshop.

Characterization of study participants

Participants’ characteristics are shown in Table 1. In most respects, they resembled the population of GPs in Hessen. The IG and CG did not differ significantly, but in the IG, there were slightly more GPs with Internet search experience and a DocCheck password (access to content provided by the pharmaceutical industry).

Table 1 Preconditions of study participants


Notes: aComplete data: 1: participation and evaluation of both workshops; 2: documentation and evaluation of seven literature searches; 3: answering 39 questions twice. bG = published statistics of the Federal Chamber of Physicians in Germany;37 cH = details of the Regional Chamber of Physicians in Hessen (Landesärztekammer Hessen);38 drepresentative sample of the Commonwealth Fund;39 ethe DocCheckTM password allows licensee to access drug formulary and labeling information; fparticipants have conducted Internet search for medical topics in the previous year; gskills as judged by participants themselves. Scale: 1 (low) to 5 (high). *P-values for between-group comparison.


Abbreviations: IG, intervention group; CG, control group; SD, standard deviation.

Acceptance

GPs rated the Continuing Medical Education WS as very positive (Figure 4). Over 80% of respondents rated all items as good or very good. We obtained similar ratings referring to WS organization and educational process (data not shown). At the end of the WS, most participants (78% in the IG and 69% in the CG) made a commitment to change their information management according to what they had learnt during the WS (Figure 5).

Figure 4 Participants’ workshop evaluation.


Notes: (N=51, 1= very good, 6= deficient). Please note that GPs in the control group are included in this process evaluation.


Abbreviations: IG, intervention group; GPs, general practitioners; CG, control group.

Figure 5 Commitment to change at the end of the workshop (n=48).


Note: Please note that GPs in the control group are included in this process evaluation.


Abbreviation: GPs, general practitioners.

Literature search experience

In order to find answers to standardized questions, the IG used more sources per topic and spent more time searching than the CG (Table 2). Members of the IG were slightly more satisfied with the quality and amount of information as well as the time spent. Ratings for the sources’ trustworthiness and relevance were almost identical in both study arms.

Table 2 Literature search experience


Notes: *P-value for between-group comparison: Mann–Whitney U test. ++Please note that physicians in the control group provided these evaluations before they were offered workshop participation. #Difference in % between IG and CG.


Abbreviations: IG, intervention group; CG, control group; IQR, interquartile range.

Literature search sources

IG and CG preferred different sources to search literature (Table 3). The IG used more Internet sources (+22%) and targeted web sites (+37%). Overall, the WS recommendations on literature review were implemented well (67%, odds ratio 9.6, 95% confidence interval 2.6–36.4; P=0.0008).

Table 3 Sources of information used for literature review


Notes: ++Please note that physicians in the control group provided these evaluations before they were offered workshop participation. #Difference in % between IG and CG.


Abbreviation: OR, odds ratio.

SAC

The improvement in answers to clinical questions was the main outcome measure. In both study arms, GPs performed better with a thorough search than at baseline (Table 4). Median improvement was 11% in the CG (Wilcoxon test: P<0.0003) and 15% in the IG (Wilcoxon test: P=0.007). Although IG GPs improved slightly more than controls, the between-group comparison was statistically not significant (P=0.29).

Table 4 Effectiveness measures of search and appraisal competence


Notes: aCorrectness: additional correct answers: before vs after search in pre–post test. bSAC corresponds with score of correctness. cConfidence: perceived safety when responding (0 to 100%: 0= completely unsafe/100%= absolutely certain that the answer is correct. P* indicates Student’s t-test used, for paired samples, two-tailed significance. ++Please note that physicians in the control group provided these evaluations before they were offered workshop participation.


Abbreviations: CI, confidence interval; SAC, search and appraisal competence; IG, intervention group; CG, control group.

Discussion

Our study shows good WS acceptance, strong learning effects, positive search experience, and significant increase of SAC in the pre–post test comparison. Also, SAC improved more in the IG than in the CG although the difference was not statistically significant in the RCT.

The potential of online evidence in order to improve SAC skills of GPs has been demonstrated in controlled laboratory settings. Previous pre–post trials have shown an improvement from 10% to 18% in GPs’ answers to typical clinical scenarios following the use of online evidence.5,25,27 Our results correspond to these findings, with the proportion of correct answers increasing by 15% in the IG (Table 4).

The learning effect on the IG could be shown through the implementation of suggestions given in the WS (Table 3, odds ratio 9.6) of using targeted Internet sources (odds ratio 4.8), pharma-critical drug bulletins (odds ratio 2.4) or guidelines (odds ratio 2.1). We encouraged our participants to use targeted sources they know well despite their limitations of scope and possible bias because accessing the Internet via unspecific search engines such as Google or Yahoo is prone to error.28,29

In many areas requiring decision making under time pressure and with limited information, fast and frugal heuristics have been shown to be valid and helpful.30 Our three-step heuristic20 (Figure 2) not only provides rules for search and appraisal of validity but also rules for when to stop a search to save time. This seems important because lack of time is one of the main reasons why questions remain unanswered.4 In our study (Table 2), the IG invested 17% more amount of time but investigated 36% more sources in this time. Moreover, GPs in the IG were more satisfied with the amount of time invested (+16%), the quality (+11%), and the quantity (+6%) of the information thus obtained.

Our recruitment rates of 12% (Figure 3, “consenting to participate”) and 6% (“complete data available”), respectively, seem rather low. However, the experience of other studies with GPs in Germany shows that “it is generally possible to enlist 3%–4% of the initially contacted practices”.31 Thus, our recruitment quota of 12% actually indicates that GPs have a relatively large interest in the topic of our study. The positive comments by participants are in line with this idea.

GPs in the CG had more previous methodological knowledge than we had expected. They more often obtained their information from the Internet (59%) than from printed media (31%) or through consultation of colleagues (10%). Literature has so far listed the relative importance of these sources of information in the reverse order.4,10,32 This may be an indication of a powerful CG.

Strengths and limitations of the study

The main strengths are the RCT study design and the inclusion of experienced GPs in private practice because they are often underrepresented in studies of appraisal skills training.3335

Only 46% (Figure 3) of those approached actually terminated the study completely. Although this limits the external validity of our findings, it reflects the reality of continuing medical education where physicians are usually free to decide which program they want to take part in. Though, study participants’ characteristics were similar to the population of GPs in Hessen and Germany.

More serious is the high dropout rate (54%) which differed between study arms (IG: 62% vs CG: 47%). As a result, the power of the study was limited and our findings therefore failed to reach statistical significance.

Apart from a lack of power due to the small sample size, a ceiling effect remains as a possible explanation for the negative result. Theoretically, even control physicians may operate at such a high level regarding their SAC that a WS of the kind evaluated in this study would not improve the outcome to a relevant degree. We regard this as unlikely. However, there remains the possibility that we recruited GPs with above-average SAC skills. This would be an alternative explanation for the negative findings of our study.

In any case, it is known that training effects become harder to detect with longer professional experience, more previous knowledge, and higher qualifications.5 Furthermore, a longer and more intense learning experience might be required to achieve a measurable increase in SAC.36

Conclusion

Conclusions regarding our learning project: the present RCT could not prove the superiority of our intervention based on the three-step heuristic. Lack of power and above-average SAC skills of the sample are the most likely explanation. Moreover, longer and more intense teaching efforts are required to improve experienced GPs’ skills and change their behavior. On the other hand, the WS was well accepted and participants were highly satisfied. We therefore suggest that educators and researchers evaluate the three-step heuristic in their respective settings.

The high acceptance and proof of effectiveness have encouraged us to continue offering WSs for GPs and to further develop our LP. However, the success of our LP is made complicated by the fact that only two to three consecutive WSs are possible for a group of GPs. We are currently working on optimizing our material for the adoption of a blended learning LP. The initiative required of each participant in a blended learning scenario could help us manage the natural process of self-selection in a constructive way.

General conclusions regarding educational projects: an RCT is an important tool to test effectiveness. Applying only a pre–post test without a CG can easily lead to an overestimation of the effectiveness of an intervention.

Acknowledgments

We wish to thank 137 GPs from the pilot and intervention phase who gave their time to participate in this study. We thank GPs in Bielefeld Giessen, Kassel, Marburg, and Siegen.

We thank Nicole Burchardi, PhD, Clinical Trials Unit at the University of Marburg for statistical advice. We also want to thank experts in EbM for their support in the development of the study questionnaire: Lise Bjerre, MD (Department of Family Medicine, Georg August University of Göttingen, Germany); Judith Günther, MSc (pharmafacts, Freiburg, Germany); Ralf Karger, MD, MSc (Department of Haematology, Philipps-University of Marburg, Germany); Monika Lelgemann, MD, MSc (HTA-Centre, Bremen, Germany); Dagmar Lühmann, PhD (Department of Social Medicine, University of Schleswig-Holstein, Campus Lübeck, Germany); and Antje Timmer, MD (German Cochrane Centre, University of Freiburg, Germany).

The project was funded by the Federal Ministry of Education and Research in Germany (BMBF).

Disclosure

The authors report no conflicts of interest in this work.

References

1.

Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med. 1985;103(4):596–599.

2.

Ely JW, Burch RJ, Vinson DC. The information needs of family physicians: case-specific clinical questions. J Fam Pract. 1992;35(3):265–269.

3.

Ely JW, Levy BT, Hartz A. What clinical information resources are available in family physicians’ offices? J Fam Pract. 1999;48(2):135–139.

4.

Del Fiol, Guilherme, Workman TE, Gorman PN. Clinical questions raised by clinicians at the point of care: a systematic review. JAMA Intern Med. 2014;174(5):710–718.

5.

Westbrook JI, Coiera EW, Gosling AS. Do online information retrieval systems help experienced clinicians answer clinical questions? J Am Med Inform Assoc. 2005;12(3):315–321.

6.

Clarke MA, Belden JL, Koopman RJ, et al. Information needs and information-seeking behavior analysis of primary care physicians and nurses: a literature review. Health Info Libr J. 2013;30(3):178–190.

7.

Kosteniuk, JG, Morgan DG, D’Arcy CK. Use and perceptions of information among family physicians: sources considered accessible, relevant, and reliable. J Med Libr Assoc. 2013;101(1):32–37.

8.

Dwairy M, Dowell AC, Stahl J. The application of foraging theory to the information searching behavior of general practitioners. BMC Fam Pract. 2011;12:90.

9.

Coumou HC, Meijman FJ. How do primary care physicians seek answers to clinical questions? A literature review. J Med Libr Assoc. 2006;94(1):55–60.

10.

Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians’ clinical questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217–224.

11.

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71–72.

12.

Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. BMJ. 1999;319(7206):358–361.

13.

Curley SP, Connelly DP, Rich EC. Physicians’ use of medical knowledge resources: preliminary theoretical framework and findings. Med Decis Making. 1990;10(4):231–241.

14.

Young JM, Ward JE. Evidence-based medicine in general practice: beliefs and barriers among Australian GPs. J Eval Clin Pract. 2001;7(2):201–210.

15.

Cogdill KW, Friedman CP, Jenkins CG, Mays BE, Sharp MC. Information needs and information seeking in community medical education. Acad Med. 2000;75(5):484–486.

16.

McCord G, Smucker WD, Selius BA, et al. Answering questions at the point of care: do residents practice EBM or manage information sources? Acad Med. 2007;82(3):298–303.

17.

Slawson DC, Shaughnessy AF. Obtaining useful information from expert based sources. BMJ. 1997;314(7085):947–949.

18.

Haynes RB, Holland J, Cotoi C, et al. McMaster PLUS: a cluster randomized clinical trial of an intervention to accelerate clinical use of evidence-based information from digital libraries. J Am Med Inform Assoc. 2006;13(6):593–600.

19.

Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep). 2007;(149):1–69.

20.

Donner-Banzhoff N, Schmidt A, Baum E. Der evidenzbasierte Praktiker: Ein Beitrag zum hausärztlichen Informationsmanagement. [The evidence based practitioner: A contribution to family medicine information management]. Z Allg Med. 2003;79(10):501–506. German.

21.

Smith R. Strategies for coping with information overload. BMJ. 2010;341:c7126.

22.

Hall A, Walton G. Information overload within the health care system: a literature review. Health Info Libr J. 2004;21(2):102–108.

23.

Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Acad Med. 2005;80(2):176–182.

24.

Donner-Banzhoff N. P E R L E N: Patientenorientierte Evidenzbasierte Recherche Lernen Entwickeln Nutzen [The PEARL-Project, PEARL: Patient-orientated Evidence-based Appraisal Research Learning]. Allgemeinmedizin-Projekte. Available from: http://www.allgemeinmedizin-projekte.de/index.php?id=22. Accessed March 3, 2016.

25.

Lowe M, Rappolt S, Jaglal S, Macdonald G. The role of reflection in implementing learning from continuing education into practice. J Contin Educ Health Prof. 2007;27(3):143–148.

26.

Brunner E, Munzel U. Nichtparametrische Datenanalyse: Unverbundene Stichproben - Statistik und ihre Anwendungen. [Nonparametric data analysis: Unconnected Sampling - Statistics and its applications]. Berlin: Springer; 2002. German.

27.

McKibbon KA, Lokker C, Keepanasseril A, Wilczynski NL, Haynes RB. Net improvement of correct answers to therapy questions after pubmed searches: pre/post comparison. J Med Internet Res. 2013;15(11):e243.

28.

Badgett, Robert G., Dylla DP, Megison SD, Harmon EG. An experimental search strategy retrieves more precise results than PubMed and Google for questions about medical interventions. PeerJ. 2015;3:e913.

29.

McKibbon KA, Fridsma DB. Effectiveness of clinician-selected electronic information resources for answering primary care physicians’ information needs. J Am Med Inform Assoc. 2006;13(6):653–659.

30.

Gigerenzer G, Todd PM. Simple heuristics that make us smart: Evolution and cognition. New York: Oxford University Press; 1999.

31.

Güthlin C, Beyer M, Erler A, et al. Rekrutierung von Hausarztpraxen für Forschungsprojekte – Erfahrungen aus fünf allgemeinmedizinischen Studien [Recruitment of Family Practitioners for Research - Experiences from Five Studies]. ZFA. 2012;88(4):173–180.

32.

Gorman PN, Yao P, Seshadri V. Finding the answers in primary care: information seeking by rural and nonrural clinicians. Stud Health Technol Inform. 2004;107(Pt 2):1133–1137.

33.

Weberschock TB, Ginn TC, Reinhold J, et al. Change in knowledge and skills of Year 3 undergraduates in evidence-based medicine seminars. Med Educ. 2005;39(7):665–671.

34.

Hersh WR, Crabtree MK, Hickam DH, et al. Factors associated with success in searching MEDLINE and applying evidence to answer clinical questions. J Am Med Inform Assoc. 2002;9(3):283–293.

35.

Cabell CH, Schardt C, Sanders L, Corey GR, Keitz SA. Resident utilization of information technology. J Gen Intern Med. 2001;16(12):838–844.

36.

Weberschock T, Dörr J, Valipour A, et al. Evidenzbasierte Medizin in Aus-, Weiter- und Fortbildung im deutschsprachigen Raum: Ein Survey. [Evidence-based medicine teaching activities in the German-speaking area: a survey]. Z Evid, Fortbild Qual Gesundhwes. 2013;107(1):5–12. German.

37.

German Medical Association. Published statistics of the Federal Chamber of Physicians in Germany. Available from: www.bundesaerztekammer.de/page.asp?his=0.3.1667.6371. Accessed May 6, 2016.

38.

Landesärztekammer Hessen. Details of the Regional Chamber of Physicians in Hessen. Available from: www.laekh.de/aerzte/mitgliedschaft/mitgliedschaft-mitgliederstatistiken. Accessed May 6, 2016.

39.

Deutscher Ärzteverlag GmbH. 37.Representative sample: Koch K Primary Care in Germany – an International Comparison. Available from: www.aerzteblatt.de/pdf/DI/104/38/a2584e.pdf. Accessed May 6, 2016.

Creative Commons License © 2016 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.