Finding a needle in the haystack: performing an in-depth literature search to answer a clinical question

Introduction

The phrase “evidence-based medicine”^1–3 has evolved into the term “evidence-based practice”⁴ (EBP) and further expanded to denote a solution to bridge the existent gap between research results and clinical practice.⁵ Three foundational components form the cornerstones of EBP: patient preferences and values, clinical expertise, and current science.^4–7 The application of evidence links all three cornerstones, most importantly, in the context of the currently published evidence. Evidence-based clinical interventions provide one strategy for improving health care quality, reducing clinical errors, and ultimately, improving health outcomes.^8,9 Thus, deep-rooted traditions without scientifically demonstrated evidence of efficacy should be questioned. For example, historically, capped intravenous lines were flushed with heparin to assure patency between infusions. Nursing investigators¹⁰ demonstrated flushing locks with saline yielded patency without the risk associated with heparin. The tightly held practice of heparin use was changed to use of saline flushes as a result of this evidence.

Regarding the use of scientific evidence, general consensus is that in EBP, there are four primary steps to integrating evidence: developing a clinical question, locating relevant published evidence, evaluating the literature found, and then applying the literature.^11–18 However, barriers to implementing EBP include: an overwhelming amount of information; lack of skill in search/retrieval; lack of time; challenges in reading, evaluating, and understanding the reports; negative attitudes toward research; and limited exposure to research-utilization strategies.^5,19–21 Locating the needed published research to serve practice purposes can be a daunting task, rather like finding a needle in the proverbial haystack, and requires skill in order to perform a search proficiently and without being overwhelmed. An understanding of electronic procedures can help to develop search expertise and skills that, not only target the most appropriate information available, but also help to manage the overwhelming amount of information available. The purpose underlying this article was to provide the clinician with a step-by-step process for completing a comprehensive literature search process, focusing on search techniques that are useful and highlighting helpful tips for streamlining the search processes.

The point at which search of the literature begins will be different for every searcher. Developing a clinical question is the first step, although searching through literature based on a general topic and population prior to developing a clinical question can provide information that also serves to develop the question. Numerous recommendations for clinical question development are available. Richardson et al²² discussed the use of a framework to guide development of well-stated clinical questions to solve practice problems, emphasizing whether the question is specific to diagnosis, prognosis, therapy, or prevention, with all areas including education. Armstrong²³ also detailed the development of clinical questions. In other work addressing the components of a template – Patient/Population, Intervention, Comparison, and Outcome – the acronym PICO evolved, subsequently with an added “T” to represent Time (PICOT).^22,24 Other frameworks have been developed as well, representing variation but ultimately serving the same purpose. The Person, Environments, Stakeholders, Intervention, Comparison, and Outcome (PESICO) framework²⁴ is an example of an alternate form. Regardless of the framework used, if any, to structure the clinical question, how precisely the clinical question is posed determines the initial steps that guide the first foray into the literature.

Use of template frameworks to structure clinical questions can be quite helpful in thinking through issues but can be limiting in terms of best evidence for the clinical problem or related interventions. It is important to avoid approaching the search with a solution already in mind as an a priori bias may limit the search prematurely, decreasing the rigor of the exercise by excessively narrowing the information retrieved. For example, searching the literature with a focus on comparing whether traditional hand washing or use of hand sanitizers is more effective in reducing spread of infection focuses on a solution rather than the problem of infection spread. Approaching from a solution orientation rather than the problem orientation potentially creates a scenario where other possible options (beside hand sanitizers) that might be appropriate for comparison to traditional hand washing could be missed and thus, not considered. Thus, initial searching must focus on the problem or clinical question to be most effective. Snowball²⁵ identified that working through clinical question development and simultaneously beginning to search, aided the process of clarifying clinical questions.

Form and location of published literature

Citations to published literature most often can be located in electronic form through searchable databases. Citation information generally includes basic information: the publication title; authors; publication date; and journal name, volume, issue, and page numbers. Most also include an article abstract. The actual articles can be found in both print journals, electronic journal copy, or may be available both in print and electronically.

One of the first challenges confronting clinicians in search of electronically available evidence is access to electronic databases. Access is a critical factor in searching, but a huge barrier for many clinicians. Electronic databases include subscription-only access and those that are open-access, which require no subscription. A graphic depiction of the overall relationships is shown in Figure 1.

Figure 1 Display of electronic search options, based on the searcher working through publicly available information and subscription only. Abbreviations: CINAHL, Cumulative Index to Nursing and Allied Health Literature; ERIC, Education Resource Information Center.

Two of the most commonly used databases in nursing include National Library of Medicine, available through PubMed^®,²⁶ and the Cumulative Index to Nursing and Allied Health Literature (CINAHL^®).²⁷ PubMed requires no subscription for access, and citation information as well as a substantial number of full-text articles are available in PubMed. However, CINAHL²⁷ is an example of a subscription-only service, which may be cost prohibitive for clinicians who are not affiliated with a subscribing agency. Databases like Educational Resources Information (ERIC^®)²⁸ include publications related to educational research that can be accessed openly for basic information, but accessing full information articles is done only by subscription in ERIC.²⁸

No search for literature related to clinical interventions would be complete without accessing the Cochrane Library²⁹ developed to provide synthesized systematic reviews of clinical studies. The Cochrane Library has a comprehensive collection of articles indexed. Overviews of a variety of research studies, a methodology register, economic evaluation, and technology databases are also available. The Cochrane Library has established a rigorous process for any reviews that are included in the collection. The Cochrane Library is available for anyone to search, but the search results are available only through professional association or institutional subscription.

An additional and important source is the Joanna Briggs Institute (JBI) from the University of Adelaide, Australia.³⁰ The JBI site includes a large and growing collection of systematic reviews and implementation science reports. Similarly to the Cochrane Library, the JBI uses rigorous procedures for all reviews wherein each report undergoes an extensive peer-review process, and clinical protocols are also available through the JBI. The JBI carries a global interest, with emphasis on the best available evidence for practice. While the Cochrane Library focuses on evaluating evidence exclusively from randomized, controlled trials (RCTs), the JBI reviews a broader range of levels of evidence. The JBI allows open access for searching and provides abstracts of the reports included in the collection but requires subscription to access complete reports. The JBI holds other information related to EBP as well.

An option for all searchers is the growing number of online sites that provide valuable access to downloadable peer-reviewed resources. These sites are listed as open-access sources in Figure 1. Open-access sites vary from indexed databases of citations and open-access journals sites to internet sites replete with articles and other items of interest. One example is MedEdPORTAL^®,³¹ which is primarily a free, peer-reviewed publication service provided by the Association of American Medical Colleges in partnership with the American Dental Education Association. However, one section of MedEdPORTAL is only accessible to individuals who have been verified to be faculty members of an academic institution.³¹

Another more general site available to all searchers is Google/Google Scholar, which can be a useful starting place to browse or access articles available in full text or to link to websites and journal publishers. The Directory of Open Access Journals³² is an index to which many open-access journals belong and can be searched free of charge. Most articles in open-access journals are downloadable free of charge as well.

Other alternatives provide single-site access to multiple databases. SUMSearch 2³³ is another source for searching that is openly available. This system is setup to simultaneously search MEDLINE^®, Database of Abstracts of Reviews of Effects (DARE), and the National Guidelines Clearinghouse. MEDLINE is a subset of PubMed, DARE provides systematic reviews emphasizing health care interventions, and the National Guidelines Clearinghouse houses published guidelines. The SUMSearch site supports substantial flexibility in searching, including the ability to toggle “on” and “off” the three databases.

The TRIP database (originally named “Turning Research Into Practice”)³⁴ provides access to up-to-date research evidence, including videos, leaflets, and news reports. The TRIP database is organized by publication topic, not by author.³⁴ Because the TRIP database overlaps published peer-reviewed articles, as well as videos, leaflets, and patient information handouts, it can be considered a database that represents traditional publications (eg, journal publications) as well as what is considered “grey literature”.

Grey literature

The importance of “grey literature” is increasing. Grey literature is a term used to denote patents, leaflets and pamphlets, government or scientific reports and white papers, and other literature considered unconventional and often overlooked. A formal definition of grey literature was established by the Fourth International Conference on Grey Literature:³⁵ “That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers.”³³

Often grey literature is overlooked by searchers because it represents information not found in the more highly regarded peer-reviewed journals or indexed in traditionally accessed databases.^36,37 Conference proceedings, some of which are indexed in CINAHL²⁷ and PubMed,²⁶ constitute a form of grey literature. A professional conference presentation is valued as a way to accelerate the implementation of the most current information into practice. A newly established repository, GreyNet, some grey literature is available,³⁸ with access by subscription. An additional site is that of the New York Academy of Medicine,³⁹ which provides a notification system for updating grey literature in the areas of health services research.

Grey literature is important to consider in completing a literature search for EBP purposes. Publication bias can occur when only studies with positive results are published or available for review. An unsuccessful intervention report or presentation completed at a conference may be just as valuable to furthering EBP. Although grey literature might or might not have undergone the meticulous peer review process, a rigorous evaluation and inclusion of relevant grey literature helps to limit bias and adds comprehensiveness to the search results.

Practice guidelines, systematic reviews, and meta-analyses

Published articles representing already synthesized information related to interventions and solutions for clinical problems are already available as practice guidelines, systematic reviews, and meta-analyses. The references used as primary sources in a meta-analysis or systematic review are always included in the reference list of the published report and can serve as a great source of information for the searcher.

The National Guideline Clearinghouse,⁴⁰ the Agency for Healthcare Research and Quality (AHRQ),^41,42 the Joanna Briggs Institute,³⁰ and the Cochrane Library²⁹ guidelines are examples of places to start looking initially. Unfortunately, sometimes clinical questions may not have research-based solutions readily available in published form.

A systematic review, such as published through the Joanna Briggs Institute³⁰ or Cochrane Library,²⁹ is generally considered one of the most robust forms of evidence because of the rigorous procedures in place to assure appropriate information and processes are met.⁴³ Systematic reviews are also indexed in PubMed and CINAHL. Meta-analysis is also considered one of the highest forms of research evidence, with a systematic review of multiple RCTs as the highest and individual RCTs the next highest.^3,43 Meta-analyses are considered to provide high-level evidence because of their objectivity and depth of statistical analysis as well as their ability to link cause and effect.⁴³

Choosing the databases

Initially, a decision regarding which database to search must be made. Traditional databases of published literature include The National Library of Medicine through PubMed²⁶ for comprehensive biomedical literature, CINAHL²⁷ for nursing and other allied health literature, and others, such as PsycINFO^®44 or ERIC.²⁸ These databases are entered through a variety of subscription-based portals, with the exception of PubMed, which is also publicly available. PubMed alone includes over 22 million citations, often with links to full-text articles.²⁶ CINAHL²⁷ also includes millions of articles. All of the databases have some overlap with the others. For example, some abstracts for dissertations are indexed in PubMed but can be accessed directly through Dissertation Abstracts Online.⁴⁵

The CINAHL²⁷ database and National Library of Medicine Database via PubMed²⁶ are commonly used by clinicians. Appropriate published information can also be found in the databases that emphasize the work of other disciplines or other major topics. These sources of published information should not be overlooked. For example, searching PsycINFO⁴⁴ for psychology literature or ERIC²⁸ for educational literature may be useful. PsycINFO and ERIC are subscription-only databases. The decision on which databases to use is important, to assure appropriate literature is located. For example, limiting a search to only PubMed or only CINAHL will produce incomplete search results.

Often multiple databases can be searched simultaneously, such as through EBSCO, provided by the searcher’s library. Searching in this manner is time efficient and effective in reducing the redundancies in search results that occur when singularly searching sequential databases. For example, PubMed and CINAHL have substantial overlap in the publications indexed in their databases. Searching both simultaneously will reduce that redundancy. However, library services software limitations determine what database subscriptions are possible. Figure 2 is a schematic demonstrating use of library services and the selection of several databases for search simultaneously. Options and appearance will vary depending on the access provider, but most access providers include similar components. Becoming familiar with the library system, the processes set up within the library, and the access provider software is critical for effective and trouble-free searching.

Figure 2 Depiction of the process for selecting multiple databases for simultaneous searching, as supported by systems such as EBSCO. Abbreviations: CINAHL, Cumulative Index to Nursing and Allied Health Literature; ERIC, Education Resource Information Center.

Search vocabularies

Regardless of the database being searched, refinement of the search through the use of pertinent key words is critical for any search. Knowing and understanding how information is categorized can make the difference between a successful or failed search. Unlike searches with engines such as Google, in which essentially any term can be used for a search, searches of electronic citation database systems, such as CINAHL and PubMed index articles, are best accomplished using a systematic application of keywords integrated into the systems themselves.

A “controlled vocabulary” is a listing of keywords and search terms that is formally developed for searching in a particular system, such as medical subject heading (MeSH) terms in PubMed,²⁶ and used in indexing each citation included in the database. Whether using a search term from a controlled vocabulary or using an alternative term, the precision and relevancy of the search terms to the topic are important and are directly proportional to the resulting search results. Search results can be maximized if the search terms are those terms included in controlled vocabularies, but the databases can be searched generally by whatever terms the searcher develops.

Familiarity with library services also means enlisting the assistance of librarians, who are well versed in the services of their libraries, in search processes, and in the means for developing the most precise search possible for the clinical question posed. Precision searching reduces inclusion of unnecessary or irrelevant citations.

Every article indexed in a citation database is indexed with the key terms used in the database. These key terms make up the controlled vocabulary used to index the articles. As many terms as needed to represent the content of the article are assigned; PubMed indicates as many as 5–15 terms, plus additional subterms and population characteristic terms.⁴⁶ There are usually additional options of searching on most any term and, in some databases, phrases. In PubMed, the terms are known as MeSH terms. In CINAHL, the term list is labeled the CINAHL/MeSH list, representing a modified CINAHL-specific listing of subject terms. These terms (“major” and “minor”) are assigned as subject headings to each article when the article is entered into the database. In PubMed and CINAHL, a searcher can view the subject headings themselves in a listing form and additionally, in PubMed, can review through a hierarchical tree-type structure. In most systems, there are detailed tutorials available for honing skill in use of the terms (eg, see http://www.nlm.nih.gov/bsd/disted/pubmedtutorial/cover.html). Becoming familiar with relevant key terms streamlines the search process, linking up the selected terms and articles in the database. Learning about the vocabulary in these databases is integral to search processes. Examples of subject headings are demonstrated in Figure 3. MeSH heading can be browsed or viewed from a hierarchical tree-type structure with links to activate the terms and subcomponents (see: http://www.nlm.nih.gov/mesh/MBrowser.html). The plus signs in Figure 3 indicate an additional listing that can be activated, and the specific numeric code for the term is included in brackets. For example, the category of “Names Groups” opens to an alphabetical list of 54 links, ranging from “Adult Children” through “Vulnerable Populations”, and “Women”,⁴⁷ with each representing a MeSH term used in PubMed for indexing articles. A sampling of these items is included in the right side of the figure. Strategic use of these terms in any search strategy will help to identify only articles indexed with the terms used and exclude extraneous articles.

Figure 3 PubMed primary subject categories, with examples of related MeSH terms. Note: + indicates option to expand item to multiple listing. Listing represents only beginning and end of the PubMed listing (http://www.nlm.nih.gov/mesh/MBrowser.html). Abbreviation: MeSH, medical subject heading.

CINAHL listings vary from PubMed, but the process is the same: selecting major topical categories and the subcategories, as relevant. Figure 4 depicts the selection of “Accidental Falls” as a major conceptual category. The subcategory list provides options for refining the search, depending on the searcher’s targeted clinical question or general topic area.

Figure 4 Subject headings categories in CINAHL® using “falls” as example. Abbreviation: CINAHL, Cumulative Index to Nursing and Allied Health Literature; IOW and IOWA, State of Iowa, USA; NOC, Nursing Outcomes Classification; NIC, Nursing Intervention Classification; SABA CCC, Virginia Saba Clinical Care Classification System.

The purpose of any search is to perform a thorough search that garners expansive results that can then be refined, through use of precise delimiters and filters, to produce a valuable and focused result. Integration of Boolean operators also supports narrowing or broadening searches to match the topic or clinical question. Using controlled vocabulary terms, like MeSH terms, is always useful. For example, using the term “geriatric”, a term in CINAHL, will yield more results than “old people” or “elderly” because when the article is entered into the system, indexers link the most appropriate related terms. While searching, use of the “suggest subject terms” checkbox is helpful. For the term “falling”, CINHAL²⁷ suggests when exploded or expanded such terms as “accidental falls” and “fall prevention”. If the database programming “maps” or relates something like the term “falls prevention” as related to “falls”, the information sought will be found; if the database programming is not set to map the two terms, the search yield will be less precise.

Boolean operators

Boolean operators are simple to use but must be understood conceptually in order to maximize search strategies. The use of Boolean operators helps to group search terms together to refine the search strategy. A graphic depiction of Boolean operators is included in Figure 5. In Figure 5, each circle represents a topic, typically represented in a search by one or more search terms. The groupings represent the action for the search, based on the operator listed. The overlapped, highlighted areas represent the actual yield for the associated Boolean operator. Thus, the use of “AND” provides a refined search yield, while “OR” provides an encompassing search yield, and “NOT” removes any publication of a particular search term.

Figure 5 Boolean operators.

In search techniques, Boolean operators (AND, OR, or NOT) are used in finessing searches so that the searchers can combine terms in a pattern that is carefully crafted, to produce the search yield to the most specific citations matching the terms. For example, if searching on “geriatric patients” and “falls”, use of AND to link the terms will produce only those citations that include both the terms. Use of OR would link citations that included either gerontology or falls, making the search yield larger because only one term (not both) would need to be in any given citation. Use of NOT is generally reserved for assuring specificity. The operator NOT should be used with caution, because the term can remove relevant citations. For a full description of using Boolean operators for searching, see tutorials in PubMed.

Delimiters

Setting the desired characteristics to limit the search by using delimiters is useful as well. Different systems utilize different options for delimiting, but most often, the year of publication or range of years, language of publication, type of report (eg, research vs published guidelines), and other items can be included to further streamline the search to target the most relevant citations. For example, a search limited to “research” AND “fall prevention” and in “English” in the last 5 years reveals 1,842 citations. Narrowing the search further to include only “systematic reviews”, 132 citations are found, which is much more manageable but, important to note, removes primary reports and limits citations to only reported citations indexed as systematic reviews. Limiting the search to systematic reviews creates a potential bias because the report presents the interpretation of the investigators doing the reviews, not the investigators who completed the original research. If a further limit of “full text only” citation is applied to this search query, the number of citations reduces to 15 (search completed September 19, 2013, using CINAHL through EBSCOhost).

Caution must be used in applying delimiters. Limiting a search to only full-text articles potentially can remove important articles that are not available in full text, introducing a substantial bias into the overall literature obtained. “Print-only available” articles need to be requested from a library or requested from an interlibrary loan program. In reviewing “systematic review” literature, caution in interpretation is needed. Scrutinizing the reference list is a helpful tactic that provides the primary evidence used in the secondary review. It is important to note that search results change from day to day as electronic databases are updated regularly.

Wildcards and delimiters

Wildcard and delimiter use help the searcher to hone the search to a high degree of precision. The following are tips for performing a literature search in a traditional bibliographic database, such as CINAHL or PubMed. First, advanced search screens provide more detailed options and control over the search, thus are recommended if available. Second, careful use of the wildcard asterisk (*) key truncates the search term to the core, and the search will automatically expand to include variations on the term in the database. For example a simple search using “ger*” as search term will yield results for “geriatric”, “geriatrics”, “gerontology”, etc, but will also pick up “Germany” and “germ cell”. If the search result included citations related to “German gerontology”, the findings may or may not be relevant to the clinical question, but citations linked to “germ cell” would likely be irrelevant. Third, using age delimiters will streamline the search to exactly the age group of interest. For example, “Aged: 65+ years” or “Aged: 80 and over”. Therefore, as shown in Figure 3, age-specific search parameters can be done through MeSH categories in PubMed or through indicators elsewhere in the search software setup. A specific research design can be used as a delimiter as well, for example, “Randomized Controlled Trial”, etc. Finally, delimiters using year of publication will provide only publications with the years designated. Using the delimiter for language (eg, English only) will limit the search yield to only those publications presented in the language selected. A blend of search terms and use of specific delimiters must be done cautiously to assure a balanced refinement of the search that targets precisely the more relevant citations.

The example of combining of the operator “NOT” with an asterisk (wildcard) “ger*” illuminates the process of finessing search terms. “Ger*” will produce, not only citations indexed on “geriatrics”, “gerontology”, “gerontologic” but also, those indexed to “German”, and “germ cell”. Combining NOT can remove all citations indexing “German” or “germ cell”, if appropriate for the search.

A thorough review of the literature demands a foundation of rigorous searching to identify relevant literature, and experienced reviewers know that basic results yielded from a search do not necessarily translate to capturing the citations needed. As with any search, more does not equal better, and less does not always equal best. Often multiple searches are done while reviewing the results on an ongoing basis as the search is refined. The information located must be focused, precise, and “on topic”. An understanding of controlled vocabularies, Boolean operators, and delimiters is helpful for any searcher. Enlisting the help of a librarian is particularly helpful. Searchers can formulate their searches in any number of ways, but the best start is to provide a structure with the clinical question or related terms, relevant delimiters, and a process that takes multiple attempts to refine the search strategies well. Although delimiters should be used in searches, using too many delimiters will overly limit the search and result in too few results. It is always better to start broadly and then narrow the search. While the original search may yield 1,500 citations, the final yield may include ≤100 focused, relevant articles. Manageability and thoroughness are paramount. The goal of any search is to locate the best evidence on the topic: appropriate, relevant literature, without capture of too many, too few, or irrelevant citations. A listing of helpful hints is demonstrated in Figure 6.

Figure 6 Ten tips for effective searching. Abbreviations: CINAHL, Cumulative Index to Nursing and Allied Health Literature; MeSH, medical subject headings.

“Really simple syndication” feed

The goal in locating relevant literature always involves searching for the most up-to-date information, and staying up-to-date on any clinical question is challenging; however, repeatedly searching a database or searching across databases can become tiresome. A Rich Site Summary (RSS) (often called “really simple syndication”) feed sets up an automatic electronic notification system on any topic selected for the feed. Setting up an RSS feed can be beneficial and supports location of the most current information available on a given topic. The RSS feed pushes information to the user as updates on the topic identified become available. For example, by setting up a search for “geriatrics” AND “falls” and then setting up an RSS feed, new information will be pushed via feed whenever new articles that match those subject terms are added to the electronic database. The result is that instead of having to do a search over and over, the information, in a manageable format, is sent as it is published. An RSS feed is helpful when developing a body of evidence in a particular area, with a plan to use the findings over months or years, or for continual updating processes in clinical venues.

Consultation with a librarian

The ability to locate all kinds of information is paramount when doing a methodical and thorough literature search. Reference librarians have high-level skills for locating needed published information, while clinicians have strength in understanding the substance of what is included in the published report. Collaboration produces a synergistic effect to maximize efficiency in searching and outcomes, in terms of what is located during the search. Although with practice, any individual can develop competency in performing literature searches, a team approach can have advantages.⁴⁸ Librarians are conversant in the electronic search and retrieval process, and familiar with the wide array of citation databases and other electronic sources available. Further, the world of articles, books, guidelines, and other sources of information is the librarian’s domain, and research librarians enjoy the challenge of developing precision searches.⁴⁸

Determining relevancy

Once the search strategy is completed and the search is done, the decision about which citations can be discarded and which need to be retained is the first step in finalizing the set of research reports located in the search yield. The process has been labeled in a number of different ways, including the easily understood process of “berry picking” for relevance.^49,50 Berry picking is a multistep culling process based on relevancy, with relevancy dependent on the focus and needs of the individual doing the search and preparing for the review.^49–51 Preliminary information to review includes the title and abstract, followed by the whole document. Principles and processes for assessment decision making are published, including a figure diagram for a qualitative metasynthesis.⁵² These procedures are relevant to any endeavor requiring assessment of citations from searches.⁵² If a decision can be made to discard based on title or abstract, the citation can be removed completely and the whole article need not be evaluated. For example, if a search was focused on geriatric patients and falls prevention, and one of the citation titles indicates falls prevention in adult, nongeriatric patients, that record can be discarded, based on identification of a sample that mismatches the intended review. If the report appears to be a clinical intervention-related study but the abstract indicates it is simply an editorial or opinion piece, the citation can be discarded. Title is a useful tool but only if the title is well written and includes sufficient information pertinent to decision making, such as variables under study, population, and type of report, as recommended for a well-written report.^21,43 However, due to the nuances of research, journal parameters, and space allocation, these items are often not included in the title. The “maybe” decision point comes into play if the title and abstract are written in a manner that does not provide sufficient information to assess; “maybe” articles cannot be assessed properly without obtaining a full article and review of the full article.⁵³ The iterative process of sifting through and culling citations should culminate in a fairly comprehensive and tight database of relevant articles for the topic.

Once the collection of highly relevant citations is selected, the full articles need to be assessed to assure relevancy and integrity of the research or project purpose, design, overall techniques and procedures, outcomes, and results.^43,53,54 The research or project design used by any investigator is often attributed a hierarchical weight in order to assess the effects of clinical interventions. However, as Polit and Beck⁴³ emphasize, the hierarchy is biased to quantitative design research, especially for interventional evidence; thus, weighting cannot be universally applied, particularly with qualitative evidence and questions for which there is little research found in the literature. Evidence produced in the qualitative design realm emphasizes meaning and understanding, and utilizes predominantly narrative text as data, rather than statistical analysis of numbers found in quantitative reports.⁴³ Generally, evidence regarding interventions included in a systematic review or meta-analysis of all relevant RCTs provides the best evidence, but is limited to quantitative research only.⁷ The evaluation of RCTs requires careful analysis for conceptual or methodological errors, which can potentially lead to bias. All forms of bias can arise in research, so the research report must be scrutinized carefully.

Managing citations

Electronic bibliographic software, such as EndNote^® (Thomson Reuters Corp, New York, NY, USA), provides a robust and highly useful mechanism to manage citations to support research synthesis.^55–57 Electronic bibliographic software packages facilitate management and organization of citations and expedite the writing process. Features include: search capabilities within the references collected in the software databases, support attaching a copy of an article (or other documents), ease of accessing the capability of appropriate and relevant information needed (abstract, DOI, etc.), and linkage directly with PubMed²⁶ and CINAHL²⁷ for electronic downloading of citation material and word processing programs to facilitate using the citations in writing As well, the National Library of Congress can be searched directly from reference manager applications, for easy identification of books; searching directly through PubMed²⁶ and indirectly through such databases as CINAHL²⁷ can also be done. Both CINAHL and PubMed include a mechanism for exporting citations into bibliographic reference manager software. Manual input is also facilitated, for example, of web pages or other source information that is not easily downloadable via electronic means. Citations can be easily deleted from or added to the software for ongoing management of citations related to a particular topic. Additional fields are available within each citation record downloaded so that additional notes or coding is facilitated in these software packages.⁵⁶ The current version of EndNote has greater capability than prior versions, including the mechanism of searching for full-text article availability and the mechanism for attaching copies of documents to the citation record. Electronic bibliographic software is quite useful for the generation of written documents as well.

Evidence table

There are numerous terms used to describe an evidence table, including “summary table”, “evidence grid”, and “evidence matrix”.^21,43,58 Regardless of the term used, the overall purpose is the same – an evidence table is a tool that is developed to help manage citation information and related content on a given topic, readying for the synthesis process. The table includes records and columns, much like a spreadsheet, with the characteristics heading each column and each row representing a single citation. Information from each citation is entered into a row, across the columns of characteristics. The characteristics vary, depending on exactly what information is deemed important. Models of evidence tables can be found in any research or EBP textbook^7,21,43 and can be developed in a word processing or spreadsheet program. Summary or evidence tables can be extremely useful in organizing information to assist in identifying patterns in the information, grouping information into categories (eg, RCT studies vs systematic reviews), and in beginning the synthesis process.^21,43,58

Access

Clinicians affiliated with hospitals or academic institutions that have subscriptions to indexing databases have a decided advantage over those individuals who do not have these relationships. When affiliations do not exist, clinicians need to be a bit more creative in gaining access to electronic databases that require subscription. One option may be to pursue relationships, such as adjunct faculty status, to gain access. Another possibility is to explore local public library resources as an access point, especially publicly funded libraries at health science centers, which should allow access by citizens. Private universities often offer continued access to library resources to their graduates. Additionally, membership in professional organizations can be a source for access to journals that are linked specifically to the organizations, and to additional literature posted in members-only portions of the organization web sites.

Conclusion

EBP is necessary for providing safe, high-quality patient care.^8,9 Knowledge of current science is one of the three cornerstones of EBP.^2–7 Thus, regardless of educational preparation, all clinicians need the skills to search the literature for credible research-based solutions. Computerized databases make access to literature more readily available. However, the more readily available literature can provide an overwhelming amount of information, and finding the appropriate data is much like searching for a needle in a haystack. Developing the skills necessary to locate relevant literature is the first critical step to access current scientific information and understand the evidence on any given topic.^14,15,17 Search strategy skills help to hone a search that will extract the most relevant information from the existing published literature. The appropriate use of Boolean operations, search vocabulary, and delimiters enhance the quality of the search and are necessary to locate published evidence to address clinical problems. This is especially important in practice, given heightened patient complexity and time constraints^.7,59,60 Evidence-based clinical interventions provide one strategy for improving health care quality, reducing clinical errors, and ultimately improving health outcomes.^8,9

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	Rosenberg W, Donald A. Evidence based medicine: An approach to clinical problem-solving. BMJ. 1995;310(6987):1122–1126.
2.	Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71–72.
3.	Straus SE. Evidence-based medicine: how to practice and teach EBM. 3rd ed. New York: Elsevier/Churchill Livingstone; 2005.
4.	Goode CJ. What constitutes the “evidence” in evidence-based practice? Appl Nurs Res. 2000;13(4):222–225.
5.	Fineout-Overholt E, Levin RF, Melnyk BM. Strategies for advancing evidence-based practice in clinical settings. J N Y State Nurses Assoc. 2004;35(2):28–32.
6.	Vincent D, Johnson C, Velasquez D, Rigney T. DNP-prepared nurses as practitioner-researchers: closing the gap between research and practice. Am J Nurse Pract. 2010;14(11–12):28–34.
7.	Melnyk BM, Fineout-Overholt E, editors. Evidence-Based Practice in Nursing and Healthcare: A Guide to Best Practice. 2nd ed. Philadelphia, PA: Wolters Kluwer/Lippincott Williams and Wilkins; 2011.
8.	Institute of Medicine. The Future of Nursing: Leading Change, Advancing Health. Washington, DC: National Academy Press; 2010.
9.	Committee on Quality of Health Care in America [Institute of Medicine]. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
10.	Goode CJ, Titler M, Rakel B, et al. A meta-analysis of effects of heparin flush and saline flush: quality and cost implications. Nurs Res. 1991;40(6):324–330.
11.	Haroon M, Phillips R. “There is nothing like looking, if you want to find something” – asking questions and searching for answers – the evidence based approach. Arch Dis Child Educ Pract Ed. 2010;95(2):34–39.
12.	Phillips R, Glasziou P. Evidence based practice: the practicalities of keeping abreast of clinical evidence while in training. Postgrad Med J. 2008;84(995):450–453.
13.	Facchiano L, Snyder CH. Evidence-based practice for the busy nurse practitioner: part three: critical appraisal process. J Am Acad Nurse Pract. 2012;24(12):704–715.
14.	Facchiano L, Snyder CH. Evidence-based practice for the busy nurse practitioner: part two: searching for the best evidence to clinical inquiries. J Am Acad Nurse Pract. 2012;24(11):640–648.
15.	Facchiano L, Snyder CH. Evidence-based practice for the busy nurse practitioner: part one: relevance to clinical practice and clinical inquiry process. J Am Acad Nurse Pract. 2012;24(10):579–586.
16.	Facchiano L, Snyder CH. Evidence-based practice for the busy nurse practitioner: part four: putting it all together. J Am Assoc Nurse Pract. 2013;25(1):24–31.
17.	Fineout-Overholt E, Melnyk BM, Schultz A. Transforming health care from the inside out: advancing evidence-based practice in the 21st century. J Prof Nurs. 2005;21(6):335–344.
18.	Fineout-Overholt E, Williamson KM, Gallagher-Ford L, Melnyk BM, Stillwell SB. Following the evidence: planning for sustainable change. Am J Nurs. 2011;111(1):54–60.
19.	Hallas D, Melnyk BM. Evidence-based practice: The paradigm shift. J Pediatr Health Care. 2003;17(1):46–49.
20.	Melnyk BM, Fineout-Overholt E, Fischbeck Feinstein N, et al. Nurses’ perceived knowledge, beliefs, skills, and needs regarding evidence-based practice: implications for accelerating the paradigm shift. Worldviews Evid Based Nurs. 2004;1(3):185–193.
21.	DiCenso A, Guyatt G, Ciliska D. Evidence-Based Nursing: A Guide to Clinical Practice. St Louis, MO: Elsevier Mosby; 2005.
22.	Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995;123(3):A12–A13.
23.	Armstrong EC. The well-built clinical question: the key to finding the best evidence efficiently. WMJ. 1999;98(2):25–28.
24.	Schlosser RW, Koul R, Costello J. Asking well-built questions for evidence-based practice in augmentative and alternative communication. J Commun Disord. 2007;40(3):225–238.
25.	Snowball R. Using the clinical question to teach search strategy: fostering transferable conceptual skills in user education by active learning. Health Info Lib J. 1997;14(3):167–172.
26.	PubMed.gov [database on the Internet]. Bethesda, MD: National Center for Biotechnology Information, US National Library of Medicine. Available from: http://www.ncbi.nlm.nih.gov/pubmed. Accessed February 5, 2013.
27.	CINAHL® [database on the Internet]. Birmingham, AL: EBSCO; 2013. Available from: http://www.ebscohost.com/biomedical-libraries/the-cinahl-database. Accessed February 5, 2013.
28.	ERIC® [database on the Internet]. Washington, DC: Institute of Education Sciences. Available from: http://eric.ed.gov/. Accessed May 14, 2013.
29.	The Cochrane Library [database on the Internet]. Hoboken, NJ: John Wiley and Sons; 2013. Available from: http://www.thecochranelibrary.com/view/0/index.html. Accessed September 18, 2013.
30.	joannabriggs.org [homepage on the Internet]. Welcome to the Joanna Briggs Institute. The University of Adelaide; 2013 [cited March 26, 2014]. Available from: http://joannabriggs.org/. Accessed April 13, 2014.
31.	MedEDPORTAL® [database on the Internet]. Washington, DC: Association of American Medical Colleges (AAMC); 2013. Available from: https://www.mededportal.org/icollaborative/. Accessed May 14, 2013.
32.	Directory of Open Access Journals [database on the Internet]. UK: Infrastructure Services for Open Access (IS4OA); 2013. Available from: http://www.doaj.org. Accessed May 14, 2013.
33.	SUMSearch 2 [database on the Internet]. Wichita, KS: University of Kansas; 2014. Available from: http://sumsearch.org/. Accessed April 13, 2014.
34.	Trip. [homepage on the Internet] Trip Database; 2014. Available from: http://www.tripdatabase.com/. Accessed May 27, 2014.
35.	greylit.org [homepage on the Internet]. What is grey literature? New York Academy of Medicine; [cited January 31, 2014]. Available from: http://www.greylit.org/about. Accessed April 13, 2014.
36.	Oermann MH, Nordstrom CK, Wilmes NA, et al. Information sources for developing the nursing literature. Int J Nurs Stud. 2008;45(4):580–587.
37.	Farace D. Peering through the review process: Towards transparency in grey literature. TGJ: The Grey Journal. 2011;7(1):25–31.
38.	greynet.org [homepage on the Internet]. GreyNet International. Grey Literature Network Service. Available from: http://www.greynet.org/thegreyjournal.html. Accessed April 13, 2013.
39.	New York Academy of Medicine. Grey Literature Report. 2014. Available from: http://www.greylit.org. Accessed February 2, 2014.
40.	National Guideline Clearing House [database on the Internet]. Rockville MD: Agency for Healthcare Research and Quality; 2013. Available from: http://www.guideline.gov. Accessed May 14, 2013.
41.	ahrq.gov [homepage on the Internet]. Innovations exchange. Agency for Healthcare Research and Quality; 2014 [updated April 9, 2014; cited May 14, 2013]. Available from: http://www.innovations.ahrq.gov/. Accessed April 13, 2014.
42.	Agency for Healthcare Research and Quality. 2013. Available from: www.ahrq.gov. Accessed May 14, 2013.
43.	Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9th ed. Philadelphia, PA: Wolters Kluwer Health/Lippincott Williams & Wilkins; 2012.
44.	PsycINFO® [database on the Internet]. Washington, DC: American Psychological Association; 2014. Available from http://www.apa.org/pubs/databases/psycinfo/index.aspx. Accessed April 13, 2014.
45.	[homepage on the Internet]. Dissertation Abstracts International; 2013. Available from: http://www.umi.com/en-US/catalogs/databases/detail/dai.shtml. Accessed February 5, 2013.
46.	nim.nih.gov [homepage on the Internet]. PubMed tutorial. Indexing with MeSH vocabulary. US National Library of Medicine; 2001 [updated September 11, 2013; cited February 1, 2014]. Available from: http://www.nlm.nih.gov/bsd/disted/pubmedtutorial/015_030.html. Accessed April 13, 2014.
47.	nim.nih.gov [homepage on the Internet]. MeSH browser. US National Library of Medicine; 2014 [updated March 11, 2014; cited February 13, 2013]. Available from: http://www.nlm.nih.gov/mesh/MBrowser.html. Accessed April 13, 2014.
48.	Hopkins B, Callister LC, Mandleco B, Lassetter JH, Carlton T, Astill M. Librarians as partners of the nursing faculty in teaching scholarly inquiry in nursing to undergraduates at Brigham Young University. Sci Technol Lib. 2011;30:267–276.
49.	Bates MJ. The design of browsing and berrypicking techniques for the online search interface. Online Review. 1989;13(5):407–424.
50.	Barry CL. Document representation and clues to document relevance. J Am Soc Inf Sci. 1998;49(14):1293–1303.
51.	Wang P, Soergel D. A cognitive model of document use during a research project. Study I. Documentation selection. J Am Soc Inf Sci. 1998;49(2):115–133.
52.	Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–178.
53.	Vance DE, Talley M, Azuero A, Pearce PF, Christian BJ. Conducting an article critique for a quantitative research study: perspectives for doctoral students and other novice readers. Nursing (Auckl). 2013;3:67–75.
54.	Magarey JM. Elements of a systematic review. Int J Nurs Pract. 2001;7(6):376−382.
55.	Smith CM, Baker B. Technology in nursing scholarship: use of citation reference managers. Int J Ment Health Nurs. 2007;16(3):156–160.
56.	King R, Hooper B, Wood W. Using bibliographic software to appraise and code data in educational systematic review research. Med Teach. 2011;33(9):719–723.
57.	Zhang Y. Comparison of select reference management tools. Med Ref Serv Q. 2012;31(1):45–60.
58.	Aveyard H. Doing a Literature Review in Health and Social Care: A Practical Guide. 2nd ed. Maidenhead: McGraw-Hill; 2010.
59.	Fineout-Overholt E, Melnyk BM, Stillwell SB, Williamson KM. Evidence-based practice step by step: Critical appraisal of the evidence: part I. Am J Nurs. 2010;110(7):47–52.
60.	Fineout-Overholt E, Melnyk BM, Stillwell SB, Williamson KM. Evidence-based practice, step by step: critical appraisal of the evidence: part II: digging deeper – examining the “keeper” studies. Am J Nurs. 2010;110(9):41–48.