Reading Between the Lines: Navigating Nuance in Medical Literature to Optimize Clinical Decision-Making and Health Care Outcomes

Introduction

The aim of clinical research is to design and implement ethical studies from which valid and meaningful results can be derived and applied to clinical practice for the purpose of increasing medical knowledge, competence, and performance, and improving health outcomes.^1–4 Research-based articles published in medical journals are key to reporting these results,⁵ but there are challenges inherent in the communication process.

While 95% of physicians want to learn about new trials, treatments, and procedures that can improve clinical outcomes, 68% report feeling overwhelmed by the amount of information they need to access to stay up to date.⁶ Current approaches to education in evidence-based medicine may contribute to the communication challenge.⁷

In 2016, Caverly et al reported that many third- and fourth-year medical students; internal medicine residents, internists, and faculty; and clinician-researchers with evidence-based medicine expertise, were unable to correctly rank the clinical importance of reported trial outcomes in terms of the “proof” they provided that “a new drug might help people”.⁸ Their inability to do so could lead to suboptimal clinical decision-making that might potentially lead to inefficient use of health care resources, avoidance of interventions that may be indicated and useful, or worse, use of interventions that do more harm than good.⁸

One possible explanation for this disconnect may be that education in evidence-based medicine does not prepare physicians adequately to make the best decisions for their patients based on the evidence they read.⁷ Researchers have found that clinicians and trainees have low to moderate understanding of important clinical trial data concepts.⁹ Studies that report surrogate endpoints, or composite outcomes that include multiple endpoints, especially if some of those endpoints are surrogates, can also be misleading.⁸ Only some family medicine clinicians report receiving formal training on clinical trial data during graduate school, medical school, residency, or fellowship.⁹ Failure to recognize the limitations of data has been shown to lead to misinterpretations of the clinical utility of that data even among specialists.¹⁰

We conducted a literature search to answer the question, what are the challenges affecting the interpretation of clinical trial results reported in the medical literature that may adversely affect clinical decision making and patient outcomes and how can those challenges be overcome? The objective of this article is to assist new and aspiring clinicians as well as experienced practitioners in critically assessing the medical literature so they can be informed about the latest medical advances in their areas of specialty and interest and confidently and appropriately integrate this knowledge into clinical practice to optimize health care outcomes for their patients. This article aspires to be a tool for residency and fellowship program directors, clerkship faculty, mentors, and other HCPs engaged in clinical education.

Methodology

A literature search of PubMed^® was conducted without limitation of publication date to identify empirical studies (preferred) and review articles on the role and influence of the medical literature in clinical decision making and the challenges and limitations of various study design and reporting techniques. Additional literature searches were conducted as necessary to more deeply explore specific topics revealed by the initial searches. The resulting articles were reviewed and synthesized by the authors to identify the challenges in assessing published clinical trial results and how these challenges can be effectively met.

Opportunities for Education

Understand the Utility and Limitations of the Various Methods by Which Medical Knowledge is Disseminated Outside the Classroom

It is difficult for HCPs to keep up with the ever-increasing volume of information that is available,^11–14 but that problem is not new.^15,16 In 1954, practitioners were having difficulty keeping up with ~400 English language medical journals.¹⁵ Today, the National Library of Medicine lists ~300,000 publications in PubMed,¹⁷ and between January 2000 and January 2022, MEDLINE cited an average of >300,000 newly published articles annually in the United States alone.¹⁸

Medical publishers are seeking new ways to make communicating information to HCPs more accessible, digestible, and memorable.^11,19–24 Innovations such as plain language or lay summaries, video/graphic/animated abstracts, talking-head and procedural videos, author interviews, audio slides, infographics, webinars, and podcasts are being incorporated into publishers’ offerings to aid in the visualization and interpretation of data.^19–23

HCPs also get information from social media channels including Facebook, Twitter, Instagram, YouTube, and podcasts. Social media deliver messages in sound bites that are easy for audiences to digest.²⁵ But sound bites cannot provide the depth of information necessary to promote critical evaluation and foster the deeper understanding necessary for clinical decision-making. Social media are also infamous for spreading misleading and false information.²⁵ With the emergence of CME-certified tweetorials, the need for cautious consideration is amplified.²⁶

Digital enhancements are just that—enhancements. The journal article is still the foundational published resource upon which the enhancements are constructed, and where readers will find the details they need to assess whether the study results can, and should, support sound clinical decision-making.^13,16,27

Know How to Critically Evaluate and Interpret Literature-Based Information

Following is a guide to help HCPs in training and in clinical practice to strategically select the most pertinent articles for optimal clinical decision-making, and to derive more value from those articles. For educators, the following information identifies areas in which more knowledge is needed to empower HCPs to critically evaluate what they read and reap the many benefits the medical literature has to offer.

Choose Appropriate Resources

Hundreds of thousands of citations are added to the library of medical literature each year.¹⁸ This is where selectivity comes in. It is not difficult to find tips on how to create strategies and systems to tame the jungle of medical literature.^{11–14,28,29} Distillation services that scan for topics of interest and provide alerts or daily, weekly, or biweekly summaries are often recommended.^11,14,28,29 However, this may reduce the likelihood that the reader will be motivated to critically consider the sources of information contained in those summaries.

We recommend the following when choosing appropriate sources from which to derive information that will be synthesized into clinical recommendations:

● Identify the leading journals in the field of interest

Identifying the top journals in a particular field of interest is not always obvious. This is where knowing impact factors, acceptance rates, altmetrics, and the EMPIRE Index, and recommendations, can help.

 ○ Impact factor is the measure of the average number of times articles in a journal have been cited by other authors.³⁰ While it may correlate with journal reliability or prestige, and suggest the journal is of high quality, impact factor is not a measure of quality. It is also not necessarily a measure of a journal’s popularity with readers. Keep these limitations in mind.

 ○ Acceptance rate is most commonly defined as the measure of the number of manuscripts accepted by a journal divided by the number of manuscripts submitted to that journal.³¹ It can serve as a proxy for the perceived prestige and demand for a particular journal compared to its availability.³¹ However, acceptance rate does not identify the reasons manuscripts have been rejected, which could have more to do with a manuscript’s applicability to the journal’s editorial mission than with quality or scientific rigor.³¹ Consider as well that the most prestigious journal in a given field may have a low acceptance rate simply due to the desire of authors to submit their manuscripts.

 ○ Altmetrics are a diverse set of alternative metrics and qualitative data that complement traditional citation-based metrics.³² Altmetric indicators include record of attention (the number of people exposed to and engaged with a scholarly output), measure of dissemination (where a piece of research is being discussed in scholarly and public venues such as in the news, social media, and blogs), and indicator of influence and impact (where and how a piece of research is affecting a field of study, public health, or society).³²

 ○ EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index is a value-based, multicomponent metric framework for measuring the impact of medical publications.³³ It comprises three component scores incorporating related altmetrics to indicate social, scholarly, and societal engagement with a publication.³³

 ○ Recommendations from a colleague, peer, or mentor on what journal(s) they find most useful can be helpful and provide appreciable savings.

● Confirm Retracted Articles

A recent cross-sectional study found 46 retracted articles on COVID-19; most of which had been published in scientific journals and more than half of which remained available as original unmarked documents despite their retraction.³⁴ Retractions attempt to remove false or erroneous information from the medical literature, but they do little to erase their impact. Over time, retracted articles may be cited more often after they have been retracted than before their retraction.³⁵ Dissemination of false or misleading information undermines confidence in the scientific community and can have potentially dangerous implications.^34–36 Readers must check regularly with the publishers of articles they have read to learn whether an article has been retracted. A faster and easier way is to subscribe to the Retraction Watch database. This searchable database currently includes more than 38,000 retracted articles.³⁷ Subscribing to this database would enable HCPs to avoid retracted articles that might possibly misguide their clinical decision-making.

● Avoid Paper Mills and Predatory Journals and Publishers

Retracted articles often originate from so-called paper mills.³⁶ Paper mills are unscrupulous companies involved in large-scale production and publication of scientifically questionable articles and unethical practices.^36,38 They sell authorships to researchers, academics, and students; fabricate databases; and falsify peer reviews.³⁶ A research report on paper mills from the Committee on Publication Ethics (COPE) and the Association of Scientific, Technical and Medical Publishers (STM) concluded that submission of suspected fake research papers is a growing problem and provides practical help to HCPs in determining whether an article may have been generated by a paper mill.³⁹

Predatory journals and publishers have been defined as “…entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices”.⁴⁰ They do a great disservice to the scientific community, to HCPs who may rely and make clinical decisions based on the information they provide, and the public. Beall’s List of potential predatory journals and publishers is a valuable tool for HCPs to identify articles with questionable content due to predatory publishing practices.⁴¹

● Understand the Differences and Similarities of Traditional and Open-Access Publishing

In the traditional model, a publisher’s revenue is generated by subscriptions or association membership, by fees charged to individuals for access to specific articles, and by advertising.⁴² In the open-access model, articles are available without fee to everyone, and the publisher’s revenue is generated by an article processing charge (APC) paid by the author or funder.⁴² Although their legitimacy has been questioned, articles in high-quality open-access journals delivered via the publisher’s platform (gold open-access) and holding the Directory of Open Access Journals (DOAJ) Seal, are comparable to similar high-quality traditionally published journals.^43,44 HCPs should always consider the quality of the journal in which an article appears whether it is published traditionally or open-access.

● Be Aware and Wary of Artificial Intelligence (AI)

AI is here to stay, and it is already making inroads into medical publishing.^45–50 Although still in its infancy, technology tends to grow up quickly. By design and intent, it teaches itself.⁴⁸ But AI authoring is not without problems—both technical and ethical.^45–50 Pharmaceutical companies are experimenting with AI to handle translations and to draft clinical study reports.⁵¹ However, ethicists and publishers are against AI authoring due to its lack of accountability and its potential to produce fraudulent and inaccurate documents.^46–50 Publishers that currently allow some degree of AI use stipulate these technologies cannot be listed as authors and that their use must be disclosed.^46,48–50 HCPs should watch for AI involvement in articles and consider the potential for bias and inaccuracy in their use.

Assess Bias

Despite the peer review process, significant errors—both unintentional and intentional—can make their way into the published medical literature.⁵² It is therefore incumbent upon HCPs who read and rely on these articles to be ever vigilant. Bias, whether introduced in the design or execution of a clinical trial (methodological bias) or in the reporting of clinical trial results (reporting bias), can influence the perceived utility of a treatment, mislead prescribers, and lead to suboptimal clinical decisions.^53,54 Here we discuss several biases that can potentially skew the results of a clinical trial and/or how HCPs may perceive those results

● Methodological Bias

When reading an article reporting the results of a clinical trial, HCPs should consider whether, and if so, how, the study design (eg, randomized controlled trial versus retrospective/real-world evidence, open-label versus blinded, cross-over versus parallel treatment arms, comparators), participant selection criteria, randomization procedures, or endpoints and outcome measures may have influenced the results.^54–59

When reading the medical literature, HCPs should ask themselves:

 ○ What is the question to be answered, and does the study design appropriately explore that question?⁵⁸

 ○ How might bias have influenced the study design in terms of the choice of intervention or comparator, the study outcomes, and the choice of analysis?⁵⁹

 ○ Might any performance biases have been introduced while the study was being conducted, such as investigators or participants being aware of the outcome measures?^54,56

 ○ During the study, might deviations from the study protocol have affected the results?^55,56

 ○ Could one study variable have influenced another study variable?⁵⁸

 ○ How might study discontinuations have influenced the results, and was enrollment sufficient for the results to be clinically and/or statistically meaningful?^54,56,60

Clinical trial registration in a publicly accessible register such as ClinicalTrials.gov has been shown to decrease the risk of bias, and HCPs can employ the Cochrane Risk of Bias (RoB2) assessment tool to assess the risk of bias in randomized controlled trials.^54–56 Numerous ways to limit or reduce the risk of methodological bias in clinical trials have been proposed.^{52–55,59,60}

● Reporting Bias

 Reporting bias is when the outcomes of a trial affect how those results are reported.⁵³ When reading the medical literature, HCPs should consider such things as whether the data reported align with the outcome measures identified in the study methodology, or whether only favorable results may have been cherry-picked for inclusion because they support the interests (financial or otherwise) of the study sponsor or the authors.^{53,55–57,59} HCPs should also consider whether the statistical methodology was selected to show the data in the best light, and whether the data may have been manipulated to show statistical significance.^52,59 Other reporting biases include publication bias, which may favor the publishing of certain types of research over others or preference for publishing studies that report favorable outcomes;^57,59 distorting or misrepresenting results to make them appear more positive;^56,59 allowing the study results to influence journal selection, the number of journals in which the results will be published, or the language in which the results will be published;⁵⁷ and whether the authors or study sponsor will seek rapid publication, normal or delayed publication, or decide not to publish the results at all.^53,56,57,59

● Conflict of Interest

 Conflict of interest (COI) can create biases that affect every stage of the clinical trial process, from study design through results reporting.⁵² Study sponsors, investigators, or authors could potentially be biased by self-interest; but as Hirsch suggested in 2009, journal editors also have the potential to inject bias into medical publishing by applying different standards to authors who are affiliated with drug companies (presuming inherent bias) versus authors who are presumably independent of drug company influence.^61,62 HCPs should consider how COI statements, authors’ affiliations, and funding sources may influence the article they are about to read.

Study Limitations

Although often not reported by study authors and frequently overlooked by readers, the statement of study limitations in research articles identifies potential weaknesses that may have influenced the outcomes and conclusions of a study.⁶³ Study limitations put results within the proper context, allow the reader to critically assess the value of the reported outcomes to clinical practice, and ensure transparency of the researchers and the research process. HCPs should be educated to spend at least as much time considering the limitations of a study as the results.

Navigate the Sea of Statistics

● P-values

 Often overemphasized in the reporting of research, p-values may not be as important, or as reliable, a gauge of statistical significance as they were once considered to be.⁶⁴ The American Statistical Association notes several limitations to the p-value that HCPs should consider when reading the medical literature.⁶⁴ A p-value alone is not a statement about the probability that a study hypothesis is true or that the data were produced by random chance alone.⁶⁴ Neither does a p-value measure the size of an effect or the importance of a result.⁶⁴ When reading the medical literature, HCPs should consider whether specific results have been cherry-picked for their statistical significance.⁶⁴ Authors should be transparent and thorough in their reporting of all hypotheses explored, all data collection decisions made, and all statistical analyses conducted.⁶⁴

● Causation Versus Association

 Treatments lead to consequences—some beneficial, some adverse, some intended, some unintended. Although an association may be observed between a treatment and a particular outcome, this does not mean the outcome was caused by the treatment. As studies conducted during the COVID-19 pandemic most recently demonstrated, factors such as study design, sample size, and randomization can make it impossible to determine whether an outcome was caused by a treatment or merely observed.⁶⁵ HCPs must be educated to critically consider whether published results support a conclusion of an observed or a causal relationship between the treatment and the outcome.^63,65

Limitations

This article is limited by the accessibility of studies published in the medical literature and the authors’ interpretation of those studies.

Conclusion

HCPs rely on an overwhelming and ever-increasing volume of medical literature throughout their careers to help inform and guide clinical decision-making. Journal articles reporting the results of clinical trials are an important source for guidance, but they are not the only source. While digital enhancements and social media serve to make the medical literature more digestible, HCPs must ultimately refer to the articles that report detailed study design and methodology, results, limitations, COI, and acknowledgements for full context. Still, it can be challenging for readers to fully appreciate the myriad nuances that can affect the accuracy, utility, and applicability of reported data.

With more than 300,000 articles being published in medical journals annually, plus snippets and soundbites delivered via social and other media channels, it is understandable that HCPs could be overwhelmed by the volume of information they may feel compelled to digest in order to make sound clinical decisions and recommendations for their patients. Narrowing the choices to those that are most pertinent and appropriate can be done with the use of Impact factors, acceptance rates, altmetrics, EMPIRE index, and recommendations of colleagues; being attentive to retractions; avoiding paper mills and predatory journals and publishers; and considering quality whether a journal is published traditionally or open access. Despite the many checks and balances in place to ensure accurate and ethical reporting of clinical trial results, readers must be vigilant in considering the potential for errors as well as unintentional and intentional methodological and reporting biases, conflicts of interest, and study limitations. Reliance on p-values as an indicator of statistical significance may lead to incorrect conclusions about a treatment’s impact, and numerous factors can blur the assessment of observed versus causal relationship between a treatment and outcome.

Future continuing professional development interventions can address these challenges and thus help to improve patient care and health outcomes by educating HCPs about the utility and limitations of the various methods by which medical information is disseminated, and by giving HCPs the knowledge to critically evaluate and interpret literature-based information. Likewise, HCPs can help themselves by seeking out only the most reputable and relevant sources of information, identifying and avoiding articles that have been retracted and articles that have been published by paper mills and predatory journals and publishers, and by being vigilant in the assessment of biases, conflicts of interest, study limitations, and the use of statistics in the reporting of clinical trial results. A list of resources has been provided to aid in this pursuit (Box 1).

Box 1 Resources for Assessing the Validity and Utility of Published Research Articles

Lessons for Practice

• Health care professionals rely on published reports of clinical trials to guide medical practice.
• Results of clinical trials are influenced by a range of study design and reporting variables; the nuances, impact, interpretation, and application of which are rarely taught inside or beyond formal training environments.
• Continuing professional development interventions can address this challenge and thus help to improve patient care and health outcomes by educating health care professionals about the utility and limitations of the various methods by which medical information is disseminated, and by providing the knowledge necessary to critically evaluate and interpret literature-based information.

Implications for Future Study

Extending the work of Caverly et al,⁸ it would be beneficial to explore the ability of trainees and attendings in various subspecialties (eg, family medicine, surgery, internal medicine) to critically interpret clinical trial results reported in the medical literature.