Factors Affecting the Use of Medical Articles for Citation and Academic Reference

Introduction

Evidence-based medicine (EBM), a set of principles and methods intended to ensure medical decisions are based on up-to-date and effective evidence, has been placed on par with antibiotics and anaesthesia by medical scientists in terms of importance.^1,2 The importance of EBM highlights how the success of a medical scientist is dependent not only on clinical excellence but also on the continued commitment to life-long learning. However, with over two million scientific articles published annually worldwide, the volume of medical literature poses a new challenge.³ In a physician survey conducted in 1989, two-thirds of respondents stated the volume of the literature was unmanageable, indicating that finding high-quality research can be challenging in light of the sheer quantity available.⁴ In addition, misleading statements and skewed representations of data found in existing literature further increase the difficulty in efficiently finding high-quality evidence.⁵

Bibliometric markers such as impact factor (IF), first coined by Eugene Garfield in 1955, have been developed to identify high-quality journals and evidence that should serve as a reliable source of information.⁶ Journal IF, currently the most widely used of bibliometric markers, is a measure of the frequency by which an article has been cited over a period of time.⁶ The benefit this provides is a quantitative method for comparing different journals. Despite its frequent use, IF is not without flaws. Articles may be cited to highlight misleading or erroneous information.⁷ Journals may also manipulate the numerator and the dominator used to calculate IF in order to make it appear more important.⁶ The inclusion of longer articles (with resulting fewer publications per issue) or the inclusion of citable articles (such as reviews that do not count towards the denominator) are examples of IF manipulations currently in place.⁷ These forms of citation and IF manipulations can lead to an inaccurate representation of a journal's importance.

A “citation classic” denotes a published peer-reviewed work receiving 100 or more citations from other peer-reviewed publications.⁸ The citations accumulated by said classics are considered to mirror the scientific merit of a piece of published work.⁹ While a useful measure, existing literature suggests that authors from higher income countries are often favoured over their lower income counterparts.^8,10 Furthermore, there is a tendency for English-speaking authors to publish, read, and cite English papers.⁸ Similarly, language barriers may prevent authors from considering papers outside of their own native language. These factors can lead to some publications, despite being of importance, being overlooked and forgotten.

The dawn of the internet has changed the face of literature access with the advent of open access (OA) journals, providing publications for free online.¹¹ While OA removes many of the financial barriers to literature access, the door then opens for so-called predatory journals masquerading as scholarly peer-reviewed journals.¹² Predatory journals are driven by profits, charge authors a fee, and do not provide a transparent peer-review or editorial process. These journals often report an unauthentic IF’s and allow manuscripts that have not gone through the rigorous peer review process to exist online.¹³ While convenient, medical scientists need to be aware of the low-quality predatory information that exists within some OA journals.

There are many benefits and drawbacks with increases in literature, bibliometric markers like citations and IF, and new methods of access such as OA. With so many factors to consider when conducting a literature search, the question remains: what factors does the medical scientist examine when considering articles for clinical and academic research?

This study aims to investigate the factors that influence authors when deciding which citations to include in their literature reviews and subsequent publications.

Methods

Study Design

Ethical approval for this study was obtained from the University of Manitoba Research Ethics board. Using SurveyMonkey (surveymonkey.com, California, United States), a questionnaire-based survey was designed. Face validity was provided by review and revisions with a group of medical research experts, with further review by potential respondents with further revisions. The questionnaire included quantitative questions with the option for participant comments on qualitative data. Once reviewed, it was distributed to participants via email. Respondents were sent follow-up reminders at one and three weeks. Participation was voluntary and anonymous. Informed consent was obtained prior to survey participation.

The survey was distributed using SurveyMonkey to medical students, residents, and faculty from multiple medical schools across Canada. Responses were collected from participants during the 2020/2021 academic year.

Survey Information

The survey questionnaire included questions about the respondent’s academic level (medical student, resident, faculty), age, academic institution to gather demographic data. Respondents were asked to rate the importance of article, journal, and accessibility factors when choosing a resource for citation. Ratings were scored on a 5-point Likert scale. Comments were allowed following completion of the survey to elaborate on responses or provide personal opinions on the subject matter. Descriptive statistics were used to analyze the data from the questionnaire with ANOVA followed by t-test analysis.

Results

The survey received 180 responses with 156 complete sets of responses. These included 78 trainees (61 medical students and 17 residents), and 78 faculty. Of the faculty 18 (23%) had been in practice for 1–5 years, 19 (24%) for 6–10 years, 20 (26%) for 11–20 years, and 21 (27%) for more than 20 years. The median age group for trainees was 25 to 29 years old, and for faculty was 40–49 years old.

The importance of factors in choosing a citation/reference were scored on a 5-point Likert scale (1-Not important, 2-Mildly important, 3-Moderately important, 4-Strongly important, 5-Very strongly important). Important article factors indicated in the data included relevance of the article (4.75), level of evidence (4.45), and language of the article (3.93), as shown in Figure 1. Country of origin and institution were indicated to be only mildly to moderately important (2.14, 2.26 respectively). Factors felt to be less important included the primary author (1.9), and the number of authors (1.51). Trainees felt that the year of publication was more important than faculty did, which was statistically significant (3.94 vs 3.47, p = 0.0016).

Figure 1 Importance for citation selection – Article factors. Note: *indicates statistically significant difference.

Journal and accessibility factors that were deemed important included availability on PubMed/Medline (3.77) and, somewhat less importantly, impact factor of the journal (2.97), as shown in Figure 2. The availability of the article in paper form was of low importance (1.52). The data showed statistically significant differences between trainees and faculty, with trainees indicating greater importance for availability on Google/Google Scholar (2.51 vs 1.88, p = 0.0013), and Open-access (2.46 vs 1.87, p = 0.0011); and faculty indicating greater importance for availability on PubMed/Medline (3.97 vs 3.56, p = 0.0078).

Figure 2 Importance for citation selection – Journal and access. Note: *Indicates statistically significant difference.

The frequency of methods used for finding citations/references was ranked on a 5-point Likert scale (1-Never, 2-Rarely, 3-Sometimes, 4-Often, 5-Very Often). Frequently used methods included PubMed/Medline (4.45), Citation lists from other articles (3.68), and Google/Google Scholar (3.42), as shown in Figure 3. Paper articles and libraries were used very rarely (1.42). PubMed/Medline was used more frequently by faculty (4.64 vs 4.23, p = 0.0009), and Google/Google Scholar was used more frequently by trainees (3.76 vs 3.08, p = 0.0002).

Figure 3 Methods for finding citations/ references. Note: *Indicates statistically significant difference.

The most commonly used type of citation was journal articles (4.77), while online videos were rarely used (1.57), as shown in Figure 4. There was a trend towards trainees using more citations of web pages compared to faculty (2.54 vs 2.29, p = 0.10), though this was not statistically significant and overall use was still low.

Figure 4 Types of citations used.

Open-ended comments from participants included some self-reported caution about using impact factor and about predatory journals.

Discussion

Faculty Vs Trainee

There are many ways to access medical literature in today’s modern age. While traditional libraries with paper journals are still available, online access via PubMed, Google, Google Scholar, Weblinks and other online databases serve an essential role with the increasing shift to online access. This survey found that respondents were more likely to use PubMed than other methods listed, while trainees were more likely than faculty to use Google/Google Scholar to complete a literature search. Compared to faculty members, trainees placed greater importance in articles being OA and free online. This is somewhat surprising as most trainees have access to paid subscription journals through their academic institution. These differences in literature access preferences are important, highlighting the potential biases individuals may be exposed to simply due to how literature is obtained.

OA journals typically incur greater publication charges than non-OA journals, subjecting them to a newer type of bias known as e-publication bias.¹⁴ Previous studies found a significantly greater proportion of industry-funded studies on OA platforms when compared to subscription access, a phenomenon attributed to e-publication bias.¹⁵ A higher proportion of industry-funded studies could lead to preferential publication of industry supported products in OA journals.¹⁵ While industry funding does not inherently reflect research quality, readers must be aware that industry sponsors have higher level of control and may have industry involvement, which can serve as a conflict of interest and influence the question being answered.^16,17 Additionally, industry-funded research tends to focus more on lines of inquiry regarding products, processes, or activities that can be commercialized.¹⁷ Previous studies have shown that researchers working with industry funding are aware of the influence said funding may have on research priorities; however, the same cannot necessarily be said for trainees.¹⁷ While OA comes with the benefit of making scientific articles freely available to all,¹⁸ its faults and limitations should not go unnoticed. It is important for faculty to be aware of the greater importance placed by trainees on an article being OA. With this knowledge, faculty can educate trainees in an attempt to mitigate the effects of these biases.

Compared to faculty, trainees placed a greater influence on the year of publication. We speculate that this difference could in part be due to the bias towards newer literature seen in some undergraduate and graduate assignment rubrics where there is a requirement for research to be published within a certain number of years. Part of this preference towards newer literature may also be due to trainees preference for OA articles as there has been a steady increase in the percentage of publications available by OA from 2000 to 2015.¹⁹ While favoring newer literature could result in important landmark publications being forgotten and overlooked simply due to the date of publication, it does come with the advantage of helping ensure that scientific literature is an accurate representation of the current developments in the field and is not outdated.

It is important to note that while our study indicates that there are differences in the ways faculty and trainees view and access literature, some of these differences may be mitigated by the close working relationship between these groups. Trainees are typically supervised by faculty members with their work reviewed and corrected. This close relationship may change the way these literature preferences are manifested when working on a publication or other form of scientific work. While our survey shows that differences exist between trainee and faculty literature preferences, the explanation for this is unclear. One can speculate that factors such as experience in literature appraisal, generational differences, or a combination of the two are at play. Future studies observing how trainee attitudes change over time as they shift into faculty roles could provide insight to explain the observed differences.

The Online Movement and the Physical Library

While differences in the use of online search methods, such as PubMed, Google, and Google Scholar did exist between faculty and trainees in our study, both groups did place high value on all online search methods listed. Respondents indicated that being able to access the paper form of a journal is of low importance and that they rarely, if ever, use paper articles. This raises questions regarding the current needs and benefits of physical libraries. One of the benefits of the online movement is the potential for environmental savings. The US book industry is responsible for 12.4 million metric tons of carbon emissions each year.²⁰ While this includes books and press outside of medical literature, it indicates substantial environment savings could be achieved with online libraries. Currently, many large clinical centers do not have all the necessary licenses to identify all relevant publications.²¹ The reallocation of the use of funds from physical to online libraries could allow such institutions to subscribe to more online services, resulting in a more comprehensive online resource. However, there are some drawbacks to the online movement. While access to a computer is common into today’s technological age, it cannot be assumed that all individuals have access. In addition to grey literature, information existing outside of traditionally publishing methods,²² or publications in non-electronic journals can be difficult to identify and often go undetected in systematic searches.²¹ The loss of physical literature resources could increase the difficulty of identifying such literature. However, we believe that the cost, environmental, and space savings outweigh the drawbacks of continuing to transition towards online libraries.

Predatory Journals

It is important to note that while there are many good resources available online, there also exist predatory journals, which may be of lower quality.^12,13 Identifying these journals can be difficult, a feeling shared by one survey participant via comment. Information such as the title of the journal, editor and editorial board member information, scope of the journal, guidelines/instructions to authors, language of the articles, publication charges, turnaround time, and contact details can all be used to help differentiate a legitimate journal from a predatory one.²³ While there are ways to identify predatory journals, it remains unclear how aware medical scientists are of their existence, specifically young trainees. An area of future study regarding medical trainees' awareness, ability to identify, and how often they check the legitimacy of their sources may be in need. The benefit of such a study is that it could identify gaps that may exist in medical trainees' abilities to appraise the quality of a body of work and guide restructuring of the supplemental aspects of their training in order to develop such skills.

Language, Country of Origin, and IF

Both faculty and trainees listed the country of origin as mildly important. While this indicates that an explicit bias towards the country of publication may not exist, it does not exclude more subtle biases that may not be as easily recognized. Respondents to our surveys listed the language of publication as highly important, an attitude that is easily understood given the additional effort to obtain a translated copy or to translate it themselves. Our study demonstrated a bias towards the literature published in English, however we recognize that the survey was offered in English in a predominantly English-speaking country. Future research could be performed with surveys in multiple languages. Not only does language bias potentially cause non-English publications to be overlooked, it can also skew the results of larger systematic reviews and meta-analyses by omitting non-English papers, in addition to affecting the IF of non-English journals.

Previous studies have found that authors are more likely to publish positive results in English-language journals and negative results in non-English-language journals.²⁴ This can lead to an unequal distribution of trials and a selection bias in some studies.²¹ The publication bias introduced by the unequal distribution of positive and negative results can decrease the quality of evidence for larger systematic reviews and meta-analyses.¹⁴ This was shown when several meta-analyses evaluating the safety of some antidepressants were found to have a therapeutic value that was significantly overestimated due to the exclusion of unpublished negative data.¹⁴ As such, systematic reviews and meta-analyses ignoring non-English trials and publications are subject to a language bias that can affect the accuracy of reported data.²⁴ We believe the strength and level of evidence is highly important in medicine, an attitude shared by respondents to our survey. Due to the potential for language bias to negatively affect the level of evidence in systematic reviews and other publications, there exists a need to reduce this form of bias and to make it easier to read non-English bodies of work.

Overall journal IF was considered to be moderately important by survey respondents. While IF can be a useful tool for assessing the quality of a body of work, it is not without its flaws. In regard to the language of publication and IF, it is important to note that the language of publications has a dramatic effect on citation-based measurements of research performance.²⁵ Non-English papers count towards the denominator of an IF but are typically cited less since fewer scientists can read them.²⁵ A study from 2001 to 2010 found that English medical journal IFs were significantly higher than non-English ones.²⁶ This discrepancy is even more present in application-oriented fields such as clinical medicine.²⁵ Citation-based rankings per staff and per paper have caused surprisingly low rankings for many German and French universities, particularly those with a medical school (2010 Leidan Ranking).²⁵ Despite these citation differences, previous studies comparing English to non-English publications found only small differences in quality between the two.²⁴ These findings further reinforce the need to be cautious and critical of IF, especially when analyzing non-English journals, even if respondents only considered IF mildly important.

Our study has shown that while an outright bias towards non-English publications may not exist, there may be an implicit bias towards excluding non-English literature. Google Translate currently enables translation of some bodies of work into English; however, these translated documents can be subject to formatting and grammatical issues, making them difficult to read. We believe that the development of translator applications that can improve upon said formatting and grammatical issues should be developed and subsequently made available in online university libraries. Hopefully, such advances will result in non-English literature being less likely to be overlooked due to the reduced effort needed to read a non-English publication.

Limitations

This study was conducted during the COVID-19 pandemic. Due to the timing of the survey, faculty and trainee attitudes towards some of the aspects questioned in the survey, such as use of online or physical resources for accessing literature, may differ from non-pandemic times. However, we believe that these attitudes are still in line with the current direction regarding literature access and any pandemic-related changes are more likely to be an acceleration of shifts that are already occurring rather than an outright change.

This study used participants’ opinions on their use of reference works for citation as a proxy for their actual behavior when selecting articles for use in citations in research works. In the future, other study designs could consider a retrospective (but may factor in memory bias), or a prospective design for further examination of this topic.

Additionally, the current level of understanding of survey respondents, particularly trainees, towards topics, such as IF, OA, and predatory journals, is unknown. It is possible that not all individuals who completed the survey were familiar with these concepts and were therefore unable to accurately rate their feelings towards them. That being said, it is a fair assumption that faculty members are familiar with these concepts and that trainees having completed an undergraduate level of education were likely familiar with them as well.

Many trainees may have a more limited experience with writing academic papers. Their responses may reflect inexperience with the publication process rather than generational differences in technology and academic culture. Future areas of study could include how citation practices change for the individual as they progress from trainee to faculty.

Conclusion

The purpose of our study was to investigate what factors are involved when authors choose citations to include in their publications. Points of interest highlighted by our data include how faculty and trainees' literature search preferences differ, the strong bias towards English language publications, and the movement towards online literature sources. The impact of our study is that it provides insight into what biases individuals may be exposed to based on their language and literature search preferences, as well as providing data showing the continued trend towards online literature resources. Our study has identified areas of future studies such as how trainees’ opinions change over time, identifying trainee ability to recognize predatory journals, and the need for better online journal article translators to mitigate the language bias that can affect research quality. The importance of this study and similar future studies is that over time we will better understand what factors are important when searching for literature and the different biases we may be exposed to when doing so. Ultimately, we hope this will lead to more comprehensive and less biased searches that will lead to high-quality evidence and allow for optimal patient care amongst healthcare workers.