Blinding of Peer Review and the Impact on Geographic Diversity of Authors in the Medical Literature

Introduction

Peer review is a crucial step in the editorial processing of submitted manuscripts in academic publishing, including in medical journals, where journal editors solicit external reviewers to evaluate the submitted work and provide their opinions. The main objective of peer review is to ensure the publication of high-quality work that is of value and benefit to practitioners and the scientific community.¹ Hence, a work that sounds scientifically sound is often accepted but sent back for a revision to further enhance its quality and presentation. Peer review can be one of three main types, single-blind, double-blind, or open.² Single-blind review is the most prevalent form where the reviewers can see the names and affiliations of the authors, whereas the reviewers are kept anonymous. Double-blind review is the second most common type and involves anonymizing the identity of both the authors and the reviewers, so the reviewers could only evaluate the work without knowing who did it or where it came from. Open review is the opposite of double-blind review, where the authors and the reviewers know each other’s identities during the peer review process. Some journals also employ what is called transparent peer review, which involves publishing the peer review report with the reviewers’ information alongside the published article. Transparent peer review is often independent of the type of blinding of the peer review process since it takes place after a manuscript has already been accepted for publication.

Blinding of reviewers is assumed to improve the peer review process by removing potential bias. However, the impact of each type of peer review blinding on the quality of peer review remains debatable.^3–5 Additionally, two studies have shown a higher likelihood of accepting a manuscript from recognized authors and/or affiliations when the peer review is single-blinded, whereas another study found lower acceptance rates of manuscripts submitted from non-English speaking countries.^6–8 A recent systematic review of eight studies that evaluated the relationship between the assessment of research quality and geographic bias found that research coming from high-income countries usually received higher review scores and were more likely accepted at conferences than works coming from low-income countries.⁹ Another aspect that has been assessed with regards to peer review blinding is gender and ethnic bias, where a number of studies found that double-blinded reviews highly ranked work from women and ethnic minorities.^10–12

The term diversity, equity, and inclusion (DEI) has become a global trend including in scientific publishing to provide an equal opportunity to researchers from developing and less publishing countries to contribute their science and learn from fellow researchers through the peer review process.¹³ Moreover, indexing in top journal databases, such as Web of Science (WOS), Scopus, and MEDLINE, requires geographic diversity of authors and research locations; though, no specific scoring criteria is listed in their journal selection guidelines.^14–16

Currently, no study was identified in the literature has addressed the degree of geographic diversity of published authors with respect to blinding of peer review. Therefore, the primary objective of this cross-sectional study was to assess whether the type of peer review blinding may have an impact on the geographic diversity of authors of published articles in MEDLINE-indexed journals. The secondary objective was to evaluate the impact of other relevant factors, such as the age of the journal, the publisher, and specialty, on the same outcome.

Methods

Journals Database and Eligibility Criteria

MEDLINE-indexed medical/clinical journals (as obtained from the National Library of Medicine’s website: https://www.ncbi.nlm.nih.gov/nlmcatalog) were evaluated for eligibility. MEDLINE database was selected because it mainly indexes peer reviewed journals in the focus of biomedicine, and it is the main component of PubMed, which is the most popular database used by biomedical researchers.¹⁷ Journals that publish in both clinical and basic sciences were included (but only clinical articles were evaluated). However, journals that only publish in basic sciences or in health systems administration, management, policy, and ethics were excluded. This was determined based on the journal’s title, aim and scope of the journal, as well as a sample of published articles. Journals that publish in languages other than English, journals that publish in English and another language but require a copy of the abstract in the journal’s native language, and journals that only publish solicited material, such as those that publish review articles by invitation only, were excluded. Since single-blinded or double-blinded peer review were the major types of peer review employed by medical journals, journals that employ open peer review were excluded.

Blinding and Diversity Evaluation

The type of blinding was assessed based on the description of the peer review process as outlined in the instructions for authors page or the review policy page of each journal. When such information was not explicitly described, other clues were looked for, such as the strict requirement for a title page containing authors’ names and affiliations that should be submitted separately from the main manuscript file. Alternatively, when information on blinding type of peer review was unavailable, the publisher was contacted to provide such information. The diversity of the geographic location of authors (based on the location of their affiliation) was evaluated for the following article types: original articles, short papers (also known as brief communications), case reports or series, review articles (solicited review articles if specified as such were excluded from the evaluation), and systematic reviews (with or without meta-analysis). Hence, letters to the editor (ie, correspondences/commentaries) and perspectives (ie, expert opinions) were not evaluated. Since some journals do not have a category for short papers and instead accept such work as a letter to the editor, articles under this category in such journals were evaluated. Multinational papers of authors from different countries (based on the location of their affiliations) were dismissed.

Diversity was calculated using two methods. The first method involved dividing the number of countries from which 20 evaluated articles come and multiplying the result by 100 to obtain a percentage (named %diversity). The second method involved calculating Simpson’s diversity index (SDI) using the following equation:

where D is the diversity index (ranged from 0 to 1, where the higher the value, the higher the diversity), n is the frequency of appearance of each country among the evaluated articles, and N is the total number of articles evaluated (fixed at 20). The number 20 for the articles to be evaluated from each journal was calculated based on an expected diversity index of 0.75, an acceptable level of precision of 20%, and an α error probability of 5%. These values resulted in a sample size of 19 (rounded to 20 articles per journal). Each journal’s website was accessed once, where data collection spanned between April 6th through October 18th, 2022. The study was cross-sectional as the latest published clinical articles at the time of accessing the journals’ websites were evaluated. These included the online first or ahead of press articles.

Statistical Analysis

Continuous variables were presented as median [interquartile range, IQR] and were compared using Mann–Whitney U-test since the Shapiro–Wilk test for normality showed that the data were not normally distributed. Chi-square test was used to compare categorical variables. Multivariate linear regression using generalized linear model was used to estimate the effect of different variables on diversity outcomes. A P value of < 0.05 was considered significant. All statistics were carried out using SPSS version 24.0 (SPSS, Inc. Chicago, IL).

Sample size calculation revealed that a total number of journals of 1054 was needed to detect a difference of 0.2 in %diversity or SDI assuming an α error probability of 0.05, a power of 90%, and an estimated allocation ratio of 2:1 (two single-blinded journals for each double-blinded journal). Therefore, the first 1054 journals of all the eligible journals, after alphabetical sorting, were included in the analysis.

Results

A total of 5287 journals were indexed in MEDLINE as of April 2022. Of these journals, 2178 met the eligibility criteria (Figure 1). Of the included 1054 journals, 766 (72.3%) employ single-blinded peer review while 288 (27.7%) employ double-blinded peer review. Table 1 lists the general characteristics of the journals and the results of geographic diversity comparisons. Journals in both groups had a median age of 28 years (P = 0.969), and a publication frequency of 8 and 6 issues per year for the single-blinded and double-blinded journals, respectively (P < 0.0001). Overall, the median number of countries contributing to MEDLINE-indexed journals was 9 [6–12] with a %diversity of 45% [30–60] and an SDI of 0.84 [0.64–0.92]. While numerous journal characteristics were significantly different between the two journal groups, the primary endpoints of median %diversity and SDI were not different (45 [35–60] vs 45 [30–55] and 0.84 [0.65–0.92] vs 0.82 [0.57–0.92]; P = 0.199 and 0.128, respectively). A similar finding was observed with the median number of countries (9 [7–12] vs 9 [6–11]; P = 0.199). Although general medicine was the most common specialty in both groups, it was noted that more than one-tenth of the double-blinded journals were nursing journals (n=32; 11.1%) compared with only six single-blinded journals (0.8%) in this field. International journals accounted for 452 (42.9%) of the 1054 journals, followed by American (n=284; 26.9%) and British (n=85; 8.1%) journals. Figure 2 illustrates the counts of the journals coming from different countries.

Table 1 Characteristics of the Included Journals and Outcomes of Diversity Evaluation

Figure 1 Initial evaluation of MEDLINE-indexed journals for inclusion in the study.

Figure 2 Count of journal original countries (n=603; 452 of the 1054 journals are international journals; hence, were not included in this chart).

Most journals in both groups were indexed in the major databases, including Science Citation Index Expanded (SCIE; also known as Information Sciences Institute [ISI]) collection of WOS, Embase, and Scopus, where more single-blinded journals were indexed in WOS and Embase compared with the double-blinded journals (Table 1). Among journals metrics, single-blinded journals had statistically higher median impact factor and CiteScore than double-blinded journals (3.671 vs 2.852 and 4.8 vs 3.5, respectively; P < 0.0001 for both comparisons).

When the impact of different factors on the diversity metrics was evaluated, the following factors were associated with at least 1% increase in %diversity: publishing both clinical and basic sciences work, indexing in SCIE, ESCI, Scopus, and Embase, as well as a one-point increase in CiteScore (Table 2). Factors that were significantly associated with an increase in SDI included the presence of a journal’s country name in the journal title, the ownership of a journal by a society, the indexing in SCIE, the indexing in Scopus, and CiteScore (Table 3). Table 4 shows the impact of the studied factors on the number of countries publishing in the included journals, where only the indexing in Scopus and CiteScore were associated with an incremental increase of at least one country.

Table 2 Effect of Different Factors on %diversity Using Multivariate Linear Regression

Table 3 Effect of Different Factors on Simpson’s Diversity Index Using Multivariate Linear Regression

Table 4 Effect of Different Factors on the Number of Countries Using Multivariate Linear Regression

Discussion

This study examined the potential impact of different forms of blinding of peer reviewers on the geographic diversity of authors. Although no difference was observed between single blinding and double blinding of peer review in the diversity metrics (%diversity, SDI, and number of countries), the indexing of journals in SCIE and Scopus was significantly associated with an incremental increase in the diversity metrics.

As English remains the predominant language in the medical and scientific literature,¹⁸ authors from non-English speaking countries may struggle to publish their work or often required to seek language editing to improve the language of their manuscripts to avoid the possibility of rejection if their writing does not convey the details of their work appropriately. Some guides are available online for such authors on the proper English writing of medical papers.¹⁹ In fact, a study by Primacket al found a lower acceptance rate of manuscripts submitted to a biology journal from non-English speaking countries or developing countries compared with manuscripts submitted from English speaking countries.⁸ The journal evaluated in this study utilizes single blinding peer review process, to which the authors attributed this potential bias and suggested switching to double blinding to tackle this issue. Nevertheless, the authors also mentioned that the overall review process is complex as it involves various factors that may contribute to the possible bias, such as the lack of blinding of editors, their countries, and specialties.⁸

An interesting finding of the present study was the positive relationship between the indexing status of journals in SCIE and Scopus with increased %diversity and SDI, but not with indexing in Embase or ESCI. The essential requirement of geographic distribution of authors by both WOS Core Collection and Scopus could potentially explain this observation.^14,15 MEDLINE also considers the geographic distribution of authors for journal selection, particularly authors from a “geographic location that is relatively under-represented in the scientific literature.”¹⁶ On the other hand, Embase aims to broaden its coverage by focusing on selecting high-quality journals from regions other than North America and Western Europe but does not have a specific criterion for the diversity of authors publishing in the journals indexed in the database.²⁰

The overall median SDI of all the evaluated journals in the present study was relatively high at 0.84 (Table 1). However, it is important to note that this index does not take into account the number of countries contributing to the journals as much as the number of papers coming from the contributing countries even when only very few countries were among the top publishers in a certain journal. For example, a high SDI could be achieved if only two countries contributed equally to one journal by publishing 10 papers each. In such case, the calculated SDI would be 0.53 compared with a zero if all the 20 papers came from a single country, which indicates a difference by 53%. To tackle this issue, the metric %diversity was developed in this study to provide a different dimension of geographic diversity estimation. In the example provided, %diversity would be 10% compared with 5% if all papers came from the same country (difference of only 5%). In the present study, the median %diversity fell below one-half at only 45% with an IQR of 30–60%. A probable explanation of this outcome could be the predominance of a very few particular countries in medical and scientific publishing, namely the United States followed by the United Kingdom, China, and Germany.^21,22 Additionally, some international authors may refrain from submitting to certain journals with high impact factors or CiteScores fearing immediate rejection or criticism.

While double blinding of peer reviewers in this study did not result in significant difference in the geographic diversity of authors, previous reports provided an insight into the impact of blinding on the quality of peer review. Two randomized controlled trials evaluated the effect of blinding of reviewers on the quality of peer review, where a manuscript was sent to two reviewers, one was aware of the authors’ identities (single-blinded review) while the other was blinded to the authors’ information (double-blinded review). The first study by McNuttet al evaluated reviewers’ reports of 127 manuscripts and found that double-blinded reviews were of a significantly higher quality than single-blinded reviews (overall grade 3.5 vs 3.1; difference 0.41; P = 0.007).³ In contrast, the other study, which evaluated the quality of review reports of 118 manuscripts, found no difference in the quality as judged by the editors between the two types of blinding (mean grade difference 0.1 95% CI −0.2 to 0.4); though, the authors of this study noted that masking of authors’ identities was unsuccessful in 32% of the manuscripts as some were written by generally known authors.⁴ In a large study by Chung, et al, blinding was successful in only 66% of 680 manuscripts due to familiarity of the blinded reviewers with the authors of the submitted manuscripts (33%) and author self-citation (25%). Such limitation of unsuccessful masking was hypothesized to have possibly impacted the results as double-blinded review reports were of a significant lower quality compared with single-blinded reports (overall grade 3 vs 4; P < 0.001).²³ Nonetheless, it should be noted that the judgement on the quality of a review report in the two former studies was solely subjective by the editors and was not based on specific objective criteria unlike the study by Chung et al. Moreover, a meta-analysis by Bruce et al favored double-blinded peer review for better quality reports and lower rejection rates with no impact on the time spent on peer review.⁵ A similar finding on better review quality with double blinding was also reported in a meta-analysis by Gaudino et al.²⁴

While a late-career reviewer considers signing a review report can provide confidence in the peer review process and to the authors as it was carried out by an expert in the field, early- and mid-career reviewers feared that such practice may compromise their career and potential relationships if they provide negative reviews.² Interestingly, in their opinion paper, the early-career reviewer also stated finding some flaws in manuscripts written by authors prominent in their field, which made the reviewer more inclined toward blinded review to provide a more objective and honest review.² The impact of peer review blinding was also studied on gender diversity. A recent systematic review of eight studies evaluating gender bias in academic publishing found mixed results where half of the studies revealed a significant difference in the review score between papers based on perceived author gender and the other half found no such difference.²⁵ Despite these findings, the authors assert that reasonable evidence exist on the possibility of gender bias and that double blinding the review process may diminish such effect. One interesting observation in the current study is that more nursing journals utilize double blinding than single blinding (n=32 vs 6). Given that gender discrimination is potentially an issue with single blinding, this could have possibly been the reason behind double blinding the peer review process of nursing journals, especially that the majority of workers in the field of nursing are women.

There are many databases that index medical journals. However, only a few are well-known among researchers in the medical field and often sought for their credibility and rigorous indexing requirements, such as MEDLINE, WOS, Scopus, and Embase. MEDLINE database was selected in this study given its popularity in medical research, its credibility, its requirement for geographic diversity of authors, and that it largely indexes medical journals unlike other databases that indexes journals from different scientific and human studies fields.¹⁷ MEDLINE is sometimes confused with PubMed when both databases are in fact different and have different indexing requirements. While all MEDLINE journals are also part of PubMed, the latter indexes more than 30,000 titles, including journals and books, and has less strict indexing criteria.²⁶

This study is the first to assess the potential impact of the type of peer review blinding on the geographic distribution of authors of published medical research. Certain strengths of this study include the inclusion of a large number of journals in order to attain a high study power and the evaluation of different journal-specific factors (beside the review blinding type) on diversity metrics. Moreover, the focus on evaluating only clinical research is assumed to have helped widening the geographic distribution of authors from middle- and low-income countries who may not afford to have sophisticated equipment and supplies to conduct basic science research but could rather conduct simple clinical studies, surveys, or reviews that do not require expensive resources. Nevertheless, this study is perhaps limited by the fact that many (or perhaps all) journals do not blind the editors, who typically make the initial evaluation and may forward the work for peer review or decide to reject it; thus, such factor could not be assessed. Furthermore, the current study did not specifically evaluate the continental distribution of the authors, or their distribution based on their countries’ income level. Additionally, some of the checked research papers did include authors from middle- or low-income countries that do not have a high rate of scientific publication; however, since many of these papers included co-authors from foreign countries, they were deemed multinational and could not be included in the diversity evaluation as the presence of the foreign authors was considered a potential confounder. Such elimination may have resulted in the underrepresentation of such countries. Moreover, the journals included in this study encompass a wide range of specialties that some may not be practiced or at least popular in some countries (eg, journals about genetics or nuclear medicine). Lastly, the fact that some journals impose article processing or page charges may also limit the diversity when authors from certain countries cannot afford paying the charges and cannot be granted waivers.^27,28 It was also noticed during the data collection process that journals that carry their country of origin’s name in their title or the title of the journal is in a country’s native language (that is not English or Latin) tend to have more authors from the journal’s particular country.

Conclusion

Although blinding of reviewers to authors’ identities and affiliations was not associated with higher geographic diversity of authors, the lack of blinding of editors, who make the initial evaluation, could not be assessed. However, editors and publishers are encouraged to enhance geographic diversity to be able to index their journals in SCIE, Scopus, and MEDLINE as geographic diversity is an essential indexing requirement. Such pursuit could also broaden the availability and reach of the journals to wider global readership. Findings from this study suggest the need for further studies that may add evidence to this important topic as it is not trivial to control for variables underlying judgment in the process of reviewing manuscripts. This work adds credence to the idea that it is challenging to assess the impact of peer review models or blinding peer review on geographic diversity.