Does a research group increase impact on the scientific community or general public discussion? Alternative metric-based evaluation

Introduction

Altmetrics creates a new approach to evaluating the impact of publications by considering the number of downloads, shares, and discussions on social networks.¹ This approach does not replace the traditional bibliometric indicators, such as Impact Factor and H-index, but rather focuses on new aspects of publication impact.² Although still in its infancy, altmetrics has the potential to become a valid assessment strategy for the evaluation of publication impact.³ Altmetric tools capture information through the use of metrics from HTML views and downloads of articles, blog posts, tweets, bookmarks, etc. All of these sources are alternative indicators of impact that go beyond traditional citation, focusing on the content and uses of the social web,⁴ with this information provided in real time. Altmetrics elucidate not only the impact of scientific research by researchers but also the impact of the research on the public through social media.^2,5 In fact, through altmetrics, the impact of research can be measured at the individual-article level, using a combination of such data as the number of times that a particular paper has been downloaded, discussed, shared, and cited.⁶ This approach allows not only researchers but also institutions to analyze postpublication activity around a paper in near-real time using various online resources.⁵

To use this new tool, we focused on the Italian researchers of the SIMPAR group, which was founded in Pavia in 2007. This group has rapidly become an eminent translational group in the pain field, with its annual meeting now considered a major international pain conference (www.simpar.eu). In addition to the authors of this study, the other nine members of the SIMPAR group are Drs Marco Baciarello, Dario Bugada, Christian Compagnone, Andrea Fanelli, Stefano Govoni, Maurizio Marchesini, Cristina E Minella, Carolina Muscoli, and William Raffaeli. SIMPAR’s multidisciplinary collaboration has included several professionals of different disciplines and has produced a number of publications on the personalization of pain therapy through a multidisciplinary approach, including traditional medical, genetic, epigenetic, and “omic” disciplines. Table S1 lists each of the 18 papers published by at least two SIMPAR members in collaboration between 2010 and 2015. As described herein, we have been able to obtain statistically significant results regarding the force of the group as a whole in both the research and public communities.

Materials and methods

For each of the 12 researchers of which our team is comprised, we created an ORCID (Open Researcher and Contributor ID) account (www.orcid.org), in addition to an Impact Story (https://impactstory.org) account that imported our data and synchronized it with the unique ORCID identifiers. Collected items were assigned to specific categories, such as “cited” (or highly cited), “saved” (or highly saved), or “discussed”. In doing so, our Impact Story provided us with data regarding the number of times an article was saved by scholars, cited by other researchers, publicly discussed (Facebook, etc), and cited by the general public (blog posts, Wikipedia). These metrics were classified along two dimensions: audience (scholars or the public) and type of engagement with the online research products (viewed, discussed, saved, cited).⁷ From Impact Story, we were able to retrieve all altmetrics data for the 12 researcher accounts (paper citations, discussions, views by the research community or public).

Statistical analysis

Through the personal profiles of altmetrics, for each member of the SIMPAR group, we were able to count the number of citations, times a paper was saved, and discussions from the public community for each paper published. Then, we compared the SIMPAR group percentages of articles cited (or highly cited), saved (or highly saved), or discussed relative to those published by single authors (either written alone or in collaboration with coauthors who were not members of the SIMPAR group) by means of Fisher’s exact test. Quantitative variables are described as median and interquartile range (IQR), ie, the 25th and 75th percentiles and compared to collective SIMPAR data or individual publication articles by means of a nonparametric Mann–Whitney test. The association between citations from Scopus and the altmetric score or its components (eg, Facebook posts, tweets, Mendeley readers) was expressed through a nonparametric Spearman’s r-correlation coefficient. A P-value <0.05 was considered statistically significant. All analyses were performed utilizing Stata 14 (Statacorp LP, College Station, TX, USA).

Results

We analyzed 169 articles, 18 of which were SIMPAR group publications. The median number of articles by the authors was 17 (IQR 10–27), and the median year of publication was 2013 (IQR 2011–2014). Altmetrics demonstrated that SIMPAR group publications were more likely to be saved (78% vs 53%, P=0.05) and publicly discussed (61% vs 4%, P<0.0001) than individual publications. However, no significant difference emerged between the SIMPAR group publications and individual publications in terms of being cited (cited 44% vs 36%, highly cited 22 vs 11%; P=0.20) and publicly viewed (11% vs 3%, P=0.25). Notably, eleven of 18 articles published collectively by the SIMPAR group received a tweet (median 1, IQR 1–3), while only 36 of 151 individual publications received a tweet. Moreover, 15 articles from the SIMPAR group collectively were accessed through Mendeley readers (median times accessed 4, IQR 1–11) versus 85 of the individual publications (median times accessed 2, IQR 0–8; P=0.01). We describe the correlation between Scopus citations and the single components of the analyzed alternative metrics in Table 1. We found that the alternative metrics were generally low, with the exception of those for Mendeley readers (r=0.47, P<0.0001) (Figure 1).

Table 1 Correlation coefficient (P-value) between Scopus citations and altmetric components Note: Data presented as correlation coefficient (P-value).

Figure 1 Correlation between Mendeley readers and Scopus citations.

Discussion and conclusion

We found significant correlations between the SIMPAR group collective publications and their impact on the indicator linked to research activity (Mendeley readers), although not to public discussion (such as Facebook and tweets). However, the impact of collective SIMPAR group articles was high also on general public items, even though it did not reach statistical significance. This is meaningful, as funders, universities, and publishers increasingly demand indicators of the impact of science on society.⁸

Moreover, we are confident that the new metrics of medical groups could have an impact on the pain-patient community as well. Through online platforms, such as Twitter and Facebook, like-minded people can form their own communities to discuss their shared experiences, problems, and more.

Our results also illustrate how collaborative multidisciplinary teams and their projects improve the overall impact of researchers’ work on the researchers themselves. If collaborative efforts are more widely disseminated than individual publications, as our study suggests, such efforts can potentially provide additional exposure to group members, which may result in greater career enhancement than individual publications. Through these avenues, researchers can leverage social media opportunities to their own professional and academic advantage.⁹ Enhanced exposure allows for the sharing of ideas and research among respective networks, spotlighting pain studies both nationally and internationally.

Finally, as the SIMPAR group take research ethics seriously, we opine that our collaborative approach is an ethical one as well as an effective one. Research resources are scarce, and becoming more so. Ways to increase the “yield” from biomedical research have been identified as an imperative.¹⁰ As our collaborative efforts have been demonstrated to enhance access to meaningful, clinically relevant research results, our approach results in more “bang for the buck” by more readily disseminating useful information to practicing clinicians, as well as to other researchers interested in building upon the fund of data obtained through our investigations. Hopefully, other pain researchers will choose to follow our lead.

Acknowledgments

The paper was funded by the European Commission (FP7 Pain-OMICS project; grant agreement number 602736) and by the Italian Ministry of Health (project code GR-2010-2318370).

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Scotti V. Recenti Progressi in Medicina [Bibliometrics and web use: the birth of altmetrics]. Recenti Prog Med. 2015;106(4):176–179. Italian.
2.		Priem J, Groth P, Taraborelli D. The altmetrics collection. PLoS One. 2012;7(11):e48753.
3.		Cress PE. Using altmetrics and social media to supplement impact factor: maximizing your article’s academic and societal impact. Aesthet Surg J. 2014;34(7):1123–1126.
4.		Levitt J, Thelwall M. From webometrics to altmetrics: one and a half decades of digital research at Wolverhampton. Proceedings of the: Libraries in the Digital Age (LIDA) 2014; June 16–20, 2014; Zadar, Croatia.
5.		Cave R. Overview of the Altmetrics Landscape. Proceedings of the: 2012 Charleston Libraries Conference; November 7–10, 2012; Charleston, SC.
6.		Tananbaum G. Article-Level Metrics: A SPARC Primer. Washington: SPARC; 2013. Available from: http://sparcopen.org/our-work/article-level-metrics/. Accessed April 1, 2013.
7.		Rodgers EP, Barbrow S. A look at altmetrics and its growing significance to research libraries. 2013. Available from: http://hdl.handle.net/2027.42/99709. Accessed April 18, 2016.
8.		Dinsmore A, Allen L, Dolby K. Alternative perspectives on impact: the potential of ALMs and altmetrics to inform funders about research impact. PLoS Biol. 2014;12(11):e1002003.
9.		Haustein S, Sugimoto CR, Larivière V. Social media in scholarly communication. Aslib J Inf Manag. 2015;67(3):01877.
10.		Chalmers I, Bracken MB, Djulbegovic B, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–165.

Supplementary material

The 18 papers published from Italian SIMPAR group (from at least two members) analyzed in the paper are as follows:

• Perotti L, Cusato M, Ingelmo P, et al. A comparison of differences between the systemic pharmacokinetics of levobupivacaine and ropivacaine during continuous epidural infusion: a prospective, randomized, multicenter, double-blind controlled trial. Anesth Analg. 2015;121(2):348–356.
• Catenacci SS, Lovisari F, Peng S, et al. Postoperative analgesia after laparoscopic ovarian cyst resection: double-blind multicenter randomized control trial comparing intraperitoneal nebulization and peritoneal instillation of ropivacaine. J Minim Invasive Gynecol. 2015;22(5):759–766.
• Casale R, Di Matteo M, Minella CE, Fanelli G, Allegri M. Reduction of painful area as new possible therapeutic target in post-herpetic neuropathic pain treated with 5% lidocaine medicated plaster: a case series. J Pain Res. 2014;7:353–357.
• Bugada D, Allegri M, Lavand’homme P, De Kock M, Fanelli G. Inflammation-based scores: a new method for patient-targeted strategies and improved perioperative outcome in cancer patients. Biomed Res Int. 2014;2014:142425.
• Gigliuto C, De Gregori M, Malafoglia V, et al. Pain assessment in animal models: do we need further studies? J Pain Res. 2014;7:227–236.
• Gardella B, Porru D, Allegri M, et al. Pharmacokinetic considerations for therapies used to treat interstitial cystitis. Expert Opin Drug Metab Toxicol. 2014;10(5):673–684.
• De Gregori S, Minella CE, De Gregori M, et al. Clinical pharmacokinetics of morphine and its metabolites during morphine dose titration for chronic cancer pain. Ther Drug Monit. 2014;36(3):335–344.
• Compagnone C, Tagliaferri F, Allegri M, Fanelli G. Ethical issues in pain and omics research. Some points to start the debate. Croat Med J. 2014;55(1):1–2.
• Bugada D, Guardia Nicola F, Carboni V, Allegri M. Transversus abdominis plane catheter infusions after major abdominal surgery in morbidly obese patients: reply to comments. Minerva Anestesiol. 2014;80(6):747.
• Fanelli A, Ghisi D, Allegri M. Is spinal anaesthesia a suitable technique for ultra-short outpatient procedures? Acta Biomed. 2013;84(1):76–80.
• Bugada D, Guardia Nicola F, Carboni V, Allegri M. TAP block for opioid-free postoperative analgesia in obese surgery. Minerva Anestesiol. 2013;79(12):1447–1448.
• De Gregori M, Garbin G, De Gregori S, et al. Genetic variability at COMT but not at OPRM1 and UGT2B7 loci modulates morphine analgesic response in acute postoperative pain. Eur J Clin Pharmacol. 2013;69(9):1651–1658.
• Mura E, Govoni S, Racchi M, et al. Consequences of the 118A>G polymorphism in the OPRM1 gene: translation from bench to bedside? J Pain Res. 2013;6:331–353.
• Allegri M, Clark MR, De Andrés J, Fanelli G. Pain treatment: a new approach to link bench to bedside – the SIMPAR meeting 2011. Eur J Pain. 2012;16(6):
  779–781.
• De Gregori S, De Gregori M, Ranzani GN, Allegri M, Minella C, Regazzi M. Morphine metabolism, transport and brain disposition. Metab Brain Dis. 2012;27(1):1–5.
• De Gregori M, Allegri M, De Gregori S, et al. How and why to screen for CYP2D6 interindividual variability in patients under pharmacological treatments. Curr Drug Metab. 2010;11(3):276–282.
• Allegri M, De Gregori M, Niebel T, et al. Pharmacogenetics and postoperative pain: a new approach to improve acute pain management. Minerva Anestesiol. 2010;76(11):937–44.
• Allegri M, Lombardi F, Custodi VM, et al. Spontaneous cervical (C1-C2) cerebrospinal fluid leakage repaired with computed tomography-guided cervical epidural blood patch. J Pain Symptom Manage. 2010;
  40(3):e9–e12. Erratum in: J Pain Symptom Manage. 2010;40(6):e9.