Quality and Impact of Survey Research Among Anesthesiologists: A Systematic Review

Introduction

Survey research has been used by investigators for studying clinical, educational, and professional topics in the field of anesthesiology. The feasibility of such research has been facilitated by the prevalence of computer technology, the increased ease of communication via the internet, the development of online survey questionnaires, and the availability of e-mail lists from various organizations for participant recruitment. However, the validity of survey data is reduced by low response rates, missing data points, and poorly designed questionnaires.¹ The importance of appropriate survey design for enhancing inference from survey data is increasingly recognized among clinical researchers.^2,3 Recent guidelines for survey research on clinical topics have emphasized the need for reducing potential sources of bias by using validated questions, pre-testing survey questionnaires, and using incentives or reminders to enhance response rates.^4–6 However, the value of adopting these “best practices” of survey research has not been examined for studies recruiting the participation of anesthesiologists. Understanding the quality and impact of data that can be generated from surveys of anesthesiologists can support development of higher-quality surveys among members of the profession and increase the value of members’ participation in surveys.

In this systematic review, we identified published survey research sampling anesthesiologists and evaluated the quality and impact of these studies. The primary measure of quality was the survey response rate.⁶ Measures of impact included article citations, publication in high impact factor journals, Altmetric scores, and mention of studies in educational materials, news media, and professional society publications (other than academic journals). We hypothesized that adherence to recommended practices of survey design was associated with higher survey response rate and higher article impact. Our secondary aim was to determine which survey characteristics were associated with improved survey response rate, or greater impact, of surveys conducted among anesthesiologists.

Methods

IRB approval was not necessary for this study because it was a review of published research articles. In April and May of 2018, we conducted a systematic search of the MEDLINE database for English-language peer-reviewed research articles published in 2010–2017, which reported original data generated by surveying anesthesiologists based on their membership in one of the following professional societies: American Society of Anesthesiologists, ASA; Association of Anaesthetists of Great Britain and Ireland, AAGBI; Canadian Anesthesiologists’ Society, CAS; Australian and New Zealand College of Anaesthetists, ANZCA; Society for Pediatric Anesthesia, SPA; Association of Paediatric Anaesthetists of Great Britain and Ireland, APAGBI; Canadian Pediatric Anesthesia Society, CPAS; and Society for Pediatric Anaesthesia in New Zealand and Australia, SPANZA. Search keywords included “survey” and “anesthesia” or “anesthesiologist”/“anesthetist” (using British and American spelling, as well as plural and singular nouns). We used PubMed to search for articles, filtering results according to journal indexing in MEDLINE.

We identified titles and abstracts of articles and evaluated each for potential inclusion in the systematic review. We obtained full texts for original research articles (full length or brief communication formats) meeting our inclusion criteria. We screened each paper for original survey research of individual anesthesiologists responding on their own behalf. We excluded studies that sampled institutions or asked respondents to participate in their study on behalf of an institution (eg, studies that targeted fellowship program directors), and qualitative research using entirely unstructured surveys or interviews. We resolved disagreements regarding whether an article met inclusion criteria through discussion among the investigators. The lead investigator reviewed articles selected for inclusion, and coded the pre-specified fields shown in Table 1. All studies meeting inclusion criteria were planned to be included in the analysis, and no a priori power calculation was performed. Survey characteristics and quality outcomes which we expected would be commonly reported, and which would be suitable for objective assessment, were pre-selected to be included in the review. Data that were deemed unlikely to be reported in published studies, such as questionnaire visual design, were not included.

Table 1 Pre-Specified Fields Coded for Each Manuscript

The primary quality outcome in this review was the reported survey response rate, defined as the number of completed responses out of the number of potential respondents who were invited to participate. Partially completed surveys were included in the response rate if they were described as part of the number of completed responses in the original studies. Secondary quality measures included the item non-response rate for the primary study outcome, if one was specified, and the fraction of missing information. The item non-response rate was calculated as the number of responses missing data on the primary study outcome out of the total number of survey responses. The fraction of missing information was defined as the number of surveys with any incomplete data on study variables, out of the total number of complete surveys. We also noted whether the studies used multiple imputations to complete any items missing data for analysis. Article impact outcomes were assessed at the time of the review and included article citation counts from Web of Science and Google Scholar; the Altmetric score of news media and social media mentions; and the impact factor of the journal in which the article was published. These characteristics have been used in prior bibliometric research assessing the impact of publications in medicine and life sciences.^7–9 Additionally, we used a Web search (Google) to determine whether articles were mentioned in news media, textbooks or edited volumes, or official publications of professional societies, excluding academic journals.

Characteristics of survey design included the use of validated questions, survey pre-testing, use of incentives for survey completion, pre-notification of invited respondents, and use of reminders, reflecting general recommendations for improving survey research on clinically relevant topics.^4–6 Additional characteristics of surveys that may have influenced the response rate, such as survey mode and duration of data collection, were coded as summarized in Table 1.¹⁰ The study was not powered for a specific primary hypothesis test but explored a range of plausible associations among survey characteristics, survey response rate, and study impact. Response rates and impact measures were compared against survey characteristics using Spearman correlation coefficients and rank-sum tests. We did not assess the risk of bias because the surveys described included studies that evaluated a diverse range of topics, so bias in the estimation of a specific quantity or association was not an area of focus for our review. Data analysis was performed using Stata/IC 14.2 (College Station, TX: StataCorp LP) and two-tailed P<0.05 was considered statistically significant.

Results

Our initial MEDLINE search identified 1448 publications which were reviewed for potential inclusion in the study. Based on title and abstract review, 1403 publications were excluded, and full texts were obtained for the remaining 45 publications. Bibliographic data and study characteristics for the included studies are summarized in Table 2. Twenty surveys were primarily conducted in the United States (US), compared to 16 in Australia/New Zealand, 5 in Canada, and 4 in Great Britain and Ireland. Sample sizes ranged from 84 to 8178, for a total of 35,177 responses among the articles that reported sample size. Most surveys elicited respondents’ opinion about various anesthetic practices, such as the use of laryngeal mask airways, the prevalence of general anesthesia without intravenous access, perioperative management of patients with obstructive sleep apnea, and use of a difficult airway cart.

Table 2 Bibliographic Data and Study Characteristics for the Included Studies

Survey methodologies for included studies are summarized in Table 3. Most surveys were completed online, although five surveys used paper questionnaires. Reported methods to ensure survey validity included using validated questions (12/45) and pre-testing the survey questionnaire (23/45). Only one survey reported using both a pre-notification and incentives to increase participation, while 4 surveys reported using incentives alone. By contrast, most surveys used one or more reminders to increase participation (33/45, using a median of 2 reminders). Survey response rates, summarized in Table 4, ranged from 7% to 95% (median [IQR]: 37% [25%, 46%]), although the highest response rate attained on an online survey was 67%. Two studies reported the nonresponse rate for the primary outcome (1.4% and 19.1%, respectively), and 12 studies reported an overall fraction of missing information, ranging from 1.3% to 9.3% (median: 5%; IQR: 2%, 6%).

Table 3 Reported Characteristics of Survey Methodologies for Selected Studies

Table 4 Survey Quality and Impact Metrics

Considering the use of reminders, incentives, pre-notification, questionnaire pre-testing, and use of validated questions, we identified 30 studies which used at least one of these methods, and 15 studies which used none of these methods. The survey response rates did not significantly differ between these two groups (median: 37% vs 36%, p = 0.544). Considering other survey characteristics, median response rates were lower in US surveys compared to non-US surveys (26% vs 39%; 95% confidence interval [CI] of difference: 2%, 23%; p = 0.021), and higher for paper surveys compared to web surveys (55% vs 30%; 95% CI of difference: 13%, 42%; p=0.002). There were not enough data to compare secondary quality outcomes, such as the fraction of missing information.

Other study impact metrics include article citation counts, which ranged from 0 to 218 in Google Scholar (median [IQR]: 10 [4, 20]), and from 0 to 135 in Web of Science (median [IQR]: 5 [2, 14]). Among 16 articles from publishers reporting Altmetric scores, these scores ranged from 0 to 105 (median [IQR]: 2 [1, 6]). All except one of the surveys were published in journals with an assigned 2016 impact factor, with scores ranging from 1.2 to 5.8. Thirteen of the surveys have been referenced in official society publications, 15 in textbooks, and nine in news media. The survey response rate was not correlated with article citations on Google Scholar (ρ = −0.17, p = 0.251) citations on Web of Science (ρ = −0.12, p = 0.450), Altmetric scores (ρ = 0.06, p = 0.826) or journal impact factor (ρ = −0.005, p = 0.976). Survey response rates did not differ between studies that were referenced in official society publications, textbooks, or news media (median [IQR]: 36% [26%, 42%]) and studies that were not (median [IQR]: 38% [25%, 51%]; p = 0.544).

Discussion

Increased feasibility of administering surveys has prompted many groups to conduct survey research of anesthesiology professional societies, in order to gather novel data on clinical practices and to explore professional issues in the field. While recent overviews have brought attention to important aspects of survey design, empirical data remain scarce on what defines high-quality research involving surveys of anesthesiologists. To address this, we reviewed published surveys of large English-language professional anesthesia societies, focusing on variation and correlation in measures of survey quality and impact. Our review identified limited reporting of survey characteristics and a wide variability in survey response rates (7–95%). Other than the use of paper surveys, there were no evident associations between elements of survey design and survey response rates.

Response rate is the primary metric used to assess the quality of survey research.^11,12 Specifically, 23 of the articles reviewed (reporting response rates of 8–55%) cited a low response rate as one of their study limitations. Current research suggests that a response rate of 50–60% could minimize the risk of non-response bias, although 35 of the 45 surveys that were published and were included in our study did not meet this threshold.¹³ The low response rates in many of the surveys reviewed may be due to lower response rates seen in online surveys as compared to paper surveys.^14,15 Nevertheless, the median response rate among surveys included in our review was similar to that found in Sheehan’s meta-analysis, which indicated a median response rate of 37% for web-based surveys.¹⁶ Efforts to increase response rates may include financial incentives, advance letters, attempts to convert respondents who refuse to participate, and follow-up reminders. However, while these techniques have produced higher response rates in some experimental settings, their use has not been definitively associated with a reduction in non-response bias.¹⁷ Thus, high response rates to a survey may be necessary, but not sufficient to assure survey validity.¹⁸ Although our study did not overtly measure non-response bias, it was notable that no survey characteristics other than survey mode were correlated with the response rate. Given the generally high and variable non-response rates in the studies reviewed, we speculate that non-response in this setting could often be caused by anesthesiologists choosing to participate in a given survey primarily based on the topic of the research and its relevance to their practice and interests.

Our review presents new data on the state of survey research in anesthesiology but is subject to some limitations. First, we have focused on surveys of anesthesiology professional society members, which are facilitated by the availability of society mailing lists, and the probable interest of society members in contributing to academic research. Therefore, our conclusions may not be generalizable to surveys of other populations, such as patients or caregivers. We also excluded surveys that sample institutions (eg, surveys of fellowship program directors about characteristics of their fellowship program), as response rates to these surveys tend to be very high, possibly owing to respondents’ perceived responsibility to complete the survey on behalf of their institution.^19,20 Furthermore, we evaluated quality and impact metrics among published studies, but did not analyze which survey characteristics influenced the likelihood of publication and did not include surveys reported only in the “grey literature” (eg, non-peer-reviewed reports) or surveys conducted internally within professional societies. An additional limitation of examining impact metrics is that only a small number of articles receive scholarly and public attention more than the typical article, such as the Baird et al paper.²¹ Our review was further limited to publications in MEDLINE-indexed journals. Among the societies included in the study, all affiliated journals (eg, Anaesthesia, Anesthesiology, Canadian Journal of Anaesthesia, and Anaesthesia and Intensive Care) are indexed in MEDLINE, and some studies have described using MEDLINE as a “white list” of high-quality medical journals.²² Therefore, our review did not address the possibility that some surveys with methodological issues may have been published in journals not indexed in MEDLINE, and may not have captured articles that were published ahead of print during the review period, but not indexed in PubMed, as of May 2018. Lastly, we observed no statistically significant associations and weak correlations for a wide range of plausible of associations among survey characteristics, survey response rate, and study impact. This supports our conclusions regarding the unpredictability of response rates, and an evident lack of association between survey response rates and study impact.

In this study, we utilized bibliometric analyses to quantify the attention scientific articles receive.^7–9,23 One of the better-known aspects of bibliometrics is citation analysis.²⁴ Our review included several highly cited articles, yet we found that the survey response rate was not correlated with the number of citations. Thus, while survey response rates may influence journals’ decision to publish an article, this study characteristic does not appear to influence academic audiences’ subsequent judgment of the study’s importance. Rather, Falagas et al determined that characteristics such as article length and journal impact factor influence citation counts.²⁵ While citation counts are a classic tool of bibliometrics, alternative metrics are gaining popularity for assessing the impact of research outside of academia. The Altmetric score of a research article indicates the amount of online attention it has received by combining information from multiple data sources, such as social media and news mentions.²⁴ In this review, one article had a very high Altmetric score while the rest of the studies received little attention according to this metric. As with citation counts, the response rate was not correlated with the Altmetric scores. This further suggests that the value of survey research in anesthesiology professional societies may be to raise novel questions or present data unobtainable in other ways, but not necessarily to generate precise and unbiased estimates dependent on a high response rate.

In previous studies, aspects of survey methodology were often reported inconsistently and only partially.²⁶ Reviewing articles in anesthesiology journals, Story et al found that the reporting of survey methods was inconsistent and potentially compromised the transparency and reproducibility of surveys.²⁷ Many sources, including but not limited to review articles published in the anesthesiology literature, outline good practices in conducting and reporting survey research, such as robust testing and development of the research instrument.^1–6 Kelley et al emphasize the importance of reporting the details of primary data collection, such as how participants were selected, as well as how data were analyzed and whether any adjustments were made to account for nonresponse, missing data, or differential probability of response.²⁸ Additionally, Davern et al have discussed the importance of estimating nonresponse bias, such as by comparing the survey with other sources, analyzing para-data, or analyzing external data sources.¹² Halbesleben et al discussed specific formulas to calculate nonresponse bias based on the response rate, proportion of non-respondents, and characteristics of non-respondents.²⁹ Despite these recommendations, one study in this review did not report the number of participants, while only 12 studies reported using previously validated questions and 23 studies reported pre-testing surveys. Only three studies used statistical weights to account for differential probability of response, and only two studies reported primary outcome nonresponse rates.

Techniques such as weighting and imputation, used in survey data analysis to address potential bias and non-response, appear to be under-used in surveys of anesthesia professional societies. Based on this review, several recommendations for survey practice can be made. Investigators should report essential information regarding data collection and analysis, such as the population targeted, the sampling frame, and whether the sampling frame included the entire society, a random subsample, or a non-random subsample (eg, attendees at a conference). Studies should also describe the development of survey research tools, the source of survey questions, the response rate, the primary outcome nonresponse rate, and the fraction of missing information, using established definitions from the survey methodology literature as appropriate. Additionally, researchers should attempt to analyze nonresponse bias, which can be significant even in the presence of high response rates.^12,29 It is particularly important to consider using weighting or analysis of nonresponse bias in the presence of low response rates. Lastly, in surveys of smaller societies, survey data analysis may utilize finite population correction when estimating standard errors of estimates, as many survey samples in our review represented significant proportions of the total society membership. Statistical guidelines on this technique recommend using a finite population correction, which deflates the standard error in proportion to the population fraction surveyed, when the sample size is more than 5% of the total population.³⁰ These recommendations could be incorporated into the development and evaluation of survey research for future studies in this area.

In summary, many surveys of anesthesiology professional societies have been conducted to study clinical, educational, and professional topics in the field. Technology has increased the ease of survey administration, as many survey invitations are distributed to all society members via email to complete on-line, with repeat reminders sent as often as once a month. Notwithstanding the ease of survey administration, response rates were often low even in published survey research. Investigators’ use of reminders and other aspects of survey methodology were not associated with higher response rates in published surveys of anesthesia professional societies. Furthermore, despite a high variability in response rates, the survey response rate was not associated with article impact as measured by article citations, Altmetric score, journal impact factor, or references in society publications, textbooks, or news media. Improving reporting of survey methodology and validating techniques for increasing the response rate specifically among members of anesthesiology professional societies may aid in increasing the quality of survey research in this area, and improve understanding of possible sources of nonresponse bias. Our findings provide a baseline for initiatives to improve survey research in anesthesiology professional societies, and a point of comparison for readers or reviewers assessing the quality of surveys in this population.