Back to Journals » Patient Related Outcome Measures » Volume 13

Characteristics of Exceptionally Good Doctors: A Protocol for a Cross-Sectional Survey of Adults

Authors Schnelle C , Jones MA

Received 24 May 2022

Accepted for publication 7 August 2022

Published 12 August 2022 Volume 2022:13 Pages 181—188


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Professor Lynne Nemeth

Christoph Schnelle, Mark A Jones

Institute of Evidence-Based Healthcare, Bond University, Robina, Queensland, Australia

Correspondence: Christoph Schnelle, Institute for Evidence-Based Healthcare, Bond University, Robina, Queensland, Australia, Email [email protected]

Background: Doctors constitute a significant proportion of a very large number of medical interactions. They are known to vary in the quality of their work, with some having an exceptionally beneficial effect on patients’ physical health. In a qualitative study, we interviewed medical doctors on their opinions and experiences of exceptionally good doctors. Their responses and the results from previous research are used as a basis for this proposed cross-sectional survey directed to members of the public on their encounters with exceptionally good doctors. The primary aim of this cross-sectional study is to describe the characteristics of exceptional doctors as reported by a large representative sample of adult patients.
Methods and Analysis: A mixed qualitative and quantitative anonymous cross-sectional survey of 500 Amazon Mechanical Turk (MTurk) respondents, who have met one or more exceptionally good doctors in their life, will be conducted. Information requested will include reasons for nominating a particular doctor; experience of how that doctor differs from other and average doctors; and 34 5-point Likert scale questions on the characteristics of that doctor and the same Likert questions for the average doctor. An opportunity to report their experience in free-text form will be provided. Sample size will be sufficient to obtain a margin of error of 4%. The authors will provide descriptive statistics, including graphs of the Likert scale question responses; conduct factor analysis for internal validity; investigate satisficing and logical inconsistencies; and explore whether there are multiple types of exceptionally good doctors.
Discussion: Previous surveys of patients’ perceptions of doctors exist though none have focused on exceptionally good doctors. The expected results will include a list of characteristics that are important to patients in determining exceptionally good doctors.

Keywords: medical practice, good doctors, doctors’ performance, patients’ opinion, cross-sectional survey

Plain Language Summary

Previous research has shown that some doctors are exceptionally good. In a qualitative study, we have interviewed doctors about what characteristics make up an exceptionally good doctor. Using the interview results, we have designed a survey to investigate the opinions of the general adult public on exceptionally good doctors. The survey will be conducted via Amazon’s Mechanical Turk (MTurk) platform, where members are offering to provide online services like answering surveys or identifying objects in images. MTurk has been used successfully in previous research surveys. The responses to this survey will provide a list of the important characteristics of an exceptionally good doctor from a patient perspective.


“[T]he most frequently used drug [intervention] in general practice is the doctor himself”.1,2 In Australia, general practitioners receive more than 150 million visits per year.3 That is an average of six visits for each Australian. Adding visits to specialists and extending this to the entire world, we can expect billions of times this intervention, “the doctor himself” is used each year.

It is possible to identify doctors who have an exceptionally beneficial impact on patients’ physical health, even after accounting for all known confounding factors such as patient risk, doctor demographics, and hospital factors.4–16 Evidence from systematic reviews has shown that surgical experience, measured in case volume or years of practice, is a key factor in improved surgical performance and outcomes, supporting the need for a continuing learning process.17,18 Provider expectations can substantially affect patient outcomes.19 For non-surgical specialties, data have suggested that doctor-related factors have a considerable effect on patients’ health outcomes in primary,20 acute care,21 and obstetrics.10 Abu-Hilal et al22 studied the characteristics of good doctors, concluding that clinical ability, knowledge, approachability, and appreciation of limitations are the most important factors. Qualitative study research has identified eight vital skills of exceptionally good doctors in their relationships with patients: “do the little things; take time; be open and listen; find something to like; remove barriers; let the patient explain; share authority; and be committed”.23 Steiner-Hofbauer et al's24 systematic review of “good doctors” included 6 studies and 2 questionnaires,25–29 and concluded that there is no clear definition on what makes a good doctor.

Although there are published opinions on what makes a good doctor,30–40 the character of a good doctor,41 divergent patient and doctor viewpoints,29 and a thesis on the good doctor in education,42 exceptionally good doctors have been investigated infrequently.23,43,44 It is not known what makes an exceptionally good doctor.

In a qualitative study, we have interviewed a sample of medical doctors on their opinions of what makes an exceptionally good doctor.45,46 These interviews have led to a set of characteristics that describe an exceptional doctor from the practitioner point of view.

A systematic review on the links between patient experience and clinical safety and effectiveness reported “consistent positive associations between patient experience, patient safety and clinical effectiveness for a wide range of disease areas, settings, outcome measures and study designs”47 and therefore “supports the case for the inclusion of patient experience as one of the central pillars of quality in healthcare.” Patient tutors now work in collaboration with clinical staff to develop the curriculum and assessment of medical students in the UK.48

Given the importance of patient input into medical education, the positive associations between patient experience and clinical effectiveness, and the knowledge gap on what makes an exceptional doctor, we plan to conduct a survey of adult patients on their experiences with exceptionally good doctors. The primary aim of the survey is to describe the characteristics of exceptional doctors from a patient perspective.

Methods and Analysis

The study design reporting follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines on observational studies.49

Study Design

This is an observational cross-sectional study, using a mixed qualitative and quantitative anonymous online survey design to collect data on patients’ perceptions of exceptionally good doctors. In particular, this is both an exploratory sequential design mixed methods study50 as the authors previously interviewed 13 medical doctors about exceptionally good doctors45,46 and used the insights gained to produce the quantitative part of this survey. It is also a convergent design mixed-method study50 as the three qualitative questions and six questions with a free-text option “other” provide a more in-depth and personal perspective of the respondents.

Participants will be sampled through Amazon Mechanical Turk.51

Data will be collected for the respondent demographic information on gender; age by decade; highest education level achieved; count of doctor visits in the previous year; and country will be deduced from the respondents’ IP address. The respondents will be asked how many exceptionally good doctors they have met in their life (1, 2, 3, 4, 5+) and how many overall doctors they have met (1–5, 6–10, 11–20, 21–50, 51–100, 101+).

Questions asked in the survey about the nominated exceptionally good doctor are the doctor’s age by decade, gender, type of doctor (GP, specialist, other), the respondent’s relationship to that doctor (patient, employer, etc), and the reason for nominating this doctor as exceptionally good, with the reasons offered for the respondents to choose from falling into the broad categories of firstly being an exceptional person or behaving exceptionally by being an exceptional communicator or empowering patients, secondly being an exceptional diagnostician or, thirdly, being an exceptional wielder of interventions. A further question asks how many characteristics are needed for a doctor to be exceptional.

In addition there are 34 5-point Likert scale questions. The Likert scale questions were chosen based on responses from a sample of doctors interviewed as part of a qualitative study on the characteristics of exceptional doctors.45 The full list of questions is shown in the Supplementary File.

With the Likert questions, the respondents are asked how much they agree with the nominated doctor having specific traits compared to an average doctor.

A further qualitative question offers the respondents the opportunity to relate their experience with the exceptionally good doctor into their own words.

One question displays all the Likert question items where the respondent gave 4.5 or more out of 5 for the exceptionally good doctor, and asks to pick the top three such questions and rank them.


The participants will be recruited from Amazon’s Mechanical Turk (MTurk) population. As per Amazon’s website,51 “Amazon Mechanical Turk (MTurk) is a crowdsourcing marketplace that makes it easier for individuals and businesses to outsource their processes and jobs to a distributed workforce who can perform these tasks virtually. This could include anything from conducting simple data validation and research to more subjective tasks like survey participation, content moderation, and more.

MTurk has been used by thousands of social scientists in research52 and is very suitable for surveying the general population.53–55 In addition, the large number of MTurk workers alleviates concerns of non-naivety of the workers.56 MTurk works by logging in if you have an Amazon account, or by registering as a requester. The requester creates a new project by providing instructions and a link to the survey, to which the workers are requested to respond, in this case a Qualtrics57 survey. The Qualtrics survey creates a code that is displayed to the worker if they finish the survey, which should be entered by the respondent into the Amazon MTurk instruction page, as proof of them having done the survey. Thereafter, the requester is enabled to download a list of the workers’ anonymous IDs and the codes the workers have submitted, compare this list to the codes that were generated, include those who submitted a correct code and exclude those who did not.

The demographics of MTurk workers are generally representative of the population, except that they are skewed younger,58 therefore, to assure a representative sample, we conducted a pilot study of 210 respondents, of which 6% were found to be over 55, and 67% were males. To obtain a gender split that is closer to 50/50 and to have a representative age-based distribution, the authors will submit four requests, also known as batches, as per Table 1. In this manner, we expect to have around one third of participants over 55 and approximate gender parity. MTurk makes it possible to ensure that no worker participates in more than one project or batch. Survey weighting59,60 will be used for age and gender if the respondents’ age and gender are more skewed than expected.

Table 1 MTurk Batches

Sample Size

We have based our sample size of 500 participants on a desire to obtain a margin of error of 4% around estimates of proportions of participants reporting particular characteristics of exceptional doctors (eg, communication skills) and reasons why they consider the doctor in question to be exceptional (eg, they saved my life).


Selection Bias

Since MTurk population tends to be skewed towards younger-aged workers and males, a random sample could lead to sampling bias. We will avoid this possibility by selecting a stratified sample to ensure 34% of the population being aged 55 years or older and gender equality.

Nonresponse Bias

In the MTurk marketplace, where the survey is accessed by Amazon’s workers, it is not possible to determine the number of potential respondents who have seen the survey but chose not to participate. Consequently, it is not possible to accurately calculate the response rates. Nevertheless, it is argued that response representativeness is more important than response rate in survey research.61 Therefore, since we will attempt to obtain a representative sample, this bias will be less of a concern.62

Response Bias

Question order bias will be addressed by randomizing the order in which the response categories within the Likert question sets are presented.

In addition, there is a potential for bias as the questions have been derived from interviews with medical doctors on what they consider to be the characteristics of exceptionally good doctors. Any putative exogenous characteristics that pertained to exceptionally good doctors were addressed in the pilot study by offering the respondents a free-text opportunity to add further characteristics; likewise, several free-text spaces have been added to the final version of the survey.

Missing Data

All questions, except for the free-text questions, will be mandatory; therefore, no missing data is expected. However, since this requirement may lead to satisficing, ie respondents not replying truthfully in order to get to the end of the survey more quickly,63 the responses will be checked for unusual patterns such as many identical responses to the Likert questions.

Confounding variables are not an issue as no cause-and-effect relationship is investigated, only opinions and experiences.

Planned Analysis

The authors will provide descriptive statistics of the demographic variables supplied, in addition to the combined graphs of the Likert questions. The 10 most commonly reported characteristics of exceptional doctors will be reported as proportions with 95% confidence intervals. The frequency of reasons why a doctor was nominated as exceptional will also be reported as proportions with 95% confidence intervals.

Factor analysis64 of the Likert questions in the pilot study showed that there were 37 factors, in which component one had an eigenvalue of 18, component two had an eigenvalue of 1.1, while all other components had smaller eigenvalues. Thereafter, as factors 22–37 were very slightly negative in their contributions, the first factor accounted for 82% of the total proportion. Therefore, the pilot survey overwhelmingly measured a single factor: the characteristics of exceptionally good doctors.

The validity of the Likert questions for identifying exceptionally good doctors will be further explored by asking the same Likert questions in relation to average doctors and then assessing whether factor analysis identifies separate factors for exceptionally good doctors and average doctors. Further, the Likert questions will be specified as dependent variables in linear regression models to explore whether the demographic and doctor characteristics show an association with the Likert question responses. As the Likert question responses may not follow a normal distribution, nonparametric regression will also be conducted.65

To facilitate visual comparisons between the responses to the Likert questions, “[k]ernel density estimates [plots], which can be considered as a smoothed form of histogram”66 will be constructed. Kernel density plots allow multiple histogram outlines to be shown in a single graph.

Subgroup comparisons will be performed for all relevant variables to explore whether responses differ by age group, gender, education levels, or count of doctor visits.

The free-text question 5 responses will be analyzed to investigate whether the traits, aspects or qualities that make a doctor exceptional listed by the respondents at the very beginning of the survey are consistent with the responses provided by the 13 medical doctors interviewed in the qualitative study. The free-text question asking respondents for their personal experience will be analyzed by the usage of an NVivo67 word cloud to detect patterns.

Ethical Considerations

Ethical approval has been received from Bond University. A preprint shows that it is ethical to use MTurk as a survey tool.68 An informed consent form will be provided at the beginning of the survey. The participants will be informed that they can stop the survey at any time though they will forego payment if they do. The data will be stored at Bond University in a secured facility by the lead researcher on a password-protected computer.


There is existing research on patients’ opinions on what constitutes a good doctor,69–71 but not on what constitutes an exceptionally good doctor. Currently, there is no clear definition on what is an exceptionally good doctor. A qualitative study aims to provide a set of characteristics that describe exceptional doctors from the doctor perspective.45 This proposed study will add to this knowledge base by providing a set of characteristics that describe exceptional doctors from the patient perspective.


A strength of this proposed survey is that the questions are based on in-depth interviews with medical doctors on what they consider to be the characteristics of exceptionally good doctors. The survey also has a qualitative element to allow the respondents to report their experience in a free-text format. The participant responses to the survey questions will be important to anyone in healthcare practice and medical education but also of interest to the general public.


A limitation is that this is a survey of respondents who speak English and are mostly from developed nations with a preponderance of respondents being from the US. Therefore, it is uncertain whether the findings could be generalized to other regions, particularly developing nations. In addition, due to the heterogeneity regarding medical/surgical specialties, types of interventions, and types of outcomes relevant to different interventions, there may be differing criteria on what makes an exceptionally good doctor for different medical/surgical specialties.


The views expressed are those of the authors and not necessarily those of Bond University.

Data Storage

Data will be stored in a secured location at Bond University for a period of 5 years after the end of this project as per 601.3/C150 of the Qld Government University Sector Retention and Disposal Schedule in accordance with the guidelines set out by the Bond University Human Research Ethics Committee.

Ethics Approval

Ethical approval CS03416 was provided on April 27th by the Bond University Human Research Ethics Committee.


The participants will only be able to preview the rest of the survey after they provide their consent.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation of data, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.


This review has been funded by the first author as part of his PhD studies. No external funding was received.


The authors report no conflicts of interest in this work.


1. Curran J. The doctor, his patient and the illness. BMJ. 2007;335(7626):941. doi:10.1136/bmj.39384.467928.94

2. Balint M. The doctor, his patient, and the illness. Lancet. 1955;265(6866):683–688. doi:10.1016/S0140-6736(55)91061-8

3. Health AGDo. Medicare in Australia. Australians make more than 150 million visits to a GP every year; 2022.

4. Begg CB, Riedel ER, Bach PB, et al. Variations in morbidity after radical prostatectomy. New Engl J Med. 2002;346(15):1138–1144. doi:10.1056/NEJMsa011788

5. Bianco FJ Jr., Riedel ER, Begg CB, Kattan MW, Scardino PT. Variations among high volume surgeons in the rate of complications after radical prostatectomy: further evidence that technique matters. J Urol. 2005;173(6):2099–2103. doi:10.1097/01.ju.0000158163.21079.66

6. Bianco FJ Jr., Vickers AJ, Cronin AM, et al. Variations among experienced surgeons in cancer control after open radical prostatectomy. J Urol. 2010;183(3):977–983. doi:10.1016/j.juro.2009.11.015

7. Brown EC, Robicsek A, Billings LK, et al. Evaluating primary care physician performance in diabetes glucose control. Am J Med Qual. 2016;31(5):392–399. doi:10.1177/1062860615585138

8. Glance LG, Dick A, Osler TM, Li Y, Mukamel DB. Impact of changing the statistical methodology on hospital and surgeon ranking: the case of the New York State cardiac surgery report card. Med Care. 2006;44(4):311–319. doi:10.1097/01.mlr.0000204106.64619.2a

9. Gossl M, Rihal CS, Lennon RJ, Singh M. Assessment of individual operator performance using a risk-adjustment model for percutaneous coronary interventions. Mayo Clin Proc. 2013;88(11):1250–1258. doi:10.1016/j.mayocp.2013.07.017

10. Harley M, Mohammed MA, Hussain S, Yates J, Almasri A. Was Rodney Ledward a statistical outlier? Retrospective analysis using routine hospital data to identify gynaecologists’ performance. Br Med J. 2005;330(7497):929–932. doi:10.1136/bmj.38377.675440.8F

11. Kunadian B, Dunning J, Roberts AP, Morley R, de Belder MA. Funnel plots for comparing performance of PCI performing hospitals and cardiologists: demonstration of utility using the New York hospital mortality data. Catheter Cardiovasc Interv. 2009;73(5):589–594. doi:10.1002/ccd.21893

12. Landercasper J, Borgert AJ, Fayanju OM, et al. Factors associated with reoperation in breast-conserving surgery for cancer: a prospective study of American Society of Breast Surgeon members. Ann Surg Oncol. 2019;26(10):3321–3336. doi:10.1245/s10434-019-07547-w

13. Rudmik L, Xu Y, Alt JA, et al. Evaluating surgeon-specific performance for endoscopic sinus surgery. JAMA Otolaryngol Head Neck Surg. 2017;143(9):891–898. doi:10.1001/jamaoto.2017.0752

14. Schnelle C, Clark J, Mascord R, Jones M. Is there a surgeons’ effect on patients’ physical health, beyond the intervention, that requires further investigation? A systematic review. Ther Clin Risk Manag. 2022;18(18):467–490. doi:10.2147/TCRM.S357934

15. Schnelle C, Clark J, Mascord R, Jones M. Is there a doctors’ effect on patients’ physical health, beyond the intervention and all known factors? A systematic review. Ther Clin Risk Manag. 2022;18:721–737. doi:10.2147/TCRM.S372464

16. Schnelle C, Jones MA. The doctors’ effect on patients’ physical health outcomes beyond the intervention. A methodological review. Clin Epidemiol. 2022;14:851–870. doi:10.2147/CLEP.S35792

17. Wilt TJ, Shamliyan TA, Taylor BC, MacDonald R, Kane RL. Association between hospital and surgeon radical prostatectomy volume and patient outcomes: a systematic review. J Urol. 2008;180(3):820–829. doi:10.1016/j.juro.2008.05.010

18. Maruthappu M, Gilbert BJ, El-Harasis MA, et al. The influence of volume and experience on individual surgical performance: a systematic review. Review. Ann Surg. 2015;261(4):642–647. doi:10.1097/SLA.0000000000000852

19. Chen P-HA, Cheong JH, Jolly E, Elhence H, Wager TD, Chang LJ. Socially transmitted placebo effects. Nat Hum Behav. 2019;3(12):1295–1305. doi:10.1038/s41562-019-0749-5

20. Moreau A, Boussageon R, Girier P, Figon S. The “doctor” effect in primary care. Presse Med. 2006;35(6I):967–973. doi:10.1016/S0755-4982(06)74729-7

21. Beckett DJ, Spears M, Thomson E. Reliable consultant level data from an acute medical unit: a powerful tool for improvement. J R Coll Physicians Edinb. 2018;48(2):108–113. doi:10.4997/jrcpe.2018.202

22. Abu-Hilal M, Morgan EC, Lewis G, McPhail M, Malik HZ, Hocken D. What makes a good doctor in the 21st century? A qualitative study. Br J Hosp Med. 2006;67(7):375–377. doi:10.12968/hmed.2006.67.7.21623

23. Churchill LR, Schenck D. Healing skills for medical practice. Ann Intern Med. 2008;149(10):720–724.

24. Steiner-Hofbauer V, Schrank B, Holzinger A. What is a good doctor? Review. Wien Med Wochenschr. 2018;168(15–16):398–405. doi:10.1007/s10354-017-0597-8

25. Herzig S, Biehl L, Stelberg H, Hick C, Schmeißer N, Koerfer A. What makes a doctor a good doctor? A content analysis of assessments by a sample of doctors. Dtsch Med Wochenschr. 2006;131(51–52):2883–2888. doi:10.1055/s-2006-957216

26. Kim JH, Tor PC, King J, Seo JS. A Korean survey on qualities and definition of a good psychiatrist. J Korean Med Sci. 2015;30(5):632–638. doi:10.3346/jkms.2015.30.5.632

27. Kliems H, Witt CM. The good doctor: a qualitative study of German homeopathic physicians. J Altern Complement Med. 2011;17(3):265–270. doi:10.1089/acm.2010.0158

28. Lambe P, Bristow D. What are the most important non-academic attributes of good doctors a Delphi survey of clinicians. Med Teach. 2010;32(8):e347–e354. doi:10.3109/0142159X.2010.490603

29. Miratashi Yazdi SN, Nedjat S, Majdzadeh R, Arbabi M. Who is a good doctor? Patients & physicians’ perspectives. Iran J Public Health. 2015;44(1):150–152.

30. Conti CR. What makes a good doctor? Clin Cardiol. 2005;28(11):496–498. doi:10.1002/clc.4960281102

31. Bates C. The good doctor. Clin Med. 2001;1(2):128–131. doi:10.7861/clinmedicine.1-2-128

32. Smith DH. How to be a good doctor in the 1990s: stand and deliver. Am J Obstet Gynecol. 1994;170(6):1724–1728. doi:10.1016/S0002-9378(94)70347-7

33. Magauran CE, Brennan M. Being a “good doctor”. J Palliat Med. 2008;11(3):506–508. doi:10.1089/jpm.2007.0206

34. Davis K, Carbone R, Fredericks J, Lujan J, Basdeo A. What makes a good doctor: a qualitative study of patient perspectives; 2021.

35. Simpson L. What makes a good surgeon? Editorial. J Natl Med Assoc. 2008;100(2):261–264. doi:10.1016/S0027-9684(15)31216-5

36. Crile GW. The most important factor in the treatment of war wounds and the most important factor in civilian surgery – the good surgeon. Ann Surg. 1919;70(4):385–387. doi:10.1097/00000658-191910000-00001

37. Darzi A, Smith S, Taffinder N.Assessing operative skill. Needs Become More Objective. BMJ. 1999;318(7188):887–888. doi:10.1136/bmj.318.7188.887

38. Gandhi J. Making of a surgeon. Al Ameen J Med Sci. 2019;12(2):54–55.

39. Arora S, Sevdalis N, Suliman I, Athanasiou T, Kneebone R, Darzi A. What makes a competent surgeon?: experts’ and trainees’ perceptions of the roles of a surgeon. Am J Surg. 2009;198(5):726–732. doi:10.1016/j.amjsurg.2009.01.015

40. Jackson B. What makes an excellent surgeon? Note. Obes Surg. 2019;29:1087–1089. doi:10.1007/s11695-019-03778-8

41. Jones D What are the character strengths of a good doctor? Jubilee centre for character and virtues: insight series; 2013.

42. Whitehead CR. The Good Doctor in Medical Education 1910–2010: A Critical Discourse Analysis. Canada: University of Toronto; 2011.

43. Hanyok LA, Hellmann DB, Rand C, Ziegelstein RC. Practicing patient-centered care: the questions clinically excellent physicians use to get to know their patients as individuals. Patient. 2012;5(3):141–145. doi:10.1007/BF03262487

44. Schenck D, Churchill L. Healers: Extraordinary Clinicians at Work. Oxford University Press; 2011.

45. Schnelle C, Jones MA. Protocol for a qualitative study on doctors’ opinions on and experiences of exceptionally good doctors. Adv Med Educ Pract. 2022;13:103–109. doi:10.2147/AMEP.S343554

46. Schnelle C, Jones MA. Qualitative study of medical doctors on their experiences and opinions of the characteristics of exceptionally good doctors. Adv Med Educ Pract. 2022;13:717–731. doi:10.2147/AMEP.S370980

47. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3(1):e001570. doi:10.1136/bmjopen-2012-001570

48. Wilkinson E. The patients who decide what makes a good doctor. BMJ. 2018;361:k1829. doi:10.1136/bmj.k1829

49. Vandenbroucke JP, von Elm E, Altman DG, et al. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4(10):e297. doi:10.1371/journal.pmed.0040297

50. Zheng M. Conceptualization of cross-sectional mixed methods studies in health science: a methodological review. Int J Qual Methods. 2015;3(2):66–87.

51. Amazon. Amazon mechanical turk; 2022. Available from: Accessed February 5, 2022.

52. Buhrmester MD, Talaifar S, Gosling SD. An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspect Psychol Sci. 2018;13(2):149–154. doi:10.1177/1745691617706516

53. Boas TC, Christenson DP, Glick DM. Recruiting large online samples in the United States and India: Facebook, mechanical turk, and qualtrics. Political Sci Res Methods. 2020;8(2):232–250. doi:10.1017/psrm.2018.28

54. Buhrmester M, Kwang T, Gosling SD. Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect Psychol Sci. 2011;6(1):3–5. doi:10.1177/1745691610393980

55. Mortensen K, Hughes TL. Comparing Amazon’s Mechanical Turk platform to conventional data collection methods in the health and medical research literature. J Gen Intern Med. 2018;33(4):533–538. doi:10.1007/s11606-017-4246-0

56. Robinson J, Rosenzweig C, Moss AJ, Litman L. Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the mechanical turk participant pool. PLoS One. 2019;14(12):e0226394. doi:10.1371/journal.pone.0226394

57. Qualtrics. Qualtrics website; 2022. Available from: Accessed February 5, 2022.

58. Moss A, Litman L Demographics of people on amazon mechanical turk. Cloud Research blog post. Available from: Accessed August 9, 2022.

59. Pfeffermann D. The role of sampling weights when modeling survey data. Int Stat Rev. 1993;61:317–337. doi:10.2307/1403631

60. Valliant R, Dever JA. Survey Weights: A Step-by-Step Guide to Calculation. TX: Stata Press College Station; 2018.

61. Cook C, Heath F, Thompson RL. A meta-analysis of response rates in web- or internet-based surveys. Educ Psychol Meas. 2000;60(6):821–836. doi:10.1177/00131640021970934

62. Cook C, Heath F, Thompson R. A meta-analysis of response rates in web- or internet-based survey. Educ Psychol Meas. 2000;60:821–836. doi:10.1177/00131640021970934

63. Krosnick JA, Narayan S, Smith WR. Satisficing in surveys: initial evidence. New Dir Eval. 1996;1996(70):29–44. doi:10.1002/ev.1033

64. Tabachnick BG, Fidell LS, Ullman JB. Using Multivariate Statistics. 7th ed. Pearson Educatoin, Inc.; 2018.

65. Altman NS. An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat. 1992;46(3):175–185. doi:10.1080/00031305.1992.10475879

66. Baxter MJ, Beardah CC, Wright RVS. Some archaeological applications of kernel density estimates. J Archaeol Sci. 1997;24(4):347–354. doi:10.1006/jasc.1996.0119

67. NVivo (version 12); 2018. Available from: Accessed August 9, 2022.

68. Moss AJ, Rosenzweig C, Robinson J, Litman L. Is it ethical to use mechanical turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. Prepr PsyArXiv Prepr. 2020. doi:10.31234/

69. Klingenberg A, Bahrs O, Szecsenyi J. Wie beurteilen Patienten Hausärzte und ihre Praxen? Z Arztl Fortbild Qualitatssich. 1999;93(6):437–445. German.

70. Mercer SW, McConnachie A, Maxwell M, Heaney D, Watt GC. Relevance and practical use of the Consultation and Relational Empathy (CARE) measure in general practice. Fam Pract. 2005;22(3):328–334. doi:10.1093/fampra/cmh730

71. Schattner A, Rudin D, Jellin N. Good physicians from the perspective of their patients. BMC Health Serv Res. 2004;4(1):26. doi:10.1186/1472-6963-4-26

Creative Commons License © 2022 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.