Developing a Quality Evaluation Index System for E-Consultation Doctor-Patient Communication Using the Delphi Method

Introduction

In recent years, with the rapid development of the Internet, Web-based consultation has gradually penetrated the medical industry, and the online clinic consulting community came into being.^1,2 These communities connect doctors and patients through the internet, providing patients with access to various medical information. This has significantly enhanced the convenience of patient participation in the medical process, streamlined the treatment procedures, and better fulfilled the medical needs of patients.^3,4 Patients can make appointments with specialists and pay for consultations online, offering a targeted and efficient approach to obtaining healthcare answers.⁵ For minor illnesses, chronic disease management, and post-diagnosis rehabilitation, patients can choose to seek immediate answers through online consultations, saving them the time spent queuing at offline clinics. It is worth noting that many of the doctors in these online clinic consulting communities come from high-level tertiary hospitals, ensuring the provision of high-quality healthcare answers to patients.^6–9 Well-known platforms such as Good Doctor Online, WeChat Medical Network, and DXY Garden have become as renowned as WebMD, Yahoo, and Your Disease Risk.

The core element of these online medical communities is e-consultation, which extends the boundaries of medical services with the aid of new technologies. It serves as the main channel for doctors and patients to connect and receive clinic services.

At present, there is a growing body of research focused on e-consultation and remote medical services, highlighting their significance as integral components of internet healthcare.^10,11 For instance, Miaojie Qi et al¹² demonstrated that e-consultation plays a crucial role in enhancing patients’ disease awareness and guiding them towards more informed medical decisions. Similarly, Bokolo et al¹³ highlighted the timely application of telemedicine, which holds immense potential in safeguarding the well-being of medical practitioners and patients, while also curbing the spread of viruses by limiting patients’ social mobility. However, some researchers have found that the quality of e-consultation is not high, and the patient’s perception of doctor-patient communication is also not ideal.^14–16 As online consultations are a relatively new form of healthcare service, there are still areas that require improvement in terms of quality. Particularly, doctors’ responses to patients’ questions may be difficult to understand, unclear, or vague. Furthermore, in the process of communication, there is often a lack of providing human care. Doctors tend to focus primarily on the disease rather than addressing the emotional needs of patients. Improving the quality of e-consultations is crucial to enhancing the overall patient experience. Nevertheless, defining what constitutes high-level doctor-patient communication and how to evaluate its quality remain significant questions. Therefore, our research endeavors to establish a comprehensive e-consultation quality indicator system. This system aims to serve as a reliable reference standard for users, enabling them to evaluate the quality of platforms from various perspectives. By doing so, users can make informed judgments about their platform usage experience, ultimately saving both time and expenses.

There have been numerous studies conducted on the evaluation of the consultation level between doctors and patients, employing various evaluation measures such as assessments from doctors, patients, others, self-evaluations, and department-specific evaluations. For instance, Internal Medicine Patient Satisfaction Questionnaire, widely used in the United States, is a patient satisfaction survey questionnaire.¹⁷ Stewart et al¹⁸ developed the Interpersonal Process Scale (Process of Care: IPC) to evaluate doctors’ communication behaviors during patient visits. Elwyn et al¹⁹ designed the OPTION (Observing Patient Involvement in Decision Making) scale to increase patients’ understanding of their consultation process. Additionally, Stewart Mercer et al²⁰ devised the CARE (Consultation and Relational Empathy) scale. In comparison to the extensive range of quality evaluation scales abroad, research on the quality evaluation of doctor-patient communication in China has been relatively nascent, primarily consisting of qualitative studies with limited quantitative research. Moreover, the existing studies have predominantly focused on face-to-face doctor-patient communication.^21–23 To date, no relevant research reports specific to our research scope have been identified. However, it is important to note that due to cultural disparities, variations in medical systems, and diverse consultation processes both domestically and internationally, it is not feasible to directly apply foreign scales in the Chinese context.

Based on this, there is a lack of specialized measurement tools for evaluating the quality of e-consultations. Moreover, there has been limited exploration and verification of applying face-to-face doctor-patient communication measurement tools to the online environment.

Therefore, this study combines the situational characteristics of e-consultation and adopts the Delphi and AHP methods to build an e-consultation quality indicator system, which aims to provide a standard and quantitative basis for evaluating the quality of e-consultation.

Materials and Methods

Study Design

With “online doctor-patient communication, quality of doctor-patient communication, and online doctor-patient communication” as keywords, about 153 relevant literature were retrieved and collected in databases, such as PubMed, Web of Science, and China National Knowledge Infrastructure After consulting relevant literature and discussion in the research group, a preliminary indicator system was established. Then, two rounds of expert consultation were conducted using the Delphi method. On the basis of expert evaluation and suggestions, the indicator system was modified through internal discussion of the research group to form the final indicator system (Figure 1).

Figure 1 Flow chart for establishing indicator system.

Since the main objects of online consultation include clinical doctors and patients, we use stratified purposive sampling, fully considering the types, experiences, and authority levels of experts, to ensure the comprehensiveness, diversity, and authority of the experts. Based on this principle, we have invited experts with experience in online doctor-patient communication, experts with experience in doctor-patient interpersonal communication, and patients who have received online medical treatment to ensure the reliability of the expert consultation process.

Expert Selection

The selection of experts for consultation was based on several key principles. Firstly, the experts needed to possess a deep familiarity with the subject matter and hold authority in the industry. They should also demonstrate a strong understanding of informed consent and possess the ability to cooperate effectively. In line with the objectives of this study, the following criteria were established for selecting experts:

Clinicians with e-consultation experience:

Doctors who have been engaged in relevant fields for at least four years.

Doctors with intermediate or higher professional titles and experience in e-consultation are required.

Individuals who possess a certain level of understanding of the Delphi method and voluntarily express their willingness to participate in the study.

physician-patient interpersonal communication specialists:

Epecialists who have been engaged in relevant fields for at least three years.

Experts who exhibit a rigorous scientific research attitude and can provide comprehensive and professional insights.

Individuals who possess a certain level of understanding of the Delphi method and voluntarily express their willingness to participate in the study.

Patients with e-consultation experience:

Having had at least three online full electronic consultationsspecialists who have been engaged in relevant fields for at least three years.

Individuals who possess a certain level of understanding of the Delphi method and voluntarily express their willingness to participate in the study.

Keeping these criteria in mind, we have selected a diverse and knowledgeable group of experts, including doctors with experience in online doctor-patient communication, experts in doctor-patient interpersonal communication, and patients who have undergone online medical service, to ensure a multidimensional and comprehensive analysis of the quality indicators for electronic consultations.

Expert Consultation Questionnaire

After conducting a thorough search and review of relevant literature, combined with the existing scale, we summarized the dimensions for evaluating the quality of online doctor-patient communication. Through extensive discussions within the research group, a preliminary indicator system was established consisting of three first-level indicators, ten second-level indicators, and twenty-nine third-level indicators. To gather expert opinions and feedback, a questionnaire was developed for the first round of expert consultation. The questionnaire consists of three parts:

Explanation of the questionnaire. This section provides the research background, purpose, and significance of the study, ensuring that experts have a clear understanding of the survey’s objectives.

Evaluation of each indicator by experts. Using a Likert 5 scale, experts rate the importance and feasibility of each indicator. The scale ranges from “not important” to “very important” for importance, and from “not feasible” to “very feasible” for feasibility. Each indicator is assigned a corresponding rating from 1 to 5. Experts are encouraged to provide suggestions during this part, such as proposing additional indicators or reasons for removing certain indicators.

Survey of experts’ basic information. In this section, experts are asked to assess their familiarity with the consultation content and the impact of different judgment bases, such as practical or scientific research experience, medical experience, theoretical knowledge, domestic and foreign references, and visual judgment.

By including these components in the questionnaire, we aim to gather comprehensive and insightful feedback from experts, ensuring that their assessments align with both their expertise and individual perspectives on the subject matter.

The Implementation of Expert Consultation

Two rounds of expert consultation were conducted from December 2022 to January 2023. For the consulting panel, we carefully selected experts in nursing, clinical medicine, and medical communication expert who possessed in-depth knowledge and authority in their respective fields. These experts were well-informed about the subject matter and displayed a willingness to collaborate as consultants. Additionally, we also invited patients with online medical experience to ensure a comprehensive perspective.

After considering the practical constraints, a total of 14 experts were ultimately consulted, reflecting a diverse range of viewpoints. The consultation process took the form of an email survey, allowing for efficient communication and data collection. After receiving the responses from the first round of questionnaires, indicators that did not meet the following basic conditions were eliminated. These criteria included an average rating for importance and feasibility below 3.5, a coefficient of variation above 0.3, and a standard deviation higher than 1. Then, according to the experts’ suggestions and the discussion of the research group, the indicator system was revised. Then the revised results were then shared with the experts for the second round of negotiations to gather their valuable insights and opinions. Finally, three first-level indicators, ten second-level indicators and thirty-two third-level indicators were determined, reflecting the consensus achieved through this comprehensive consultation process.

Ethical Aspects

The study protocol was approved by the Ethics Committee of Chongqing Medical University (record number 2018011). The participants gave their informed consent before answering the questionnaire.

Statistical Analysis

The collected data were processed using Microsoft Excel before being input into a database for further analysis. Data analysis was performed using SPSS 26.0 (IBM Corporation, Armonk, NY, US). Descriptive statistics are used to calculate the mean, standard deviation, and coefficient of variation of the importance and feasibility of each indicator.

The reliability of the expert’s correspondence is measured by means of authority, enthusiasm, and coordination. The consistency of the two rounds of consulting experts was determined by Kendall’s coefficient and P values <0.05 were considered statistically significant. The weights of indicators were determined using the Analytical Hierarchy Process (AHP). Through assigning importance values to each indicator, we established the Saaty scale, constructed the judgment matrix, and conducted a hierarchical ranking and consistency test, and conduct a hierarchical ranking and consistency test. These steps allowed us to determine the weight of each indicator at all levels of the hierarchy.

Result

In the first round of survey, the participant demographics included 7 (50%) patient experts and 7 (50%) doctor experts. Additionally, there were 7 (50%) female participants and 7 (50%) male participants. Regarding age, 5 (36%) participants fell into the 20–30 age group, while 9 (64%) fell into the 30–50 age group. The number of participants in the second round decreased by one compared to the first round (Table 1). By utilizing these various analytical techniques and considering demographic factors, we aimed to ensure a comprehensive and rigorous evaluation process for the expert consultation phase of our study.

Table 1 Demographic Characteristics of Participants

Round 1

Fourteen participants were invited to partake in the first round of consultation, and all of them agreed to participate, resulting in a response rate of 100% (Table 1), nine experts (64.29%) provided valuable suggestions during the consultation, indicating a high level of enthusiasm and engagement. It is worth noting that the self-evaluation of the experts revealed a familiarity coefficient (Cs) of 0.86 and a judgment coefficient (Ca) of 0.91. Based on these coefficients, the expert opinion authority (Cr) was calculated as 0.89 (Table 2), highlighting the credibility and expertise of the participating experts.

Table 2 Degree of Authority of Experts

To assess the degree of coordination among experts’ opinions, we examined both the coefficient of variation (CV) and Kendall’s consistency coefficient (Kendall’s W). Following the first round of expert consultation, the Kendall’s W values for indicators at all levels ranged from 0.134 to 0.307 (Table 3). These values indicate a reasonable level of agreement and consistency among the experts’ opinions.

Table 3 Concordance Coefficients of Respondent Experts

According to the experts’ suggestions and the discussion of the research group, a consensus was reached on the three Primary indicators, ten secondary indicators and twenty-nine tertiary indicators proposed in the first round. During the second round of consultation, all primary indicators, secondary indicators and twenty-eight tertiary indicators were retained. However, one tertiary indicator did not meet the basic conditions and was excluded, while four new tertiary indicators were added. In addition, based on the experts’ suggestions, appropriate modifications were made to the content of various indicators to enhance their relevance and effectiveness.

Overall, these iterative rounds of consultation and refinement help ensure that the indicator system is comprehensive, reliable, and well-aligned with the expert opinions and knowledge in the field.

For example, in consideration of the corresponding diagnosis and treatment process in online doctor-patient communication, we have made adjustments to the order of the five secondary indicators under the primary dimension. This change aligns with the sequential flow of diagnosis and treatment, where patients first report their clinical conditions, followed by further communication, and finally receiving treatment suggestions (Table 4).

Table 4 A Change in the Content Order

Additionally, we have revised the definition of the primary indicator “responsiveness” to capture a more comprehensive understanding. It now encompasses not only the timeliness and pertinence of the patient or doctor’s reply message but also the patient’s compliance with the doctor’s diagnosis and treatment suggestions. Furthermore, under the Primary indicator “responsiveness”, we revised the secondary indicator “whether the patient refers to the doctor’s advice (the patient follows the doctor’s advice)” to “whether the patient referred to the doctor’s advice regarding medication, dietary guidelines, and follow-up visits”. This modification reflects the three specific areas where patient compliance is assessed.

Finally, the modified indicator system, along with the data from all the indicators in the first round, was carried forward to the second round.

Round 2

Finally, thirteen experts participated in the second round of voting, resulting in a response rate of 92.86%, and three of them proposed amendments (21.42%). The experts demonstrated a high level of authority, with a familiarity coefficient (Cs) of 0.860 and a judgment coefficient (Ca) of 0.88, resulting in a combined coefficient (Cr) of 0.880. This indicates a strong level of expertise among the experts. After the second round of expert consultation, the Kendall W value was 0.133–0.372 (Table 3), with statistical significance (P<0.05).

In the second round of negotiation, the importance and feasibility scores of various indicators met the basic conditions (see Table 5 for specific data). The indicator system now consists of three first-level indicators, ten second-level indicators, and thirty-two third-level indicators. We also made some modifications to the content of the indicator. For example, the secondary indicator “special reminder given by the doctor” was refined to include additional specifics, such as the doctor told the patient the contraindications, side effects, and possible sequelae of drug use etc. (especially the part easily misunderstood by the general population).

Table 5 Importance and Feasibility Coefficient of Variation of Indicators

These iterative rounds of expert consultation and negotiation have ensured the comprehensiveness, relevance, and accuracy of the indicator system. The modifications made during the second round have not only enhanced the content of specific indicators but also addressed potential areas of misunderstanding, providing a more comprehensive assessment of online doctor-patient communication quality.

Generation of e-Consultation Quality Evaluation Indicator System

The Delphi process involved extensive research and evaluation of three first-level indicators, ten second-level indicators, and thirty-two third-level indicators. After constantly revising and updating the indicators, the importance and feasibility scores of the indicators satisfied the basic conditions (See Table 5 for specific data).

As a result of this rigorous process, a final set of three first-level indicators, ten second-level indicators, and thirty-two third-level indicators were determined to form the e-consultation quality evaluation indicator system (Table 6).The results of AHP showed that the weights of three first-level indicators from high to low were Joint decision-making between doctors and patients (0.6232), Patient responsiveness (0.2395), and Interpersonal relationship between doctors and patients (0.1373). The weights of each level index are listed in Table 7. It is worth noting that the consistency ratio (CR) for each judgment matrix was found to be lower than 0.10, indicating a high level of consistency among the judgments made.

Table 6 Quality Evaluation Index of Online Doctor-Patient Communication

Table 7 Weight Calculation Results of Online Doctor-Patient Communication Quality Evaluation Index System

Overall, the Delphi process, combined with the AHP analysis, has led to the development of a comprehensive and well-structured indicator system for evaluating the quality of e-consultations. The thoroughness of the process, as evidenced by the consistency of the judgment matrices, ensures the reliability and accuracy of the evaluation results.

Discussion

With the advancement of medical informatization, there is a growing expectation for doctors to enhance their communication skills during doctor-patient interactions.^24,25 The communication between doctors and patients in online medical communities is complex, diverse, and the time span is relatively large.^26–28 In order to quantify and evaluate the quality of doctor-patient communication, it is essential to employ scientifically validated evaluation methods. This not only encourages doctors to actively engage in communication but also facilitates the improvement of their communication skills. In summary, this study uses the Delphi method to construct an electronic consultation quality evaluation scale.

The key to predicting success or failure using the Delphi method is the selection of expert consultation. In this study, experts were selected who had more than 5 years of work experience and a bachelor’s degree or higher. They included clinical doctors, experts in doctor-patient communication, and patients with online consultation experience, all of whom had rich experience in electronic consultation. The reply rates of experts in the two rounds of consultations are 100% and 92.86%, and 12 experts provided opinions or suggestions, indicating a high level of enthusiasm and cooperation among the experts, and a relatively high importance attached to this research work. The authority coefficient of the two rounds of consultation experts was both >0.80, indicating a high level of expertise. The average importance rating of all indicators in the e-consultation quality evaluation scale constructed in this study was ≥3.5, and the coefficient of variation was <0.3, indicating a high level of consensus among the experts. In summary, this study is scientific and reasonable, and the e-consultation quality evaluation scale constructed based on the Delphi method is relatively scientific and reliable.

This study sought to gather insights from both doctors and patients’ experts in order to obtain a comprehensive and multi-faceted assessment of e-consultation quality. In the first and second rounds of the Delphi process, the Kendall coefficients for the feasibility of the third-level indicators were found to be relatively low, measuring 0.134 and 0.133 respectively. This indicates that the feasibility of the tertiary indicators is low in terms of the harmony between doctors and patients, considering that it may be due to different positions of both parties in the medical service process. We conducted a separate data analysis on the consultation form of doctors and patient experts, and the results showed that Kendall coefficients, which separately counted the feasibility of the second round of entries for doctors and patient specialists, were not low, at 0.274 and 0.355 (Table 8). This indicates that the consistency between doctors and the consistency between patients is relatively high. Research shows that doctors and patients have different positions in the process of medical services.^29–31 As the payers of the online consultation service, patients seek high-quality and targeted health information while also considering the efficiency of doctors’ responses and their overall satisfaction with the treatment process. On the other hand, doctors, as service providers in online consultations, doctors aim to deliver quality medical services to their patients.³² However, due to work commitments, doctors may not always be able to respond promptly, leading to longer wait times for users. Additionally, users may not be able to describe the condition completely, resulting in low efficiency of consultation. These different perspectives of doctors and patients may contribute to the variation in their evaluation of the quality of doctor-patient communication which is consistent with our research results.

Table 8 Kendal Coefficient of Doctors and Patients

The e-consultation quality evaluation index system, tailored to the characteristics of Internet medical treatment, plays a vital role in enriching the theoretical framework for assessing the quality of online doctor-patient communication. It also broadens the theoretical perspectives within the field of Internet medical treatment, thereby advancing the research progress of online medical consultation services. From a practical standpoint, the evaluation and research of e-consultation quality offer valuable guidance for both online medical community operators and users. For operators, utilizing the evaluation system to assess the quality of e-consultation services aids in identifying platform issues, promptly improving service quality, enhancing user satisfaction and loyalty, and expanding the user base. Furthermore, the system provides users with a reference standard for quality evaluation, enabling them to assess the platform usage experience from multiple perspectives. This facilitates effective selection of e-consultation services, resulting in energy and cost savings.

Conclusion

Online consultation is a relatively new and evolving medical service that has some limitations in terms of quality, especially when it comes to non-specialist doctors answering specialist questions. One common shortcoming is the lack of clear and understandable responses to user queries, often lacking in human care and focusing solely on the disease, neglecting the emotional experiences of the users. Effective communication between doctors and patients is crucial in online consultation services, yet there is a lack of objective evaluation tools to assess the quality of electronic consultations in clinical practice. Therefore, it is essential to develop an electronic consultation quality evaluation scale that can serve as a basis for assessing the quality of dialogue in medical consultation services and further enrich the theoretical framework of online medical evaluation. Additionally, this scale will help improve the efficiency of communication between doctors and patients, enhance the quality of platform consultation services, advance the technology of platform electronic consultation services, and optimize the overall platform consultation environment.

In constructing the electronic consultation quality evaluation scale, the Delphi method is employed. It is important to acknowledge that the research results may have a certain degree of subjectivity. Currently, in China’s internet medical environment, the emphasis is primarily on the consultation process, particularly the interaction with clinical doctors.It is worth noting that in China, online electronic consultation service platforms do not have dedicated pharmaceutical malls, and there is almost no communication with clinical pharmacists, which, to some extent, limits the selection of experts for this study. It is important to note that the evaluation indicators may evolve over time as new evidence emerges and our understanding of the quality of doctor-patient communication continues to advance. Therefore, continuous research and updates to the index system in the future will be necessary to ensure its relevance and effectiveness in evaluating the quality of e-consultation.