Accuracy of Medically Experienced Emergency Dispatchers – A Unique Nationwide Model

Introduction

Background

In the event of a pre-hospital medical emergency, a patient or bystander initiates the emergency response by contacting local or national emergency services. These calls are handled by specially trained dispatchers who determine the nature of the emergency.¹

The professional identity of dispatchers varies significantly worldwide. Typically, dispatchers are specifically trained for their position and do not possess a medical background. In Israel, Magen David Adom (MDA) functions as the national Emergency Medical Services (EMS) organization.^2,3 The MDA dispatch team consists of emergency medical technicians (EMTs) and paramedics, most of whom have field experience. Israel has nine regional dispatch centers, along with a centralized national emergency medical dispatch center (NEMDC).⁴ Calls are directed to the nearest available center or the NEMDC, where a dispatcher assesses the urgency of the call and determines the appropriate response level.⁵ MDA dispatch center staff do not follow strict protocols for dispatch triage. Instead, they classify cases based on general guidelines taught during their training, combined with their personal judgment and clinical acumen. Continuous performance feedback is a crucial component of their ongoing training, reinforcing sound decision-making skills⁴ (Figure 1).

Figure 1 Magen David Adom dispatch and ambulance crew method of operation.

First responders are typically motorbike-based MDA civilian volunteers or ad-hoc medical professionals who have subscribed to the MDA emergency phone application, “Life Guardians,” and are not part of MDA.⁵ The ambulance response is organized into a two-tier medical system: basic and advanced. The basic team consists of an EMT ambulance driver, who serves as the team leader, and an additional team member, usually another EMT. They operate an ambulance equipped to provide simple medical treatment and Basic Life Support (BLS) before transporting the patient to the hospital. EMTs in Israel are skilled in BLS, can operate Automated External Defibrillators, prepare medications for use by paramedics, and assist in endotracheal intubation.^2,3

The more advanced response unit consists of a paramedic paired with an EMT ambulance driver, operating an Advanced Life Support (ALS) ambulance. These units are reserved for more severe cases. Paramedics undergo extensive training, including courses in ALS, pre-hospital trauma life support, and pediatric advanced life support. Many paramedics elect to undertake a bachelor’s degree.⁶

Globally, dispatching systems can be broadly classified into two types: the Medical Priority Dispatch System (MPDS), predominantly used in America, and the criteria-based dispatch (CBD) system, used in Nordic and European countries.^7,8 Both systems employ a list of chief complaints and dispatchers code each case based on the most appropriate complaint. While MPDS uses codes and scripted questions for callers, the CBD system relies on the call taker’s experience to conduct the interview.^7,8 In the United States, most 911 center “public safety answering point” (PSAP) call takers are not medically trained but handle all emergency calls, including police, fire, and medical. In some high-performance EMS systems, a primary PSAP operator forwards medical calls to a medically trained “secondary PSAP”.^7,8 However, in both MPDS and CBD systems, dispatchers typically do not have field medical experience.^7–15

In Africa, there is no consistent dispatch method across countries. While in some countries, such as South Africa, the system may be homegrown and generally based on the CBD system,¹⁶ others may use methods such as the computer-aided dispatch (CAD) system, which may provide a series of prompts based on specific conditions protocols (such as FAST [Face, Arm, Speech, Time] in cases of suspected stroke) for prioritizing patients and making the assessment. At the same time, the dispatcher himself obtains minimal medical literacy training.¹⁷

Accurately diagnosing and triaging patients at the dispatcher center is of utmost importance in low-income countries, such as in Africa, where medical systems operate with limited resources. These systems should focus on recognizing time-critical conditions, and reduce over-triage, which may cause strain on hospitals and the already-constrained EMS. Therefore, in those countries, it is essential to invest productively in efficient and concise methods to maximize the dispatching capabilities.^16,17

Literature on the accuracy of medical diagnoses by emergency dispatchers generally focuses on a few critical conditions that are either life-threatening or significantly impact quality of life, such as stroke, acute coronary syndrome (ACS), cardiac arrest, and trauma.^7–15 This research is essential for EMS systems to evaluate their performance and function effectively as a liaison for secondary, and tertiary care. For instance, early diagnosis of stroke or complex trauma is crucial for directing patients to specialized centers designed to improve prognostic outcomes, such as hyperacute stroke units and specialist trauma centers.^{7,13,14,18,19} However, the sensitivity of dispatchers in making accurate diagnoses varies widely in the literature, depending on the condition being diagnosed and the protocols used in the specific area of study. For example, in the case of acute strokes, sensitivity ranged from 48.9% in the study by Mould-Millman et al to 86.4% in the study by Clawson et al^7,20,21 For cardiac arrest, sensitivity was found to be 46% by Lee et al and 93% by Fukushima et al.^7,22,23

Additionally, there is a paucity of literature addressing many other acute diagnoses, including anaphylaxis, medication or drug overdose, poisoning, hypoglycemia, and meningitis. In these cases, the level of ambulance response can significantly influence outcomes, as ALS units are equipped with the knowledge and medications such as adrenaline, naloxone, antidotes for cyanide poisoning, and IV dextrose, among others. There may also be prehospital interventions that could be performed before the ambulance crew’s arrival, potentially improving prognostic outcomes.

There is an inherent conflict between achieving high sensitivity and maintaining specificity in dispatch triage. Missed or delayed emergent events result in poorer outcomes, such as early mortality, making a highly sensitive system ideal.²⁴ However, poor specificity can lead to the misuse of precious resources. Overestimation of the significance of trauma or head injury is a common reason for over-triage, which may result in the unnecessary deployment of multiple teams, including helicopter EMS.^7,25 Studies have shown that there is a high cancellation rate for helicopter launches, with up to half of the calls being canceled, typically by ground EMS crews.^7,26 Having a clinician on dispatch has proven to increase the accuracy of detecting major trauma events, as evidenced by tests conducted in Scotland’s Scottish Ambulance Service.¹¹

Published literature in this field generally focuses on the predictive power of dispatch diagnoses, evaluating various tools and protocols designed to help EMTs make quick and accurate diagnoses.^7,8,13,27

Objectives

This study explores the unique model used by Israel’s national medical emergency dispatch system, which relies on both higher levels of medical training for dispatchers and general guidelines rather than strict protocols or scripted questions. To the best of the authors’ knowledge, this is the first research to compare a wide range of EMS preliminary diagnoses made by medically experienced emergency dispatchers with those made by ambulance crew paramedics.

Material and Methods

Study Design

In this retrospective comparative study, de-identified, pre-processed data representing all medical calls received by MDA between January 1^st 2019 and December 31^st 2019 were requested and provided from their central database. The study was approved by the MDA Research Committee, and the committee waived MDA Staff and patient approval as the data were unidentified. The study included 317,667 Medical preliminary diagnoses made by MDA personnel and led to the dispatch of an ambulance crew. Due to data sharing restrictions, data provided was limited to dispatch team preliminary diagnosis and ambulance crew paramedic working diagnosis. Calls resulting in the dispatch of a medical team were sorted for further analysis. Duplicated calls were excluded from the results. These were then sorted into 53 preliminary diagnoses that were then matched against MDA ambulance crew diagnoses. We compared the dispatcher’s preliminary diagnosis with the ambulance crew paramedic working diagnosis.

Study Outcomes

Primary analysis was conducted on raw data to determine the degree to which dispatch and ambulance crew diagnoses matched. A secondary analysis was then conducted on the dataset comparing the system affected in the dispatcher preliminary diagnosis to the system identified in the ambulance crew diagnosis. Finally, a tertiary analysis was conducted by subgrouping symptoms and syndromes by an expert panel to create a new preliminary diagnosis which was then compared with paramedic diagnosis. For example, in a case where the ambulance crew diagnosis was a myocardial infarction, complaints of shortness of breath may be considered reasonably similar and coded as a match, while complaints of a headache would be considered a misdiagnosis. By doing three separate analyses we were able to come to four potential outcomes for each preliminary diagnosis made by dispatch: there was an exact match, the preliminary diagnosis matched the same system, the preliminary diagnosis did not match but an expert panel deemed it compatible with the given descriptions, and the diagnoses were incompatible with each other. To conduct tertiary analysis on the dataset, dispatch preliminary diagnoses were modified with similar descriptors and subsequently indirectly compared. The data was reviewed by an expert panel consisting of MDA personnel and ED doctors, including the medical management of the ED department of Shamir Medical Center. After reviewing relevant literature, the panel reached an agreement on a range of descriptors for the 43 preliminary diagnoses that could reasonably reflect ambulance crew diagnosis (Table 1). Members of the panel were blinded to the diagnostic codes of each case.

Table 1 Statistical Analysis of Comparison Between Medically Experienced Dispatcher Preliminary Diagnoses and Paramedic Diagnoses by Symptoms

Analysis

Dispatch preliminary diagnoses were compared to ambulance crew diagnoses using McNemar’s test. The sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV) and accuracy were compared for statistical significance. P-values <0.05 were interpreted as statistically significant. The data was analyzed using R and the “DTComPair” Package.^25,26 To compare results, ambulance crew diagnoses were considered the reference point for a positive result. For example, in the setting of hypoglycemia: a true positive (TP) was recorded when both dispatch and the ambulance crew diagnosed hypoglycemia; a false positive (FP) was recorded when dispatch recorded hypoglycemia but the ambulance crew recorded any other diagnosis; a false negative (FN) was recorded when dispatch did not record hypoglycemia where the ambulance crew did; and true negative (TN) was recorded when neither dispatch nor ambulance crews recorded a diagnosis of hypoglycemia.

Results

During 2019, MDA made 317,667 Medical preliminary diagnoses from all their cases that led to the dispatch of an ambulance crew. Each of these diagnoses were one of 53 different preliminary diagnoses and were identified and compared to ambulance crew diagnoses. No other patient defining characteristics such as age or sex were included in the data provided by MDA.

Primary analysis was conducted on all of the 155,124 preliminary diagnoses made by medically experienced dispatchers. Those were comprised of 53 unique preliminary diagnoses that were directly matched to ambulance crew paramedic diagnoses. (Table 2). Overall sensitivity across all diagnoses was 48.8% (0.9%–84.2%), specificity was 97.3% (93.9%–100%), PPV was 53.0% (1.9%–90.9%) and NPV was 98.0% (96.4%–100%).

Table 2 Statistical Analysis of the Direct Comparison Between Medically Experienced Dispatch Preliminary Diagnoses and Paramedic Diagnoses

Secondary analysis was conducted for all the 155,124 medically experienced dispatcher preliminary diagnoses. Those were comprised of 53 unique preliminary diagnoses that were matched to the system found in the ambulance crew paramedic diagnoses. (Table 3).

Table 3 Statistical Analysis of Comparison Between Medically Experienced Dispatch Preliminary Diagnoses and Paramedic Diagnoses by Organ System

Dispatcher preliminary diagnosis matched the system identified by the ambulance crew in 138,235 cases. Sensitivity was 78.6% (18.6%–97.6%), specificity 88.36% (78.4%–100%), PPV 26.5% (0.1%–88%) and NPV 99.2% (96.3%–100%).

Tertiary analysis was then conducted on all the 312,122 diagnoses made by the panel of MDA experts and ED physicians. Those were comprised of 53 unique preliminary diagnoses that were matched to the ambulance crew diagnoses. (Table 1).

Panel preliminary diagnoses matched ambulance crew diagnosis in 155,433 cases. Sensitivity was 49.8% (range 0.9%–86.4%), specificity 97.3% (range 93.5%–100%), PPV 54.4% (range 2.1%–88.0%) and NPV 95.6% (96.1%–100%) (Figure 2).

Figure 2 Weighted Totals for the Three types of analyses conducted: Primary analysis which compared diagnoses directly, secondary analysis which compared diagnoses by systems and tertiary analysis which compared dispatcher diagnoses by expert panel match for syndromes.

Discussion

Through analysis of the results of directly comparable dispatches, specificity and NPV were both encouragingly high with weighted averages of 97.3% and 98.0%, respectively. This implies that medically experienced dispatchers were ruling out incorrect preliminary diagnoses most of the time. Sensitivity and PPV, however, were comparatively lower for all dispatch outcomes with a weighted average 48.8% and 53.0%, respectively, implying that medically experienced dispatchers were not correct as often when determining the preliminary diagnosis.

When looking at the secondary analysis (by systems) interestingly the sensitivity goes up from 48.8% to 78.6% while the PPV drops from 53.0% to 26.5%. This implies that when a system was affected by a preliminary diagnosis the dispatchers were more likely to correctly identify the system, but for any given preliminary diagnosis made by dispatchers, they were less likely to be correct in what system was affected. Specificity slightly decreased from 97.3% to 88.4 and NPV slightly increased from 98.0% to 99.2%, but relatively they stayed much steadier indicating that the incorrect systems were still ruled out most of the time.

The indirect analysis surprisingly showed very similar results to the direct comparison of medically experienced dispatchers’ preliminary diagnosis to paramedic working diagnosis. Sensitivity and PPV increased very slightly to 48.8% and 54.4% where specificity and NPV decreased very slightly to 97.3% and 97.9%, respectively. This shows that when the expert panel was given the same set of descriptors that the medically experienced dispatchers had, they were frequently choosing the same preliminary diagnosis.

When comparing the accuracy of medically experienced dispatcher’s preliminary diagnosis of strokes to other published literature, sensitivity falls in the middle of the range at 60.6%. However, in the studies with sensitivity on the high end of the range such as the one done by Clawson et al, specificity was allowed to drop significantly, where the specificity for the medically experienced dispatchers’ diagnosis of stroke remained very high at 98.4%.²¹ Other studies where specificity was maintained often showed a lower sensitivity.^7,28 Medically experienced dispatchers’ preliminary diagnosis of cardiopulmonary arrest tells a very similar story with a sensitivity of 54.3% lying in the lower end of the range, but a maintained sensitivity of 99.4%. Fukushima et al found a sensitivity of 93%, but their specificity did drop down to 50%.²³ Very few papers were found discussing dispatcher accuracy for acute coronary syndrome, but among those found it was seen that a sensitivity over 80% was achieved, while specificity remained below 40%.²⁹ On the other hand, medically experienced dispatchers were able to reach a sensitivity of 76.1% with a specificity kept at 93.9%.

Taken together, these indicate that compared to the published literature, the MDA medically experienced dispatchers tend to be more accurate than their less-medically trained international colleagues who set a similarly high threshold for suggesting a significant diagnosis.

Interestingly, we found that the sensitivity of a few medically trained dispatcher preliminary diagnosis of a few life-threatening diagnoses was very low. Among these were drug overdose at 28.2%, myocardial infarction at 9.8%, and apnea at 2.7%. Myocardial infarction and apnea are diagnoses only made when physician or someone on behalf of a physician calls emergency dispatch from a tertiary facility stating that diagnosis specifically. As a result, these are not true representations of the dispatches accuracy when making a preliminary diagnosis. These misdiagnoses are likely done either due to miscommunication between the physician and the person calling dispatch, or cases of acute coronary syndrome prematurely being declared a myocardial infarction. Acute coronary syndrome however had a significantly higher sensitivity of 76.1%. For drug overdoses, it was likely assumed the person reporting might have been withholding information and an assessment would be difficult to do without a person at the scene. When considering these diagnoses, it should also be noted that the sensitivity in the secondary analysis was higher indicating that the dispatchers were making preliminary diagnoses close to the paramedic working diagnosis. Unfortunately, we were unable to find studies published discussing the accuracy of dispatcher preliminary diagnosis of myocardial infarction, apnea, of drug overdose from other dispatch systems.

Additionally, we found a very large variation in the sensitivity seen for different trauma cases depending on location. During primary analysis, this ranged from head trauma having a sensitivity of 44.3% to neck trauma sensitivity dropping to 2.8%. Secondary analysis sensitivity for trauma to different locations ranged from 88.5% to 96% showing that information given during the emergent call was highly consistent with trauma, however the specific location was frequently misdiagnosed. This is particularly problematic in the case of neck trauma when dispatchers would otherwise be able to give instructions to help those calling better immobilize the cervical spine and prevent additional injury and provide the proper ambulance response type. This too was unfortunately not found in our review of the literature, so we were unable to compare our data to other national health care systems.

The study done here was compared to available published data, which to date has focus primarily on a few life-threatening diagnoses. The findings are consistent with the results of our system, which showed that dispatcher and paramedics generally arrive at similar diagnoses; however, the degree of agreement varies by diagnosis type, as some diagnoses are easier to identify whereas others are more difficult to determine. Tables 2 and 3 show that sensitivity improves after review by the expert panel. This indicates that even when the dispatcher did not assign the correct preliminary diagnosis, the affected organ system was correctly identified, and an appropriate urgency and level of response (ALS vs BLS) were dispatched.

However, we were unable to compare out results for most diagnoses done against any other study due to the paucity of literature. We suggest that future studies looking at the accuracy of emergency medical dispatch include and compare data from all medical calls to try and improve diagnoses for all cases.

As there is an inherent difference in the way that dispatchers and field crews record diagnoses, comparison was made somewhat challenging. This was addressed by performing both direct and indirect comparisons; however, this does introduce a level of subjectivity into the analysis. As this is not an issue encountered in international EMS systems, it is hard to evaluate the true impact it had on our results.

Final medical diagnosis was not available for review on request of data, meaning field team diagnosis was taken as the ground truth for comparison. Further research is needed to assess the accuracies of these ground truths for comparison. However, to assess the appropriateness of an EMS response, paramedic diagnosis is a valuable tool as any suspected case of; for example, ACS or stroke should be treated with the same amount of urgency regardless of final diagnosis.

Conclusion

In the results of the comparative study we conducted, medically experienced call takers established preliminary diagnoses with a good level of resemblance to ambulance crews in the field. Further research is required in this area to determine whether MDA’s modus operandi, which requires that medical staff in the field work shifts both as dispatchers and ambulance crew, results in more accurate and specific service when compared to their international colleagues. However, we believe that increasing the experience and field training of EMS dispatchers could benefit the international community.