National Early Warning Score in Predicting Severe Adverse Outcomes of Emergency Medicine Patients: A Retrospective Cohort Study

Introduction

The emergency department (ED) of a general hospital has to deal with a large number of patients with varying illness severities every day. Therefore, triage is important in identifying high-risk patients amongst many less urgent patients as ED overcrowding has recently turned into a national crisis.^1,2 However, the situation of emergency patients is complex, and information is limited. Overreliance on subjective judgment based on triage nurse experience may lead to results with low reliability and delayed treatment.^3,4 Over-triage, where the actual critical level of the patient’s condition is lower than the nurse’s triage, could lead to a waste of medical resources and patient retention; under-triage, where the critical level of the patient’s condition is higher than the nurse’s triage, leads to delayed care for more severely ill patients because they were inappropriately assigned lower triage categories.⁵ Therefore, a simple tool based on readily available parameters is needed to assist nurses in rapid and accurate triage.

The Modified Early Warning Score (MEWS), based on systolic blood pressure, pulse rate, temperature, respiration rate, and level of consciousness, is a simple bedside tool that nurses use in a busy clinical area. The MEWS calculation for emergency admissions may be useful in triage to identify some of those patients at risk of deterioration and need for more active intervention.⁶ The quick Sepsis Related Organ Failure Assessment (qSOFA) was recommended by the Third International Consensus Definitions Task Force.⁷ Its predictive validity for in-hospital mortality was found to be statistically greater than the SOFA and systemic inflammatory response syndrome (SIRS) among encounters with suspected infection outside of the ICU.⁸ The National Early Warning Score (NEWS), which was developed by the National Early Warning Score Development and Implementation Group (NEWSDIG) in the United Kingdom (UK) in 2012, is used by the National Health Service (NHS) to assess the deterioration of in-patient conditions and to predict in-patient mortality or ICU admissions.⁹ NEWS measures physiological parameters (systolic blood pressure, pulse rate, respiratory rate, temperature, oxygen saturation), level of consciousness (Alert, Voice, Pain, Unresponsive), and supplement of oxygen—which are all simple and easily accessible.^9,10 NEWS is now popular in many countries since it shows a greater capacity than other early warning scores to discriminate patients at risk of the combined outcome of cardiac arrest, unanticipated ICU admission, or death within 24 hours.^9,11 In recent years, NEWS use in specific populations of EDs (geriatric patients or patients with suspected sepsis or dyspnea) has also attracted attention.^11–16 Some studies focused on the performance of NEWS in severity assessment and mortality prediction for category 2 (Emergency) and 3 (Urgent) patients.^17,18 These studies show that NEWS is a beneficial ED tool. However, since the patient cohorts of some of these studies were rather small, the predictive effect of NEWS may not have been accurate. In addition, other studies focused on pre-selected sub-cohorts of patients, which limited the applicability of NEWS scores to other patients.

The aim of this study was to investigate the predictive effect of NEWS on outcomes, such as emergency mortality, ICU admission, and in-hospital mortality in unselected emergency patients (without any selected sub-cohorts). We also compared the prognostic value of NEWS alongside the commonly used MEWS and qSOFA. NEWS performance on sub-cohorts was also evaluated. We hypothesized that the NEWS obtained from triage can reliably predict ED patient outcomes.

Methods

Design and Setting

A retrospective cohort study was performed in the ED of a tertiary care center in Jinhua, China. This hospital has more than 2600 in-patient beds, 26 emergency beds, and 32 observation beds with more than 120,000 ED visits per year. The Ethics Committee of the hospital approved the study; informed consent was waived as the data were anonymous and routine care was not influenced. This study was conducted in accordance with the Declaration of Helsinki.

Study Population

Patients presented with the ED during the year-long study period (January 1, 2018 to December 31, 2018) were eligible to participate in this study if they met the following inclusion criteria: (1) Aged 18 years or older and (2) Emergency medicine patients. The exclusion criteria were as follows: (1) Patients attending the chest pain center, (2) Patients attending the stroke center, and (3) If the data manually inputted into the triage system by nurses (ie, temperature, respiratory rate, and pain score) was significantly abnormal but could not be verified. If the patients were lost to follow-up after ED discharge, they were removed from the study analysis. A total of 62,403 patients were included in this study.

Data Collection

In the sample hospital, the ED includes internal medicine, surgery, obstetrics and gynecology, and pediatrics. The internal medicine emergency department is divided into general emergency medicine, the chest pain center, and the stroke center. The patients come from the following three sources: (1) Delivery by ambulance (about 14 patients per day); (2) Transferred after outpatient treatment (2 to 4 patients per day); and (3) Self-admitted ED patients. All patients must undergo triage before they are admitted to a specific area. After the patient arrives at the ED, a triage nurse with more than three years of ED experience, measures the patient’s pulse rate, blood pressure, and oxygen saturation with a mobile monitor (Mindray V900), which automatically imports the data into the triage information system. The temperature, respiratory rate, consciousness (AVPU), pain score, and reason for visit are recorded manually by nursing staff. After all data is entered, the system automatically calculates the total score according to the calculation rules. The nursing staff determine the triage classification based on the score combined with medical history, symptoms, and vital signs. Only when all of the above items are completed can the patient’s information be entered into the doctor information system. After triage, patients are then classified into four levels, Class I (critical), Class II (emergency), Class III (urgent), and Class IV (sub-urgent), before being admitted to the resuscitation room or the emergency clinic. For example, if a patient’s NEWS≥5 points, combined with medical history, symptoms, and signs, the triage grade of the patient is most likely to be Class I. NEWS 3–4 points, or single score greater than 3 points for Class II; NEWS 2 points for Class III; NEWS 0–1 points for Class IV. The outcomes of patients, including discharge, hospitalization (general ward, ICU) or death, are uniformly summarized in the emergency patient information system and can be directly exported. The above procedures ensure the integrity of patient sources and the accuracy of data. The NEWS, MEWS, and qSOFA values were calculated for each vital signs obtained during the triage (Supplementary Table 1).

However, there was no record of supplemental oxygen in the triage system. In addition, the proportion of patients admitted by pre-hospital emergency care was not large, and the oxygen intake of these patients was mainly based on the patient’s chief complaint, rather than the strict principle of oxygen use. Due to this, the NEWS used in this study did not include the component of supplementary oxygen used. We categorized V, P, or U (Voice, Pain, or Unresponsive levels of consciousness) as having “altered mental status” when calculating qSOFA scores.¹⁹

Outcomes

The primary outcome of the study was ED mortality, and the secondary outcomes were observation in the ED; composite of ED mortality and ICU admission; and in-hospital mortality. ED mortality is defined as death in the ED, including death within 24 hours of being discharged from the ED. Emergency observation refers to a patient requiring being observed and treated in the emergency room, resuscitation room, or emergency observation room. The end point of emergency observation is when a patient is discharged or hospitalized from ED or dies in ED. Admission to the ICU includes patients who were admitted directly from the ED to the ICU or who were admitted to the ICU after the emergency intervention. Last, in-hospital mortality refers to death during hospitalization in this visit or death within 24 hours of hospital discharge.

Statistical Analysis

Data were summarized using descriptive statistics (ie, mean, standard deviation, median, and interquartile) when appropriate. To assess group differences, we used the Kruskal–Wallis test for continuous and skewed variables, and the Pearson’s χ2 test for categorical and binary variables.

Multivariate logistic regression analysis was performed to evaluate the associations between the NEWS, MEWS, and qSOFA, as well as those between other parameters with ED mortality. We developed different models with stepwise adjustments for potential confounders. The qSOFA and the main reason leading to ED admission were used as categorical variables. Results of the regression analyses were presented as odds ratios (ORs) with 95% confidence intervals (CIs). The predictive performances of NEWS, MEWS and qSOFA were compared by the area under the receiver operating characteristic curve (AUROC). Sensitivity, specificity, positive and negative predictive values, and Youden’s index were calculated for the different cut-off points. The optimal cut-off point for the different scores was determined by Youden’s index and this was used to assess the data. Calibration was measured by the Hosmer–Lemeshow goodness-of-fit test and expressed as a χ2-value and accessory p-value, which represent how outcome predictions resemble the observed outcomes. A two-tailed p-value of <0.05 was considered statistically significant. Further, we also used the AUROC to evaluate the ability of the NEWS to predict outcomes in different subgroups. Analyses were conducted using Statistical Package for the Social Science (SPSS) version 21.0, R version 3.4.3 (2017-11-30), and EmpowerStats (www.empowerstats.com) and MedCalc version 15 for Windows (MedCalc Software bvba).

Results

Basic Characteristics

There were 70,638 patients who visited the emergency medicine department during the study period (Figure 1). After exclusions, 62,403 patients remained eligible to participate in the study. Only 315 (0.5%) were excluded owing to incomplete data (Figure 1). Out of all those who were admitted, 6502 (10.4%) were under emergency observation, 106 (0.2%) experienced ED death, 638 (1.0%) were for ICU admission, and 324 (0.5%) suffered an in-hospital death (Table 1). Respiratory and digestive system, respectively, were the primary complaints. Since patients visiting the chest pain centers and stroke centers were excluded, the number of patients with cardiovascular and neurological disorders was decreased. Generally, the risk of all outcomes increased with patient age; men were more likely to suffer adverse consequences than women; and diseases of different systems had varying degrees of risk. Among the patients who died in the ED, there were 67 males (63.2%), 71.4 ± 14.3 years old, and 79 (74.52%) with NWS > 3 points. The time from visiting ED to death was 15.0 (6.0–27.0) hours. As for the main reason for referral, 35 (33.0%) was respiratory system, 23 (21.7%) digestive system, and 23 (21.7%) nervous system.

Table 1 Baseline Characteristics of the Study Population

Figure 1 Emergency department patient flow and the final cohort.

In the multivariate logistic regression analysis, we found a significant association between ED mortality and the NEWS (OR 1.5, 95% CI: 1.5 to 1.6, p < 0.001), MEWS (OR 1.8, 95% CI: 1.6 to 1.9, p < 0.001), and qSOFA (OR 126.4, 95% CI: 51.3 to 311.1, p < 0.001, qSOFA=3 compared to qSOFA=1). Additionally, age (OR 1.014, 95% CI: 1.029 to 1.053, p < 0.001) and gender (OR 0.7, 95% CI: 0.4 to 1.0, p < 0.001) were also associated with ED mortality (Table 2).

Table 2 Multivariate Logistic Regression Analysis of Factors Independently Associated with ED Mortality

Prognostic Value of NEWS, MEWS and qSOFA

ED Mortality

Tables 3 and 4 and Figure 2 show the prognostic values and calibration of the NEWS, MEWS, and qSOFA for the prediction of ED mortality. The area under the curve (AUC) of the NEWS, MEWS, and qSOFA to predict ED mortality was 0.862 (95% CI: 0.859–0.865), 0.806 (95% CI: 0.803–0.809), and 0.792 (95% CI: 0.789–0.795), respectively. Using Youden’s J statistic, the optimal cut-off points for ED mortality were found to be: NEWS >3, MEWS >2, and qSOFA >0. The pairwise comparison of receiver operating characteristic (ROC) curves showed that the NEWS was better at predicting ED mortality than the MEWS and qSOFA. Calibration for the NEWS showed a χ2 = 0.00 and p = 1.0. The non-significant p-value indicates that the mortality rates between the observed and the predicted values were statistically equivalent.

Table 3 Optimal Cut-Off Values, Sensitivity, Specificity, PPV, NPV, Positive Likelihood Ratio, Negative Likelihood Ratio and Corresponding 95% Confidence Intervals of Each Scale for Outcomes

Table 4 AUC and Hosmer–Lemeshow of Fit Test for the Prediction of Outcomes

Figure 2 ROC curves of NEWS, MEWS and qSOFA for predicting outcomes. (A) ROC curves for ED mortality; (B) ROC curves for predicting emergency observation; (C) ROC curves for predicting composite of ED mortality and ICU admission; (D) ROC curves for predicting in-hospital mortality. Abbreviations: NEWS, National Early Warning Score; MEWS, Modified Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment.

Emergency Observation

The prognostic values of the NEWS, MEWS, and qSOFA for predicting emergency observation were also analyzed as shown in Tables 3 and 4 and Figure 2. The prognostic values for the prediction of emergency observation with the AUCs for NEWS, MEWS, and qSOFA were 0.691 (95% CI: 0.687–0.695), 0.638 (95% CI: 0.634–0.642), and 0.629 (95% CI: 0.626–0.633), respectively. The optimal cut-off points of the NEWS, MEWS and qSOFA were >2, >2, and >0, respectively. In addition, when the ROC curves were compared, the AUC of the NEWS was better than MEWS and qSOFA for predicting emergency observation in ED patients. The Hosmer–Lemeshow C statistic also showed that the NEWS has good calibration for predicting emergency observation.

Composite of ED Mortality and ICU Admission

The AUC of NEWS, MEWS, and qSOFA to predict the composite of ED mortality and ICU admission was 0.859 (95% CI: 0.856–0.861), 0.781 (95% CI: 0.778–0.785), and 0.760 (95% CI: 0.757–0.764), respectively (Tables 3 and 4; Figure 2). The optimal cut-off points were as follows: NEWS >3, MEWS >2, and qSOFA >0. NEWS also showed the best performance out of the three scores to predict the composite of ED mortality and ICU admission in ED patients, further proving NEWS’ good calibration.

In-Hospital Mortality

As for in-hospital mortality, the AUC for the NEWS, MEWS, and qSOFA was 0.805 (95% CI: 0.802–0.808), 0.739 (95% CI: 0.735–0.742), and 0.730 (95% CI: 0.727–0.734), respectively (Tables 3 and 4; Figure 2). The optimal cut-off points for in-hospital mortality of the NEWS, MEWS, and qSOFA were >2, >2, and >0, respectively. Compared to MEWS and qSOFA, NEWS had the best predictive effect.

Sensitivity Analyses

The prognostic values of the NEWS for the prediction of outcomes in subgroups are shown in Supplementary Table 2. For the primary outcome (ED mortality), the AUC of the NEWS was 0.854 (95% CI: 0.850–0.858) in male patients, 0.874 (95% CI: 0.871–0.878) in female patients, 0.864 (95% CI: 0.858–0.870) when patient age was ≥65 years, 0.800 (95% CI: 0.796–0.804) when patient age was <65 years, and 0.899 (95% CI: 0.896–0.903) in patients with respiratory system disorders. Additionally, the NEWS predictions of all outcomes appeared to be better for patients with respiratory diseases than for the total number of patients.

Discussion

To the best of our knowledge, this is the first retrospective study conducted to evaluate the performance of NEWS for unselected emergency medicine patients in a predominantly Chinese population. We found that NEWS correlates significantly with patient outcomes, including ED mortality, ICU admission, and in-hospital mortality. Irrespective of different outcomes, NEWS was the most accurate tool for prediction of outcomes when compared with the MEWS and qSOFA. The results showed that NEWS was the model with the highest agreement between the expected and observed outcomes. NEWS also performed well in pre-selected sub-cohorts of patients, especially those with respiratory system conditions.

There are several scoring systems that can be used to identify patients at risk of deterioration and predict clinical outcomes. The AIMS65 score, and Glasgow Blatchford score for example, can be used to assess severity in patients with acute gastrointestinal bleeding.^20–22 Systemic inflammatory response syndrome (SIRS) is often used to predict infection or septic shock.^23–27 However, many of these scores include laboratory results and are used primarily for specific patients, limiting their value in emergency triage. However, general early warning scores, such as the MEWS and NEWS, are based only on readily available parameters, and thus have a good range of applicability.

An obvious advantage of NEWS, MEWS, and qSOFA is that they do not require any biological tests, which together with their good performance make their ED use possible. In recent years, several studies have focused on their value in predicting the severity and prognosis of emergency patients. MEWS is used as one of the standard bases for triage classification judgment.²⁸ NEWS on ED arrival performs well in recognizing severity and prognosis of patients with severe sepsis and septic shock, patients with respiratory distress, elderly patients, and unselected emergency patients.^10,12,17,29 NEWS can serve as a useful tool for managing patients in a complex ED environment, enhancing patient care quality.¹⁸ Compared with MEWS, qSOFA, and other warning scores, NEWS is generally more accurate in predicting admission and mortality for all patient types, showing that NEWS may have greater ED application value.^{10,12,15,17,18,30}

However, these studies contain several limitations. The patient cohorts of these studies were rather small, from several hundreds to thousands;^14,17,30,31 thus, the prediction effect for NEWS may not have been stable. Furthermore, most focused on a pre-selected sub-cohort of patients, such as patients suspected of sepsis,^14,15,30 elderly patients,¹² or category 2 (Emergency) and 3 (Urgent) patients,¹³ which limits the applicability of NEWS scores to other patients. However, it is impossible to divide patients in sub-cohorts ahead of emergency triage. With the exception of one study conducted in Hong Kong,¹⁷ none of the aforementioned studies focused on a predominantly Chinese population.

The participants for the present study were continuously recruited during the entire year and were from both urban and rural areas to minimize selection bias.³² In order to guarantee the representativeness of the sample and improve the stability of results, more than 60,000 participants were included. At the same time, all emergency patients needed to go through triage before they could see a doctor, which ensured the integrity of the patient queue. In the present study, only a small number of patients (315) were excluded owing to abnormal data or were lost to follow-up after discharge. Moreover, this study was aimed at unselected emergency department patients, which is in line with the actual situation of emergency triage. We chose the particular outcomes over longer term mortality (90 days) because, for emergency care, it is essential to predict both the current level of severity and the immediate risk of death to provide more active and effective interventions. Moreover, the longer the follow-up time, the higher the incidence of other influential factors affecting the prognosis of patients.

Our results showed that NEWS had a high predictive effect for emergency death. The predictive effect of ICU admission and in-hospital mortality was also good. Whether patients need to be observed in the ED depends not only on the disease severity but also on many other factors, such as the need for further evaluation or diagnosis, the need to wait for crowded hospital beds, etc., so the prediction effect of NEWS on emergency observation was only near moderate. The optimal cut-off points of NEWS for ED mortality and in-hospital mortality were >3, which differ from Brinks et al’s research findings.¹⁵ This may be because the subjects were different, and the NEWS of our research did not include oxygen supplementation. In general, all acute deteriorating patients would not be checked and screened using the oxygen supplement category. If patients have oxygen supplementation, they were assigned a score of 2 on the NEWS score. Our results showed that NEWS can serve as a useful tool to help triage nurse’s grade patients more quickly, objectively, and accurately. Based on the triage level, then medical staff can determine the patient’s zoning, emergency response time, and implementation of first aid measures. For example, for patients with Class I (such as cardiac, respiratory arrest, heart failure, respiratory failure, etc.), emergency staff should immediately take emergency measures such as cardiopulmonary resuscitation, respiratory support, circulatory support, various tests, etc., to prevent acute deterioration.

The Hosmer–Lemeshow test showed that the results between observed and predicted values were statistically equivalent, which was consistent with Brinks et al’s results.¹⁵ The results of multiple regression analysis showed that in addition to warning scores, age and gender were independent risk factors for emergency death. The effect of gender on mortality has been a concern.³³ Studies have shown that males are significantly associated with a higher risk of death in critically ill or elderly patients.^34,35 In our cohort, the male patients were older than female patients (49.2 ± 20.0 VS 45.8 ± 19.3). However, the male patients who died in the emergency department were younger than female patients (69.6 ± 14.0 VS 74.6 ± 14.5). Males also had higher BP (132.9 ± 22.7 VS 126.9 ± 23.6). In addition, the incidence of circulatory disease was higher among males (1146, 3.9% VS 866, 2.6%). These factors may lead to gender being an independent risk factor for emergency death. The predictive effect of NEWS on some specific groups was also studied, and the results were stable. Therefore, we conducted studies on sub-cohorts of different ages and genders, and the results showed that NEWS still had a good predictive effect on outcomes. At the same time, it was found that NEWS had a better predictive effect for patients with respiratory diseases, possibly because the score increased the respiratory-related parameter, that is, oxygen saturation. In the research cohort, NEWS predictive performance on all outcomes was superior compared to MEWS and qSOFA, which is consistent with other studies.^{18,24,27,29,31}

Limitations

Our study had several limitations. First, this was a single-center investigation in a tertiary care center hospital in China; therefore, the results may not be generalizable to other settings and populations. Second, for ED mortality, we only followed critically ill patients who were discharged from the ED, owing to the large volume of patients. Therefore, a small percentage of “mild” patients who died within 24 hours of discharge may be missing. Third, because there are many influencing factors, there may be some selection bias in the definition of emergency observation. In addition, the NEWS we studied did not include oxygen supplementation, so its results may differ from the standard NEWS. It should be noted that excluding the oxygen parameters only weakens the predictive effect. Therefore, the value of the standard NEWS should be better than the NEWS in this study. Moreover, because of differing intervention processes, the patients admitted to the chest pain center and stroke center were excluded. Hence, the predictive value of the NEWS for these patients needs to be examined further. Finally, we did not stratify NEWS in more detail and compare it with the triage categories. Thus, further studies are needed to accurately determine the role of the NEWS in reducing the under-triage or over-triage rates of patients.

Conclusion

The study found that NEWS calculated according to the data obtained during the triage was significantly correlated with the prognosis of emergency medicine patients, including ED mortality, emergency observation, ICU admission, and in-hospital mortality. The study revealed that NEWS was the most accurate tool for predicting adverse outcomes when compared with MEWS and qSOFA. These findings have important implications for optimizing the triage process in EDs of general hospitals. Through the use of NEWS, the ED staff have a better indication of which patients are at greater risk. Moreover, NEWS can serve as a useful tool to help triage nurse’s grade patients more quickly, objectively, and accurately, and thus manage patients more effectively in a complex ED environment, enhancing patient care quality.

Future studies should focus on the value of NEWS in multi-center EDs. At the same time, the predictive value of NEWS for emergency patients admitted to chest pain centers or stroke centers, as well as its effect on improving triage accuracy, should be studied.