Back to Journals » Risk Management and Healthcare Policy » Volume 15

Developing and Validating an Emergency Triage Model Using Machine Learning Algorithms with Medical Big Data

Authors Gao Z, Qi X, Zhang X, Gao X, He X, Guo S, Li P

Received 21 December 2021

Accepted for publication 8 July 2022

Published 19 August 2022 Volume 2022:15 Pages 1545—1551

DOI https://doi.org/10.2147/RMHP.S355176

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 4

Editor who approved publication: Dr Kent Rondeau



ZhenZhen Gao,1,* Xuan Qi,1,* XingTing Zhang,2 XinZhen Gao,2 XinHua He,1 ShuBin Guo,1 Peng Li1

1Department of Emergency, Beijing Chao-yang Hospital, Capital Medical University, Beijing, 100008, People’s Republic of China; 2LIANREN Digital Health Co., Ltd, Beijing, 102208, People’s Republic of China

*These authors contributed equally to this work

Correspondence: Peng Li, Department of Emergency, Beijing Chao-yang Hospital, Capital Medical University, Beijing, 100008, People’s Republic of China, Tel +86 010 85231280, Email [email protected]

Objective: To establish an emergency triage model through the statistical analysis of big data during a particular time period from a hospital information system to improve the accuracy of triage in emergency department (ED).
Methods: A total of 276,164 patients who visited the Emergency Medicine Department of Beijing Chao-Yang Hospital from 2017 to 2020 were included in this study, including 123,392 men and 152,772 women aged from 14 to 112 years. The baseline characteristics (age and gender) and medical records (patient’s condition, body temperature, heart rate, breathing, blood pressure, consciousness, and oxygen saturation) of the patients was collected. The data samples were randomly allocated, with 80% as the training set and 20% as the testing set. The patients were divided into levels I, II, III, and IV in accordance with a four-level triage standard. We selected the effective Extreme Gradient Boosting (XGBoost) algorithm as our emergency classification prediction model. The XGBoost model was applied to simulate the thinking process of triage nurses, and the De Long’s test was used to compare the receiver operating characteristic (ROC) curve of different models. The P value was obtained by calculating the variance and covariance of area under the curve (AUC) values of different ROC curves.
Results: Level I had 4960 (1.8%) patients, level II had 25,646 (9.29%), level III had 130,664 (47.31%), and level IV had 114,894 (41.6%). The XGBoost model was built following a logic exercise based on the traditional manual pre-inspection and triage results. After verification, the prediction accuracy was 82.57%. The AUC of each disease severity level (levels I, II, III, and IV) was 0.9629, 0.9554, 0.9120, and 0.9296, respectively.
Conclusion: The emergency triage prediction model, which achieved a relatively strong accuracy rate, can reduce the work intensity of medical workers and improve their working efficiency.

Keywords: emergency, triage, XGBoost model, triage model

Introduction

As the health-care system improves, emergency medicine has become a core and valuable specialized area. The development of emergency medicine has changed rapidly and no longer consists of a simple triage desk and transfer station. Instead, it has become a diagnosis and treatment center for emergency medicine and severe diseases, integrating multidisciplinary resources to complement medical, teaching, scientific research, social service, and other functions.1 Since the 1990s, different emergency department (ED) triage systems have been used, such as the Australian Triage Scale,2 Canadian Emergency Department Triage and Acuity Scale,3 Manchester Triage Scale,4 and the Emergency Severity Index.5 Although these triage systems have been widely adopted by many EDs globally, limitations still exist, such as inaccurate triage.6 Failure to distinguish the risk level of critically ill patients7–9 can easily lead to undertriage or overtriage, both of which have a negative impact on the accuracy and efficiency of emergency resource allocation. The undertriage of patients with severe diseases can lead to an insufficient allocation of medical resources, resulting in increased clinical deterioration, morbidity, and mortality.10,11 The overtriage of patients with non-urgent presentations can lead to the excessive occupation of medical resources.12

The importance of the ED in providing platform support and technology leadership for the health-care system in China is becoming increasingly prominent.13 However, with the increase in patients attending EDs annually, triage has become a key link in emergency treatment; but it is a time-consuming and labor-intensive task. In addition, there is no standard emergency triage system in China.14–17 Implementing safe and effective triage is of great importance to accurately identifying emergency patients and those with severe diseases, guaranteeing their safety and improving emergency diagnoses and treatment efficiency.18–20

At present, most hospitals in China implement triage based on their own criteria within the framework of “hierarchical diagnosis and treatment” and without a uniform standard.21 To reduce time-consuming medical consultations, a large number of non-emergency patients attend the ED, putting enormous pressure on limited emergency resources. EDs not being able to provide emergency diagnoses and treatment has become a common problem.22 To address these problems, Beijing Chao-Yang Hospital started to construct an emergency information system in early 2017 and launched an intelligent triage system based on modified FRENCH guidelines,23 a reference for the second French triage guide. Currently, the emergency triage scale/standard, which is also called the four-tier triage system and was proposed by the Chinese National Ministry of Health in 2011,24 is used in the People’s Republic of China. Information-based triage and a hierarchical diagnosis and treatment system have been implemented. Thus, a diagnosis is made and treatment provided to ED patients based on the severity of disease rather than the patients’ order of arrival, greatly improving the order of diagnosis and treatment and ensuring that emergency patients and those with severe diseases can be assessed and treated without delay.

However, at present, triage in hospitals mainly relies on medical workers’ personal experience, without a scientific triage model, and the level of knowledge and skill is insufficient. In hospitals with emergency triage, there is always a senior nurse at the emergency triage station. Patients attending the ED are often in a critical condition, and therefore, a patient’s triage must generally be completed within one minute. As a result, misjudgments may occur. This study aims to train and develop an emergency triage prediction model based on historical data that reduces the number of triage nurses at emergency triage stations and, to some extent, improves their working efficiency.

Materials and Methods

General Materials

This is a retrospective study using data from a hospital data registry. A single physician independent of the study collected the data from the Computer Center of Beijing Chao-Yang Hospital. Another physician independent of the study, who was not involved in the collection of the data, validated the data gathering. The triage information of all patients who came to the ED of the Beijing Chao-Yang Hospital, Capital Medical University, between March 1, 2017, and June 30, 2020, was collected, including their sex, age, body temperature, heart rate, respiration rate, systolic and diastolic blood pressure, consciousness, and blood oxygen saturation level. Information on a total of 289,243 cases was collected. From this data, 13,079 cases were removed because of information loss, and 276,164 cases were used as the samples in the study. Inclusion criteria: ① patients with complete medical records in the ED from between March 1, 2017, and June 30, 2020; ② age ≥ 14 years. Exclusion criteria: ① age < 14 years; ② key data missing (patient visit identification number, differentiation of patients with illness). This study was approved by the Ethics Committee of Beijing Chao-Yang Hospital with a waiver of informed consent from the patient.

With the second edition of triage guidelines in France referenced and the practical clinical work considered, modified FRENCH guidelines were developed after two rounds of expert consultation by letter. A four-level triage standard was created. Level I refers to critical patients, who should be immediately assessed and treated in the emergency room; level II refers to patients with a severe condition, who should be prioritized for diagnosis and treatment in the outpatient area, with a waiting time of less than 15 min; level III refers to emergency patients, with a waiting time of less than 30 min; and level IV refers to general patients, who should wait to see a doctor in the outpatient area in the order of arrival.25

Research Method

An unbiased random sample allocation method was adopted, and a new “partition variable” was generated based on a calculation using SPSS Statistics software. The data samples were randomly allocated, with 80% as the training set and 20% as the testing set, to facilitate the subsequent construction of models, model screening, and model validation and evaluation. The training and testing samples randomly obtained were used for the training and testing of Extreme Gradient Boosting (XGBoost) software, respectively.

To select the most effective model, we applied a 10-fold cross validation method to train the model, adjust the model parameters, and then evaluate the model on the development set.

For the hyperparametric optimization of each model, we chose the HyperOpt optimization algorithm, which is faster and more effective than a grid search, so that we could quickly build the optimal state of each model.

The XGBoost algorithm and HyperOpt technology were used to build relevant models. The HyperOpt hyperparameter optimization technology was applied to optimize all hyperparameters in the XGBoost model. Local interpretable model-agnostic explanations (LIMEs) were used for the model explanation of the established XGBoost prediction model and to explain the prediction results based on feature importance and other related information. The constructed XGBoost model was not only able to predict the classification probability of a single patient but also provide an evaluation of the importance of each feature, determining the contribution of each feature to a classification in relation to data.

For the trained emergency classification prediction model, we used LIME to build the interpretation model based on the development set to enable it to provide an output analysis for the predicted patient, allowing the machine learning model, XGBoost, to become an explainable black box.

For the model evaluation, the prediction performance of each machine learning model was calculated based on the test set (the remaining 20% of data). As an expression of predictive performance, the area under the working characteristic curve (AUC), net reclassification improvement (NRI), and confusion matrix results (ie, sensitivity, specificity, positive predictive value, and negative predictive value) were used. To compare the receiver operating characteristic (ROC) curves of different models, the De Long’s test is used. The P value was obtained by calculating the variance and covariance of the AUC values of different ROC curves. The NRI was the difference between the sum of sensitivity specificity of one diagnostic test and the sum of sensitivity specificity of another diagnostic test.

Statistical Method

The model was established and analyzed using the SPSS Statistics v23.0 software package, and the enumeration data were expressed as n (%). After operating the XGBoost, the quasi-probability of triage preparation prediction was obtained. With 0.50 as the critical value of the prediction probability, the sensitivity, specificity, and overall prediction accuracy were calculated. The ROC curve and AUC were drawn to evaluate the prediction accuracy.

Results

General Information

A total of 276,164 patients were included in this study, including 123,392 men (44.68%) and 152,772 women (55.32%). The ages ranged from 14 to 112 years, with a mean of 54.4 ± 20.5 years and a median of 56 (35, 79) years. There were 4960 (1.8%) patients at level I, 25,646 (9.29%) at level II, 130,664 (47.31%) at level III, and 114,894 (41.6%) at level IV. In total, 246,062 (89.1%) patients had a normal body temperature, 216,512 (78.4%) had a normal heart rate, and 180,335 (65.3%) had normal blood pressure. In addition, 272,574 (98.7%) patients had normal respiration, 272,021 (98.5%) had a conscious mind, and 262,080 (94.9%) had normal oxygen saturation (Table 1).

Table 1 Patients’ General Conditions

Information-Based Modeling Results

The triage data of 276,164 patients were randomly allocated, with 220,931 (80%) patients as the training set and 55,233 (20%) as the testing set. Among them, the coincidence rate between the triage results in the testing set predicted by the model and the actual triage results was 0.8257. To evaluate the prediction efficiency of the model, the ROC curve was drawn and the AUC calculated. The AUC of each disease severity level (levels I, II, III, and IV) was 0.9629, 0.9554, 0.9120, and 0.9296, respectively (Figure 1).

Figure 1 XGBoost multi-class receiver operating characteristic curve.

Discussion

At present, emergency triage relies primarily on the triage nurse’s experience and triage rules. The existing emergency triage process of our hospital is as follows: (1) the nurse measures the patient’s body temperature, respiration rate, blood pressure, and blood oxygen saturation and then evaluates their consciousness; (2) the triage system obtains data for these vital signs through hardware interfaces; (3) the computer performs a preliminary triage; and (4) the nurse slightly adjusts the triage result based on the patients’ actual condition.25 The triage result is provided by the senior nurse (with more than 3 years’ experience) after evaluating the patient’s vital signs and relevant personal information. Because of the large amount of information and limited time, there is some deviation in the accuracy of triage results, resulting in anxiety and complaints from doctors and patients and increasing the probability of patient–doctor conflict.

Improving the triage nurse’s ability to evaluate patients’ conditions and the accuracy of the risk stratification in the ED is not only the main objective for triage nurses in the ED but is also a key topic explored by emergency and critical care medical workers.26,27 If emergency patients do not receive timely and effective intervention and treatment, their condition may change rapidly and become aggravated within minutes or hours, even becoming life threatening. Effective triage is of great importance to the diagnosis and treatment of emergency patients. It not only helps to ensure the timely assessment and treatment of emergency patients and those with a severe condition but also greatly reduces their waiting time. Therefore, the work of the ED should be busy but well managed, and the medical resources and medical space should be appropriately and scientifically allocated.

Currently, the triage standards in domestic hospitals lack quantitative objective indicators and operability. Therefore, triage indicators should be further detailed and the methods and tools described to form the basis of a uniform standard in accordance with the national basic standards for triage and the actual conditions in China’s hospitals. Triage standards should be science-based and operable, allowing the effective identification of potential critical patients without excessive triage. In addition, triage standards should be continuously improved during their implementation. Therefore, based on big data, and with a machine learning method adopted, a set of emergency triage standards that are feasible and conform to China’s national conditions have been summarized, further optimizing emergency triage and achieving intelligent triage.

The study results, with the XGBoost model applied, reveal that the prediction accuracy rate of this model was 0.8257, and the AUC of each disease severity level (levels I, II, III, and IV) was 0.9629, 0.9554, 0.9120, and 0.9296, respectively, which is better than the threshold value set for this model representing random chance. This indicates that the model has predictive value and the established emergency triage prediction model is relatively accurate. This triage model, based on machine learning and established by training the XGBoost model, can automatically learn the complicated relationships between the target patients’ characteristics and the triage results; thus, a well-trained triage prediction model can be used to improve the accuracy of an emergency triage evaluation. A patent has been obtained for this model.28

Based on medical big data, this study established an emergency classification prediction model to reduce the allocation of nursing triage to some extent and improve work efficiency. This model will quickly evaluate the emergency patients’ conditions and then determine their priority level based on their criticality. The XGBoost model is an innovative tool for the intelligent transfer of triage. It can provide overall information management, use uniform emergency triage software to make an analysis, and then quickly provide an accurate triage result, reducing the time emergency nurses spend on triage and ensuring patients can be quickly and effectively treated. Furthermore, this tool can improve the human resource allocation in the ED and optimize the diagnosis and treatment process. This would not only address the overtriage and undertriage problem but also achieve the effective utilization of emergency space and rational allocation of medical resources.

This model has some limitations. First, the medical data of patients in this study was obtained from a single institution, and we did not evaluate whether differences existed in China between urban and rural, public and private, or university and community hospitals. More medical data from a larger cohort should be accumulated to improve the model. Second, the model has high subjectivity. Triage standards provided at different times, in diverse scenarios, by various triage staff, and by the same triage staff with different physical and mental states may not be the same, resulting in poor stability in the triage results. Therefore, the emergency triage model constructed using XGBoost should be further optimized. In the future, as the machine algorithm is optimized with the development of mathematical models and probability theory studies, the emergency triage model will be improved.

Conclusion

In this study, we established and verified an artificial intelligence triage model using a machine learning algorithm to reduce the work intensity of medical workers and improve work efficiency. The accurate emergency triage system could allocate the limited space and medical resources appropriately according to a patient’s condition and alleviate emergency congestion.

Ethics Approval and Consent to Participate

This study was conducted with approval from the Ethics Committee of Beijing Chao-yang hospital with a waiver of informed consent from the patient. This study was conducted in accordance with the declaration of Helsinki.

Acknowledgments

We would like to acknowledge the hard and dedicated work of all the staff that implemented the intervention and evaluation components of the study.

Funding

There is no funding to report.

Disclosure

The authors declare that they have no competing interests.

References

1. Dugas AF, Kirsch TD, Toerper M, et al. An electronic emergency triage system to improve patient distribution by critical outcomes. J Emerg Med. 2016;50(6):910–918. doi:10.1016/j.jemermed.2016.02.026

2. Australasian College for Emergency Medicine. Guidelines on the Implementation of the Australasian Triage Scale in Emergency Department. Doc No G24. Australasian College for Emergency Medicine; 2013:1–8.

3. Beveridge R, Clarke B, Janes L, et al. Canadian emergency department triage and acuity scale: implementation guidelines. Can J Emerg Med. 1999;1(3 suppl):S2–28.

4. Mackway-Jones K. Emergency Triage. 1st ed. Manchester: Manchester Triage Group; 1996:168.

5. Gilboy N, Tanabe P, Travers D, Rosenau AM. Emergency Severity Index (ESI): a triage tool for emergency department care. 4th ed. Implement Handbook. 2012;2012:12–14.

6. Mazer M, Deroos F, Hollander JE, McCusker C, Peacock N, Perrone J. Medication history taking in emergency department triage is inaccurate and incomplete. Acad Emerg Med. 2011;18(1):102–104. doi:10.1111/j.1553-2712.2010.00959.x

7. Seiger N, van Veen M, Steyerberg EW, Ruige M, van Meurs AHJ, Moll HA. Undertriage in the Manchester triage system: an assessment of severity and options for improvement. Arch Dis Child. 2011;96(7):653–657. doi:10.1136/adc.2010.206797

8. Hitchcock M, Gillespie B, Crilly J, Chaboyer W. Triage: an investigation of the process and potential vulnerabilities. J Adv Nurs. 2014;70(7):1532–1541. doi:10.1111/jan.12304

9. Oredsson S, Jonsson H, Rognes J, et al. A systematic review of triage-related interventions to improve patient flow in emergency departments. Scand J Trauma Resusc Emerg Med. 2011;19:43. doi:10.1186/1757-7241-19-43

10. Haas B, Gomez D, Zagorski B, Stukel TA, Rubenfeld GD, Nathens AB. Survival of the fittest: the hidden cost of undertriage of major trauma. J Am Coll Surg. 2010;211(6):804–811. doi:10.1016/j.jamcollsurg.2010.08.014

11. Schull MJ, Vermeulen M, Slaughter G, Morrison L, Daly P. Emergency department crowding and thrombolysis delays in acute myocardial infarction. Ann Emerg Med. 2004;44(6):577–585. doi:10.1016/j.annemergmed.2004.05.004

12. Calder LA, Forster AJ, Stiell IG, et al. Mapping out the emergency department disposition decision for high-acuity patients. Ann Emerg Med. 2012;60(5):567–576.e4. doi:10.1016/j.annemergmed.2012.04.013

13. Li L, Wei G, Yue H, et al. Boosting health-the development history and prospects of emergency medicine in China. J Med Res. 2020;49:1–4.

14. Expert Consensus Group on Emergency Pre-Examination and Triage. Expert consensus on emergency pre-examination and triage. Chin J Emerg Med. 2018;6:599–604.

15. Xu TD, Wang Z, Yu XZ. Topic: interpretation of 4 medical service standards-clear work process, scientific diagnosis-interpretation of “standardized process of hospital emergency department”. China Sanitation Standard Manag. 2012;2012(7):32–36.

16. Hu YL, Huang JY, Lu YL, Hua XX, Shi DL. Investigation on status of triage standards in emergency departments among two-level and above hospitals in China. J Nurs Admin. 2014;14(5):334–336.

17. Jin JF, Guo ZT. Study on the status quo and countermeasures of emergency pre-examination and triage in tertiary hospitals in China. Chin J Emerg Med. 2015;24(4):458–461.

18. Saleh A. Cross-sectional study of emergency department presentation triage categories at Goondiwindi Hospital and their effect on the treatment of acute emergencies. Aust J Rural Health. 2017;25(4):235–240. doi:10.1111/ajr.12337

19. Mirhaghi A, Ebrahimi M. The Australasian triage scale level 5 criteria may need to be revised. Emerg. 2017;5(1):e50.

20. Buschhorn HM, Strout TD, Sholl JM, et al. Emergency medical services triage using the emergency severity index: is it reliable and valid? J Emerg Nurs. 2013;39(5):e55–63. doi:10.1016/j.jen.2011.11.003

21. Ministry of Health, PRC. Guiding principles for grading the condition of emergency patients (Exposure Draft). Chin J Crit Care Med. 2011;04(4):241–243.

22. Deng Y, Geng N, Jiang Z, et al. Brief introduction on pre-examination and triage standard and process in emergency care in China and foreign countries. Chin Hosp. 2017;021(008):63–65.

23. Yancey CC, O’Rourke MC. Emergency department triage. In: StatPearls. Treasure Island (FL): StatPearls Publishing; 2022.

24. Chinese National Ministry of Health. Guiding principles for emergency patient triage (draf for comments). Chin J Crit Care Med. 2011;04:241–243.

25. Wang HQ, You ZY, Ding S, et al. Design and application of the intelligent emergency preview and triage system. China Digit Med. 2019;2019(7):64–66.

26. Fan H, Zhang H, Yao L, et al. Application of intelligent clinical decision support system in emergency triage: a review. J Nurs Train. 2021;36(2):147–149.

27. Guo MY, Shen ZY, Xiao T, et al. Application of intelligent clinical decision support system in emergency triage: a review. J Nurs Sci. 2021;36(1):109–112.

28. Zhang XT, Gao XZ, Wang L, et al. Hierarchical processing methods, devices, electronic equipment and storage media: China. 2021. in Chinese.

Creative Commons License © 2022 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.