Validation of the Cipto Triage Method: A Single-Centre Study from Indonesia
Received 19 January 2020
Accepted for publication 30 April 2020
Published 18 May 2020 Volume 2020:12 Pages 137—143
Checked for plagiarism Yes
Review by Single-blind
Peer reviewer comments 3
Editor who approved publication: Professor Bo Løfgren
Hadiki Habib, Septo Sulistio, Imamul Aziz Albar, Radi Muharris Mulyana, Nova Yundiarto
Emergency Unit, Cipto Mangunkusumo Hospital, Jakarta, Indonesia
Correspondence: Hadiki Habib
Cipto Mangunkusumo Hospital, Jakarta Pusat, Indonesia
10430 Tel +62 81263488115
Fax +62 213905840
Purpose: A national referral hospital in Indonesia developed a three-category triage acuity method called the Cipto Triage Method (CTM) for emergency departments (ED) in developing countries. This was a validation study to assess the performance of the triage method.
Methods: This cohort, retrospective, single-centre study was conducted in the ED of Cipto Mangunkusumo Hospital that receives approximately 30,000 patient visits per year. The ED medical records throughout the year 2017 were randomly selected as the study sample. Completely written forms of triage and ED initial assessment were included in this study. Validation of the CTM decision was done by using expert panel opinion as reference standard, and also using surrogate conditions such as patient outcome for hospital admission and in-hospital mortality.
Results: There were 1348 samples assigned to the following three categories: resuscitation (14.9%), urgent (63.8%) and non-urgent (21.3%). Overall accuracy was more than 80%, positive predictive value and negative predictive value for resuscitation category were 99% (95% confidence interval [CI], 96.5– 99.9) and 96.9% (95% CI, 95.7− 97.8), respectively. Resuscitation category had a relative risk (RR) for admission of 1.341 (95% CI, 1.259– 1.429) and a RR for mortality of 4.294 (95% CI, 3.180– 5.799). Undertriage increases the risk of mortality compared to correct triage (RR, 3.1; 95% CI, 2.11– 4.54).
Conclusion: CTM has a good criterion and construct validity; it is also easy to understand and can accommodate a simple ED design in the majority of hospitals in Indonesia.
Keywords: triage, emergency department, validation
Triage is an important process in the emergency department (ED).1 It helps the emergency healthcare providers in prioritising various patients present in the ED with different clinical acuities.2,3 Sorting the patients based on their clinical severity or urgency will increase the safety of the patients and healthcare providers, promote effective utilisation of ED resources, and increase ED patient flow.4,5
Prominent ED triage methods have already been validated and are practised worldwide.6–9 Triage method specifically designed for developing countries is still rare, and some of them have been validated such as the South African Triage Scale (SATS) and Turkey Triage System.10,11
Validation of triage performance in various countries showed a moderate validity and reliability on the five-level triage acuity.12,13 It is also understood that different country settings (developed-, developing-, and under-developed countries), or country policies on healthcare systems contribute to the different characteristics of the patients seeking emergency care.3 Thus, in Indonesia, adjustment of triage method needs to be performed, because triage systems will work better on populations they were developed for.14
The ED of Cipto Mangunkusumo Hospital (CMH) had been practising triage since 1986.15 Initially, the triage decision-making was done using the common sense. Moreover, there were no proper education or training for medical doctors or nurses before they undertook the role of a triage officer. Since 2012, the ED practiced triage based on the ATS.16,17
In mid-2016, Cipto Triage Method (CTM) was developed, consisting the following three categories: resuscitation, urgent and non-urgent. The definitions of these three categories are derived mainly from the ATS categories. ATS categories 1 and 2 were merged into resuscitation category, while categories 4 and 5 were merged into non-urgent category. ATS category 3 remained as a single category (urgent).
To evaluate the sensitivity and specificity of the CTM after 1 year of implementation, we conducted a validation study based on the principles of diagnostic research to assess CTM’s criterion and construct validity. Our hypothesis for this study was that CTM has good sensitivity, specificity, and predictive value to predict ED patients’ clinical severity and urgency in Indonesia.
Design and Setting
This cohort retrospective, single-centre study was conducted between January 2017 and- December 2017 in the ED of CMH, an academic and referral hospital in Jakarta, Indonesia. The ED receives about 30,000 patient visits annually.16 The characteristics of the patients’ conditions range from simple tropical disease infection to chronic and terminally ill diseases. The ED team consisted of multidisciplinary specialised physicians and was managed by a specialist trained in emergency medicine. The CTM had been used for at least 6 months before the study initiation. The triage officers were nurses who have been practising triage more than 1 year. Triage training of CTM was performed continuously once a month in the ED continuing medical education forum. This study was approved by the institutional ethical board of the hospital. All patient data was treated with confidentiality, in accordance with the Declaration of Helsinki.
Every month, there were 128 ED medical record extracted randomly from the database, the sampling frame was total ED patients per month, and sample selection was done by using randomization application. Stratified random sampling was not performed in order to reflect the distribution of patients admitted to the ED of CMH according to their clinical severity level. All age and medical conditions were eligible for this research. Triage and initial assessment forms were collected from the extracted medical records. Incomplete forms-, or patients who left without being examined were excluded from this study. Formal sample size calculations based on power assumptions for diagnostic modelling cohort research did not exist; thus, we defined the sample size according to the previous diagnostic research in triage by using relative risk assumption = 2, power= 80%, confidence level 95%, and the sample size result was 960.11,18,19
The triage process of CTM was performed in a dedicated triage room at the main entry of the ED building by a nurse per shift, and one shift lasted for 8 hours. All patients who presented to the ED will undergo single triage process followed by disposition to the treatment zone. The CTM aims to determine patient priority based on their clinical severity and urgency. Assessment of the patients’ clinical severity and urgency was based on syndromic approach (chief complaint, comorbid conditions, or mechanism of injury)-, and vital signs (mental status, peripheral pulse rate and quality, respiratory rate, capillary refill time, temperature). Example of CTM decision making process is available in supplementary Figure S1 and S2. Triage process was not designed to establish a diagnosis or further assess any complications of the patient’s condition.16 No scoring system was utilised in prioritising the patient’s condition; instead, clinical judgement was made qualitatively using the mandatory information to establish triage decision.
Every triage decision has its significance. Resuscitation category means that the patient has a critically ill condition, therefore he/she needs immediate intervention from the emergency medical team. Any delay in diagnostics and treatments could decrease the patient’s survival potential. Urgent category means that the patient does not have obvious critical conditions, but has symptoms or conditions that, if not treated early, can result in morbidity or deterioration. The patient can wait until 30 minutes to get medical assessment and treatment. Non-urgent category means that the patient has signs and symptoms that are not associated with severe or urgent conditions, but he/she could still have the diagnostic study or treatment but could wait up to 1 hour to receive medical treatment.
Triage decision that was made by a triage officer is called index test. Triage decision from the selected medical record was evaluated by the expert panels, in which their decision was the reference standard of the triage decision (criterion validity). Expert panels consist of three specialists (internist, anaesthesiologist, and orthopaedic surgeon). All of them already had additional 1 year of clinical training in emergency medicine after finishing their respective specialist training. They practised daily as emergency physicians in the ED of CMH since early 2016.
The expert panels evaluated retrospectively the triage decision based on chart reviews of triage forms and initial assessment forms.20 Panel consensus was made after individual adjudication based on whether they agreed with the triage decision or not and established clinical diagnosis, and they also provided correction of the triage decision. Expert panels and triage officers were kept blinded from the patient outcome (admission and mortality). The outcome was identified based on information from electronic medical record.
Medical records were collected by trained general physicians. They extracted the following data: patient demography, the actual triage decision, the corrected triage decision by the expert panels, and patient outcome. This study calculated triage performance (sensitivity, specificity, predictive value, and likelihood ratios) with 95% confidence interval (CI) using the normal approximation. Statistical analysis was performed using chi-squared test to assess the relative risk (RR) of each triage category to the surrogate outcomes, that is, hospital admission and in-hospital mortality. Stata version 15 software was used to perform the analysis. Microsoft Excel was used to perform confidence interval for proportion of accuracy, undertriage, and overtriage. We presented this article following guideline from Standards for Reporting of Diagnostic Accuracy Studies (STARD) (supplementary Table S1)
There were 1536 ED medical records collected, and 1348 were included in this study since the triage and initial assessment forms were written completely (supplementary Figure S3). Table 1 illustrates the characteristics of the patients. Most patients who presented to the ED had non-traumatic cases, and the most common triage was urgent category. No adverse event was observed during the collection of index test data or during expert panel assessment.
Table 1 Characteristic of ED Patients
Distribution of the triage decision is shown in Table 2, the index test was actual triage done by triage officer and reference standard was panel judgement assessment.
Table 2 Distribution of Triage Decision Using Cipto Triage Method Based on Panel Judgement and Triage Officer and Its Accuracy
Expert panel review showed that the overall accuracy rate of the triage decision was more than 80% in every category. There was more overtriage compared to undertriage (Table 2). Accuracy was measured from each triage category using 2x2 table (supplementary Table S2) derived from Table 2.
Table 3 displays triage performance compared to the standard reference. Most triage category had good to excellent sensitivity and specificity except for the sensitivity of the non-urgent category.
Table 3 Diagnostic Performance of CTM (Criterion Validity)
This condition happened because there were considerable number of the non-urgent patients overtriaged to urgent category. Each triage category had excellent PPV and NPV.
Construct validity of each triage category was made by using admission and in-hospital mortality as surrogate markers (Table 4). Risk of admission was similar between resuscitation and urgent category, and risk of mortality was higher in resuscitation category compared to that of urgent and non-urgent (Table 4) categories. Chi squared analysis showed undertriage increased the risk of mortality with a RR of 3.1 (95% CI, 2.11 −4.54; p = 0.000). Overtriage had a RR to mortality of 0.247 (95% CI, 0.089 −0.848; p = 0.000).
Table 4 Construct Validity of Triage Category Compared to the Risk of Admission and in-Hospital Mortality
The triage systems should be designed based on the population characteristics, and also resource capability in the ED. Triage method created in the developed countries may need to be modified when applied to developing countries or poor resource settings. Development and application of the triage system should be followed by validation to make sure the methods are effective and efficient.21
The validation of triage should follow diagnostic research principles. Twomey suggested that triage validation is conducted by combining the criterion validity (using reference standard such as expert panel decision) and structural validity (eg admission and mortality).14
Consideration to divide CTM priority into three categories as oppose to five in the developed nations was based on its simplicity and easiness to understand for the ED staff. Most of the patients in Indonesia who presented to the ED are not via the ambulance services; hence, condition is fully undifferentiated when triage was performed.
These considerations are also applied by the World Health Organization. The organisation already developed three categories of emergency triage for children in developing country and poor resources settings.3 Research in other developing countries showed that three-triage category such as in Turkey had better performance than 5 categories such as in South Africa.10,11,22 One of the reasons why three-triage category was developed was that it is in line with the simple ED zone design due to space restrictions, conditions that can be found in majority of non-referral hospitals in Indonesia.
Diagnostic performance of CTM was excellent, and its overall accuracy was more than 80%, which is higher compared to Turkey Triage System (inter-rater agreement, 0.725; 95% CI, 0.68 −0.77) and Emergency Severity Index (overall accuracy is 59.2%).11,23
Sensitivity and specificity of the resuscitation category are comparable with the relevant three-level triage category from Turkey (red zone sensitivity, 80.9%; specificity, 92.7%).11 Overall triage accuracy of CTM was better than that of the previous five level acuity triage implemented in CMH before 2016. A study reported in 2016 regarding triage in CMH using ATS method stated that the ATS method had good performance in predicting 24 -hour mortality (area under the curve [AUC] = 0.787; 95% CI, 0.69 −0.88) and fair performance in predicting 7-day mortality (AUC = 0.66; 95% CI, 0.59 −0.72). Positive likelihood ratios of ATS method in CMH regarding mortality were 11.36, 1.11, 1.69, 0.4, and 0.23 for level 1 to 5 categories, respectively.24
Another triage category from developing country such as SATS that has five triage categories showed 91% sensitivity and 54.5% specificity for resuscitation category.22 Compared to that of MTS that showed, 0.47 sensitivity and 0.84 specificity for adult patients and 0.65 sensitivity and 0.83 specificity for children.25
Cipto triage system has more overtriage compared to undertriage. Overtriage leads to overuse of ED resource, but is better for safety. In ESI, undertriage rate was 27.6%, and overtriage rate was 13.2%.23 In Botswana that use SATS, overtriage rates were 38.4%, while undertriage rate was 16%;22 meanwhile, Turkey triage for under- and overtriage occurred at similar rates in 11% and 10.7% of cases.
We also evaluated the construct validity based on admission and mortality in one episode of ED visit. In 2017, there were a total of 69.7% admission rates to ward and intensive care unit (ICU), and 10.3% in-hospital mortality. In Botswana, there were 41.9% admission for non-ICU, and 0.19% of the patients died in the ED. Turkey had 15.5% patient admitted from the ED and 2.8% in-hospital mortality.11,22
Cipto triage showed that the higher the triage level, the higher the risk of mortality. This is relevant to other three triage category in Turkey that showed the RR for admission were 10.2 (95% CI, 4.2–24.9; p < 0.001), and 3.58 (95% CI, 1.45–8.83; p = 0.006) for resuscitation and urgent category, respectively. The resuscitation category had a significantly higher risk of in-hospital mortality with a RR of 30 (95% CI, 1.8–499.2; p = 0.018) compared to the non-urgent category.11
Triage method from developed country such as MTS showed similar pattern according to the admission odds ratio (level 1, 21; level 2, 9; level 3, 4.77) and mortality odds ratios were 75.9% for level 1, 20.7%, for level 2, 3.4% for level 3, and 0 for levels 4 and 5.12,13
Overcrowding and access block contribute to the lower risk of admission in our resuscitation category compared to the urgent category. Some of the patients in the resuscitation zone were unexpectedly boarded in the ED because the ICU is full. While waiting for admission, some of them were dead and were not able to complete the admission process.
Evaluation of triage agreement by using medical record could lead to bias, because evaluators only have information based on what is written in the forms and could not perform visual assessment. Triage assessment was done by nurse, and the expert panel only included physicians, this could contribute to limitation in judgemental process. This study was also not generalisable for all hospitals in Indonesia, since it was performed in a single hospital. Similar practice and subsequent validation studies should be conducted in the ED of various levels of hospital in Indonesia. Further prospective and reliability study should be conducted in a validation study of the CTM for specific age (neonates, paediatric, geriatric).
As a tertiary referral centre in Indonesia, the ED of CMH has different patient characteristics compared to other EDs in non-referral centres or in rural areas in Indonesia. Proportion of exacerbation is significantly higher in chronic non-communicable disease in the ED of CMH compared to communicable disease.
Cipto Triage Method showed excellent criterion and construct validity and can be used as predictive tool for patient’s clinical severity or urgency in emergency department. Larger studies should be conducted to assess its performance in other hospitals in Indonesia.
ATS, Australian Triage Scale; AUC, Area Under the Curve; CI, Confidence Interval; CMH, Cipto Mangunkusumo Hospital; CTAS, Canadian Triage and Acuity Scale; CTM, Cipto Triage Method; ED, Emergency Department; ESI, Emergency Severity Index; ICU, Intensive Care Unit; LR, Likelihood Ratio; MTS, Manchester Triage Scale; NPV, Negative Predictive Value; PPV, Positive Predictive Value; RR, Relative Risk; SATS, South African Triage Scale.
Ethics and Consent Statement
We certify the data presentation has complied with the policy of the journal of ethical consent and approved by local ethical committee from the Universitas Indonesia and Research Unit of Cipto Mangunkusumo Hospital. Consent to participate is not applicable because no human subject involved.
Hadiki Habib planned the study and submitted the study and responsible for the overall content. Septo Sulistio, Imamul Aziz Albar, and Radi Muharris Mulyana conducted a panel judgement evaluation, Nova Yundiarto conducted a sample selection and data analysis. All authors made substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; took part in drafting the article or revising it critically for important intellectual content; gave final approval of the version to be published; and agree to be accountable for all aspects of the work.
The authors report no conflicts of interest in this work.
1. Fitzgerald G, Jelinek GA, Scott D, Gerdtz MF. Emergency department triage revisited. Emerg Med J. 2010;27:85–92. doi:10.1136/emj.2009.077081
2. Pitts SR, Niska RW, Xu J, CW B. National Hospital Ambulatory Medical Care Survey: 2006 emergency department summary. Natl Health Stat Report. 2008;(7):1–38.
3. Robertson MA, Molyneux EM. Triage in the developing world-can it be done? Arch Dis Child. 2001;85:208–213. doi:10.1136/adc.85.3.208
4. Moskop JC, Sklar DP, Geiderman JM, Schears RM, Bookman KJ. Emergency department crowding, part 1- concept, causes, and moral consequences. Ann Emerg Med. 2009;53(5):605–611. doi:10.1016/j.annemergmed.2008.09.019
5. Nugus P, Holdgate A, Fry M, Forero R, McCarthy S, Braithwaite J. Work pressure and patient flow management in the emergency department: finding from an ethnographic study. Acad Emerg Med. 2011;18:1045–1052. doi:10.1111/j.1553-2712.2011.01171.x
6. Mackway-Jones K, Marsden J, Windle J. Emergency triage: manchester triage group. BMJ Books. 2006;2:1–20.
7. Australian Government Department of Health and Aging. Emergency Triage Education Kit Department of Health and Aging. 2009.
8. Farrohknia N, Castren M, Ehrenberg A, et al. Emergency department triage scales and their components: a systematic review of the scientific evidence. Scand J Trauma Resusc Emerg Med. 2011;19(1):1–13. doi:10.1186/1757-7241-19-42
9. Christ M, Grossman F, Winter D, Bingisser R, Platz E. Modern triage in emergency department. Dtsch Arztebl Int. 2010;107(50):892–898. doi:10.3238/arztebl.2010.0892
10. Gottschalk SB, Wood D, DeVries S, Wallis LA, Bruijns S. The cape triage score: a new triage system South Africa. Proposal from the cape triage group. Emerg Med J. 2006;23:149–153.
11. Erimşah ME, Yaka E, Yilmaz S, Kama A, Pekdemir M. Inter-rater reliability and validity of the Ministry of Health of Turkey’s mandatory emergency triage instrument. Emerg Med Australas. 2015;27(3):210–215. doi:10.1111/1742-6723.12385
12. Parenti N, Reggiani ML, Iannone P, Dowding. A systematic review on the validity and reliability of an emergency department triage scale, the manchester triage system. Int J Nurs Stud. 2014;51(7):1062–1069.
13. van der Wulp I, Schrijvers AJ. Predicting admission and mortality with the emergency severity index and the manchester triage system: a retrospective observational study. Emerg Med J. 2009;26(7):506–509.
14. Kuriyama A, Urushidani S, Nakayama T. Five-level emergency triage systems: variation in assessment of validity. Emerg Med J. 2017:1–8.
15. Kartowisastro H Instalasi Gawat Darurat. Sejarah dan Perjuangan RSCM-FKUI. 1989:282–284.
16. Habib H, Mulyana RM, Sulistio S, Albar IA. Triase modern Rumah Sakit dan Aplikasinya di Indonesia. Medika. 2016.
17. Habib H, Asaari A. System analysis of triage in emergency department Hospital Kuala Lumpur. Thesis of Clinical Attachment in Emergency Medicine. 2016.
18. Jenson A, Hansoti B, Rothman R, Ramirez SS, Lobner K, Wallis LA. Reliability and validity of emergency department triage tools in low- and middle-income countries: a systematic review. Eur J Emerg Med. 2017;00(00):1–10.
19. Hajian-Tilaki K. Sample size estimation in epidemiologic studies. Caspian J Intern Med. 2011;2(4):289–298.
20. Hodge A, Hugman A, Varndell W, Howes K. A review of the quality assurance processes for the Australasian Triage Scale (ATS) and implications for future practice. Australas Emerg Nurs J. 2013;16:21–29. doi:10.1016/j.aenj.2012.12.003
21. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140(3):189–202. doi:10.7326/0003-4819-140-3-200402030-00010
22. Mullan PC, Torrey SB, Chandra A, Caruso N, Kestler A. Reduced overtriage and undertriage with a new triage system in an urban accident and emergency department in Botswana: a cohort study. Emerg Med J. 2017;31(5):356–360. doi:10.1136/emermed-2012-201900
23. Mistry B, Stewart De Ramirez S, Kelen G, et al. Accuracy and reliability of emergency department triage using the Emergency Severity Index: an international multicenter assessment. Ann Emerg Med. 2018;71(5):581–587. doi:10.1016/j.annemergmed.2017.09.036
24. Tonda T, Ramlan A. R F. Precision of Triage Module in the ED of Cipto Mangunkusumo Hospital in Predicting Mortality Rate (Abstract). Indonesia: Library Universitas; 2016.
25. Zachariasse JM, Seiger N, Rood PPM, et al. Validity of the manchester triage system in emergency care. A Prospective Observational Study. PLoS ONE. 2017;12(2):1–14.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]