Validation of Dutch Obstetric Telephone Triage System: A Prospective Validation Study

Introduction

The increased volume of obstetric emergency care and the pursuit of high-quality interpretation and documentation of unplanned obstetric care consultations, require improvement of current care processes.^1–3 A triage system by telephone that prioritizes care according to urgency is known to have a favorable effect on safety and efficiency of emergency care.^4–8

The Manchester Triage system (MTS), Canadian Triage and Acuity Scale (CTAS) and Emergency Severity Index (ESI) are commonly used in emergency departments worldwide.^5,9 However, triage systems for emergency departments are not specific enough for unplanned obstetric care. Therefore, in several countries obstetric physical triage systems have been developed, eg, in Canada the Obstetric Triage Acuity Scale (OTAS),^10–12 in the United States the Maternal Fetal Triage Index (MFTI) and in the United Kingdom, the Birmingham Symptoms specific Obstetric Triage System (BSOTS).^2,13,14 In Switzerland, an obstetric section has been added to the general Swiss Emergency Triage Scale (SETS).^3,15 Recently, the Iranian Obstetric Triage Index (IOTI) has been developed in Iran (2020).¹⁶

All these triage systems are based on consensus of opinion by experts. Evaluation of triage systems involves assessments of validity and reliability. Triage can be viewed as a diagnostic assessment; therefore, the methodology of diagnostic studies is applicable. However, no diagnostic validation studies have been performed for obstetric physical triage systems.^{2,3,10,11,13–16} Diagnostic validation studies of triage should ideally evaluate whether the triage process accurately predicts the correct level of urgency. However, there is no single outcome measure that captures the concept of urgency.¹⁷ Validation of (non-obstetric) triage systems in different studies is commonly based on some of the following outcomes; the agreement of the system with expert opinions, assessment of vital signs, admission to intensive care unit (ICU), death, need for operations and/or follow-up.⁵

In addition to assessment of internal validity, where studies are performed in a single setting, external validity can also be studied. External validity is based on studies in different settings.¹⁷ Diagnostic validity of triage systems used in different emergency departments show a wide variation of existing scales and also a wide variation of results.^5,9 Diagnostic external validity of MTS shows an agreement of triage of 49.7–61.6% with a 6.2 −14.1% range of undertriage and a 26.9–44.0% range of overtriage; sensitivity is reported to be between 0.47 and 0.87 and specificity between 0.83 and 0.89.⁹ Analyses of the diagnostic validity of MTS, ESI and CTAS show sensitivity values of 0.58–0.88 and specificity values of 0.59–0.84 for ICU admission. And hospitalization or discharge after the emergency visit had a sensitivity of 0.08–0.65 and a specificity of 0.64–0.98 for low urgency patients.⁵ The obstetric triage systems SETS and MTFI show higher agreement. Agreement was 78.4% with SETS and 72.9% with MFTI, both had no purpose to search for diagnostic validity. No diagnostic validation studies have been performed for OTAS, BSOTS, IOTI.^10,11,14,16

It is important to note that all these triage systems^{2,3,5,9–11,13–16} have been developed to determine the urgency of care requests by means of physical (face-to-face) triage. In practice, most women call first to ask whether it is necessary to have a consultation.^18,19 To provide a uniform and practical basis for estimating the severity of symptoms of unplanned obstetric or other emergency obstetric care requests by telephone, the Dutch obstetric telephone triage system (DOTTS) was developed. DOTTS is an evidence-based guideline for obstetric telephone triage and is developed through a multi-phase multi-center study with relevant stakeholders. DOTTS was introduced in 2015 and is currently used in 25% of all Dutch hospitals (n=20/78).^20–22 The purpose of the present study was to determine the diagnostic and external validity of DOTTS in obstetric emergency care.

Patients and Methods

Design

The diagnostic and external validity of DOTTS were studied in a prospective observational study. Diagnostic validity was determined by comparing the assigned urgency level of DOTTS with a reference standard. This reference standard was the urgency level of DOTTS determined by a medical doctor (obstetrician in training) or hospital midwife, after clinical assessment during follow-up in the hospital. Also, patients’ follow-up after assessment was studied. Patients’ follow-up was recorded in two classifications: 1) urgent care (hospitalization – life threatening situation or hospitalization with treatment or preterm labor) and 2) non-urgent care (hospitalization without treatment or in labor after 37 weeks or home after consultation). The external validity was determined by comparing the results of four hospitals.

DOTTS consists of five presenting symptoms: fluid loss, vaginal bleeding, abdominal pain, concerned pregnant/non-somatic symptoms and other physical symptoms. DOTTS is comparable to other triage system, in that it consists of five urgency levels: resuscitation and life threatening, emergency, urgent, non-urgent and self-care advice. Staff is given the opportunity to overrule the DOTTS urgency category.²²

Participants, Organizational Context and Study Period

To test external validity, four different hospitals were included in this study. Hospitals A, B and C are teaching hospitals, hospitals A and C each with approximately 3100 deliveries per year. Hospital B is one of the largest in the Netherlands, with over 6100 deliveries per year. Hospital D is a smaller, non-teaching hospital, with about 1100 deliveries a year. Three hospitals have a stand-alone triage department next to the delivery ward. In one hospital, the triage consultations are performed in the delivery ward. The four hospitals are geographically spread out throughout the Netherlands.

In all hospitals DOTTS was implemented into usual care by way of protocol change. The implementation process of DOTTS was individually guided: a digital application was built in the main hospital’s information system and was accessible in the patient’s record. In addition, specific training was given to the staff responsible of triage (obstetrical nurses or doctor’s assistants). Furthermore, all professionals from the reference standard followed an information session and received written information about this research. A special application was added to the patient’s digital record in which entering of the reference standard items was obligatory. In each hospital, a medical professional was available for any questions to the research.

The study was conducted in Hospital A between April 2018 and September 2019. In Hospital B between March and December 2018. In Hospital C between July 2017 and December 2018, and Hospital D between June 2018 and November 2019. This study period depended on the date of implementation of DOTTS.

Data Collection and Statistical Analysis

During every telephone call triage staff used DOTTS to record patients’ characteristics, categories of presenting symptoms and urgency levels in a digital application in patient’s record. Patient characteristics gathered were age, gestational age, gravity/parity, singleton/multiple pregnancy, presenting symptoms and urgency levels. Triage staff were able to overrule DOTTS, if they did not agree with the classification. Urgency levels assigned by triage staff, even overruled, was used during analysis. Only comparison of urgency categories U1-U4 (resuscitation and life threatening, emergency, urgent, non-urgent) could be studied. As category U5 (self-care advice) resulted in the patient not being referred to the hospital.

Following face-to-face clinical assessment, the urgency of referral (reference standard) based on presenting symptoms, urgency levels (U1-U5) and follow-up after assessment were also recorded in the patient’s record. In the application, the reference standard was guided to prevent them from personal interpretations of the complaints by using three question with fixed answers. In this way, urgency levels and follow-up were structured into fixed categories.

The validity of DOTTS was determined by comparing the outcome of the urgency level assigned by DOTTS with the independent reference standard. Agreed, over- and undertriage using DOTTS were calculated. Agreed triage was defined as the proportion of patients who had exactly the same urgency level as a result of assessment via DOTTS, compared to the assessed urgency level from the reference standard. Over- and undertriage were defined as the proportion of patients who respectively had a higher or lower urgency level as a result of assessment via DOTTS, compared to the assessed urgency level from the reference standard. In addition, sensitivity, specificity, positive predictive value, negative predictive value and likelihood ratios were determined for cases classified as high urgency. Patients’ urgency levels were dichotomized into high urgency (U1, U2) and intermediate urgency (U3, U4). Also, likelihood ratios were calculated to assess the goodness of fit of DOTTS versus the reference standard.

The results from all hospitals were analyzed collectively to determine external validity, and per hospital to determine internal validity. Weighted analyses were performed to correct for over or under representation of characteristics caused by different numbers of cases per hospital.

To test the clinical relevance of DOTTS, follow-up data were also analyzed. Patients’ follow-up was recorded in two classifications: 1) urgent care (hospitalization – life threatening situation or hospitalization with treatment or preterm labor) and 2) non-urgent care (hospitalization without treatment or in labor after 37 weeks or home after consultation). Analysis was by, which care was provided after the consultation (urgent or non-urgent), in comparison with the classification according to DOTTS (high or intermediate urgency). During the follow-up analysis, we expected cases categorized in DOTTS as intermediate urgency, would result in little-to-no (clinical) classification of patients as requiring urgent care (ie, hospitalization).

All analyses were performed using SPSS, version 26.

Ethical Approval

The study was submitted to and approved by the daily Boards of the Medical Research Ethics Committees United (MEC-U) and the Medical Ethics Committee of Leiden University Medical Center (LUMC) Act (W.16.053 and P17.075/PG/pg). As a result of these reviews, the boards declared that the rules laid down in the Medical Research Involving Human Subjects Act (also known by its Dutch abbreviation WMO) do not apply to the study and according to their judgement there was no ethical objection to perform the study.

In accordance with Dutch legal agreements, an application statement has been requested and obtained within each hospital. In this way, permission was obtained at the local level ethical committees for conducting the research. The study started when DOTTS was offered regularly.

In the Netherlands, all pregnant women are informed during the first antenatal checkup about the use of their perinatal data for scientific research and about the opting out procedure. Withdrawal can be asked at any moment and without explanation or reason and without consequences for delivered care. Withdrawal was recorded in the Electronic Patient File. Privacy is guaranteed in accordance with Dutch legislation. Clients’ anonymity was maintained by using anonymous patient identifiers.

Results

The outcomes of DOTTS were compared with the reference standard in 983 triage consultations (hospital A: 624 cases, hospital B: 193 cases, hospital C: 116 and hospital D: 50). Mean age of patients was 31 years (SD 5) and mean gestational age 32+4 weeks. A total of 382 nulliparas (38.9%) and 597 multiparous (60.7%) women were included. In four cases, parity was unknown (0.4%). Most pregnancies were singleton: 878 (89.3%); 91 (9.3%) were multiple (14 missing/unknown (1.4%)) (Table 1). In 13 (1.3%) cases, triage staff overruled the urgency level of DOTTS.

Table 1 Characteristics of Study Population and Presenting Symptoms

The urgency level U2 (emergency) was the level most often registered (n=622, 63%). In total 625 (64%) cases were categorized as high urgency and 358 (36%) as intermediate urgency. Abdominal pain (n=263, 27%) was the most common presenting symptom followed by anxious pregnant woman/non-somatic symptoms (n=231, 23%) (Table 1).

DOTTS’ urgency level fully agreed with the reference standard in 53% (n=525; 95% CI 50–57) of cases. In total, undertriage by DOTTS was seen in 16% of cases (n=160) and overtriage in 30% (n=298) according to the reference standard. In 85% (n=135) of cases the amount of undertriage in DOTTS compared to the reference standard was one category and in 15% (n=25) more than one category. In 74% (n=220) of the cases the amount of overtriage by DOTTS was one category, and in 26% (n=78) more than one category (Figure 1).

Figure 1 DOTTS compared with reference standard.

Overall sensitivity of DOTTS was 76% (95% CI 72–80), and specificity 49% (95% CI 44–53). The overall positive predictive value (PPV) was 60% (95% CI 56–63) and the overall negative predictive value (NPV) 67% (95% CI 62–72) (Table 2). Weighted analysis did not reveal significant differences (sensitivity) 75% (95% CI 69–80), specificity, 50% (95% CI 46–53), PPV 59% (95% CI 56–62) and NPV 67% (95% CI 63–71) (Table 2). Likelihood ratios were LR+ 1.49 LR- 0.49 which indicates that DOTTS is a well-fitting triage system.

Table 2 Diagnostic Validity of the Dutch Obstetric Telephone Triage System for the Category “High Urgency” per Hospital

There were no life-threatening cases classified with DOTTS as intermediate urgency. However, 31 (9%) patients who were classified by DOTTS as intermediate urgency received hospitalization with treatment. Overall, when high and intermediate urgency were combined, in most cases (n=753, 77%) only a consultation was needed (Table 3).

Table 3 Urgency of Care After Clinical Assessment

Discussion

To the best of our knowledge, the present study is the first diagnostic validity study of an obstetric triage by telephone. DOTTS compared to a reference standard has an agreement of 53% (95% CI 50–57), and there was overtriage in 30% and undertriage in 16% of the cases. The overall sensitivity and specificity were 76% and 49%. After clinical assessment urgent care was needed in 8.7% (n=31) of the intermediate-urgency cases, none of these cases were life threatening situations. Due to absence of diagnostic validation studies for obstetric triage systems, we compared the results of our study with previous studies of physical triage systems, such as MTS, CTAS and ESI, used in general emergency. The validity of DOTTS is comparable to, or slightly better than the results of these systems. Diagnostic external validity of MTS showed an agreement of 50–62% with a range of 6–14% undertriage and a range of 27–44% overtriage; sensitivity was found to be 0.47–0.87 and specificity 0.83–0.89.⁹ Analyses of the diagnostic validity of MTS, ESI and CTAS showed a sensitivity of 0.58–0.88 and specificity 0.59–0.84 for ICU admission.⁵ However, due to the level of heterogeneity demonstrated in the literature, these figures are difficult to compare.

Validity studies of MFTI and SETS, both physical triage systems at obstetric departments, with agreement percentages of 72.9% and 78.4%, respectively, show better agreement compared to DOTTS. Sensitivity and specificity have not been studied for these systems.^13,15 These studies are about a different type of validity and due to heterogenicity of methods and quality of studies, it is difficult to compare these studies.³

In this study, we compared triage by telephone with a reference standard after clinical (physical) assessment. Telephone triage has more challenges compared to physical triage; such as the lack of clinical assessment and initial diagnostic examinations.^6,18,19 In our study clinical assessment occurred later, after some waiting time (for example, at home and/or during transport to the hospital) and after being able to do diagnostic tests, such as measuring blood pressure and monitoring fetal condition. It is unclear what effect waiting time and the results of availability of these tests have on the reference standard. However, we followed the method of Moll (2009), which advises to stay close to the reality of clinical practice. Moll argues that, as compared to diagnostic tests, there is no single outcome measure that captures the concept and therefore, researchers have to select the best proxy as a reference standard. This proxy, in our study and comparable studies, is the ultimate clinical decision that has been made, based upon the doctor’s opinion, including physical examination and (laboratory) tests.¹⁷

The sensitivity in our study is higher compared to specificity, indicating that DOTTS is able to classify highly urgent cases better than intermediate urgent cases. The likelihood ratio supported this outcome. This is of high clinical importance, as a triage system is meant to classify the need for highly urgent care. In studies on physical triage in general emergency care, the opposite was observed. Further research is essential to confirm a hypothetical explanation regarding factors such as the difference in number of cases examined in this study and the difference in the amount of presenting symptoms. To increase agreement of triage sensitivity, specificity, PPV and NPV follow-up research, with special attention for sub-analysis, is needed. Undertriage should be avoided in a triage system, as it can be assumed that this could cause irreversible health damage as a result of waiting time. Every single case involving undertriage can indicate factors, which can be used to consider improvement of quality in future.^23,24

In addition to a good assessment of medical urgency, a telephone triage system is also intended to provide a good distribution of resources and to avoid unnecessary consultations. With a 49% level of specificity (95% CI 44–53) it can be said that DOTTS does not provide sufficient differentiation in the “intermediate urgency” category. These findings are consistent with the literature.^4,25 While some overtriage can be explained by the lack of diagnostic examinations; we should review this aspect critically. Clearly, further research is needed to explore specificity, especially what is required to reduce overtriage.

Various studies have been performed to test the validity of triage systems. However, the lack of homogeneity, especially the variety in outcome measures, makes this challenging.^16,17 Current evidence suggests that the use of a multivariate approach with external validation can be seen as the highest achievable goal.^4,5,17 The results of this study, which was conducted in line with this approach, as well as (earlier) confirmation of DOTTS content validity²² support this conclusion. Nevertheless, we need to stay critical about whether triage can be compared to screening tool studies.

Separate analysis of the outcomes in each of the four included hospitals showed no significant differences. Therefore, the external validity of the system is sufficient.¹⁷ The results for specificity, PPV and NPV were almost identical in the hospitals studied (Table 2). However, the sensitivity between the hospitals varied between 63% and 77%. Reasons for this are not studied but could include differences in the type (teaching or non-teaching), location and/or size of the hospitals studied. It is known that regular exposure to triage is required to use it properly.⁵ However, more research is necessary in order to confirm whether additional factors, such as those we suggest, can also influence the sensitivity of telephone triage systems.

In follow-up research, special attention should also be given to urgency level 5 (U5), which is self-care advice. It is a weakness in this research that it was not possible to include this category. It can lead to overestimating or underestimating of the accuracy of DOTTS. At this moment, we can state that in obstetric practice of Dutch hospitals, people who contact for the same complaint a second time will always be present in this study. The reason is that the clinical procedure is that if a patient calls again about the same complaint within 24 hours, she will always come to the hospital for clinical assessment. Special attention was given to this clinical procedure during the specific training.

In addition to validity, evaluations of triage systems also involve assessment of reliability. Reliability refers to the degree of intra-observer and inter-observer variability of the system.^3,17 Results of studies, which consider both reliability and validity, can indicate what improvements are needed in the future. Also, insight into sensitivity and specificity per presenting symptom could lead to improvement of the system.²⁶ In addition, more information regarding the quality of the telephone conversation is important. Knowledge of which social skills and medical knowledge triage staff needs are of particular value for future improvements. Recordings of telephone calls for training purposes and audits could provide more insight.^6–8

This was a prospective study conducted in daily practice, in which we used the usual care outcomes as reference standard. In comparable studies, specific individuals or simulators were selected as reference standard.^4,15 In our study, any medical doctor or midwife could provide the reference standard. This makes our reference standard representative of daily practice. However, in a validation study a reference standard must identify the true urgency and there is debate concerning the best way to choose the reference standard.⁹ Other studies²⁷ have shown that experience plays a role in decision-making in triage. Therefore, a potential limitation of our study could be that this might have unfavorably influenced our results: especially in the case where a study participant had limited work experience.

The results showed that by using DOTTS an estimate of urgency level can be made. Therefore, the use of DOTTS can be encouraged in obstetric practice. In near future, additional research of DOTTS is necessary. If the urgency of care can be adequately estimated by telephone, this contributes to better distribution of human and financial resources. Ultimately, the increasing volume of urgent care within maternity service provision makes this necessary. This is comparable with any crisis, such as during the current COVID-19 pandemic, in which scarcity of medical professionals necessitates evaluation of symptoms by telephone.

Conclusion

DOTTS shows an acceptable diagnostic validity with room for improvement. The overall sensitivity was 76%, and DOTTS compared to a reference standard has an agreement of 53%, and overtriage in 30% and undertriage in 16% of the cases. Future studies need to establish reliability and diagnostic validity of self-care advice (U5). Also, more insight into specificity could lead to improvement of the system.