Validity and reliability of the Thai version of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU)
Received 18 February 2014
Accepted for publication 3 April 2014
Published 29 May 2014 Volume 2014:9 Pages 879—885
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 3
Tanyong Pipanmekaporn,1 Nahathai Wongpakaran,2 Sirirat Mueankwan,3 Piyawat Dendumrongkul,2 Kaweesak Chittawatanarat,3 Nantiya Khongpheng,3 Nongnut Duangsoy3
1Department of Anesthesiology, Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand; 2Department of Psychiatry, Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand; 3Division of Surgical Critical Care and Trauma, Department of Surgery, Chiang Mai University Hospital, Chiang Mai, Thailand
Purpose: The purpose of this study was to determine the validity and reliability of the Thai version of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU), when compared to the diagnoses made by delirium experts.
Patients and methods: This was a cross-sectional study conducted in both surgical intensive care and subintensive care units in Thailand between February–June 2011. Seventy patients aged 60 years or older who had been admitted to the units were enrolled into the study within the first 48 hours of admission. Each patient was randomly assessed as to whether they had delirium by a nurse using the Thai version of the CAM-ICU algorithm (Thai CAM-ICU) or by a delirium expert using the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision.
Results: The prevalence of delirium was found to be 18.6% (n=13) by the delirium experts. The sensitivity of the Thai CAM-ICU’s algorithms was found to be 92.3% (95% confidence interval [CI] =64.0%-99.8%), while the specificity was 94.7% (95% CI =85.4%-98.9%). The instrument displayed good interrater reliability (Cohen’s κ=0.81; 95% CI =0.64-0.99). The
time taken to complete the Thai CAM-ICU was 1 minute (interquatile range, 1-2 minutes).
Conclusion: The Thai CAM-ICU demonstrated good validity, reliability, and ease of use when diagnosing delirium in a surgical intensive care unit setting. The use of this diagnostic tool should be encouraged for daily, routine use, so as to promote the early detection of delirium and its rapid treatment.
Keywords: delirium, surgical intensive care unit, Confusion Assessment Method for the intensive care unit, validity, reliability
Delirium is a disturbance in a person’s consciousness and a change in cognition that develops over a short period and usually fluctuates over time.1 Delirium is a complication found commonly in intensive care units (ICUs); its incidence in ICUs varies from between 20%-80%, depending on the types of screening tool used and the populations involved.2–15 Patients who suffer from delirium incur higher medical costs,16 require longer hospital stays,4,6,17 and have higher mortality rates than those who do not.4,6,14,15 In addition, delirium is associated with long-term cognitive impairment.17
A diagnosis of delirium is quite a challenge for nonpsychiatric medical personnel. Many critically ill patients often require mechanical ventilation, meaning that the symptoms normally presented by delirious patients cannot be diagnosed using standard tools normally used with patients who can communicate verbally. Delirium can go unrecognized when diagnosis relies on clinical presentation alone;13 therefore, delirium screening tools are required in order to detect symptoms early and provide for immediate treatment.
In Thailand, delirium screening is not routinely performed by medical staff in ICUs due to a lack of available, standardized tools, meaning that most diagnoses rely on clinical symptoms only. The Confusion Assessment Method algorithm (CAM algorithm) has been available for clinical diagnosis of delirium by nurses or physicians who are nondelirium experts; however, its poor reliability limits the use of this tool among patients in the ICU.7,18 The Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) is a highly sensitive and reliable tool for the diagnosis of delirium in the ICU.4 The CAM-ICU has been translated into several languages, including German,3 Dutch,19 Swedish,20 Greek,8 Portuguese,10 Spanish,21 Chinese,22 and Korean.9 A Thai version of the CAM-ICU (Thai CAM-ICU) has already been produced, however, it has not yet been validated. The objective of this study was to determine the validity and reliability of the Thai CAM-ICU when compared to a standard reference diagnostic criteria, as used by delirium experts, in a surgical intensive care unit (SICU).
Materials and methods
This cross-sectional study was approved ethically by the independent Institutional Review Board at the Faculty of Medicine, Chiang Mai University. Three well-trained, critical-care study nurses informed and obtained consent from the participants or their surrogate decision-makers prior to the assessment. The clinicaltrials.gov registration number of this study is NCT 01267630.
Participants and procedures
Based on the prevalence of delirium within ICUs (50%)3 and the acceptable levels of sensitivity and specificity for the CAM-ICU (about 90%),23 70 participants were recruited. Those recruited were elderly people 60 years of age and older, who had been admitted within the previous 48 hours to a 14-bed SICU and sub-SICU at Maharaj Nakorn Chiang Mai Hospital. Recruitment took place from February 2011 to June 2011. The participants were excluded if they had severe visual or auditory impairment, were likely to pass away within 24 hours, had a poor command of Thai, were unwilling to participate, or were comatose.
A research nurse and a delirium expert were responsible for assessing selected admissions within the first 48 hours of arrival at the ICU between the times of 8:30 am and 5 pm on weekdays. The nurse carried out the assessments using the Thai CAM-ICU, while the delirium expert used criteria from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR). The assessments were randomly assigned at interrater intervals of no more than 30 minutes. Each participant was evaluated twice, once by a nurse (SM or ND) and once by a delirium expert (NW or PD), independently.
Before screening for delirium, each participant was assessed using the Richmond Agitation and Sedation Scale (RASS) to determine the level of sedation.24 RASS is a ten-point scale that ranges from +4 (combative) to -5 (deeply sedated or unresponsive). If participants are alert, agitated, or combative, they are given a score of +4 to 0. If participants are drowsy or awake only briefly, but are able to respond to verbal stimuli, they are given a score of -1 to -3. If patients do not respond to verbal stimuli at all and do not respond to any physical stimuli, they are scored -4 to -5. In this study, all participants needed to score -3 or better using RASS in order to take part. A positive mental status change was defined as one where a participant showed evidence of an acute onset of mental status change when compared to the baseline upon admission, or any fluctuation in their RASS score, within 24 hours of being initially assessed.
The Attention Screening Examination (ASE) tool was used to assess inattention. The ASE consists of visual and auditory components through the recognition of ten letters. Inattention was diagnosed when patients scored less than eight on any component. To test disorganized thinking, participants were asked to answer a set of four questions correctly. An alteration of consciousness was rated when the RASS score was not zero.
Development of the Thai version of the CAM-ICU
The Thai version of the CAM-ICU was developed by the third author (SM) of this paper, following the “Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes Measures.”25 First, SM translated the CAM-ICU into Thai. Then a bilingual translator, who had never been exposed to the original CAM-ICU, did a back translation. Both translators reached a consensus after comparing and adapting the instrument so that its meaning corresponded to Thai culture.
The CAM-ICU tool assesses four features: (1) the acute onset of mental status changes, or a fluctuating course, (2) inattention, (3) disorganized thinking, and (4) altered levels of consciousness. Delirium is diagnosed when a participant manifests features 1 and 2, plus either feature 3 or 4. The training manual for the Thai CAM-ICU is currently available from the website http://www.icudelirium.org.
CAM-ICU training and interrater reliability
To ensure the reliability of the CAM-ICU, two critical care nurses (SM and ND) who are authors of this study performed Thai CAM-ICU assessments on ten patients. A gold standard evaluation was also performed by two delirium experts (NW and PD) on another ten patients using DSM-IV-TR criteria. There was found to be perfect agreement in each group of researchers (Cohen’s κ =1 for both).
Demographic variables are presented as mean ± standard deviation for continuous data, and frequency and percentage for categorical data. The performance of the Thai version of CAM-ICU algorithms was calculated by using sensitivity, specificity, and predictive values (positive predictive value and negative predictive value). The likelihood ratio of the test (sensitivity/1 - specificity) including 95% confidence intervals (CIs) were estimated. A receiver-operating characteristic (ROC) curve was presented to demonstrate the performance of the Thai version of the CAM-ICU.26 Cohen’s κ coefficient was used to demonstrate interrater reliability between critical care nurses and delirium experts. All statistics were analyzed using STATA 11.0 (StataCorp LP, College Station, TX, USA). A P-value <0.05 was considered statisticaly significant.
Over a 5-month period, 92 patients were enrolled in the study. Three were excluded because they did not want to participate, or their relatives did not want them to. Seventeen patients were discharged and two patients died before an assessment could take place. Figure 1 shows the details of the study flow.
Figure 1 Study patient selection process.
A total of 70 patients completed the study. The baseline patient characteristics are presented in Table 1. The majority of the study patients were male (64.3%) and most had also graduated from primary school (64.7%). The percentage of delirious patients who required mechanical ventilation was 22.7% and the mortality rate during hospitalization was 18.2%.
Validity and interrater reliability of the Thai CAM-ICU
The assessments made by the two critical care nurses and the delirium experts are shown in Table 2. The prevalence of delirium in SICU patients during the first 48 hours after admission was found to be 18.6%. The sensitivity of the Thai CAM-ICU’s algorithms was 92.3% (95% CI =64.0%-99.8%), while their specificity was 94.7% (95% CI =85.4%-98.9%). The positive and negative predictive values were 80.0% (95% CI =51.9%-95.7%) and 98.2% (95% CI =90.3%-100.0%), respectively. The likelihood ratio of Thai CAM-ICU was 17.5 (95% CI =5.8-53.3). The results of the ROC analysis showed the area under the curve was 0.93 (95% CI =0.85-1.00). Cohen’s κ between the study nurses and the delirium experts was 0.81 (95% CI =0.64-0.99), with a P-value of <0.001.
Table 3 presents the results of previous CAM-ICU validations, as shown in the literature. In this study, four CAM-ICU rating misclassifications were made by the two critical care nurses, while three false-positive assessments and one false negative assessment were identified. Among the three patients who gave false positives, two had hypovolemic shock and one developed septic shock during the ICU admission. One patient who gave a false negative was diagnosed with delirium plus a hypoalert condition by a delirium expert.
Time spent by nurses using the Thai CAM-ICU algorithms
The average time taken by the nurses to perform the Thai CAM-ICU algorithm was significantly less than the time taken by the psychiatric staff to do the full psychiatric assessments using DSM-IV-TR (1.59±0.86 minutes versus 8.48±4.81 minutes) (P<0.001). The minimum and maximum times used to calculate the Thai CAM-ICU algorithms were 0.5 minutes and 5 minutes, while the minimum and maximum times used for the DSM-IV-TR calculations were 2 minutes and 25 minutes.
This study demonstrated that the Thai version of CAM- ICU has a high validity and reliability, and that it takes less time to complete than the full psychiatric assessments. The validity of the Thai CAM-ICU’s algorithms is comparable to the original CAM-ICU developed by Ely et al4 showing sensitivities of between 95%-100%, and specificities of between 89%-93%. Previous studies have validated the CAM-ICU in different settings, including in an SICU, where Guenther et al3 reported the validity of the CAM-ICU Flowsheet in a SICU. In their study, the CAM-ICU Flowsheet was modified from the original, which meant changing the numbering for Features 3 and 4 to allow rapid assessments to take place. Their results show a high sensitivity, specificity, and reliability (88%-92% versus 100%, Cohen’s κ =0.96).3 In this study, and when used by the two critical care nurses, the sensitivity and the specificity of the Thai CAM-ICU algorithm were 92.3% and 94.7%, respectively. The Thai CAM-ICU algorithms revealed a high diagnostic value when taking into account the value of the likelihood ratio (LLR) of the test when given by the two nurses (LLR =11.8 and 30.0) and the area under the ROC. When comparing the performance of the test, the LLR of the test was superior to the positive and negative predictive values and it was not affected by the prevalence of disease.27
In addition, the area under the ROC confirmed the power of the Thai CAM-ICU algorithm in terms of discriminating between delirious and nondelirious patients.26 The present study demonstrated that the Thai CAM-ICU had a higher sensitivity and specificity than other validated versions of the CAM-ICU algorithms, including the Portuguese10 (sensitivity =72.5% and specificity =96.2%), Korean9 (sensitivity =77.4% for rater 1 and 89.8% for rater 2 and specificity =72.4% for rater 1 and 75.8% for rater 2), and Greek versions8 (sensitivity =95.2% and specificity =85.0%). In addition, the validity of the Thai version of CAM-ICU was comparable to the Chinese version.22 A good agreement was shown between the two critical care nurses when diagnosing for delirium (Cohen’s κ =0.82). The level of agreement between the two sets of raters (critical care nurses and psychiatrists) was comparable to that found with the Korean version9 (Cohen’s κ =0.81 and 0.75, respectively), but lower than the Greek and Spanish versions.8,21
The present study revealed some misclassifications within the Thai CAM-ICU ratings (three false positives and one false negative). In the three cases of patients showing false positives with the Thai CAM-ICU, delirium was diagnosed based on a positive test for Features 1, 2, and 4. For the patient with a false negative, a “nondelirious” state was diagnosed due to a positive result for Feature 1 and a negative result for Feature 2, while the psychiatrist gave positive results for all the reference standard domains. There are two possible explanations for these misclassifications. First, the use of sedative drugs may have impaired participants’ consciousness levels, which would have increased the chances of delirium being diagnosed by the CAM-ICU.28 Haenggi et al28 recommended that sedation should be discontinued, and the level of sedation should be re-evaluated before any assessment for delirium was made. One previous study demonstrated that a patient’s level of education may have an influence on the results for Feature 2: ASE.22
The present study found that 64.7% of patients had graduated from primary school. Wang et al22 concluded that the reliability of the ASE test is lower for illiterate patients (Cohen’s κ =0.61) and patients with only a primary school education (Cohen’s κ =0.70). They stated that giving an explanation of the test methods before doing the ASE auditory evaluation is very important for illiterate patients. Use of the Thai CAM-ICU in routine, clinical practice will improve the accuracy of delirium diagnosis for critically ill patients, and particularly in ICU settings, providing a standardized tool and serving as a benchmark when comparing against CAM-ICU tools from other countries.29
This study had a number of strengths. First, we compared the delirium diagnosis performance of the Thai CAM-ICU against a standard reference used by psychiatrists, and second, evaluations were carried out by both critical care nurses and delirium experts in order to reduce individual variability during the assessment process. Also, the evaluations carried out by the two sets of raters (study nurses and psychiatrists) on the same patient were never more than 30 minutes apart. This approach reduced the risk of variability in terms of diagnosis due to fluctuations in the condition.
However, the study also had a few limitations. First, the two nurses used in the study were not able to evaluate all the patients over the study period, due to the manpower situation in the SICU. As a result, we could not fully demonstrate the differences between the nurses’ and psychiatrists’ assessments. However, the authors did perform an interrater reliability assessment of the two critical care nurses, until perfect agreement had been achieved, before recruiting the study patients. Next, this study did not demonstrate motoric subtypes of delirium, which can be categorized by the level of RASS found. The presence of delirium symptoms is more important than the diagnosis of delirium, because patients with different subtypes of delirium, such as hypoactive or hyperactive delirium, require different treatments and prognoses.
The Thai CAM-ICU demonstrated good validity and reliability, and proved easy to use when diagnosing delirium in a surgical intensive care unit setting. The use of this diagnostic tool in daily practice should therefore be promoted, in order to facilitate the early detection and rapid treatment of delirium.
All authors made substantial contributions to conception and design, and interpretation of data. All but KC and TP collected the data. Statistical analysis was carried out by TP. All authors took part in drafting the article or revising it critically for important intellectual content. All authors approved the final manuscript.
The authors would like to thank the Faculty of Medicine, Chiang Mai University, Chiang Mai, Thailand, for its grant support. The authors also thank staff at the Division of Surgical Critical Care and Trauma in the Department of Surgery for their cooperation.
The authors report no conflicts of interest in this work.
American Psychiatric Association. Diagnostic and Statistical Manaul of Mental Disorders (DSM-IV-TR). 4th ed, text revision. Washingon, DC: American Psychiatric Association; 2000.
van Eijk MM, van Marum RJ, Klijn IA, de Wit N, Kesecioglu J, Slooter AJ. Comparison of delirium assessment tools in a mixed intensive care unit. Crit Care Med. 2009;37(6):1881–1885.
Guenther U, Popp J, Koecher L, et al. Validity and reliability of the CAM-ICU Flowsheet to diagnose delirium in surgical ICU patients. J Crit Care. 2010;25(1):144–151.
Ely EW, Shintani A, Truman B, et al. Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA. 2004;291(14):1753–1762.
Luetz A, Heymann A, Radtke FM, et al. Different assessment tools for intensive care unit delirium: which score to use? Crit Care Med. 2010;38(2):409–418.
van den Boogaard M, Pickkers P, van der Hoeven H, Roodbol G, van Achterberg T, Schoonhoven L. Implementation of a delirium assessment tool in the ICU can influence haloperidol use. Crit Care. 2009; 13(4):R131.
Ely EW, Margolin R, Francis J, et al. Evaluation of delirium in critically ill patients: validation of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Crit Care Med. 2001;29(7): 1370–1379.
Adamis D, Dimitriou C, Anifantaki S, et al. Validation of the Greek version of Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Intensive Crit Care Nurs. 2012;28(6):337–343.
Heo EY, Lee BJ, Hahm BJ, et al. Translation and validation of the Korean Confusion Assessment Method for the Intensive Care Unit. BMC Psychiatry. 2011;11:94.
Gusmao-Flores D, Salluh JI, Dal-Pizzol F, et al. The validity and reliability of the Portuguese versions of three tools used to diagnose delirium in critically ill patients. Clinics (Sao Paulo). 2011;66(11):1917–1922.
Pandharipande P, Cotton BA, Shintani A, et al. Prevalence and risk factors for development of delirium in surgical and trauma intensive care unit patients. J Trauma. 2008;65(1):34–41.
McNicoll L, Pisani MA, Zhang Y, Ely EW, Siegel MD, Inouye SK. Delirium in the intensive care unit: occurrence and clinical course in older patients. J Am Geriatr Soc. 2003;51(5):591–598.
Spronk PE, Riekerk B, Hofhuis J, Rommes JH. Occurrence of delirium is severely underestimated in the ICU during daily care. Intensive Care Med. 2009;35(7):1276–1280.
Ouimet S, Kavanagh BP, Gottfried SB, Skrobik Y. Incidence, risk factors and consequences of ICU delirium. Intensive Care Med. 2007;33(1): 66–73.
Thomason JW, Shintani A, Peterson JF, Pun BT, Jackson JC, Ely EW. Intensive care unit delirium is an independent predictor of longer hospital stay: a prospective analysis of 261 non-ventilated patients. Crit Care. 2005;9(4):R375–R381.
Milbrandt EB, Deppen S, Harrison PL, et al. Costs associated with delirium in mechanically ventilated patients. Crit Care Med. 2004;32(4):955–962.
Girard TD, Jackson JC, Pandharipande PP, et al. Delirium as a predictor of long-term cognitive impairment in survivors of critical illness. Crit Care Med. 2010;38(7):1513–1520.
Wongpakaran N, Wongpakaran T, Bookamana P, et al. Diagnosing delirium in elderly Thai patients: utilization of the CAM algorithm. BMC Fam Pract. 2011;12:65.
Vreeswijk R, Toornvliet A, Honing M, et al. Validation of the Dutch version of the Confusion Assessment Method (CAM-ICU) for delirium screening in the Intensive Care Unit. Netherlands Journal of Critical Care. 2009;13(2):73–78.
Larsson C, Axell AG, Ersson A. Confusion assessment method for the intensive care unit (CAM-ICU): translation, retranslation and validation into Swedish intensive care settings. Acta Anaesthesiol Scand. 2007; 51(7):888–892.
Toro AC, Escobar LM, Franco JG, et al. [Spanish version of the CAM-ICU (Confusion Assessment Method for the Intensive Care Unit). Pilot study of validation]. Med Intensiva. 2010;34(1):14–21. Spanish.
Wang C, Wu Y, Yue P, et al. Delirium assessment using Confusion Assessment Method for the Intensive Care Unit in Chinese critically ill patients. J Crit Care. 2013;28(3):223–229.
Jones SR, Carley S, Harrison M. An introduction to power and sample size estimation. Emerg Med J. 2003;20(5):453–458.
Ely EW, Truman B, Shintani A, et al. Monitoring sedation status over time in ICU patients: reliability and validity of the Richmond Agitation-Sedation Scale (RASS). JAMA. 2003;289(22):2983–2991.
Wild D, Grove A, Martin M, et al; ISPOR Task Force for Translation and Cultural Adaptation. Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Health. 2005;8(2):94–104.
Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39(4):561–577.
Grimes DA, Schulz KF. Refining clinical diagnosis with likelihood ratios. Lancet. 2005;365(9469):1500–1505.
Haenggi M, Blum S, Brechbuehl R, Brunello A, Jakob SM, Takala J. Effect of sedation level on the prevalence of delirium when assessed with CAM-ICU and ICDSC. Intensive Care Med. 2013;39(12): 2171–2179.
Pun BT, Gordon SM, Peterson JF, et al. Large-scale implementation of sedation and delirium monitoring in the intensive care unit: a report from two medical centers. Crit Care Med. 2005;33(6):1199–1205.
Ely EW, Inouye SK, Bernard GR, et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA. 2001;286(21): 2703–2710.