Reliability and Efficacy of the Epworth Sleepiness Scale: Is There Still a Place for It?

Evaluation of Excessive Daytime Sleepiness

Excessive daytime sleepiness (EDS) is a common complaint for which patients are referred to sleep clinic. EDS is important not only because it is a dysphoric feeling for patients and is associated with impairments in functioning such as an increased risk for motor vehicle crashes¹ and occupational injury,² but also because it is associated with increased mortality.^3,4 In one poignant study that assessed the contribution of EDS and obstructive sleep apnea (OSA) to mortality in elderly patients, those with obstructive sleep apnea (OSA) and EDS had increased mortality; those with either OSA or EDS alone did not have increased mortality.⁵ Similarly, in patients with moderate-severe OSA, the risk of major adverse cardiac events was higher in those with EDS.⁶ These studies suggest that EDS may be a critical component in linking certain diseases, such as OSA, to deleterious outcomes including mortality. In fact, exclusion of patients with EDS in certain randomized controlled trials of OSA treatment has cast doubts on the generalizability of the findings.⁷ However, despite the fact that EDS is clearly important, there is no consensus on the optimal way to assess EDS. There are objective measures such as the multiple sleep latency test (MSLT) and maintenance of wakefulness test (MWT), and self-reported measures including single-item questions,^3,4 two-item questionnaires,^8,9 and more comprehensive validated questionnaires including the Epworth Sleepiness Scale (ESS).¹⁰ The ESS has been used extensively in clinical and research settings and will be the focus of this review.

Development of the Epworth Sleepiness Scale

The ESS was developed by Murray Johns at Epworth Hospital in Australia and was first reported in 1991.¹⁰ The ESS was designed to be a simple test to administer and interpret. It asked patients to rate their likelihood of “dozing” in eight different scenarios with a minimum score of 0 indicating “would never doze” to a maximum score of 3 indicating a “high chance of dozing.” The total score can range from a minimum of 0 indicating a low level of sleepiness to a maximum of 24 indicating a very high level of sleepiness. Patients were asked to refer to “your usual way of life in recent times” rather than how they are feeling at the moment.

In the original validation study, ESS scores were higher in patients with OSA, narcolepsy, and idiopathic hypersomnia compared to control subjects and were lower in patients with insomnia. Furthermore, the ESS score was associated with OSA severity such that patients with worse OSA had higher ESS scores. In a small subset of patients for whom MSLTs were available, the ESS score was inversely correlated with mean sleep latency.¹⁰ The associations between the ESS and OSA and the ESS and the MSLT were further detailed in subsequent studies by Johns.^11,12

One important consideration with any measure is test–retest reliability. In other words, in the absence of an intervention, one would expect the score to be stable over time. Johns showed that in healthy medical students, ESS scores were very similar when done at two separate time points spaced 5 months apart. He also demonstrated that in patients with OSA who were subsequently treated with continuous positive airway pressure (CPAP), their ESS scores decreased.¹³ These observations suggest that in healthy individuals, the ESS scores are stable and that in sleepy patients, the scores decrease with appropriate intervention. Of note, test–retest reliability in non-treated OSA patients or other clinical populations was not assessed.

Test-Retest Reliability in Clinical Settings

To determine whether Johns’ observations of test–retest reliability in healthy medical students are similar in clinical populations, a number of studies have assessed test–retest reliability of the ESS in patients presenting for clinical care. The ESS score observed on one visit was compared to the ESS score observed on a subsequent visit. These studies have demonstrated some important differences when comparing ESS test–retest reliability in clinical populations to healthy subjects (Table 1; Supplemental Document).^14–17

Table 1 Studies Comparing Test–Retest Reliability of the ESS

Features that are common to Johns' original study and studies on clinical populations are that the average scores between the two administrations are similar and the correlation coefficients are generally similar. The divergence between these two groups emerges on measures of intra-individual variability. For example, in the study of Johns, only 3% of subjects had a discrepancy in the ESS scores of ≥5 between the two administrations of the ESS. In the clinical populations included in Table 1, a discrepancy in the ESS scores of ≥5 was seen in 14% of subjects in the study with the lowest intra-individual variability.¹⁶ Similarly, in the study of Johns, 18% of subjects had a discrepancy of ≥3 between the two administrations of the ESS, but in the clinical populations, there was a discrepancy of ≥3 in 37% of subjects in the study with the lowest intra-individual variability.¹⁶

Since there was higher test–retest variability observed in clinical populations compared to healthy, young subjects, it would be logical that some covariates or factors prevalent in clinical populations could explain this finding. However, age,^14,16,17 apnea–hypopnea index,^14–17 body mass index (BMI),^15,17 sex,^14,15,17 race,¹⁷ presence of diabetes,¹⁵ presence of hypertension,¹⁵ occupation,¹⁶ type of OSA testing,¹⁷ and time interval between ESS administrations^14,16,17 did not account for the test–retest variability on ESS scores. In fact, only one study identified any covariates or factors associated with the variability on ESS scores, age, and that contribution was very small in magnitude such that the entire model, including age, only explained 3% of the variability in ESS scores.¹⁵ Therefore, the contributors to the test–retest variability in ESS scores observed in clinical populations remain unknown. Furthermore, the reasons why there was relatively high variability observed in one part of one of the included studies (Taylor oximetry/general practitioner), compared to the other part of that study (Taylor oximetry/specialist) as well as the other included studies (Nguyen, Campbell, and Walker) remain unknown (Table 1).

Test–Retest Variability with the Epworth Sleepiness Scale as a Categorical Measure (Normal versus High)

In research settings, the ESS score is often evaluated as a continuous measure; there is no specified cutoff and essentially, ESS scores are viewed as higher or lower. In contrast, in some research settings and many clinical settings, ESS scores are assessed as normal versus high; a cutoff is specified and scores above that point are considered to indicate EDS. In the original validation study of Johns, the control subjects had an ESS score of 2–10, and scores ≥11 were only found in the evaluated subjects with a sleep disorder.¹⁰ Johns and Hocking made a similar observation in a subsequent study.¹⁸ Since then, there is a common convention of using an ESS ≥11 to indicate EDS.^9,19–26 In clinical settings, diagnostic testing and treatment are often predicated upon an “abnormal” score so the use of cutoff is often necessary.

With use of a cutoff, one could ask if it is the use of a measure as a categorical rather than a continuous variable that explains the results or is it the use of that particular cutoff? In other words, if a different cutoff is specified, will the results be substantially different? In a study assessing ESS normalization, defined as improvement of the ESS from ≥11 to <11, in patients with OSA treated with CPAP, the authors found a threshold of hours of CPAP use beyond which there was relatively little improvement in the rate of ESS normalization. A similar threshold for hours of CPAP use was obtained with use of other ESS cutoffs (8 or 12) to define normalization of the ESS.¹⁹ These results suggest that it is the use of the ESS as a categorical variable, rather than the use of the specific cutoff of 11, that is more important with this type of use of the ESS.

An important question to consider when assessing test–retest variability of the ESS in clinical populations is as follows: are the differences meaningful? Specifically, what is the chance that an ESS score remains normal or abnormal with two administrations of the ESS with no intervention between tests? One way to address this question is with use of Cohen’s kappa with use of a cutoff score of the ESS. Cohen’s kappa (for categorical variables) measures how often scores will remain on either side of a cutoff value with separate testing. In this context, if there was a group of patients for whom the ESS was repeated and the values that were <11 remained <11 every time, and the values that were ≥11 remained ≥11 every time, Cohen’s kappa would be 1, indicating excellent agreement. If, however, the values <11 increased to ≥11 half the time and the values that started ≥11 decreased to <11 half the time (i.e. no better than a coin flip), Cohen’s kappa would be 0, indicating poor agreement. Using the convention of defining EDS as an ESS≥11, values for Cohen’s kappa with repeat administrations of the ESS in clinical populations generally range between 0.5 and 0.7, suggesting moderate to substantial agreement²⁷ in classifying the scores as normal versus high (Table 1). Interestingly, in data derived from one of these studies,¹⁷ Cohen’s kappa was in the range of 0.5–0.7 with ESS cut-offs of either 8, 9, 10, 11, 12, 13 or 14, suggesting that there is moderate to substantial agreement with use of a threshold within this range, not necessarily due to the specific threshold of 11. This author has previously argued that these values are similar to those observed in a wide array of clinically-used measures including polysomnographic determination of OSA severity.¹⁷ This suggests that the test–retest reliability of the ESS in determining EDS with the use of a cutoff of an ESS≥11 is consistent with other tests in widespread clinical use.

Perhaps most illuminating would be to compare the ESS to another test used to assess EDS in clinical populations. The MSLT is in widespread clinical use and is part of the diagnostic evaluation for narcolepsy and idiopathic hypersomnia.²⁸ An MSLT is performed by giving a patient five nap opportunities during the day spaced 2 hours apart. The sleep latency is measured for each nap, and a mean sleep latency is calculated from these naps. Whether rapid eye movement (REM) sleep is observed in each nap is also noted and sleep-onset REM periods are reported. A mean sleep latency of less than 8 minutes is considered to indicate an abnormal degree of sleepiness, and the number of sleep-onset REM periods is further used to identify narcolepsy. Obviously, for the results to be meaningful, subsequent testing in the absence of an intervention should reveal similar results. However, ample evidence has cast doubt on this. In one study of patients with either narcolepsy without cataplexy, idiopathic hypersomnia or physiologic hypersomnia, the results of two separate MSLTs were compared. There was no correlation between the two tests, a change in mean sleep latency crossing the 8-minute threshold occurred in 42% of patients and a change in sleep-onset REM periods crossing the threshold of 2 occurred in 31% of patients.²⁹ From the data presented in the study, Cohen’s kappa for mean sleep latency for normal versus abnormal (8 minute cutoff) was 0.11 indicating poor agreement. Similar results were obtained in a subsequent study examining test–retest reliability of the MSLT in patients with narcolepsy without cataplexy, idiopathic hypersomnia and unspecified EDS. Furthermore, a longer test–retest interval was associated with an improved mean sleep latency and MSLT normalization. Interestingly, there was better test–retest reliability in patients with narcolepsy with cataplexy.³⁰ It should be noted that the inter-test interval in these studies was 4.2 (mean) and 1.8 (median) years, respectively, and whether there would be similar variability if the inter-test interval was substantially shorter is unknown. Nonetheless, these observations and other similar results suggest a limited utility for the MSLT in testing patients with disorders other than narcolepsy without cataplexy.³¹ More broadly, this highlights the variability in measuring EDS over time and demonstrates that even this “objective” measure has important caveats.

A fundamental question to consider when assessing test–retest variability is as follows: are the observed changes a limitation of the test, or alternatively, reflective of actual variability in the object of measure? In other words, with serial measurements of EDS, is there some natural variability that occurs over time that is being identified by EDS-assessment tools? Changes that may occur in everyday life including changes in sleep duration, sleep timing, use of stimulants such as caffeine, use of medications and psychosocial stressors likely vary over time and some of these changes may be reflected in measures of EDS. It is likely that this will explain at least part of the variability observed in serial measurements of the ESS over time. It would also be interesting to know whether there is less test-rest variability associated with very high or very low scores on the ESS.

Limitations of the ESS

There is no clear definition of EDS and, as noted above, there is no real “gold standard” test. It would therefore seem logical to assume that individual tools used to assess EDS may perform differently in certain groups and populations. One area that has received attention in this regard is sex. In adjusted analyses of data from the large community-based Sleep Heart Health Study, women and men showed a similar rate of EDS in a simple sleepiness question, but women were less likely to have an abnormal ESS score of ≥11 suggesting that the ESS may not have identified sleepiness in women as well as in men.⁸ When looking at ESS normalization (improvement from ≥11 to <11) in patients with OSA started on CPAP therapy, one study showed female sex was associated with an increased likelihood of ESS normalization,²¹ while another study showed that female sex was associated with a decreased likelihood of ESS normalization.²⁶ Consistent with sex-specific effects on the ESS, in a study of patients with epilepsy, the ESS was associated with OSA risk and antiepileptic drugs in men, but in women it was not associated with these covariates and was associated with depression.³² These studies suggest that the ESS may not be an identical measure in women and men. It therefore seems likely that the ESS score may be due to different variables in different groups and the nature of these interactions requires further elucidation.

Conclusion

The question of what constitutes EDS remains enigmatic. There are various tools to measure EDS, but there is no ideal method. The ESS is an inexpensive, widely available and commonly used questionnaire to assess EDS. Although the ESS is a well-validated measure of EDS, a number of studies have demonstrated higher test–retest variability in clinical populations compared to the original validation study, calling into question the appropriateness of the routine use of ESS in clinical settings. With categorical use of the ESS where a cutoff of ≥11 is used to classify patients as normal versus high, test–retest variability of the ESS is consistent with other tools in clinical use and may even compare favorably to the MSLT. One limitation of the ESS is that it is likely to be a measure of different variables in different populations, and a given score in one group may be comprised differently than a similar score in a different group. Clinically, determining whether a patient has EDS remains a challenge. The ESS is a useful tool but should be viewed in a larger clinical context along with the patient’s presentation, history and other diagnostic testing.