Actigraphy-Based Sleep Detection: Validation with Polysomnography and Comparison of Performance for Nighttime and Daytime Sleep During Simulated Shift Work

Plain Language Summary

Actigraphy is widely used in both research and clinical settings to estimate sleep. Two commonly used algorithms for actigraphy-based sleep scoring are the built-in algorithm of the Actiware software (Philips Respironics) and the Cole-Kripke algorithm. The Cole-Kripke algorithm implements re-scoring rules to reduce the overestimation of sleep time. Existing validations of both algorithms predominantly focused on nighttime sleep, whereas it is unknown how accurate they are in detecting daytime sleep. Here, we reported that the performance of the two algorithms in identifying sleep epochs remained similar for daytime and nighttime sleep, and both algorithms had increased errors in identifying wake epochs during daytime. These findings may provide useful guidance for objective evaluation of daytime sleep using ambulatory actigraphy and justify the necessity for improving actigraphy-based algorithms in scoring daytime sleep.

Introduction

While polysomnography (PSG) is the gold standard for measuring sleep, it requires costly equipment, electrode placement and equipment calibrations by knowledgeable sleep technicians, and time-consuming manual scoring.¹ A widely used alternative sleep assessment method is actigraphy. Unlike PSG-derived sleep scores that are based on brain activity (EEG), eye movements (EOG), muscle activity and skeletal muscle activation (EMG), actigraphy measures body (wrist) movement and actigraphy-based algorithms use movement activity data to estimate sleep.^2,3 Compared to PSG, actigraphy-based sleep assessment has several advantages, including greater ecological validity, less influence on sleep,⁴ and the utility for continuous monitoring across days.² Thus, actigraphy may offer a convenient and naturalistic solution of long-term sleep monitoring in the home environment. In 2007, the American Academy of Sleep Medicine published guidelines on using actigraphy to diagnose sleep disorders.⁵ Later, a systematic review and meta-analysis concluded that actigraphy is generally a valid and useful tool for diagnosing and assessing treatment outcomes for some sleep disorders, such as delayed sleep-wake phase disorder.⁶

However, because actigraphy distinguishes wake from sleep based on movements in the forms of activity counts and/or raw acceleration, it may erroneously categorize quiet rest as sleep, especially as occurred to insomnia patients who may stay awake while lying still in bed⁷ and patients who are bedbound due to illnesses.⁸ Indeed, the widely used Actiware (Philips Respironics) algorithm shows high sensitivity (ie, correctly scoring sleep epochs) but low specificity (ie, correctly scoring wake epochs). For example, specificity of Actiware for overnight sleep was 0.48 among a group of 38 healthy young adults,⁹ 0.28–0.48 in sleep-disordered patients across different thresholds,¹⁰ and 0.67 in 33 middle-aged adults with lower back pain;¹¹ specificity of Actiware for daytime naps ranged from 0.36 to 0.64 in healthy young adults;¹² sensitivity of Actiware was above 0.87 and up to 0.96 in all these studies. Due to low specificity, Actiware as well as other actigraphy-based algorithms may generate ambiguous inference about sleep/wake states, resulting in overestimation of sleep.²

To address the issue of overestimation of sleep, re-scoring rules have been introduced in the Cole-Kripke algorithm to score actigraphy data.^13,14 For instance, based on one physiological consideration that people may stop moving for a few minutes before falling asleep (ie, the onset of sleep), the first few epochs (eg, 3 minutes) after a certain amount of wakefulness (eg, 10 minutes) scored as sleep are re-scored to wake. A study of 40 young adults showed that applying the re-scoring rules in the Cole-Kripke algorithm improved the specificity from 0.441 to 0.486 while reducing the sensitivity slightly from 0.975 to 0.970.¹⁵ Though the Cole-Kripke algorithm is a commonly used algorithm for scoring sleep based on actigraphy (eg, Gu et al 2020;¹⁶ Widome et al 2020;¹⁷ and epidemiological studies such as the Jackson Heart Sleep Study),¹⁸ especially for data from accelerometer devices other than the Actiwatch (Bend, OR), its performance as comparison to Actiware is still not clear. Certain studies showed that the Cole-Kripke algorithm can achieve a specificity up to 0.65¹³ and can reduce the overestimation of total sleep time from 1.89% to 0.81%,¹⁴ while other studies showed the specificity of the Cole-Kripke algorithm is still low (eg, 0.34 among healthy young adults).¹⁹

Previous validation studies of actigraphy-based sleep scoring have predominantly used data collected during nighttime sleep.⁶ Sleep during daytime, such as recovery sleep during the morning hours after night shift and napping in the afternoon before night shift, is common in today’s society and an important component of the 24-hour sleep-wake cycle. Many studies have provided convincing evidence for the physiological importance of daytime sleep, showing that, for instance, mid-afternoon recovery sleep improves vigilance.²⁰ The performance of actigraphy-based algorithms for detecting daytime sleep has rarely been tested.²¹ It is unclear whether these algorithms perform similarly or differently for detecting daytime sleep as for nighttime sleep. Addressing the question is crucial for the design of large-scale field sleep studies, in which actigraphy may be the only feasible tool for sleep assessment.

A related question is whether it is appropriate to perform sleep scoring on 24-h actigraphy data when scheduled sleep episodes are unknown (eg, sleep/wake diary is not available or not reliable). Though sleep diary/log is often used in conjunction with actigraphy,²² completing sleep diaries adds extra burden to participants and compliance rates vary across studies. For example, a two-week study by Maich et al found 99.8% completion rate of sleep diary.²³ However, in Thurman’s study of 30 healthy adults who were asked to wear actiwatches and keep sleep logs for up to 16 consecutive weeks,²⁴ the overall completion rate of sleep logs was 73% and only 60% of participants completed ≥75% of their 16-week sleep logs, with lower diary completion rates associated with lower agreement between sleep log and actigraphy. On the other hand, considering the health relevance of sleep, long-term sleep monitoring should become more and more common in health care as well as in clinical research, especially for older adults who may not be able to complete sleep diaries (eg, people with Alzheimer’s disease). Therefore, testing whether Actiware and the Cole-Kripke algorithms can perform well with unknown sleep episodes is also of importance for sleep study design using actigraphy.

To fill the gaps in the literature, we examined the performance of Actiware and the Cole-Kripke algorithms using the data collected from the same participants during daytime and nighttime sleep. Objective 1 of this study was to test whether actigraphy-based sleep assessment detected sleep epochs during daytime and nighttime sleep episodes equally accurately against polysomnography. Objective 2 of this study was to compare the performance of the two algorithms using the same dataset. If their performance in detecting sleep (ie, sensitivity) differed, we would further explore whether the differences were due to differences in detecting light sleep stages, because the Cole-Kripke’s re-scoring rules apply to the first few sleep epochs following wakefulness. Objective 3 of this study was to test the importance of sleep/wake diary by determining the differences in the performance of actigraphy-based sleep assessment when 24-hour actigraphy data were analyzed compared to when only scheduled sleep episodes were analyzed.

Methods

Participants and Protocols

We used data from two experiments each with two multi-day in-laboratory protocols. Other aspects of these protocols, designed to test independent hypotheses, have been published previously.^25–38 Experiment 1 included 14 non-shift worker adults (mean age = 28.16y, SD = 9.20y, age range: 20–50y; eight women) who did not perform any shift work in the past three years, were not involved in more than six months of cumulative lifetime shift work, and did not cross more than one time zone in the past three months. Experiment 2 included 11 shift workers (mean age = 34.46y, SD = 7.64y, age range: 24–48y; six women) who had on average 4.52y (SD = 7.74y, range: 1.11–25.09y) of consecutive shift work experience and on average 5.30y (SD = 7.66y, range: 1.11–25.09y) of lifetime cumulative shift work experience. In both experiments, participants were non-smokers and not taking drugs or medications (except for oral contraceptives). Each participant completed a circadian alignment protocol (Experiment 1: Figure 1A; Experiment 2: Figure 1C) and a circadian misalignment protocol (Experiment 1: Figure 1B; Experiment 2: Figure 1D). The two protocols were in randomized order and separated by 2–8 weeks (Experiment 1: Mean = 4, SD = 2; Experiment 2: Mean = 4, SD = 1). During the circadian alignment protocol of both experiments, participants had a daily scheduled sleep episode between 11pm-7am. A simulated shift work schedule was used to introduce circadian misalignment in both experiments. Specifically, in Experiment 1 (non-shift workers), following nighttime sleep episodes between 11pm-7am on Days 1–3, participants were scheduled to sleep between 3pm-7pm on Day 4 and 11am-7pm on Days 5–8; in Experiment 2 (chronic shift workers), participants had scheduled sleep episodes between 3pm-7pm on Day 1 and between 11am-7pm on Days 2–3.

Figure 1 Experiment protocols. (A and B) Two 8-day protocols in Experiment 1 were designed for non-shift workers, including (A) a circadian alignment protocol with nighttime sleep and (B) a misalignment protocol with nighttime sleep from Day 1 to Day 3, an afternoon nap on Day 4, and daytime sleep from Day 5 to Day 8. (C and D) Two 3-day protocols in Experiment 2 were designed for chronic shift workers, including (C) a circadian alignment protocol with nighttime sleep and (D) a misalignment protocol with an afternoon nap on Day 1 followed by nighttime sleep on Days 2–3.

Experiment 1 consisted of two 8-day (with 7 nights) laboratory protocols. Sleep was recorded via PSG during the scheduled nighttime sleep episodes (~11pm-7am) on Day 4 and Day 6 of the circadian alignment protocol and during the scheduled daytime sleep episodes (~11am-7pm) on Day 5 and Day 7 of the circadian misalignment protocol (Figure 1A and Figure 1). Experiment 2 consisted of two 3-day (with 2 nights) laboratory protocols. Participants were instructed to sleep from 11pm to 7am on the night before the in-laboratory protocols to reduce prior sleep debt. During the study, PSG was recorded during the scheduled nighttime sleep episode (~11pm-7am) on Day 1 of the circadian alignment protocol and during the scheduled daytime sleep episode (~11am-7pm) on Day 2 of the circadian misalignment protocol (Figure 1C and Figure 1). For both experiments, each scheduled sleep opportunity episode was 8 hours to allow sufficient sleep and increase generalizability to the general population with normal sleep duration. Wrist actigraphy (Actiwatch Spectrum, Bend, OR) was continuously worn during the whole study period. More details on the study methods have been previously reported.^37,38 Both experiments complied with the Declaration of Helsinki, were approved by the Partners Human Research Committee, and were conducted in the Center for Clinical Investigation at Brigham and Women’s Hospital (Boston, Massachusetts). All participants provided written informed consent.

Data Collection and Analysis

Polysomnography

During each PSG assessment, data were collected continuously using Vitaport-3 (Temec Instruments, Kerkrade, B.V., The Netherlands) with a sampling rate of 256 Hz, including left and right electrooculography (EOG), bipolar submental electromyography (EMG), bipolar electrocardiography (ECG), and electroencephalography (EEG; F3, F4, C3, C4, O1 and O2 channels, referenced to contralateral mastoids). Research personnel inspected and scored the PSG recordings in 30-second epochs using the Vitascore software (Temec Instruments) based on the American Academy of Sleep Medicine guidelines.³⁹ Participants were monitored and kept awake outside the scheduled sleep opportunities, based on study protocols.

Actigraphy

The Actiwatch monitored raw acceleration with a sample frequency of 32 Hz and integrated the data to a proprietary “count” value for every epoch (1 minute). A built-in capacitive sensor determined whether the device was on-wrist or off-wrist. On-wrist epochs of the activity count data were loaded into the following two software packages for sleep/wake scoring. (1) Actiware software (version 5.52; Respironics, Inc. Murrysville, PA)⁴⁰ describes the scoring approach as follows:

Whether a particular epoch is scored as wake is determined by comparing [the weighted value of] activity counts [in 5 consecutive epochs centered at] the [current] epoch in question, to a threshold value which can be set by the researcher … If the number of counts exceeds the threshold the epoch is scored as wake. If it falls below, or is equal to, the threshold the epoch is scored as sleep.

There are three levels for the threshold, 20, 40, and 80 activity counts, that correspond to low-, medium-, and high-sensitivity settings, respectively. (2) The ezActi software was developed by the Medical Biodynamics Program at Brigham and Women’s Hospital to provide a graphical user interface in the MATLAB platform for activity data analysis, and the Cole-Kripke algorithm^13,14 was implemented in this software to determine sleep or wake epochs. This software can be accessed at: https://github.com/pliphd/Actigraphy.

Temporal Alignment of PSG and Actigraphy Measurements

Because of the difference in length and start time of epochs between PSG and actigraphy, the beginning of each 1-minute actigraphy epoch did not align perfectly with the beginning of a 30-second PSG epoch. To resolve this, we identified three 30-second PSG epochs that overlapped with each 1-minute actigraphy epoch and considered a 1-minute actigraphy epoch as a true sleep epoch when the three 30-second PSG epochs were all sleep epochs. We further categorized the true sleep epochs into PSG-light sleep (ie, N1 and N2), PSG-slow-wave sleep (SWS; ie, N3), or PSG-rapid eye movement (REM) sleep if their corresponding PSG epochs were all light sleep, SWS, or REM sleep, respectively. Likewise, a 1-min actigraphy epoch was considered a true wake epoch if the corresponding three PSG 30-second epochs were wake. Otherwise, we categorized an epoch as a transitional epoch (1.93% of all epochs), which was excluded from analyses.

Epoch-by-Epoch Analysis

We compared the epoch-by-epoch scoring from Actiware and the Cole-Kripke algorithm to sleep/wake status obtained from PSG and study protocols (scheduled wakeful episodes). For each participant during each study protocol, we computed sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1 score (ie, a better measure of accuracy in the case of imbalanced data, which is less biased than the traditional accuracy), and Cohen’s kappa using data during the scheduled sleep episodes with PSG. Then, we repeated the calculation of these measures for the 24-h time windows (the 24 hours following the beginning of a sleep opportunity period) in those days with PSG (Figure 1). Sensitivity refers to the proportion of true sleep epochs that was correctly scored as sleep by Actiware/Cole-Kripke algorithm; specificity refers to the proportion of true wake epochs that was correctly scored as wake by Actiware/Cole-Kripke algorithm.⁴¹ PPV is the probability that an epoch scored as sleep by Actiware/Cole-Kripke algorithm was truly a sleep epoch and NPV is the probability that an epoch scored as wake by Actiware/Cole-Kripke algorithm was truly a wake epoch. F1 score is a measure of accuracy that is less affected by the imbalanced samples in a data set.⁴² It is the harmonic mean of sensitivity and PPV, specifically:

Cohen’s kappa measures agreement between PSG and Actiware/Cole-Kripke algorithm, with values ≤0.2 indicating no or slight agreement, 0.2–0.4 indicating fair, 0.4–0.6 indicating moderate, 0.6–0.8 indicating substantial, and 0.8–1.0 indicating perfect agreement.⁴³ For statistical analyses, we focused on the commonly used metrics (ie, sensitivity, specificity, and F1).

Summary Sleep Parameters

In addition to epoch-by-epoch analyses, we compared total sleep time (TST) and wake after sleep onset (WASO) measured by the Cole-Kripke algorithm, Actiware, and PSG, because TST and WASO are widely used in both research and clinical settings.

Statistical Analysis

We first conducted independent-samples t-tests (for continuous variables) and chi-square (for dichotomous variables) to compare participants and their sleep characteristics in the two experiments. Because Experiment 1 and Experiment 2 participants were demographically similar and they had similar sleep characteristics, we combined the samples of two experiments for the following analyses. To compare nighttime and daytime sleep characteristics of the same individuals, we conducted mixed-effect models with protocol (2 levels: nighttime/daytime sleep) as the fixed-effect factor and individual subject as the random-effect factor. Outcomes were PSG-measured TST, light sleep, SWS, REM sleep, and activity counts per epoch during each sleep stage. Mixed-effect models were further used to investigate whether activity counts per epoch were associated with sensitivity.

To evaluate the performance of actigraphy-based sleep, we conducted two models for epoch-by-epoch analyses. Model 1: For objective 1 (nighttime vs daytime sleep) and objective 2 (Cole-Kripke vs Actiware), we conducted a mixed-effect model with protocol (2 levels: nighttime/daytime sleep), Algorithm (4 levels: Actiware low/medium/high threshold or the Cole-Kripke algorithm), and Protocol × Algorithm interaction as fixed-effect factors. Individual subject was included in the model as a random-effect factor. Sensitivity, specificity, and F1 scores for scheduled sleep episodes and sensitivity during light sleep, SWS, and REM sleep were the outcomes. Post-hoc analyses (pair-wise Tukey LSD test) were conducted for significant main and interaction effects.

Model 2: For objective 3 (scheduled sleep vs 24-hour windows), we added Analysis Window (2 levels: scheduled sleep or 24-h time windows), Analysis Window × Protocol interaction, Analysis Window × Algorithm interaction, and the three-way interaction of Analysis Window × Protocol × Algorithm as fixed-effect factors to model 1. Post-hoc analyses (pair-wise Tukey LSD test) were conducted for significant main and interaction effects. Specificity and F1 scores were the outcomes. Sensitivity was not tested as an outcome, because it is not influenced by analysis window.

For analyses on TST and WASO, we computed the mean difference between the Cole-Kripke/Actiware and PSG as a measure of bias and the standard deviation of such difference as a measure of imprecision. We used paired-samples t-test to examine whether TST/WASO measured by the Cole-Kripke or Actiware was different from PSG-measured TST/WASO. Complimentarily, we computed the absolute difference between TST/WASO measured by PSG and the Cole-Kripke algorithm or Actiware and used one-sample t-test to examine whether such difference was different from 0. To test the agreement in TST between the algorithms and PSG, we computed intraclass correlation coefficients (ICC); we used two-way mixed-effect models to compute ICCs and estimated the reliability of a single measure to examine the consistency between measures.⁴⁴ ICC values <0.50, 0.50–0.75, 0.75–0.90, and >0.90 indicate poor, moderate, good, and excellent agreement, respectively. We computed ICCs in SPSS 28 and performed all other analyses in JMP Pro 16; all statistical tests were two-tailed, and statistical significance was accepted at alpha = 0.05.

Results

Participant and Sleep Characteristics

Participants from Experiment 1 and Experiment 2 had similar age (t=1.87, p=0.074) and gender distribution (χ²=0.02, p=0.897), as well as PSG-measured TST, light sleep, SWS, and REM sleep minutes per night during nighttime (ps>0.05) and daytime sleep (ps>0.05; Table 1). Next, we combined participants from the two Experiments and found that TST (F=22.85, p<0.001), light sleep (F=10.47, p=0.004), and REM sleep (F=14.24, p=0.001) were shorter during daytime than nighttime sleep. SWS minutes were similar between nighttime and daytime sleep (F=1.78, p=0.198).

Table 1 Participant Characteristics and Data Analyzed

Epoch-by-Epoch Analysis

Supplemental Table 1 and Figure 2 present the epoch-by-epoch analysis results for scheduled sleep episodes. Supplemental Table 2 presents the number of epochs scored as sleep or wake.

Figure 2 (A) Sensitivity, (B) specificity, and (C) F1 score comparison of actigraphy-based sleep scoring algorithms for scheduled sleep episodes. Each circle represents the value from an individual study protocol of a participant. The horizontal lines represent the mean and 1 standard deviation above/below the mean. *Indicates p<0.05.

Model 1 showed a significant difference in sensitivity across algorithms (F=248.46, p<0.001), no significant difference in sensitivity between nighttime and daytime sleep (F=2.41, p=0.122), and no significant interaction effect on sensitivity between protocol and algorithms (F=1.28, p=0.285). Post-hoc analyses showed that Cole-Kripke yielded lower sensitivity than Actiware with different thresholds (ps<0.05), and Actiware (low) yielded lower sensitivity than Actiware (high) (p<0.05). The difference in sensitivity across algorithms was also observed for each sleep stage (Light: F=234.88, p<0.001; SWS: F=37.58, p<0.001; REM: F=149.83, p<0.001; Supplemental Table 3; Figure 3). Cole-Kripke was the least sensitive in detecting sleep epochs during any sleep stage (ps<0.05). During both light sleep and SWS, sensitivity was similar for nighttime and daytime sleep (light sleep: F=0.02, p=0.887; SWS: F=0.82, p=0.367), which was consistent across all algorithms (Protocol × Algorithm interactions for light sleep: F=0.003, p>0.999; SWS: F=1.34, p=0.265). In contrast, within REM sleep, the algorithms were overall more sensitive for daytime than nighttime sleep (F=6.10, p=0.015), and this difference in sensitivity was similar across algorithms (Protocol × Algorithm interaction: F=2.12, p=0.100). Furthermore, there was no difference in activity counts per epoch between circadian alignment and misalignment protocols during wakefulness, NREM light sleep, or SWS (Supplemental Table 4), but activity counts were higher during nighttime than daytime REM sleep (p=0.010). During REM sleep, higher activity counts per epoch were associated with lower sensitivity in all algorithms (ps<0.001).

Figure 3 Comparison of sensitivity of actigraphy-based sleep scoring algorithms in detecting sleep during (A) light sleep (ie, N1 and N2), (B) slow-wave sleep (N3), and (C) REM sleep stages. Each circle represents the value from an individual study protocol of a participant. The horizontal lines represent the mean and 1 standard deviation above/below the mean. *p<0.05. *Indicates p<0.05. Abbreviation: REM, rapid eye movement.

Specificity was lower for daytime than nighttime sleep (F=8.37, p=0.004), differed across algorithms (F=124.05, p<0.001), and showed no Protocol × Algorithm interaction (F=0.59, p=0.622). Post-hoc analyses revealed that specificity was the highest in Cole-Kripke, and it decreased as the Actiware threshold increased from low to high (all pairwise ps<0.05). To explore the factors that contributed to the difference in specificity between daytime and nighttime sleep, we further examined those PSG-based wake epochs within sleep episodes (Supplemental Table 4). We found that, despite longer duration of wake bouts during daytime sleep, mean activity counts in each wake bout during daytime sleep were much lower than those during nighttime sleep (p<0.001). In addition, lower activity accounts were associated with lower specificity within both daytime and nighttime sleep episodes for all algorithms (ps<0.001).

F1 differed across algorithms (F=65.72, p<0.001; Supplemental Table 1; Figure 2C) and the difference appeared to depend on scheduled sleep time (Protocol × Algorithm interaction: F=4.96, p=0.003). Post-hoc analyses showed that Cole-Kripke yielded lower F1 than Actiware with different thresholds for both daytime and nighttime sleep (ps<0.05), F1 of Cole-Kripke showed an improving trend for scoring daytime sleep, and F1 of Actiware showed an improving trend for scoring nighttime sleep. Consistently, the difference in F1 between Cole-Kripke and Actiware was more pronounced for nighttime sleep.

Scheduled Sleep vs 24-Hour Analysis Window

Overall specificity increased when using 24-h time windows as compared to that based on scheduled sleep episodes (F=49.18, p<0.001; Figure 4A). This effect was mainly driven by Actiware rather than Cole-Kripke (Analysis Window × Algorithm interaction: F=5.57, p=0.001), ie, Actiware had improved specificity for 24-hour analysis window than scheduled sleep episodes (ps<0.05), while no improvement was observed in Cole-Kripke (p>0.05). Additionally, the difference in specificity between 24-hour and scheduled sleep episodes was similar for daytime and nighttime sleep (Analysis Window × Protocol interaction: F=1.49, p=0.223; Analysis Window × Algorithm × Protocol three-way interaction: F=0.22, p=0.882). Though the change of specificity was different for different algorithms when using 24-h windows, Cole-Kripke still had the highest specificity and Actiware (high) had the lowest specificity for both nighttime and daytime sleep.

Figure 4 (A) Specificity and (B) F1 score comparison of actigraphy-based sleep scoring algorithms using data during scheduled sleep episodes and using 24-h data. Each circle represents the value from an individual study Protocol of a participant. The horizontal lines represent the mean and 1 standard deviation above/below the mean. *Indicates p<0.05.

F1 score decreased when using data across 24 h as compared to that using data during scheduled sleep episodes (F=789.19, p<0.001). This decrease in F1 was observed in all algorithms (ps<0.05), but it was larger in Actiware than Cole-Kripke (Analysis Window × Algorithm interaction: F=44.74, p<0.001; Figure 4B). Protocol (daytime/nighttime sleep) had no significant influences on these changes in F1 score (Analysis Window × Protocol interaction: F=0.28, p=0.594; Analysis Window × Algorithm × Protocol three-way interaction: F=0.07, p=0.974).

Summary Sleep Parameters

Complementary to the epoch-by-epoch analyses, we examined whether TST scored by actigraphy algorithms agreed with TST measured by PSG. As shown in Table 2 and Figure 5, during the scheduled sleep episodes, Cole-Kripke agreed with PSG poorly (ICC = 0.29) for nighttime sleep and moderately (ICC = 0.57) for daytime sleep (Supplemental Figure 1A and B). Actiware (low) showed moderate agreement with PSG for nighttime sleep (ICC = 0.65) and poor agreement for daytime sleep (ICC = 0.49; Supplemental Figure 1C and D). Actiware (medium) scored nighttime sleep with moderate agreement (ICC = 0.72) and daytime sleep with poor agreement with PSG (ICC = 0.47; Supplemental Figure 1E and F). Actiware (high) showed moderate agreement with PSG for nighttime (ICC = 0.66) and poor agreement for daytime sleep (ICC = 0.38; Supplemental Figure 1G and H). In summary, Cole-Kripke underestimated TST for both nighttime and daytime sleep (Table 2). Actiware (medium) showed the least bias and highest agreement with PSG for nighttime sleep; Actiware (low) showed the least bias and Cole-Kripke showed highest agreement with PSG for daytime sleep.

Table 2 Agreement Between the Cole-Kripke Algorithm, Actiware, and PSG on Total Sleep Time (TST)

Figure 5 Agreement in total sleep time during scheduled sleep episodes between PSG- and (A) Cole-Kripke algorithm, (B) Actiware with low threshold, (C) Actiware with medium threshold, and (D) Actiware with high threshold. The dashed lines represent perfect agreement. Abbreviation: TST, total sleep time.

Next, during 24-hour time windows, all algorithms showed poor agreement with PSG for both nighttime and daytime sleep (ICCs <0.50; Table 2; Figure 6; Supplemental Figure 2), but Cole-Kripke yielded the least bias. All Actiware thresholds overestimated TST, with greater overestimation in higher thresholds.

Figure 6 Agreement in total sleep time during 24-hour periods between PSG- and (A) Cole-Kripke algorithm, (B) Actiware with low threshold, (C) Actiware with medium threshold, and (D) Actiware with high threshold. The dashed lines represent perfect agreement. Abbreviation: TST, total sleep time.

As shown in Supplemental Table 6, all algorithms yielded poor agreement with PSG in scoring WASO for nighttime and daytime sleep (ICCs <0.50).

Discussion

Using simulated shift work data collected from the same individuals with both daytime and nighttime sleep, we compared the performance of Cole-Kripke algorithm and Actiware software in identifying sleep/wake epochs and achieved three objectives.

For objective 1 (nighttime vs daytime sleep), overall sensitivity for both algorithms was similar between daytime and nighttime except that sensitivity in detecting sleep epochs within REM sleep was higher for daytime sleep than that for nighttime sleep. The lower sensitivity during nighttime for REM sleep may be explained by greater activity counts across REM epochs during nighttime than those of daytime because actigraphy-based sleep scoring is mainly based on activity levels (ie, low levels for sleep). Different from sensitivity, specificity was lower for daytime sleep, which is consistent for both algorithms. This finding is consistent with previous studies. For example, using a within-subject design, Paquet found that specificity of Actiware (as compared to PSG) was lower for two daytime recovery sleep episodes than that during a nocturnal sleep episode;⁴⁵ and another actigraphy study of a diverse sample of participants showed lower specificity of the Cole-Kripke algorithm during daytime sleep in night workers as compared to that during nighttime sleep in sleep-restricted individuals.⁴⁶ We showed that the daytime-nighttime difference in specificity was caused by lower mean activity levels within wake bouts during daytime sleep that led to lower specificity despite longer duration of wake bouts during daytime sleep. Regarding TST, we found that Actiware (medium) showed the least bias for nighttime sleep and Actiware (low) showed the least bias for daytime sleep. Though the Cole-Kripke algorithm showed the highest agreement with PSG for daytime sleep, it showed significant bias. In conjunction with previous studies showing the least bias of Actiware low threshold (compared to medium/high) in estimation of TST among patients with sleep disorders,^7,10 Actiware low-threshold (rather than medium/high) should be recommended for estimating TST when sleep may be disrupted.

Our objective 2 was to compare the Cole-Kripke algorithm with Actiware. The Cole-Kripke algorithm yielded lower sensitivity, higher specificity, and lower F1 scores than Actiware (all three thresholds). Our findings on Actiware (ie, high sensitivity and low specificity) were consistent with past observations.⁶ However, our findings on the Cole-Kripke algorithm were different from past observations of high sensitivity, low specificity, and accurate estimation of TST.^13,15,19 The differences may be caused by different accelerometer devices (eg, Motionlogger Actigraph,¹³ Motionlogger Micro Watch Actigraph,¹⁵ Mini Motionlogger Actigraph – Basic 32 C¹⁹ vs Actiwatch Spectrum in this study) with different configurations to obtain activity counts (eg, zero-crossing mode^13,15,19 vs proportional integrated mode in this study), PSG scoring criteria (eg, standard criteria of Rechtschaffen and Kales^13,19 vs AASM guidelines in this study), populations (eg, middle-aged adults,¹³ healthy young adults,^15,19 vs healthy shift workers and non-shift workers in this study), and/or the study environment (eg, at home¹⁵ vs laboratory environment in this study).

Our objective 3 was to compare the algorithms’ performance between scheduled sleep episodes and 24-h time windows. For the Cole-Kripke algorithm, specificity was similar for scheduled sleep and 24-hour periods, but F1 was lower for 24-hour windows. For Actiware, specificity was higher but F1 was lower for 24-hour windows than scheduled sleep episodes. The improvement in specificity is likely due to the inclusion of more wake epochs with higher activity counts for 24-hour windows. Researchers and clinicians often employ different methods to assess sleep via actigraphy. For example, in some studies, actigraphy was collected continuously across multiple days and nights (eg, Gao et al 2019),⁴⁷ whereas in other studies, actigraphy was only collected during scheduled sleep episodes at night (eg, Alsaadi et al 2014;¹¹ common among studies that only used actigraphy for one night).² The proportion of sleep epochs across all available epochs would influence the estimation of specificity and accuracy of actigraphy-based algorithms, especially when participants were relatively sedentary. Therefore, limiting analyses to sleep episodes, for example via sleep diary, appears to improve the accuracy of sleep assessment.

Strengths of the current study include the diverse samples of non-shift workers and shift workers, within-subject design for both circadian alignment and misalignment protocols (nighttime and daytime sleep), and the comparison of algorithms using the same data sets. In a laboratory environment, we were also able to closely monitor sleep/wake using both video and PSG while minimizing potential confounders during simulated shift work (eg, differences in sleep environment). The compromise of both algorithms in focusing on only one performance measure (either sensitivity or specificity) while sacrificing the other calls for the development of new approaches that can balance and/or improve both measures. One of the limitations is that all participants were young and healthy. The performance of the algorithms should also be tested in older populations and clinical populations with sleep, circadian, or movement disorders and during sedentary periods. We combined non-shift workers and shift workers in our experiments as they were demographically similar, but future studies with sufficient sample sizes should investigate the populations separately. Moreover, in the current study, the scheduled sleep opportunities were regular across days and the duration of each sleep opportunity was 8 hours during daytime/nighttime while no spontaneous sleep was allowed outside of this window. Our findings based on 8-h scheduled sleep episodes in the laboratory may not be generalizable to short sleep episodes (eg, napping). For instance, Cole-Kripke’s re-scoring rules should be reconsidered for detection of short naps. Given that naps are linked to diseases (eg, dementia),⁴⁸ algorithms and validation studies are needed on detecting naps via actiwatches to facilitate investigations of objective naps and diseases. Last, we only evaluated two actigraphy algorithms using activity counts collected with the Actiwatch Spectrum devices that automatically detect and exclude off-wrist periods.⁴⁹ Future studies are needed to examine other actigraphy-scoring algorithms (eg, the Sadeh algorithm)⁵⁰ or other sleep-monitoring devices (eg, other wristband accelerometers or non-wearables)⁵¹ especially those without on-wrist/in-use detection.

In conclusion, across Actiware and Cole-Kripke algorithms, sensitivity of sleep scoring was similar between daytime and nighttime sleep and specificity was lower for daytime sleep. Actiware outperformed the Cole-Kripke algorithm in sleep scoring of actigraphic recordings for both daytime and nighttime sleep episodes scheduled in laboratory. On average, Actiware with low-threshold yielded the least bias in estimating daytime TST and Actiware with medium-threshold yielded the least bias in estimating nighttime TST. Additionally, using sleep diary and limiting the actigraphy-based analyses to known periods in bed should improve sleep assessment.