Defining the rest interval associated with the main sleep period in actigraph scoring
Received 14 June 2016
Accepted for publication 8 September 2016
Published 21 November 2016 Volume 2016:8 Pages 321—328
Checked for plagiarism Yes
Review by Single-blind
Peer reviewers approved by Prof. Dr. Roumen Kirov
Peer reviewer comments 3
Editor who approved publication: Professor Steven A Shea
Chin Moi Chow,1 Shi Ngar Wong,1 Mirim Shin,1 Rebecca G Maddox,2 Kristy-Lee Feilds,3 Karen Paxton,4 Catherine Hawke,4 Philip Hazell,3 Katharine Steinbeck5
1Discipline of Exercise and Sport Science, Faculty of Health Sciences, 2Sydney Medical School, 3Discipline of Psychiatry, 4School of Rural Health, 5Discipline of Paediatrics and Child Health, Sydney Medical School, The University of Sydney, Sydney, NSW, Australia
Abstract: Actigraphy is increasingly used for sleep monitoring. However, there is a lack of standardized methodology for data processing and analysis, which often makes between study comparisons difficult, if not impossible, and thus open to flawed interpretation. This study evaluated a manual method for detection of the rest interval in actigraph data collected with Actiwatch 2. The rest interval (time in bed), defined as the bedtime and rise time and set by proprietary software, is an essential requirement for the estimation of sleep indices. This study manually and systematically detected the rest interval of 187 nights of recording from seven healthy males and three females, aged 13.5±0.7 (mean ± standard deviation) years. Data were analyzed for agreement between software default algorithm and manual scoring. Inter-rater reliability in manual scoring was also tested between two scorers. Data showed consistency between default settings and manual scorers for bedtime and rise time, but only moderate agreement for the rest interval duration and poor agreement for activity level at bedtime and rise time. Manual detection of rest intervals between scorers showed a high degree of agreement for all parameters (intraclass correlations range 0.864 to 0.995). The findings demonstrate that the default algorithm on occasions was unable to detect rest intervals or set the exact interval. Participant issues and inter-scorer issues also made difficult the detection of rest intervals. These findings have led to a manual detection protocol to define bedtime and rise time, supplemented with an event diary.
Keywords: actigraph recording, activity level, light level
Actigraph recording is a convenient method for detecting activity and rest periods.1 When validated against polysomnography (PSG), actigraph records have been found in general to be valid and reliable for estimating sleep in healthy adults with normal sleep patterns,1–3 in pediatric4,5 and elderly populations,6 and in those with sleep disorders (advanced or delayed sleep phase syndrome and insomnia).2,7 A limitation of actigraphy in sleep research is the lack of standardized methodology for data handling, leaving researchers to independently make decisions regarding data sampling, processing, and analysis.8 As a consequence, comparison between studies may be open to flawed interpretation. In addition, other factors also limit such comparisons, since differences exist in how movement is quantified with different devices (eg, Motionlogger Sleep Watch [AMI] versus Phillips Respironics Actiwatch 2; Murrysville, PA, USA), in how proprietary software algorithms score records for sleep and wake estimates, and in elected settings within the Actiwatch actigraph (eg, low, medium, or high sensitivity, and epoch length). Indeed, a scoring and instructional manual has been commissioned by the Society of Behavioral Sleep Medicine to standardize the use of actigraphy devices.9
A range of actigraphs is commercially available. Although the Motionlogger (AMI) and AMA-32 (AMI) actigraph scoring algorithms for characterizing epochs of sleep and wake have demonstrated agreement with PSG of 88% to over 90%,10,11 further improvements in scoring rules have been proposed.12 A commonly used and well-validated brand of actigraph in sleep research is the Actiwatch 2 with light sensor and event marker by Philips Respironics (previously owned by Mini-Mitter Co., Inc., Sunriver, OR, USA).4,13–15 The present study will examine the automatically scored data collected using the Actiwatch 2 and analyzed with its Actiware software.
Actigraphs utilize an in-built accelerometer to record movement, and proprietary software use algorithms to score each recorded epoch as sleep or wake, yielding sleep indices such as sleep onset latency, wake after sleep onset, total sleep time, and sleep efficiency. In addition, a light sensor and an event marker (a button that can be pressed by the wearer) are useful features that can be employed to define the bedtime and rise time.
The rest interval demarcates the “time in bed” in Actiwatch 2. It is defined as the time the participant got into bed (bedtime) with the intention of sleeping and the time the participant eventually gets out of bed in the morning (rise time). After the rest interval is defined, the proprietary software automatically detects the time spent asleep within the rest period. All sleep indices are calculated from these measures. Thus, the marked rest interval has a direct impact on the estimated sleep indices, including sleep efficiency, which is the proportion of time spent asleep relative to the time spent in bed, and sleep onset latency, the time taken to fall asleep after going to bed.
The proprietary software can be used to automatically detect the rest interval. Because of potential limitations with location and marking of the rest intervals by the proprietary software, the software’s instructions advise the researcher to first inspect the data and then manually set or alter rest intervals that seem to have been missed by the algorithm, before generating sleep statistics.16 However, the software does not suggest a standard method for doing this, and to date there has been no published standardized methodology for defining the rest interval in Actiwatch 2. Thus, automatic detection may be further defined by manual intervention for estimation of sleep indices. Notably, such standardization has been in use for AMI Motionlogger device and software and verified by way of sleep diary17,18 or time-stamped phone-in reporting of bedtime and wake-up time.19
While actigraphy appears as a simple, nonuser dependent method, a number of specific technical considerations need to be accounted for when evaluating sleep. Therefore, the aims of this study were to 1) establish a protocol for manual scoring of the rest intervals in actigraph data collected from Actiwatch 2 by comparing the rest intervals yielded by a standardized manual scoring method with those yielded by the default algorithm and 2) test the inter-rater reliability of this standardized manual scoring method.
Actigraphy data were collected from a pilot study on sleep, depression, and puberty in adolescents. Written consent was obtained from all participants and their parent or guardian as per our study protocol that was approved by the University of Sydney Human Research Ethics Committee approval number 12502. A total of 187 nights of recording from 7 healthy males and 3 females, aged 13.5±0.7 years, were available for analysis. Each participant wore the Actiwatch 2 on the nondominant wrist for 4–12 days on 2–4 occasions. The participants were instructed to press the event marker when getting into bed at night and on waking up in the morning. Data were collected in 30-second epochs. This sampling rate, which has been applied in studies in children and adolescents,20 yields logging time of 15 days compared to the setting of 15 seconds, which yields 7.5 days. Default sensitivity (medium) was selected since this setting has been found to yield the least overestimation or underestimation of sleep or wakefulness for total sleep time and wake after sleep onset compared to PSG in children aged 9–11 years21 and 6–12 years.22 The participants were 10–12 years old when they were enrolled in this study. The rest interval was detected by 1) the standard default algorithm using the Respironics Actiware v5.71.0 (ENU) and 2) manually scored according to the standardized protocol as discussed later in this article.
Default algorithm detection of rest intervals
Rest intervals were defined automatically by the Respironics Actiware software using the default algorithm. The default algorithm does not take event markers or light level into account, but rather sets the rest interval according to activity data only. The algorithm automatically detects the major rest intervals of low activity that are longer than 3 hours. However, a low activity definition is not provided by the manufacturer. In this study, the major rest interval per day (the nocturnal sleep period) was used, and minor rest intervals (eg, daytime naps) were not selected.
Standardized manual scoring of rest intervals
The bedtime was defined using the following information: drop in activity level (Figure 1A), drop in light level to zero, and the event marker. The rise time was defined using the following indicators: increase in activity level (Figure 1B), solid wake bar (red bar) on the actogram, and increase in light level above 1.0 µW/cm2. The rise time was described as the first epoch after the sleep/wake bar (red bar) on the actogram had become solid (indicating waking), where there was increased activity. The instructions for scoring the rest interval are depicted by a decision tree in Figure 1. With no empirical data to suggest a cutoff point for drop in light or activity level, we have chosen an arbitrary but realistic cutoff of 10 minutes.
Figure 1 Decision tree for defining the rest interval.
Notes: (A) Bedtime criteria and (B) rise time criteria.
Abbreviations: A, activity level; L, light level; M, event marker; W, wake.
Agreement rates between default algorithm and manual detection, and between scorers
A two-way, mixed consistency, average-measures intraclass correlation (ICC) was applied to assess the degree of consistency and agreement between standard default setting and manual scoring, and between the two scorers in defining bedtime, rise time, duration of the rest interval, activity level at bedtime, and activity level at rise time (SPSS v21; Chicago, IL, USA). Two experienced scorers (SNW and MS) analyzed the actigraph data according to the proposed manual scoring protocol. One hundred and eighty three eligible rest interval records were included in the ICC reliability analysis. Eleven records had already been excluded owing to technical failures (no data had been collected). Four rest intervals, not detected by the default algorithm and independently detected by both scorers, yielded a total of N=187. Following analysis of the data independently by the scorers, problems or issues arising from the manual scoring of the rest intervals were identified and resolved.
ICCs between default algorithm and manual scoring are reported in Table 1. These data show consistency between the default algorithm and manual scoring in identifying bedtime and rise time, moderate agreement for duration of the rest interval, and poor agreement for activity level at bedtime and rise time. The activity level in the epochs, which the default algorithm selected as the bedtime and rise time, was exceedingly low (Table 1), whereas the manual scorers selected bedtimes and rise times in epochs with 12%–41% higher activity levels than the default readings. There was a high level of agreement between the scorers across all parameters (Table 1).
Figure 2 displays examples of errors associated with rest interval detection by the default algorithm. Figure 3 displays examples of those concerned with participant-related issues, and Figure 4 shows examples related to inter-scorer issues.
The original findings of this study conducted with Actiwatch 2 (v5.71.0 ENU) suggest that there are a number of problems with the default algorithm’s estimation of the bedtime and rise time. Thus, the protocol used in this study aimed to improve the marking of this demarcated period and detection of any rest periods missed by the default algorithm. It is not clear in some cases why rest intervals were not detected. However, it is known that the default algorithm uses activity level as a primary criterion for determining the start and end of a rest interval, for example, an epoch is marked as sleep following an immobility of 10 minutes. A light sensor has been incorporated into the Actiwatch as a result of research interest in circadian rhythms and measurements, but the default rules have not taken advantage of the light signal. Our manual detection rules include not only the level of activity but also consider the light level and the event marker. A further improvement to this protocol could be the addition of an event diary.
The highlighted examples of limitations produced by the default algorithm demonstrate the importance of manually checking actigraph data, rather than sole reliance on the default algorithm. Challenging data sets, which apparently arose from participant behavior and also from the potential for inter-scorer disagreement, demonstrate that manual scoring is not straightforward and that a structured approach is required to obtain the optimal definition of rest intervals. This standardized manual scoring process offered could add value to the Actiwatch 2 as an indispensable research and clinical tool, but it does require additional and experienced human resources.
A strength of the study is the high agreement between the two scorers. The manual scoring process suggests high reproducibility and thus reliability of the criteria used to detect bedtime and rise time. It does not demonstrate external validity; there is no objective verification of rest intervals. However, the manual protocol is directed at imprecisions generated by the default algorithm. Through human visual inspection and by means of the criteria set out in the decision tree, the bedtime and rise time are determined by taking into account the drop in activity and light levels, in addition to event markers where present. Accordingly, this manual scoring process has the potential to improve the detection of rest intervals.
A sleep diary and interview of the subject may reconcile participant-related issues. Although sleep diary of 1-week duration was recorded at three time points over 6 months, along with actigraph recordings by our participants, these data have not been used since we observed improved sleep diary recording longitudinally (unpublished data). Thus, in this group of mid-adolescents we did not feel confident that the sleep diaries were helpful in addressing manual scoring of rest intervals in Actiwatch 2. These data came from a field study and the opportunity to debrief adolescents in an interview was not available to the investigators.
The current findings illustrate many of the difficulties that investigators and clinicians face in determining rest intervals from actigraph recordings. Hence, we recommend that actigraph wearers keep an up-to-date self-report diary of relevant activities and mark events related to sleep routines to resolve ambiguities for better rest interval detection. An event diary that captures the sleep schedules (as indicated by Respironics) but furnished with additional sleep-related events is provided in Table 2. Clear instructions to participants regarding when to press the event marker are also essential.
Table 2 An example of an event diary
Abbreviation: N/A, not applicable.
In addition to the aforementioned issues, some researchers may define bedtime and rise time differently compared to how we defined them in this study. We defined the beginning of the interval as being the time the participant got into bed with the intention of sleeping (bedtime). Our definition highlights the fact pre-bedtime activities are often intentional. Similarly, the end of the interval may also be defined differently. While we defined it as the time the participant eventually gets out of bed in the morning (rise time), other researchers may define it as the time that the individual was no longer trying to sleep.
Given that the study tested only one brand of actigraph, the Actiwatch 2 and its automated algorithm, these findings cannot be generalized to other devices or other automated algorithms. Furthermore, future investigations to determine an optimized, standardized approach to defining rest intervals would benefit from including a larger number and a variety of participants, apart from the normal, healthy adolescents tested in this study. Subjects with sleep disorders or healthy sleepers whose rest-activity schedule is atypical could be included. In addition, the manual scoring system would need to be assessed using different actigraphs and automated software. A further important element of future research would be to validate the manual detection method through direct human observation, such as through video recording. Although PSG is a reference standard for sleep assessment, its simultaneous recording with actigraphs should make PSG unnecessary as a validation tool in this situation, since these studies often take place under standardized conditions with lights out at a fixed time. Thus, these conditions remove the flexibility seen in ambulatory home studies due to random activities such as reading, use of mobile phones and other electronic devices, or falling asleep with lights on.
The present study reports evidence of several instances where the default algorithm may yield imprecise rest intervals. These challenges may be avoided by employing a manual scoring protocol that defines the bedtime and rise time using activity level, light level, and event markers in a systematic way. This manual scoring method has a high inter-scorer reliability. While this process is more time consuming and increases participant load, the current proposed scoring rules for rest intervals translate to better estimated sleep indices.
This study on sleep, depression, and puberty in adolescents was supported by the Australian Rotary Health Research grant. The recipients were PH, CMC, KS, and CH.
The authors report no conflicts of interest in this work.
Sadeh A. The role and validity of actigraphy in sleep medicine: an update. Sleep Med Rev. 2011;15(4):259–267.
Ancoli-Israel S, Cole R, Alessi C, Chambers M, Moorcroft W, Pollak CP. The role of actigraphy in the study of sleep and circadian rhythms. Sleep. 2003;26(3):342–392.
Littner M, Kushida CA, Anderson WM, et al. Practice parameters for the role of actigraphy in the study of sleep and circadian rhythms: an update for 2002. Sleep. 2003;26(3):337–341.
Meltzer LJ, Montgomery-Downs HE, Insana SP, Walsh CM. Use of actigraphy for assessment in pediatric sleep research. Sleep Med Rev. 2012;16(5):463–475.
Belanger ME, Bernier A, Paquet J, Simard V, Carrier J. Validating actigraphy as a measure of sleep for preschool children. J Clin Sleep Med JCSM Off Publ Am Acad Sleep Med. 2013;9(7):701–706.
Luik AI, Zuurbier LA, Hofman A, Van Someren EJ, Tiemeier H. Stability and fragmentation of the activity rhythm across the sleep-wake cycle: the importance of age, lifestyle, and mental health. Chronobiol Int. 2013;30:1223–1230.
Morgenthaler T, Alessi C, Friedman L, et al. Practice parameters for the use of actigraphy in the assessment of sleep and sleep disorders: an update for 2007. Sleep. 2007;30(4):519–529.
Berger AM, Wielgus KK, Young-McCaughan S, Fischer P, Farr L, Lee KA. Methodological challenges when using actigraphy in research. J Pain Symptom Manage. 2008;36(2):191–199.
Ancoli-Israel S, Martin JL, Blackwell T, et al. The SBSM guide to actigraphy monitoring: Clinical and research applications. Behav Sleep Med. 2015;13:S4–S38.
Cole RJ, Kripke DF, Gruen W, Mullaney DJ, Gillin JC. Automatic sleep/wake identification from wrist activity. Sleep. 1992;15(5):461–469.
Sadeh A, Sharkey KM, Carskadon MA. Activity-based sleep-wake identification: an empirical test of methodological issues. Sleep. 1994;17(3):201–207.
Webster JB, Kripke DF, Messin S, Mullaney DJ, Wyborney G. An activity-based sleep monitor system for ambulatory use. Sleep. 1982;5(4):389–399.
Weiss AR, Johnson NL, Berger NA, Redline S. Validity of activity-based devices to estimate sleep. J Clin Sleep Med. 2010;6(4):336–342.
Lichstein KL, Stone KC, Donaldson J, et al. Actigraphy validation with insomnia. Sleep. 2006;29(2):232–239.
Hyde M, O’Driscoll DM, Binette S, et al. Validation of actigraphy for determining sleep and wake in children with sleep disordered breathing. J Sleep Res. 2007;16(2):213–216.
Respironics. Practice aspects of sleep diagnostic. Sleep Technician Guide. 2009.
Acebo C, Sadeh A, Seifer R, et al. Estimating sleep patterns with activity monitoring in children and adolescence: How many nights are necessary for reliable measures? Sleep. 1999;22:95–103.
Wolfson AR, Carskadon MA, Acebo C, et al. Evidence for the validity of a sleep habits survey for adolescents. Sleep. 2003;26(2):213–216.
Crowley SJ, Van Reen E, LeBourgeois MK, et al. A longitudinal assessment of sleep timing, circadian phase, and phase angle of entrainment across human adolescence. PloS ONE. 2014;9(11):e112199.
Gaina A, Sekine M, Chen X, Hamanishi S, Kagamimori S. Sleep parameters recorded by Actiwatch in elementary school children and junior high school adolescents: schooldays vs. weekends. Sleep Hypnosis. 2004;6:66–77.
Ward TM, Lentz M, Kieckhefer GM, Landis CA. Polysomnography and actigraphy concordance in juvenile idiopathic arthritis, asthma and healthy children. J Sleep Res. 2012;21(1):113–121.
Meltzer LJ, Walsh CM, Traylor J, Westin AM. Direct comparison of two new actigraphs and polysomnography in children and adolescents. Sleep. 2012;35(1):159–166.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]