Quantification of Differences in Sleep Measurement by a Wrist-Worn Consumer Wearable Compared to Research-Grade Accelerometry and Sleep Diaries of Female Adults in Free-Living Conditions

Introduction

Sleep is an essential process of restoration for the human body.^1,2 Due to its links to outcomes across mental, physical, and chronic disease dimensions of health, human sleep is a subject of epidemiologic interest and importance.³

The value of population-level insights on sleep health underscores the importance of sleep measurement tools that are robust and feasible in large studies under non-laboratory conditions with long-term follow-up. Epidemiological studies of sleep often rely on self-reported measures. While self-reported measures are easier to integrate into large studies, device-based measures of sleep are valuable because self-reported measures of duration and quality have been shown to differ from objective sleep measures. In a study by Lauderdale et al, sleep duration was consistently overestimated in self-reported measures, with the difference between the two measures widening as sleep duration decreased.^4–6 The gold-standard of sleep measurement, polysomnography (PSG), is complex and requires expertise and specialized equipment. This typically limits its application to a single night and is not only expensive but also does not capture typical at-home sleep for the average person.⁷ Research-grade actigraphy devices that are worn on a participant’s wrist are often used, which have been shown to yield valid measures of sleep relative to PSG.^8,9 However, research-grade objective sleep measurement tools like actigraphy watches are not well-suited to long-term individual-use and can be cumbersome to wear.

The advent of mobile health tools and commercially available wearable devices presents a potentially cost-effective, more accessible option for objective sleep measurement of large or geographically dispersed study populations, as well as enabling longer-term monitoring in patients.¹⁰ Consumer-wearable sleep trackers can provide new avenues of sleep health assessment for patients in clinical settings.¹¹ Like in research settings, sleep medicine tools like PSG have constrained periods of possible measurement, while consumer-wearables offer possibilities of sleep monitoring over longer windows that can be useful for chronic disease prevention. In some settings, providing patients with consumer wearables can be a part of care, as it gives patients a way to track their own health behaviors.¹² Additionally, tracking sleep over longer windows of time can be valuable for the promotion of health in specific populations such as women, who experience cyclical and lifetime hormonal changes that impact sleep.¹³

The development of mobile health technologies like wrist-worn sleep and activity monitors provides an opportunity to quantitatively assess sleep in ecologically valid and free-living (ie, non-laboratory) conditions on a larger scale and over longer time periods.^14,15 There has been a marked increase in use of commercial-grade wearables in NIH-sponsored research, including the All of Us Research Program and Risk Underlying Rural Areas Longitudinal Heart and Lung Study.¹⁶ However, their novelty and the proprietary nature of their product development and scoring algorithms raise questions about their interchangeability with research-grade accelerometers.¹⁷

In previous validation studies of mobile health sleep measurement tools, consumer wearables have demonstrated high levels of accuracy and sensitivity for correctly identifying sleep episodes compared to PSG but have been studied only in laboratory settings rather than free-living conditions that conform to everyday sleep.^18–23 In this study, we aim to quantify differences in sleep measurement by a consumer wearable and research-grade accelerometry in the healthy free-living female US-based adult population of the Nurses’ Health Study 3 Sleep and Physical Activity Validation Substudy.

Materials and Methods

Nurses’ Health Study 3 Sleep and Physical Activity Validation Substudy

The Nurses’ Health Study 3 (NHS3) is a prospective cohort of nurses and nursing students in the United States and Canada that began enrollment in 2010. Participants in NHS3 were born January 1^st, 1965 or later and fill out internet-based surveys on health and health behaviors approximately every six months. Within NHS3, the Sleep and Physical Activity Validation Substudy (SPAVS) was conducted with a goal of enrolling 50 female NHS3 participants, with data collection beginning in September 2023 and concluding in May 2024. In SPAVS, participants were asked to wear a research-grade accelerometer and a consumer wearable device concurrently on their wrist for fourteen consecutive days and to complete daily sleep logs on their smartphone using a research application called Beiwe.²⁴ The eligibility criteria for participation in NHS3 SPAVS was as follows: owned a smartphone, lived in the contiguous United States, had reliable access to Wi-Fi at least once a week, did not work night shifts during the collection period, had ability to wear wrist-worn devices 24-hours a day for 14 days, did not use sleep medications or have a sleep disorder, did not use a continuous positive airway pressure (CPAP) machine, was not pregnant, and had no health conditions that result in a tremor. For their participation in the study, participants were able to keep the consumer wearable. We sent study kits to 63 participants. We excluded participants with no days with simultaneous research-grade accelerometry, sleep diary, and consumer wearable sleep data (N = 12); this was due to device (ie, research-grade accelerometer, consumer wearable, or smartphone application) issues (N = 9), participant non-adherence with the protocol (eg, not wearing the research-grade accelerometer and consumer wearable on the same wrist) (N = 2), or never returning the research-grade accelerometry device (N = 1) (Figure 1). Our final analytic sample size comprised 47 participants.

Figure 1 Recruitment, data collection and derivation of the analytic dataset of 47 in the Nurses’ Health Study 3 (NHS3) Sleep and Physical Activity Validation Substudy (SPAVS).

Sleep Questionnaires and Sleep Diary

After completing an online screener and completing an online consent form, participants in SPAVS completed a baseline web questionnaire that included the Pittsburgh Sleep Quality Index (PSQI) and the reduced version of the Morningness-Eveningness Questionnaire (rMEQ); the PSQI and rMEQ questionnaires and scoring are accessible in their introductory publications by Buysse et al and Adan and Almirall, respectively.^25,26 At the beginning of the device data collection period, participants were asked to download the research smartphone application, which served as the platform to complete daily surveys. During the study period, participants completed an electronic sleep diary every morning, reporting their bedtime from the previous night and waketime. Participants also reported any naps taken in the previous day. Sleep diaries are often collected with research actigraphy, to assist with annotation. On the seventh day of the study period, participants filled out the Patient-Reported Outcomes Measurement Information System (PROMIS) Sleep-Related Impairment scale (Version 8a), whose contents are accessible in the publication by Yu et al, on the smartphone application; their scores were tabulated based on the cut-points provided by the Health Measures scoring service with the calibration sample set to default PROMIS Sleep Wave 1 (accessible from PROMIS Health Organization via https://www.assessmentcenter.net/ac_scoringservice).²⁷ As a backup in case, there were any issues with the smartphone application, participants were able to document their sleep periods on a provided paper sleep diary.

Research-Grade Accelerometer

Participants received wrist-worn research-grade accelerometry devices to wear on their non-dominant wrist for all 14 days of the study. We used GT3X+ and Actisleep models from Actigraph (Pensacola, FL) that were configured for 60-second epochs, with a sampling rate of 30 Hertz. The Actigraph GT3X+ and Actisleep are both models of actigraphy watch that have a tri-axial accelerometer; they primarily differ in additional research features (ie, light exposure sensor on the Actisleep) that were not utilized in this study. The actigraphy data were downloaded and scored using Actilife software (Actilife 6; Version 6.13.4); sleep periods were scored blinded to consumer wearable data. Sleep metrics were generated by inputting the start and end times of sleep periods from the sleep diaries into the scoring software, applying the Cole-Kripke algorithm, and exporting the resulting sleep period-level summary measures.²⁸

Consumer Wearable Device

Participants wore a Fitbit (San Francisco, CA) alongside the research-grade accelerometer on their non-dominant wrist for the study period with consistent wrist placement for all participants (Figure S1). Due to the device availability, both Charge 3 and Charge 5 models were used; among the 47 participants in the analysis, 9 participants wore a Charge 3, and 38 participants wore a Charge 5. “According to manufacturer, there are some differences between Charge 3 and Charge 5 devices; the Charge 5 has additional GPS capabilities, and electrodermal activity sensors, which measure sweat levels on skin to assess stress levels. However, no explicitly advertised differences in hardware or software for sleep estimation. Compared to the Charge 3, the Charge 5 has some additional sleep features for users, such as a sleep score metric and sleep mode; however, no use of sleep mode was included in our study protocol, and no sleep score measures were included in our analysis.^29,30 Sleep data were synced to and downloaded from Fitabase. Fitabase is a platform that supports the collection and management of data from their consumer wearable devices. We used Fitabase to monitor participants’ data on a daily basis and download their sleep data.

Statistical Analysis

All statistical analyses were conducted in R (Version 4.4.0; Vienna, Austria). Our inclusion criteria for sleep periods in this analysis were having concurrent valid research-grade accelerometer and consumer wearable measures for a given sleep period. We included the main sleep periods and naps in this analysis. We conducted sleep period-level comparisons for three sleep metrics: total sleep time (TST), time in bed (TIB), and sleep efficiency (SE). For TST, we compared the research-grade accelerometer’s measure of TST to the consumer wearable’s provided minutesAsleep measure. For TIB, we used the time elapsed from the bedtime and waketime reported in the sleep diary and the consumer wearable’s TimeInBed metric, for the research-grade accelerometer and the consumer-grade device, respectively. For SE, we used the research-grade accelerometer’s SE measure (TST*100/TIB) and calculated SE for the consumer wearable using their derived minutesAsleep and TimeInBed measures using the following formula: minutesAsleep*100/TimeInBed. For all three sleep outcomes, we calculated mean absolute percent error (MAPE) for each sleep period; the MAPE was calculated as the average of the absolute values of: (consumer wearable’s measure – research-grade accelerometer’s measure)*100/research-grade accelerometer’s measure. We also calculated intra-class correlations (ICCs), for which we specified a random intercept for participant due to the within-participant repeated measures of the study and an autoregressive correlation structure to account for correlation in adjacent days of observations. ICCs served as our sole measure that accounts for clustering attributable to longitudinal measurements by participant and consecutive days of observation. We used Bland-Altman analyses to calculate limits of agreement (LOA) for each measure, specifying a 0.95 significance level; thus, LOAs denote the range in which 95% of differences lie. The Bland-Altman analyses were calculated so that a positive mean difference value signified overestimation by the consumer wearable, and negative mean differences indicated consumer wearable underestimation. These differences in sleep metrics were also visualized using Bland-Altman plots.³¹ For the primary analyses, we pooled observations of the two models of consumer wearable (Charge 3 and Charge 5) but also conducted secondary analyses stratified by device to assess if data from a particular device model were driving our observed results. We also conducted stratified analyses by PSQI, rMEQ, and PROMIS sleep-related impairment score.

Results

SPAVS participants were female and predominantly white, with an average age of 40.4 years (SD = 6.9) (Table 1). There were no differences in demographic characteristics of participants who were and were not included in the analysis (Table S1). The average number of sleep periods contributed during the 14-day study period was 12.5, with a sleep duration and sleep efficiency of 7.2 hours and 90.2%, respectively, as measured by the research-grade accelerometer (Table 2). For the consumer wearable, the average sleep duration was 6.9 hours and average efficiency was 89.1%. There were no major differences between models of consumer wearable used in the study for demographics of participants in the study (Table S2), nor average TST, TIB, and SE (Table S3).

Table 1 Demographic Characteristics of 47 Female Participants in the Nurses’ Health Study 3 Sleep and Physical Activity Validation Substudy

Table 2 Sleep Period Characteristics of 47 Female Participants in the Nurses’ Health Study 3 Sleep and Physical Activity Validation Substudy

The MAPE and ICCs for consumer wearable compared to research-grade accelerometer and sleep diary measures are presented in Table 3. The MAPE for TST was 11.18%, with an ICC of 0.79. The MAPE and ICC for TIB were of similar magnitude, at 10.45% and 0.75, respectively. For SE, the MAPE was 5.09%, and the ICC was 0.39. In stratified analyses, we did not observe consistent patterns of differential agreement across strata of higher/lower PSQI scores, chronotype, or higher/lower PROMIS score for MAPEs or ICCs (Table 3). In sensitivity analyses by model of consumer wearable, we did not observe meaningful differences in MAPEs and ICCs (Table S4).

Table 3 Mean Absolute Percent Error and Intraclass Correlation for Total Sleep Time, Total Time in Bed, and Sleep Efficiency Overall and by Sleep Characteristics in the Nurses’ Health Study 3 Sleep and Physical Activity Validation Substudy

From the Bland-Altman analysis (Figure 2), we observed that the consumer wearable underestimated TST, TIB, and SE. TST had a mean difference of −16.0 minutes (95% LOAs: −143.2, 111.3). Similarly, in TIB, the mean difference was −11.2 minutes (95% LOAs: −167.0, 144.7 minutes). For SE, the mean difference of −1.0% (95% LOAs: −12.6%, 10.7%) indicated underestimation by the consumer wearable relative to research-grade actigraphy. In our Bland-Altman plots, we observed no clear visual differences in spread between the two models of consumer wearable included in the study (Figure S2). We did see a pattern of smaller biases for the Charge 5 device compared to the Charge 3 device (Figure S2). However, due to the smaller sample size of Charge 3 device users (N = 9), we have limited power to interpret this result as indicative of differential performance by device model.

Figure 2 Bland-Altman plots for total sleep time in Panel (A), total time in bed in Panel (B), and sleep efficiency in Panel (C) in the Nurses’ Health Study 3 (NHS3) Sleep and Physical Activity Validation Substudy (SPAVS).

Discussion

In this study, we compared measurements of TST, TIB, and SE from a consumer wearable with research-standard wrist actigraphy combined with sleep diaries in a population of healthy female adults. Overall, we observed high agreement between TST, TIB, and SE measurements from the consumer wearable and our research standard measures of actigraphy and daily self-report sleep diary, as well as a modest underestimating bias by the consumer wearable for all sleep outcomes.

Our findings of relatively small biases in TST and SE parallel prior studies examining sleep measures of similar consumer wearable models, such as products from the same company. Within NHS3 SPAVS and relative to wrist actigraphy, the consumer wearable underestimated the sleep metrics we investigated: TST by 16 mins and just short of 1% for SE. However, the calculated LOAs with 95% significance for all three outcomes included zero. The magnitude of differences we observed was comparable to other studies (ie, within a range of less than 20 minutes for TST, and less than 5% for SE and LOAs that similarly encompass zero), though there are mixed findings on whether this model series of consumer wearable devices overestimate or underestimate research standard measures. Some prior studies using PSG and using actigraphy also reported that consumer wearables of this series model underestimated the reference measure, with biases ranging from approximately 7 to 14 minutes for TST.^{19–21,32,33} However, some studies comparing other models of this consumer wearable to PSG and EEG reported the consumer-grade device overestimating TST, with a similar magnitude of bias (9 to 17 minutes).^18,22 Compared to TST, SE is understudied; our study contributes to the extant three studies, for which the SEs range from −4% to 4%.^20,22,33 Our study of healthy adult females supports the literature that suggests that there are no consistent patterns in the direction of bias nor meaningful differences in magnitude across types of study populations (eg, healthy adults, Huntington disease patients, adolescents), device models, or reference measures (ie, PSG or actigraphy).^{18,20,22,32,33} Several previous studies with models of this consumer wearable were in laboratory settings, and our findings from a field-based study design provide the insight that the magnitude of differences observed is like that of lab studies. Our study also contributes to the limited literature on TIB for this model series of consumer wearable. A study of consumer wearables from the same company compared with wrist actigraphy by McMahon et al found that the consumer-grade device underestimated TST and TIB by 71 and 77 minutes, respectively; however, this study had participants wear research-grade actigraphy watches and consumer wearable devices on opposite wrists, which may contribute to the observed differences in bias.³⁴

We also conducted stratified analyses to assess if there were differences in consumer wearable agreement between participants with different status across dimensions of sleep health. Within NHS3 SPAVS, we observed elevated PSQI and PROMIS scores in a majority (57% and 62% for PSQI and PROMIS, respectively) of participants; while we may describe this population as “healthy” regarding lack of sleep disorder diagnoses, these scores indicate poor sleep or sleep-related impairment. Based on the PSQI and PROMIS metrics, we saw that participants with poorer sleep quality and participants with greater sleep-related impairment had greater agreement in TST and less agreement in SE. From our stratified analyses, there did not appear to be consistent differential performance for “better” sleepers within a population of healthy adults. The consumer wearable’s consistency in performance by study population is additionally supported by the parity of findings for this model series of consumer wearables in sleep-disordered and sleep-impacted populations.^19,20

For the time-based sleep measures like TST and TIB, we observed a small number of observations (<4%) with meaningful discrepancies (ie, over 3 hours) between consumer wearable and the reference measure (research-grade accelerometry for TST, self-report diary for TIB), which we kept in the analysis; among these observations, there was an even distribution of observations in which the consumer wearable was overestimating and was underestimating. When we investigated these discrepant observations via visual inspection, we saw that the timing of onset of both TIB and TST periods agreed very well. These factors suggested that the discrepancies in these particular observations comes from when a consumer wearable device is either premature or delayed in detecting the end of a sleep period.

Across all three outcomes, although the LOAs include zero, their ranges are relatively wide, reflecting variation among the calculated biases of individual sleep observations. The wide LOAs suggest reduced precision, but the modest mean biases indicate that there is not a strong systematic bias, which reflects accuracy. The lower precision of the consumer wearable would introduce non-differential measurement error. In regards to use in clinical and epidemiologic applications, the implications of this measurement error depend on whether sleep is used as an exposure or outcome; for analyses with sleep as an exposure, potential effects would be attenuated towards the null, while for analyses with sleep as an outcome, there would be no introduced bias.

One of the central limitations of this study is intrinsic to studies using consumer wearable devices. Our findings are generalizable to observations from this consumer wearable model with the algorithms and firmware active at the time of data collection; since the consumer wearable’s algorithms and firmware are proprietary, we cannot definitively know if or when these software or firmware features were or may be updated in ways that impact the generation of sleep metrics. Another limitation of this study is that NHS3 SPAVS was an all-female study population, so we are unable to assess sleep-related sex differences in consumer wearable measurement. Additionally, our study sample is very racially homogenous, with a predominance of White participants. With the limited demographic diversity of our study sample, we are unable to assess how the performance of the consumer wearable relative to research-grade accelerometry may vary in other populations. Due to the proprietary nature of the sleep estimation process, it is difficult to speculate on whether certain behaviors or characteristics like skin tone may impact the accuracy of sleep measurement.

The strengths of this study lie within the study design. First, we were able to study individuals in free-living conditions, providing insights applicable to future epidemiologic studies of sleep that have non-laboratory conditions. Second, we were able to conduct our studies in a group of nearly 50 participants, with 14 days of longitudinal measurements, which augmented the power of our findings. Relative to previous studies, this study design has increased power from a larger sample size (N = 47) and longer study period (14 consecutive days); prior studies comparing sleep measures from research-grade accelerometers and consumer wearables of free-living adults, the study sample size has been approximately 20 participants who were followed for 3 days.^35,36

Consumer wearables often provide measures of sleep staging, in addition to the metrics evaluated in this study. Rather than using research-grade actigraphy in free-living conditions as a reference, sleep staging measures are compared to PSG measures in laboratory studies. Sleep staging measures from this model series of consumer wearable have been compared to PSG in several smaller scale studies that report mixed results indicating moderate accuracy.^18,19,33,37 Since we rely on the insights of studying a larger group of individuals in conditions representative of daily life to determine a sleep metric’s robustness, it can be more challenging to evaluate the suitability for use in large scale studies for these sleep staging measures.

Consumer wearables represent exciting opportunities to advance epidemiologic studies of sleep and expand assessment in sleep medicine but require careful consideration for integration into research settings.³⁸ Ideally, the process of evaluating consumer wearables for research comprises three stages: first, a laboratory-based study with comparison to PSG, then, a field-based study with comparison to a validated measure (ie, actigraphy), and finally, studies of specific subpopulations.³⁹ In context of the several extant studies comparing models of this consumer wearable to PSG in lab settings, this study contributes to the advancement of the second stage by characterizing differences from research standard measures in non-laboratory conditions in a well-defined sample.

Conclusion

The emergence of mobile health wearable technologies like wrist-worn sleep and physical activity tracking devices has implications for increasing feasibility of sleep data collection; these data can be used in population-level sleep studies or in clinical settings to assist in monitoring patient sleep health. Our comparison of a consumer wearable and wrist-worn actigraphy among free-living healthy adult females demonstrates that generally high agreement is retained in non-laboratory conditions. Sleep insights into a female population can be valuable for advancing women’s sleep health; women’s sleep has specific social and biological considerations, such as social obligations from gender roles for women that shape their sleep opportunities as well as biological considerations like hormonal cycles that impact sleep physiology for females. More broadly, improving our understanding of how consumer wearable sleep monitors in free-living conditions facilitates their implementation into sleep medicine as well as into population-level studies, which are becoming an integral part of future investigations of the links of various epidemiologic exposures including environmental, contextual, and psychosocial factors with sleep.