Exploratory Validation of Sleep-Tracking Devices in Patients with Psychiatric Disorders

Introduction

Assessment of sleep problems is an important clinical component of psychiatric treatment, as sleep problems coexist with a high percentage of psychiatric disorders. A cross-sectional study reported insomnia symptoms in 50% of patients with schizophrenia spectrum disorders and generalized anxiety disorder and in more than 80% of patients with depressive disorder.^1–3 In addition, sleep disorders in psychiatric conditions are not limited to the acute phase of the disease but could also persist for longer periods of time. This could impact the patients’ quality of life, leading to impaired cognitive and social functioning, and increasing relapse and recurrence. In the STAR*D trial, which was conducted in the United States, insomnia was the most common residual symptom among patients with major depression who responded to antidepressant therapy but did not achieve remission. It was observed in 94.6% of these patients.⁴ A study evaluating quality of life measurements reported that patients with post-remission residual insomnia exhibited lower physical and mental daily role functioning compared to those without insomnia.⁵ An association has also been reported between sleep disturbances and the risk of recurrence of depressive disorders, bipolar disorders, and schizophrenia, suggesting that maintaining good sleep status is important for good clinical outcomes.^6–12 Alternatively, cases of sleep state misrecognition have also been described, in which subjective insomnia symptoms persist despite objective sleep maintenance.¹³ Insomnia frequently persists after the psychiatric disease that triggered the sleep problems has resolved, hence becoming the main symptom and primary theme in these patients’ lives. Prolonged experiences of insomnia could lead to escalated anxiety. Thus, it is necessary to understand and evaluate both subjective and objective symptoms when assessing sleep in patients with psychiatric disorders.

In particular, it is essential to observe the relationship between psychiatric symptoms and sleep architecture in studies of the pathophysiology and clinical outcomes of mental disorders. For example, previous polysomnography (PSG) studies have reported rapid eye movement (REM) sleep abnormalities and alterations in slow wave sleep in schizophrenia patients, which are associated with psychiatric symptoms.¹⁴ It has also been reported that PSG REM sleep abnormalities are found in patients with depressive disorder and change after drug treatment.¹⁵ Continuous measurement of these sleep abnormalities by wearable devices is valuable, but the validity of the measurements needs to be verified.

The recent development of wearable sleep-tracking devices has made it possible to measure sleep parameters and sleep stages easily at home. Various types of devices that measure sleep, including wristband-type, ring-type, and under-mattress devices, are available. Portable electroencephalograph (EEG) devices that can be used to measure sleep at home are also being further developed. The American Academy of Sleep Medicine (AASM) stated that validation of device accuracy is needed for these devices to be considered part of medical evaluation and treatment.¹⁶

PSG is the main reference used to validate the performance of sleep-tracking devices. Several studies have demonstrated its use in healthy subjects. The validity of sleep-wake determination with sleep-tracking devices has been reported to be higher than that with actigraphy, which has been used to measure sleep at home.^17,18 On the contrary, few studies have focused on the performance of sleep-tracking devices in patients with psychiatric illnesses, and this remains to be elucidated. Patients with psychiatric disorders have different sleep architecture compared to healthy subjects.^19,20 Since consumer sleep technology, such as Fitbit, evaluates sleep based on machine learning using healthy subjects as training data, the performance of these devices may be reduced when patients with psychiatric disorders use them. We therefore proposed an exploratory evaluation of the sleep-tracking performance of three wearable devices against PSG in patients with psychiatric disorders and aimed to compare performance across the devices. We tested a portable EEG and two devices relying primarily on an accelerometer and hypothesized that the portable EEG would show better agreement than the other devices.

Methods

The flow diagram of the study is shown in Figure 1.

Figure 1 Flow diagram of the study. Notes: Fifty-two patients aged 16 years or older, diagnosed with schizophrenia, depressive disorder, bipolar disorder, or sleep disorder, who were admitted to Akita University Hospital for a sleep study between December 2020 to November 2021 were included in the study. Three sleep-tracking devices were tested with PSG as a reference: Sleepgraph, Fitbit Sense, and MTN-220/221. PSG recordings were performed for two nights in 26 patients who gave consent. When two nights of data were available, the first night was used in the epoch-by-epoch analysis and Bland-Altman analysis, and the association of accuracy between the two nights was analyzed using Spearman correlation coefficient.

Participants

Fifty-two patients (20 males, 32 females) aged 16 years or older (mean age 48.1 ± 17.2 years, range 16–83 years) diagnosed with schizophrenia, depressive disorder, bipolar disorder, or sleep disorder who were admitted to Akita University Hospital for a sleep study between December 2020 to November 2021 were included in the study. The admissions were specifically for sleep tests and were of short duration. The diagnosis of psychiatric disorders was established by clinically experienced psychiatrists based on the Diagnostic and Statistical Manual of Mental Disorders-5 (DSM-5). Schizophrenia was diagnosed in 18 patients, depression in 19 patients, bipolar disorder in 3 patients, and sleep disorder in 12 patients. Exclusion criteria were as follows: patients under 16 years of age, and patients whose participation in the study was deemed inappropriate by the treating physician (eg, due to anticipated worsening of medical condition). Since sleep apnea is a common comorbidity in psychiatric patients, we did not exclude these patients.^21,22 Data regarding body mass index (BMI) and the medications taken by the patient on the day of recording were collected. Medications were compared and classified: chlorpromazine equivalents for antipsychotics, imipramine equivalents for antidepressants, and diazepam equivalents for sleeping pills and anti-anxiety medications.²³ All patients were considered fit to provide informed consent for participation in research by their treating physician. Informed consent was obtained prior to participation in the study, and all patients received compensation. Parental or legal guardian consent was obtained for all study participants under the age of 18 years. The Ethics Commission of the National Center of Neurology and Psychiatry, set up as the Central Ethics Review Board, approved all the study protocols and experimental procedures (approval no. B2019-001). The ethics commission of the Akita University gave permission to conduct the study. The study was conducted in accordance with the Declaration of Helsinki.

Sleep-Tracking Devices

Three sleep-tracking devices were tested: Sleepgraph (Proassist, Ltd.; Osaka-shi, Osaka, Japan),²⁴ Fitbit Sense (Fitbit, Inc.; San Francisco, CA, USA),²⁵ and MTN-220/221 (Kissei Comtec Co., Ltd.; Matsumoto-shi, Nagano, Japan).²⁶ The basic characteristics of each device have been demonstrated in previous studies. The Sleepgraph is a portable EEG device, which requires the attachment of electrodes at four locations: right frontal region, below the left eye, right mandible, and left mastoid process. EEG data were recorded at two channels: right frontal (Fp2)-mastoid (M1) and left ocular (EOG)-right mandible (EMG). Each sleep EEG was scored by a technologist skilled in the evaluation of minority-channel sleep EEG. The same technologist also scored the PSG. The sleep technologist was blinded to the study design. Sleep stages by Sleepgraph were used to score sleep at four stages, in accordance with the AASM criteria for PSG scoring.²⁷ However, N1 and N2 were integrated as light sleep to compare scoring with the data obtained from Fitbit Sense that scores at three stages: light sleep, deep sleep and REM sleep. Fitbit Sense is a device worn on the wrist that uses machine learning to detect sleep using acceleration sensors and heart rate estimations obtained by photoplethysmography (PPG).²⁵ It was worn on the non-dominant arm and measured simultaneously with PSG. MTN-220/221 is an accelerometer worn on the waist. It determines sleep/wake status based on the activity intensity calculated from the activity value.²⁶ Epoch duration for sleep data output by Sleepgraph and Fitbit Sense was 30 seconds. However, MTN-220/221 only produced output 2 minutes of sleep data per epoch.

PSG Recording and Scoring

PSG recordings were acquired using a digital PSG recorder (Somno HD; Fukuda Denshi, Bunkyo-ku, Tokyo, Japan or Polymate Pro; Miyuki giken, Bunkyo-ku, Tokyo, Japan) in a hospital room. PSG sleep stages (N1, N2, N3, and REM) and wake status were manually scored by a polysomnographic technologist certified by the Japanese Society of Sleep Research. They were scored at another facility in accordance with AASM manual.²⁷ Light-off time and light-on time were determined based on position sensors. PSG recordings were performed for one night on all patients, and for two consecutive nights on 26 patients who gave consent.

Recordings were started at 0 or 30 seconds for comparison with other sleep-tracking devices. PSG data were scored in 30-s epochs and were compared epoch-by-epoch (EBE) with sleep data obtained from Sleepgraph and Fitbit Sense devices. To align with the MTN-220/221 sleep data, the PSG data were additionally scored in 2-minutes epochs according to the following rule: If there were two or more occurrences of stage wake in four continuous epochs, this was defined as wake.

Statistical Analysis

For demographic data and data following a normal distribution, multiple comparisons were performed by one-way analysis of variance. The Scheffe method was used for between-group comparisons. The Steel-Dwass method was used for data not following a normal distribution.

Analysis of the sleep data was performed using the open source R code by Menghini et al (https://github.com/SRI-human-sleep/sleep-trackers-performance).²⁸ The following EBE agreement statistics were calculated between PSG and each individual device for every night; sensitivity (the percentage of epochs scored as sleep by PSG that were also detected as sleep by the device), specificity (the proportion of epochs scored as wake by PSG that were also detected as wake by the device), positive predictive value (PPV; the proportion of epochs detected as sleep by the device that were also scored as sleep by PSG), negative predictive value (NPV; the proportion of epochs detected as wake by the device that were also scored as wake by PSG), accuracy (proportion of all PSG epochs correctly detected by the device), and bias-adjusted kappa (PABAK; Cohen’s kappa weighted to account for the amount of inequality between the number of sleep and wake epochs). For the Sleepgraph and Fitbit Sense devices, an error matrix was created that exhibited sensitivity and misclassification for each sleep stage. The mean, standard deviation, and 95% confidence interval for each parameter were calculated. The following sleep parameters were calculated for each device; total sleep time (TST; total time spent in all sleep stages), sleep efficiency (SE; percentage of TST divided by time in bed), sleep latency (SL; time taken to fall asleep from bedtime), and wake after sleep onset (WASO; total time awake after sleep onset), and REM latency (REML; time from sleep onset to first determination of REM). The light off/light on time for each device was set at the same time as the PSG. Sleep parameters were subjected to Bland-Altman analysis to estimate systematic and chance errors between PSG and each device. Bland-Altman plots were created to visualize discrepancies between the PSG and sleep measurement devices. For patients whose sleep data were recorded over two nights, the association of accuracy between the two nights was analyzed using Spearman correlation coefficient. All analyses were conducted by R version 4.2.0 (R Foundation, Vienna, Austria, 2022). Significance level was set at p<0.05.

Results

Fifty-two patients participated in the study, with a combined total of 80 nights of recorded sleep data. After excluding total insomnia (1 sleep data) and incorrectly set PSG recording time (1 sleep data), PSG data for 78 nights in 52 patients was included in the analysis. Data loss occurred due to reasons such as failure to wear the device or loss of the device: Sleepgraph (15, 19.2%), Fitbit Sense (11, 14.1%), and MTN-220/221 (12, 15.4%). PSG recording was performed for two nights in 26 patients who gave consent. When two nights of data were available, the first night was used in the analysis. The disease distribution and characteristics of the participants are demonstrated in Table 1. The patients included in the depression group were significantly older than the patients included in the schizophrenia and sleep disorder groups. Individual clinical information and medications at the time of study participation are listed in Supplementary Table S1.

Table 1 Participants’ Demographic Data

Epoch-by-Epoch Analysis

Table 2 and Table 3 demonstrate the results of the EBE analysis. In the sleep-wake assessment, the sensitivity was high for all devices, and the specificity of Sleepgraph was higher than that of Fitbit Sense and MTN-220/221. In the EBE analysis of sleep stages, Sleepgraph’s REM sleep demonstrated high sensitivity and specificity, while the sensitivity of deep sleep and light sleep was relatively low. Fitbit Sense exhibited lower sensitivity than Sleepgraph for all sleep stages. The specificity of Fitbit Sense was high for deep sleep and REM sleep, but low for light sleep. Fitbit Sense over-judged light sleep and demonstrated decreasing sensitivity for other sleep stages. The error matrix is displayed in Table 4. Sleepgraph incorrectly classified stage of wake and deep sleep as light sleep, and Fitbit Sense also incorrectly classified stage of deep sleep and REM sleep as light sleep.

Table 2 Epoch-by-Epoch Analysis for Sleep/Wake Detection

Table 3 Epoch-by-Epoch Analysis for Sleep Stages Detection

Table 4 Error Matrix of Epoch-by-Epoch Analysis

Sleep Parameters

Bland-Altman plots of sleep parameters are demonstrated in Figure 2A–E. Detailed results of Bland-Altman analysis are displayed in Supplementary Tables S2 and S3. The TST measured by Sleepgraph exhibited a low mean difference from PSG. Alternatively, Fitbit Sense and MTN-220/221 demonstrated a proportional bias, overestimating TST on nights when PSG TST was low. SE, SL and WASO demonstrated a proportional bias for Fitbit Sense and MTN-220/221, overestimating SE and underestimating SL and WASO in cases of low SE and long SL and WASO as determined by PSG, respectively. REML exhibited a proportional bias with Sleepgraph and Fitbit Sense, underestimating REML in cases of long PSG REML.

Figure 2 Bland-Altman plots. Notes: Plots depict the mean bias (solid red line) and upper and lower limits of agreement (solid gray lines) for deviation in sleep parameters for the devices compared with the reference PSG. Black circles depict individual nights. Dashed lines represent the 95% confidence intervals around the bias and limits of agreement lines. Gray shaded regions on the right y-axis are density plots demonstrate the distribution of individual night biases. Zero on the y-axis represents no difference, with positive and negative y-axis values indicating an overestimation or underestimation, respectively, compared with the reference. Diagonal mean bias lines indicate significant proportional bias. Non-parallel limits of agreement lines indicate significant heteroscedasticity.

Intra-Subject Comparison

A positive correlation for all devices was observed when the accuracy of PSG and each device was compared for the same patient, with measurements obtained over 2 nights (Figure 3, Sleepgraph: r = 0.78, p<0.001; Fitbit Sense: r = 0.70 p<0.001; MTN-220/221: r = 0.68, p<0.01).

Figure 3 Plots of accuracy within the same patient measured over 2 nights. Notes: The x-axis indicates the first night’s accuracy, and the y-axis indicates the second night’s accuracy. Higher values for accuracy indicate higher agreement of sleep/wake detection.

Discussion

In the present study, we validated the performance of sleep tracking devices by using PSG as a reference. This allowed us to directly compare the validity of each device’s sleep assessment in patients with psychiatric disorders. The sleep-tracking devices tested in this study demonstrated high sensitivity and moderate to high specificity in determining sleep/wake. Among the devices assessed, Sleepgraph, which is a portable EEG device, exhibited higher specificity than the other devices and was able to reliably determine sleep/wake in patients with psychiatric disorders. MTN-220/221 demonstrated a relatively low specificity, indicating that it underestimates wake status.

In studies validating sleep tracking devices for healthy adults,¹⁸ the sensitivity of actigraphy, which determines sleep using only an accelerometer, was 0.91 to 0.96 and the specificity was 0.34 to 0.63. The sensitivity of devices that utilize PPG in addition to using accelerometers such as the Fitbit, Apple Watch, and Oura Ring is reported to be 0.93–0.98, and the specificity is 0.41–0.69. In the present study, Sleepgraph, Fitbit Sense, and MTN-220/221 validated in patients with psychiatric disorders also demonstrated similar or higher agreement than that of actigraphy. Our results support the at home use of these wearable devices for determining sleep/wake for patients with psychiatric disorders.

In the EBE analysis of sleep stages, both devices often misclassified other sleep stages as light sleep (Table 4). In a given sleep period, REM sleep, light sleep, and deep sleep periods compete with each other. Therefore, the sensitivity and specificity of each sleep stage in a device may be related, and each device may have certain associations, such as a higher sensitivity for REM sleep and a lower specificity of light sleep. Fitbit has been found to over-score light sleep in previous study, which may be a feature of the scoring algorithm.²⁵

The performance of the Fitbit Sense in determining each sleep stage in psychiatric patients was relatively low compared to the results of previous study conducted in healthy adults.¹⁷ Most wearable devices developed in the recent years use data obtained from heart rate variability (HRV) and acceleration sensors to determine sleep status. In healthy adults, HRV is useful for estimating sleep stages.²⁹ HRV information is also used in Fitbit’s sleep assessment.²⁵ There are several factors in patients with psychiatric disorders that physiologically affect the functional correlation between sleep stage and HRV. For example, HF power, which reflects parasympathetic activity in HRV analysis, is decreased in patients with depression and schizophrenia. It is also associated with disease severity.^30,31 In addition, some antidepressants, such as tricyclic antidepressants, modify HRV through their anticholinergic and other effects.³¹ Although the characteristics of training data and features of classifiers on sleep stage of Fitbit Sense determination have not been disclosed, the accuracy of sleep stage determination was measured in this study using existing algorithms and found to be lower for patients with psychiatric disorders, as compared with studies that included healthy subjects. This could have resulted from the clinical factors modifying the HRV parameters. However, we have not included a healthy comparison group, so this hypothesis needs to be tested in future studies.

The performance of all devices in measuring sleep parameters decreased with poor sleep: short TST, low SE, long SL, and long WASO. Similarly, low sleep efficiency decreased the validity of sleep parameters in a validation study that included healthy adults.³² REML demonstrated a proportional bias for both devices, and a study comparing PSG and Fitbit Alta HR in patients with hypersomnia exhibited a similar trend.³³ A meta-analysis of PSG for psychiatric disorders reported that patients with depression and schizophrenia had lower TST and SE and longer SL and WASO compared with healthy controls.³⁴ Participants in the present study included patients with poor sleep quality, which may have reduced the validity of sleep parameters related to sleep/wake.

Intra-subject accuracy of PSG and each device was positively correlated for all devices. This suggests that for patients with low sleep-wake accuracy on the first night, it is also low on the second night. Patient-related factors led to the reduction in accuracy.

Sensor accuracy is a major concern in consumer sleep technology, as inaccurate data and inaccurate recommendations for improving sleep from sleep-tracking devices have been reported to cause orthosomnia.³⁵ Compared with Fitbit Sense and MTN-220/221, Sleepgraph demonstrated higher accuracy and greater sensitivity and specificity in determining each sleep stage, even in patients with psychiatric disorders. It also had a higher agreement rate for sleep parameters. This indicates the superiority of using a portable EEG device with a small number of channels for determining sleep quality in psychiatric patients. It is also less burdensome than PSG and can be performed at home. A study comparing portable EEG and actigraphy in healthy adults also demonstrated that portable EEG was more reliable for assessing sleep parameters, similar to our results, although different portable EEG devices were used.³⁶ Portable EEG devices that can determine sleep using automatic determination are now available. They have been reported to be highly reliable in validation studies including psychiatric patients.³⁷ These EEG-based sleep assessment devices are expected to facilitate clinical and epidemiological studies that seek to assess natural sleep over consecutive nights in the home setting in patients with psychiatric disorders.

This study had the following limitations: (1) As the measurements were obtained in a laboratory, they do not reflect the validity of measurements acquired in a natural environment, such as at home. The hospital environment may reduce the validity of these assessments by being more disruptive than normal everyday conditions. (2) As with many previous studies evaluating device performance, there were measurement failures for each device and not all measurements could be analyzed. (3) Due to the lack of drug restrictions, EEG and HRV could be impacted by pharmacological effects. (4) Consumer sleep technologies such as Fitbit may update their device models, apps, and sleep assessment algorithms, so these results only correspond to the versions available during the study period.

Conclusion

Our validation results indicate that sleep-tracking devices can be a promising tool for objective sleep state assessment in patients with psychiatric disorders. However, the validity of these devices may be compromised in patients with psychiatric disorders due to factors that affect the wearable device’s sleep assessment algorithm, such as changes in autonomic nervous system activity and drugs/medications taken by the patient. Sleepgraph, a portable EEG device with a few channels, demonstrated higher validity in sleep assessment than actigraphy and the Fitbit Sense.