The Accuracy and Reliability of Sleep Staging and Sleep Biomarkers in Patients with Isolated Rapid Eye Movement Sleep Behavior Disorder

Introduction

Isolated rapid eye movement (REM) sleep behavior disorder (iRBD) is a parasomnia recognized as a prodromal biomarker for alpha-synuclein pathologies, including Parkinson disease, dementia with Lewy bodies, and multiple system atrophy.^1,2 iRBD has also been variably associated with progressive supranuclear palsy, a tauopathy. Dream enactment behaviors, characterized by complex vocal or motor behaviors during REM sleep, are the primary iRBD symptom that leads to its recognition. iRBD-related dream enactment ranges from subtle hand gestures to violent punching and kicking that may lead to self or bed partner injury.^1–4 Within a 10- to 15-year period after diagnosis of iRBD, approximately 70–75% of patients develop a defined neurodegenerative disease.^1,4 Between 40% and 50% of patients with Parkinson’s disease also have RBD, particularly those who are older with a longer disease duration, and who exhibit greater disease severity and/or more severe motor fluctuations.⁵

Despite the injury potential and neurodegenerative consequences, many iRBD patients are unaware of their dream enactment symptoms,⁶ leading to delayed diagnosis. Over 1% of the adult community population has iRBD with the prevalence increasing with age.⁷ The ratio iRBD is 9 to 1 in men versus women;⁸ however, IRBD is likely underdiagnosed in women, possibly a result of more subtle and less violent dream enactment behaviors.^9,10 It is common for clinicians to overlook the relatively frequent occurrence of REM sleep without atonia (RSWA) without clinical dream enactment behaviors, which can be a harbinger of future iRBD or other manifest synucleinopathies.^11,12

Diagnosis of iRBD is based on identification of its neurophysiologic signature, RSWA. In current practice, patient reports of dream enactment behaviors should be promptly referred for in-laboratory video-polysomnography (PSG) for the assessment of RSWA. During routine PSG, RSWA can be challenging to recognize in the setting of elevated submental muscle activity associated with obstructive sleep apnea and antidepressant medications.^13,14 In patients with suspected neurodegenerative disorders, the staging of sleep can be especially challenging due to elevated muscle activity and reduced spindle activity, which makes accurate stage distinction difficult.^15,16 Additionally, scoring of RSWA requires specialized training and is laborious and time intensive.^17,18 To date, estimates of the prevalence and clinical significance of iRBD have been limited by the difficulty and cost of screening larger community samples with PSG.^6,7,19

This study was designed to evaluate the comparative accuracy of Sleep Profiler (SP) sleep staging to the simultaneously acquired gold standard laboratory PSG. We investigated the agreement between PSG and SP sleep staging in records which contained excessive muscle activity during REM that can confound automated sleep staging. We also explored the diagnostic concordance between iRBD and non-REM hypertonia (NRH) a SP biomarker found to be significantly associated with Parkinsonian spectrum neurodegenerative disorders independent of age, sex and antidepressant medications.¹⁶ NRH was discovered in the frontal electroencephalography signal during what visually appeared to be stage N2 or N3 sleep, but with periodic, persistently elevated EMG power relative to delta, theta, and sigma bands.

Materials and Methods

Patients with a history of dream enactment behavior (DEB) underwent a diagnostic PSG. Institution Review Board (IRB) approvals were obtained by the Mayo Clinic and by the University of Arizona for the Banner University Health Center site. Patients enrolled under approval and informed consents were obtained prior the PSG at Site-1 (six women and 10 men with a mean age of 64.6+13.0 years) and Site-2 (one woman and nine men, age 63.2+12.7 years).

During the PSG, SP recordings were simultaneously acquired from electroencephalography (EEG) sensor sites AF7-AF8, AF7-Fpz and AF8-Fpz and labeled EEG, LEOG and REOG, respectively. Chin electromyography (EMG) was recorded with sensors placed near those used to acquire the PSG EMG, however the signal was only used for technical review, ie, it was not needed for auto-staging. Figure 1 shows a patient wearing the SP with chin EMG sensors.

Figure 1 Patient wearing the Sleep Profiler with chin EMG.

The SP records were synchronized with the PSG and then auto-staged using previously described and validated machine learning algorithms that apply within-epoch temporal power spectral characterization with combined detection of individual slow waves, sleep spindles and cortical arousals.²⁰ Following auto-staging, the SP records were technically reviewed for final sleep stage assignments. Corrections were directed toward epochs auto-staged wake, N1, and REM based on rules described in the manual for technical review of records of patients with neurodegenerative disorders.²¹ The most common edits were to convert epochs auto-stage wake as a result of arousals from sleep disordered breathing to N1, and to stage REM in epochs with EMG intrusions resulting from RBD. No corrections were made to change auto-staged N3 to N2 or N2 to N3.

The PSG records were scored at each site by a registered sleep technologist experienced in staging iRBD patient studies according to the American Academy of Sleep Medicine (AASM) criteria.²² PSG and SP records were initially sleep staged while blinded to their respective counterpart. RSWA events from both sites were scored by one technician, with consideration taken for untreated obstructive sleep apnea.

After the Kappa scores were computed for the initial staging, each site then reedited the PSG studies for stage N3. At Site-1, only the PSG epochs that conflicted with SP staging of N2 and N3 were reedited. At Site-2, the technologist reedited the PSG records with greater attention given to consistent application of the stage N3 slow-wave amplitude, frequency, and duration criteria,²² while blinded to the SP results.

NRH was detected using automated algorithms designed to detect the distinctive pattern of periodic and persistent elevation in EMG power relative to delta, theta, and sigma bands in the frontal EEG. The characteristics of the NRH signal pattern are visually depicted Figure 2. A detailed description of algorithms developed for auto-detection of NRH episodes and the association between abnormal NRH and Parkinsonian spectrum disorder subgroups were previously described.¹⁶ Briefly, the automated detection of NRH utilized variability thresholds applied to each 30-second epoch to ensure EMG bursts attributed to sleep disordered breathing arousals were not mischaracterized as NRH. A NRH block required four of six contiguous epochs to have satisfied the threshold criteria. Finally, NRH blocks were extended to link blocks with <2-second gaps. The percent-time NRH was based solely on auto-detected blocks with no corrections made to add or remove NRH. The duration of auto-detected NRH was tallied and scaled relative to total sleep time (ie, proportionally shared with stages N1, N2, N3 and REM). Abnormal NRH was based on the empirically derived threshold >5% of sleep time.¹⁶

Figure 2 (a) A 1-hour epoch presenting the Chin EMG, LEOG, REOG and EEG signals, with the relative EEG power spectral distributions for alpha (pink), sigma (green), beta (brown) and EMG (black). The increase in relative EMG power corresponds with two reassigned to stage NRH. (b) Shows one-half of a 30-sec epoch without visual evidence of NRH. Panel c illustrates NRH with visually apparent fast frequency activity evident in the LEOG, REOG, and EEG channels. The dashed lines and arrows highlight the average relative EMG power extracted from the EEG signal in 15-sec panel without and with NRH.

Unweighted positive and negative percent agreements (PPA and NPA) and Cohen’s kappa coefficients (k) were used to assess staging concordance and Chi-square analysis was used to identify between-site staging differences.

Two boarded sleep neurologists independently characterized iRBD by visually inspecting the PSG records for abnormal qualitative RSWA in the submental, arm and leg EMG channels. Kappa scores, PPA, NPA, and positive and negative predictive values (PPV and NPV) were used to assess iRBD detection agreement among the physicians, and between the physicians and abnormal NRH.

Results

Agreement in Sleep Staging

In total, 8.8% of the SP auto-staged epochs were corrected during technical review. The proportional distributions of auto-staged epochs that were corrected included Wake = 1.7%, N1 = 41.4%, N2 = 2.7%, N3 = 0.6%, and REM = 28.1%. Significant differences were observed in the proportions of SP auto-scored epochs that were corrected in the Site-1 and −2 records for N1 (28.0% vs 52.4%) and N2 (1.8 vs 4.0%)(both P<0.01).

After the PSG records were reedited, the percent agreement across all staged epochs at Sites-1 increased from 74% to 79% and at Site-2 from 67% to 73%, while the kappa scores increased from 0.65 to 0.71 and from 0.59 to 0.66, respectively.

Table 1 presents the PPA and NPA values observed by site and stage, before and after PSG reediting for stage N3. The agreement between the initial PSG staging and SP were significantly different at Sites-1 vs −2 for Wake (k=0.82 vs 0.65), N1 (k=0.15 vs 0.23), N2 (k= 0.63 vs 0.72), N3 (k=0.72 vs 0.37) and REM (k=0.83 vs 0.72)(all P<0.001). The divergence in kappa values from Sites-1 vs −2 for stage REM was attributed to differences in the proportion of PSG-REM epochs staged SP-N2 (8.5% vs 16.8%, respectively, P<0.001). A greater proportion of epochs staged N1 by PSG were staged SP-Wake at Sites-1 when compared to Site-2 (41.6% vs 19.8%, P<0.001). Conversely, the Wake kappa value was superior at Site-1 as a result of fewer PSG-Wake epochs staged SP-N1 (6.6 vs 15.6%, P < 0.001) or SP-N2 (5.9 vs 12.0%, P<0.001).

Table 1 Positive and Negative Percent Agreement (PPA and NPA) by Sleep Stage and Site, Before and After Reedits

After PSG reedits, the kappa values were substantial or greater at Sites-1 and −2 for stage N2 (k=0.71 and 0.78) and stage N3 (k=0.91 and 0.74). Significant differences were observed at Sites-1 and −2 in the proportion of PSG epochs that were initially staged N2 and reedited to N3 (6.5% and 1.7%), or initially staged N3 and reedited to N2 (15.9% and 47.9%)(both P<0.001). After reedits, Site-1 had significantly fewer PSG = N3 and SP = N2 conflicts as compared to Site-2 (5.6% vs 20.8%, P<0.001).

Agreement in iRBD Characterization

The agreements between sleep neurologist determined RSWA and abnormal NRH are presented in Table 2 using pooled comparisons from 14 Site-1 records with polysomnography-determined REM sleep and all Site-2 records. Based on the kappa scores, fair agreement was observed between a) the two-sleep neurologist’s reporting of abnormal RSWA, and b) each physician’s iRBD diagnosis and abnormal NRH. The overall classification accuracies for the three comparisons were each 0.67; however, the between-physician PPA was stronger and NPA weaker as compared to each physician’s iRBD diagnosis and abnormal NRH. There were no significant differences in the number of women’s records that physicians agreed (3/16) vs disagreed (3/8) on the presence of RSWA (P=0.62), nor in proportion of records by site in which the physicians agreed vs disagreed in the presence of RSWA (Site-1: agreed = 63%, Site-2: agreed = 50%, P=0.67).

Table 2 Agreement in the Characterization of iRBD Based on Physician-1 and 2’s Detection of REM Sleep Without Atonia and in Comparison to Abnormal Non-REM Hypertonia (NRH)

Abnormal NRH was observed in 63% of the records. The NRH classification (ie, normal or abnormal) corresponded with at least one of the sleep neurologist’s iRBD diagnosis in 83% of the records. NRH was normal when both physicians observed RSWA in three records, and in one record NRH was abnormal when RSWA consensus was normal. Each sleep neurologist characterized two non-overlapping cases as “borderline” which was classified normal for iRBD. The borderline classifications corresponded with normal NRH in three of the four cases. NRH was observed in one of the two Site-1 records with insufficient PSG-based REM time to obtain an iRBD diagnosis.

Discussion

In records with suspected iRBD, perfect agreement was achieved between PSG and manually corrected SP for stage REM at Site-1, and substantial agreement at Site-2. The PPA and NPA for REM at Sites-1 and −2 were superior and equivalent to the accuracy achieved in comparison to majority agreement among human scorers in patients with sleep disordered breathing²⁰ and/or taking medications that impact sleep architecture.¹⁸ Across all records, 9% of auto-staged epochs were corrected using the full-disclosure recordings in combination with presentation of the relative alpha, sigma, beta and EMG power spectral characteristics and identification of epochs that likely needed to be inspected. Due to the atypical EMG activity associated with iRBD patients, 28% of the epochs ultimately tallied as REM resulted from manual corrections of the auto-staging.

After PSG reedits, the agreement with SP for stage N3 was perfect at Site-1 and substantial at Site-2. The discrepancies in N3 epochs after the initial staging confirmed the difficulty sleep technologists face in differentiating stages N2 and N3. In the AASM inter-scorer reliability program, the agreement across users in N3 staging of PSG records was only 63%²³. In a second study, the proportion of total PSG-based epochs staged N3 across 47 records by five blinded raters ranged from 7.5 to 26.5%.²⁰ In this study, results from the initial PSG N3 staging by a single rater triggered a second review with more careful and consistent application of the AASM N3 scoring rules. After reedits, the Site-2 PPA increased by more than 35% with almost 50% of the epochs initially staged N3 reedited to N2. Directing the technologist to review epochs that disagreed with SP resulted in a final N3 staging discrepancies 15% lower than when the technician reedited the PSG records blinded to the SP staging. SP auto-staging was used exclusively to differentiate N2 vs N3 epochs, with the EEG signal colored to visually identify regions that met the slow wave activity threshold to assist in making corrections to epochs contaminated by movement or other artifacts. Given exposure to or interruptions in slow wave sleep has been associated with memory consolidation,²⁴ glucose intolerance,²⁵ insulin resistance,²⁶ and beta amyloid accumulation,^27,28 methods that improve the calibrated consistency of stage N3 could increase its across-study utility and repeatability as a sleep biomarker.

Based on visual detection of RSWA, the sleep neurologists’ iRBD diagnoses were consistent in 67% of the pooled records, with sex nor site biasing the outcome.²⁹ Given the same technician scored RSWA events at both sites and the cardio-respiratory signals were available for review by the physicians in conjunction with the chin EMG, it is unlikely untreated OSA was responsible for the differences in iRBD diagnoses. The between-physician agreement in detection of RSWA was equivalent to the agreement between each physician’s iRBD diagnosis and abnormal NRH. NRH aligned with at least one sleep neurologist’s characterization of iRBD-related RSWA in 83% of the records.

It should be noted that abnormal NRH was detected in one of the two records where RSWA could not be determined given absence of REM sleep, suggesting that NRH could be particularly helpful for detecting iRBD diagnosis when REM sleep is absent, or in patients with progressive synucleinopathies when accurate staging of REM is challenging.

This study had a number of limitations. First, interpretation of the staging accuracy required partitioning of the bias introduced by use of a single technologist per site, rather than consensus agreement across multiple scorers.²⁰ Second, each physician’s iRBD diagnosis was based on qualitative assessments of RSWA, typical at most clinical sleep laboratories, rather than quantitative analyses.^11,13,18,30 Third, the iRBD diagnoses were made using the PSG recordings without consideration of the videos, similar to what might be obtained in an ambulatory, in-home environment. Access to the video recordings might have influenced the agreement among the sleep neurologists in their diagnosis of iRBD. Finally, these findings were based on a relatively small sample size.

Recently, isolated RSWA has been shown to occur in between 14% and 32% of patients without iRBD symptoms, and a subset of these patients may be at risk to develop RBD or another synucleinopathy in the future.^11,12,30 The proportion of suspected iRBD patients in this study with abnormal NRH (63%) was consistent with symptom severities observed cross other Parkinsonian spectrum disorders, ie, progressive supranuclear palsy (92%), dementia with Lewy bodies/Parkinson disease dementia (81%), and Parkinson’s disease (56%).¹⁵ Thus, longitudinal neurological follow up is warranted in patients with isolated RSWA or abnormal NRH.^4,13,30 Patients with iRBD symptoms confirmed by abnormal NRH but with insufficient REM sleep to assess RSWA could likely have iRBD and merit routine reassessments. Given the elevated risk of developing neurodegenerative disease and the long latency between iRBD symptom onset and phenoconversion, improved iRBD detection could enable more timely administration of future neuroprotective therapies.³¹

Conclusions

iRBD biomarkers capable of being recorded in the home hold promise as a clinical tool for recognition of patients that need to be more carefully examined. SP was accurate in its detection of REM and in the differentiation between stages N2 and N3 in patients suspected of iRBD. The biomarker NRH corresponded with physician’s diagnosis of iRBD based on detection of RSWA. The inconsistencies in clinician agreement of iRBD diagnosis may be improved through the use of quantitative detection of RSWA events, similar to the methodologies deployed in this study for automated detection and assignment of stages N3 and NRH.