Major Depressive Disorder Prediction Based on Sleep-Wake Disorders Symptoms in US Adolescents: A Machine Learning Approach from National Sleep Research Resource

Introduction

Adolescence is a period of development characterized by marked physical and social changes, during which the young person’s brain and cognition gradually matures, including growing social understanding and self-awareness, and increased levels of perceived stress.¹ At the same time, this leads to a substantial increase in the prevalence of depression in adolescence compared to childhood, with a one-year prevalence rate of up to 4–5% in mid- to late adolescence.² There are longitudinal studies showing that while 60–90% of adolescent depressive episodes resolve within one year, 50–70% of patients experience a re-emergence within 5 years.^3–5 In addition, there is a large number of potential sufferers who go undetected and untreated in a timely manner, as they are often overlooked and incorrectly attributed to being going through the necessary teenage phase.⁶

It is important to be alert that the neglect of the onset of depression in adolescence often leads to serious consequences. There is a high correlation between adolescent depression and adolescent suicide rates, which are the second to third leading cause of death at that age phase.⁷ According to relevant research,⁸ more than half of teenage suicide victims were suffering from depression at the time of death. In addition, early episodes of depression in adolescence can present as longer episodes and higher recurrence rates of major depression in adulthood, leading to a subsequent series of adverse social outcomes associated with low educational attainment, high unemployment and divorce rates.^9–11

Sleep-wake disorder symptoms are one of the key symptoms identified by the Diagnostic and Statistical Manual of Mental Disorders (DSM)¹² and the International Classification of Diseases (ICD),¹³ and there are complex and paradoxical correlations between it and depression. These sleep-wake disorder symptoms are both precursor and independent risk factor for the development of depression.¹⁴ Insomnia is a classic symptom of sleep-wake disorders, and according to the epidemiological study by Ford et al,¹⁵ patients with insomnia symptoms have a much higher risk of depression compared to those without, and patients who are subsequently cured of their insomnia symptoms have a lower risk of depression recurrence. Furthermore, obstruct sleep apnea (OSA), and narcolepsy were prevalent in most adolescent depressed population.^16,17 The depression-related sleep parameter abnormalities presenting in almost all stages of sleep architecture¹⁴ show that it has great potential in predicting the onset of depression in adolescents.

Adolescent depression screening at present is usually performed using self-report scales assessing immediate feelings related to depression. However, the social neglect and lack of social support resulting from the difficulty in identifying abnormal adolescent psychological problems calls for the need of more intuitive and accurate predictive models to refine primary care and prevention goals for adolescent depression. The application of advanced statistical techniques based on artificial intelligence, represented by machine learning (ML), is expected to perform depression diagnosis through more objective sleep-wake disorder symptom data.¹⁸ The ML distinguishes itself from previous approaches solving inference problems and understanding relationships between variables, it learns directly from data and searches for complex, non-linear patterns to achieve predictions.¹⁹ Therefore, the aim of the present study is to establish and compare the performance of multiple ML models based on baseline characteristics and sleep-wake disorder symptoms data and to select the optimal model to predict the occurrence of depressive symptoms in adolescents.

Methods

Description of Database

The data in this study were derived from national sleep research resource (NSRR): 2020–2021 Assessing Nocturnal Sleep/Wake Effects on Risk of Suicide Phase I Study (ANSWERS). The NSRR was the first and largest of its kind national repository of sleep data, which was funded by National Institutes of Health.²⁰ The ANSWERS data were collected between May 2020 and May 2021 from undergraduate students aged 18–25 at the University of Arizona to evaluate sleep continuity, timing, quality, and disorders in conjunction with general mental health and suicidal thoughts and behaviors.²¹ The ANSWERS study was approved by the University of Arizona Institutional Review Board (IRB) (Protocol #2005675654). This machine learning model analysis was a secondary data analysis based on ANSWERS observational data and therefore according to the relevant law of the corresponding author’s country, item 32 of the “Notification on the Issuance of Measures for Ethical Review of Life Science and Medical Research Involving Human Beings”²² states that ethical review may be waived for research using legally available public data, or data generated by observation without interfering with public behavior.

Depression Definition and Measurement

The study’s definition of major depressive disorder (MDD) was based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), published by the American Psychiatric Association (APA), which defines depressive symptoms as depressed mood and loss of interest/pleasure in activities (lack of pleasure) for more than two weeks.²³ The center for epidemiologic studies depression scale (CES-D) 20-item²⁴ was used to measure whether MDD occurrence. The items are summed to obtain a total score using the 0 (rarely or none of the time) to 3 (most or all of the time) scores for individual items and A higher score reflects greater symptoms of MDD, CES-D score ≥16 is typically employed as a cutoff for clinical MDD.²⁵

Predictors Selection and Measurement

The variable selection was guided by the previous literature review and model structure.^26,27 In total, thirteen predictor variables included specifically categorized as demographic variables (age, gender, race, orientation and education) and clinical sleep-wake disorder symptom variables, including sleep quality, insomnia severity, OSA symptoms, narcolepsy symptoms, intense daytime drowsiness symptoms, night terrors, REM sleep-wake disorder symptoms and nightmare severity. Demographic data, with the exception of participant age, were included as categorical variables in this ML prediction model. Insomnia Severity Index (ISI),²⁸ Disturbing Dream and Nightmare Severity Index (DDNSI),²⁹ Sleep-wake Disorder Symptoms Checklist-25 (SDSC-25)³⁰ and Pittsburgh Sleep Quality Index (PSQI)³¹ were used to measure the participants sleep-wake disorders symptoms. Detailed tool information and scale descriptions are provided in Table 1.

Table 1 Instrument to Measure Depression and Sleep-Wake Disorders Symptoms in Adolescents

Descriptive and Inferential Analysis

All demographic variables, sleep-wake disorders symptoms variables and MDD categorical variables included in this study were conducted descriptive analysis to explore their central tendencies and frequencies distribution. Spearman correlation test was used to measure the association and collinearity between predictors and the Chi-square test and Mann–Whitney-U-test was conducted to access whether the individuals with/without MDD have significant difference in the predictors.

Statistical Analysis

Data Pre-Processing

The K-Nearest Neighbor (K-NN) interpolation algorithm was used to fill in the missing data in our study, with the missing values being replaced by values obtained from the relevant cases in the entire record set.³² MDD scores based on the CES-D assessment were cut into suffering from MDD (Yes = 1) and non-suffering from MDD (No = 0) at a cutoff of 16 score. The Boderline-1 SMOTE algorithm was used for sample equalization to avoid confounding accuracy due to possible imbalance of dichotomous sample categories, and models before and after sample equalization were compared. Then, the balanced dataset was randomly split into training (70%) and test (30%) set.

Model Selection

In this study, six machine learning algorithms were chosen to predict MDD in adolescents based on their sleep-wake disorder symptoms. The six models that we have developed are as follows: eXtreme Gradient Boosting (XGBoost) model, Light Gradient Boosting model (LightGBM), AdaBoost, Gaussian Naïve Bayes (GNB), Complement Naïve Bayes (CNB), and multilayer perceptron (MLP). XGBoost, LightGBM and AdaBoost models are optimized distributed gradient boosting libraries with excellent predictive performance by converting a set of weak variables into strong ones.^33–35 The Gaussian and Complement Naïve Bayes model are classifier on the basis of the Bayes theorem and assume that categorical features are independent of each other.³⁶ The MLP is a neural network with input layers, multiple hidden layers and output layers that excels at modelling complex patterns and can be adapted to suit different task complexities.³⁷

Model Establishment and Evaluation

A 10-fold standard nested cross-validation (CV) approach was applied to model training for this study to obtain unbiased estimates of the generalization performance and minimal overfitting hyperparameters of the ML model. The final hyperparameters definite was provided in eTable 1. The standard nested CV is a better choice compared to CV in the finite data case.³⁸ It creates a double loop by dividing the dataset into k outer folds and merging and splitting the remaining k-1 folds into inner folds to ensuring that every piece of data is utilized, while avoiding data leakage due to the data sharing between the training and validation sets,³⁹ as shown in Figure 1.

Figure 1 Schematic representation of the 10-fold standard nested CV approach.

The area under the receiver operating curve (AUC-ROC) and area under the precision-recall (PR) curve (AP) were used to quantify model discrimination. The Delong test was used to determine whether there was significantly difference in AUC between models. Subsequently, the calibration plot and decision curve analysis (DCA) were used to assess the extent to which model predictions deviated from actual events and to calculate the net benefit of different threshold probabilities, respectively. Moreover, the sensitivity (1), specificity (2), accuracy (3), positive predictive value (PPV) (4), and negative predictive value (NPV) (5) were used to evaluate the ML model performance. Ultimately, the model with the best composite metrics is considered the optimal model and entered the test.

The true positives (TP) and true negatives (TN) mean the adolescences that were accurately identified respectively as suffering MDD and not suffering MDD and the false negatives (FN) and false positives (FP) are opposite.

(1)

Model Interpretation

Given the persistent difficulty in interpreting the black boxes of machine learning models, we applied the Shapley additive explanations (SHAP) technique to help better explain the results of the predictive models. The SHAP method, as a model-agnostic technique based on cooperative game theory, is able to quantify the SHAP values of each sleep-wake disorder symptom predictor feature and rank the importance by the sum or average SHAP values of each individual.⁴⁰ Moreover, the SHAP method is also able to measure the influence of features on predicted outcomes, with positive SHAP values indicating that features in that state lead to a higher risk of MDD in adolescents and negative SHAP values means opposite results.⁴⁰ All Statistical analyses were performed using R version 3.6.3 and python version 3.7.

Results

Descriptive and Inferential Analysis

In this study, a sample from ANSWERS was used, which contained a total of 971 undergraduate students from the University of Arizona with a median age of 19 years, most of whom were female (73.43%) and white (76.83%). There were 557 participants who were suffering from MDD and 414 participants who were not, based on the CES-D delineation criteria. According to the results of the Chi-square test and Mann–Whitney-U-test, there was a significant difference on all sleep-wake disorders symptoms assessments between the suffering and non-suffering MDD participants. In demographic comparisons, there were significant differences between suffering from MDD and not suffering from MDD only in terms of gender (P<0.001) and sexual orientation (P<0.001), but the differences between the other dimensions were not significant (Table 2). The Spearman correlation analysis results also showed that all sleep-wake disorders symptoms scores were significantly and positively correlated with MDD scores (P<0.05) and no co-linearity among predictors (r<0.8), with insomnia severity having the highest correlation (r = 0.54). Other more detail descriptive statistics and correlation analysis results can be accessed in Table 2 and Figure 2.

Table 2 Demographic and Sleep-Wake Disorders Symptoms Characteristics of the Study Sample

Figure 2 The heatmap of Spearman correlation analysis results.

Prediction Performance of Different ML Models for Adolescents’ MDD

Figure 3A–D provides the comparison of ROC curves and PR curves for the predictive efficacy of different ML models for adolescent MDD in the training and validation sets. In the performance of the 6 ML models, XGBoost has the highest AUC value (AUC _T = 0.862; AUC _V = 0.798), AP value (AP _T = 0.873; AP _V = 0.802), PPV value (PPV _T = 0.815; PPV _V = 0.749) and F1-Score (F1-Score _T = 0.783; F1-Score _V = 0.751), while LightGBM has the highest Sensitivity value (Sensitivity _T = 0.762; Sensitivity _V = 0.767) followed by XGB (Sensitivity _T = 0.754; Sensitivity _V = 0.758). In addition, although the highest values of accuracy, specificity and NPV in the training set are still from XGBoost (Accuracy _T = 0.788; Specificity _T = 0.825; NPV _T = 0.766), the highest values of them in the validation set are from GNB (Accuracy _V = 0.728; Specificity _V = 0.784; NPV _V = 0.723). All other scores for each of the model training and validation sets are provided in Table 3. The results of comparing all ML models with real event perfect calibration curves are provided in the calibration plots, where XGBoost is suggested to have the smallest degree of deviation (Figure 3E). Subsequently, the DCA curve (Figure 3F) results suggest that all ML models have similar net benefit curves except for GNB and CNB. The Delong test did not show any significant difference in AUC between different models (eTable 2), but considering all the metrics together, we believed that XGBoost may have a more comprehensive performance in the comparing of ML prediction models based on sleep-wake disorders symptoms for adolescent MDD. Figure 4 had shown the XGBoost models’ AUC and AP performance from training, validation and testing set, and other evaluation metrics results were supplied in eTable 3. The calibration plot and DCA curve of XGBoost model were supplied in eFigures 1 and 2.

Table 3 Prediction Performance of Adolescents with Depression Using 6 Machine Learning Models

Figure 3 Performance measurements of different ML models: the ROC and PR curve of train sets were supplied on (A) and (B); the ROC and PR curve of validation sets were supplied on (C) and (D); the calibration and DCA plots of validation sets were supplied on (E) and (F).

Figure 4 Performance measurements of XGBoost model from different sets: the ROC curve of train set was supplied on (A); the ROC curve of validation set was supplied on (B); the ROC curve of test set was supplied on (C).

Model Interpretation

To identify the features that have the greatest impact on the predictive models, we show the SHAP summary plot for XGBoost and the four example adversarial force diagrams (Figure 5A and B). According to the SHAP summary plot (Figure 5A), insomnia severity, OSA and narcolepsy were the first three most critical predictive features of the XGBoost model, which had the most significant impact on the predicted outcomes. Where the red indicates high feature values, purple indicates feature values close to the overall mean, and blue indicates low feature values, while the higher the SHAP summary value of the feature, the more important for predicting MDD.⁴⁰ Figure 5B provides four examples of predicted antagonism. In these cases, the XGBoost model predicted risks of developing MDD of 0.789, 0.289, 0.408, and 0.710 (base value: 0.511), respectively. For example, in case 1, severe insomnia, poorer sleep quality, and occasional night terrors increased the likelihood of developing MDD, whereas mild OSA reduced the probability of developing MDD in this sample.

Figure 5 The SHAP summary value plots and adversarial force diagrams.

Discussion

There is a growing number of studies exploring the use of ML models in predicting MDD, from the onset of MDD to the treatment outcomes, with populations covering children, pregnant women, and middle-aged and older adults, and with predictors covering daily activity data, neuroimaging data, and general clinical data.^41–45 The study by Gomes et al²⁶ predicted the onset of MDD symptoms in middle-aged and older adults by using NHENES sleep data from 2015 to 2016 and concluded that XGBoost is outperforms other machine learning models. To our knowledge, this study is the first ML prediction model to predict the MDD risk in adolescents based on sleep-wake disorder symptom data. The aim of our investigation was to develop this predictive model to identify individuals at increased risk for MDD in US adolescents through sleep-wake disorder symptom data.

In our study, 557 (57.4%) of the total 971 participants were at risk of MDD. There was a significant positive correlation between all sleep-wake disorder symptoms and the occurrence of MDD and a significant difference between the groups of US adolescents who did and did not suffering MDD. In our primary machine learning model comparisons, XGBoost achieved consistently best performance in AUC, AP, PPV, and F1 scores, and LightGBM achieved consistently best performance in sensitivity. Combining all the metrics, XGBoost was considered to be the best performing model, and also had a good performance in the test set with AUC value of 0.804.

Insomnia severity was identified as the most important feature predicting the presence of MDD risk in US adolescents in this study. The result is supported by the research findings of Sun et al,⁴⁶ which concluded a significant causal association between insomnia and MDD by bidirectional Mendelian randomization research. However, there is a difference between this and the conclusion of Gomes et al’s model²⁶ that DDS being the most important feature triggering MDD (Accuracy = 0.87; Sensitivity = 0.81) in middle-aged and older adults. Much of this difference is due to different social responsibilities and social interests resulting from age difference. Compared to middle-aged and older adults, adolescents have a heavier daytime academic load, cumbersome social relationships, and curiosity to explore society.⁴⁷ The sleep restriction resulting from this social activity attribute reduces the significance of their DDS manifestations. Whereas one possible explanation for insomnia as the most important predictor of the onset of adolescent MDD is that the onset of adolescence is significantly associated with a reduction in homeostatic sleep pressure during wakefulness.^48,49 The homeostatic sleep-wakefulness system controls sleep demand in humans, and this change leads to a reduction in sleep drive.^48,49 As stress perception increases during adolescence, they remain awake longer at night, but are inevitably subject to sleep restrictions (fixed school hours), creating a vicious cycle of “perceived stress-over wakefulness-perceived stress”. Moreover, related studies have shown that sleep deprivation reduces the availability of dopamine D2/D3 receptors in the striatum and leads to inactive reward-seeking behavior.^50,51 This increases susceptibility to internalized symptoms, which can further lead adolescents into a state of “insomnia-depression”. Another possible explanation is that the other sleep-wake disorder symptoms in this study were not as easily perceived by the participants compared to insomnia, thereby may underestimating the ratings of these symptoms when receiving symptom evaluations. Such self-assessment scales magnify the effect of some symptoms that can be consciously perceived by the participant while may hiding the truly important features, and addressing this bias requires more objective data on sleep-wake disorders symptoms.

The model suggested that, except for insomnia, obstructive sleep breathing disorder and narcolepsy were also important predictors for the development of adolescent MDD. According to previous studies,^52–54 patients with OSA and narcolepsy had MDD symptoms in 63% and 32%, respectively, which is a common co-morbid symptom. One possible explanation is that both sleep-wake disorder symptoms are associated with sleep deprivation and impaired daytime functioning in adolescents. Sleep deprivation leads to disrupted sleep architecture and symptoms of sleep fragmentation associated with sleep disruption and oxygenation deficits.^53–55 This chronically inefficient quality of sleep affects adolescents’ mood regulation and cognitive functioning, leading to the occurrence of low mood, sadness, emotional numbness, and lack of pleasure in daily life,^54,56 greatly increasing the risk of MDD. Similarly, impaired daytime functioning can further increase the frequency of negative life events in adolescents’ schooling and lives, including academic difficulties, poor quality of life, and social stigma.⁵³ The high frequency of these negative life events can further increase the social pressure perceived by adolescents, which in turn interferes with the normal homeostatic sleep-wake system of adolescents, leading to a vicious cycle of over-wakefulness and MDD. In addition, there are some potential mechanistic evidences that provided support for the MDD prediction by OSA symptoms and narcolepsy. It has been suggested by previous studies that the abnormal changes in cerebral white matter may be a potential predisposing mechanism for the development of MDD due to OSA symptom.⁵⁷ The findings of Kim et al⁵⁸ showed that the AHI index was significantly positively correlated with the severity of white matter hyperintensities (WMHs) and that 89% of WMHs in patients with OSA were in the frontal lobes. The OSA may cause functional white matter injuries, and disruption of neural connections involving the dorsolateral prefrontal white matter contributes to the risk of MDD.^59,60 Meanwhile, deficiencies in hypothalamic secretin neurotransmission may be a key mechanism for MDD induced by narcolepsy. Narcolepsy is significantly associated with abnormalities in hypothalamic secretin neurotransmission, and hypothalamic secretin deficiency may trigger affective and psychological disorders including MDD through cholinergic-monoaminergic imbalance.^61,62

As with OSA symptoms and narcolepsy symptoms, nightmare symptom and night terrors symptom also are responsible for the fragmentation of sleep that occurs in adolescents. The occurrence of nightmare severity and night terrors is a complex bidirectional relationship with the development of adolescent MDD. The night terrors and frequent nightmares triggered by neurobiological factors, such as neurotransmitter imbalances, abnormalities in the functioning of brain regions, and genetic factors may induce MDD symptoms by exerting great psychological stress and causing sleep fragmentation, which may affect the quality of life and social attributes of adolescents.^63,64 However, it should be noted that the occurrence of these two sleep-wake disorders stems largely from the psychological trauma suffered by adolescents in their daily lives, which may be a manifestation rather than a cause of the MDD already present in adolescents.

In addition, in this study, although DDS and REM symptoms were the two sleep-wake disorder symptoms of low importance from SHAP charts, their significant differences between adolescents suffering from MDD or not also had a positive effect on identifying the occurrence of MDD in adolescents. This further validated the conclusion of previous research that DDS and REM sleep-wake disorders are reliable observational indicators of the development of MDD.⁶⁵ Similarly, while most demographically relevant predictors were not significant in this model, some meaningful indicators, such as gender and sexual orientation, also informed the prediction of the developing of adolescent MDD.

Finally, some limitations of this study need to be explained. First, due to the cross-sectional study design of ANSWERS, there are some limitations in interpreting the causality or direction of the relationship between sleep-wake disorder symptoms and the onset of adolescent MDD. Secondly, we lack more objective data on the measurement of sleep-wake disorder symptoms in adolescents, such as sleep electroencephalograms, which may lead to confounding effects and misinterpretation of true associations due to overlapping features between sleep-wake disorder symptoms measured by the scales. Thirdly, the CES-D 20-item was used in this study as a measure of MDD only to reflect the risk of its occurrence, which may not meet the need for clinical diagnostic prediction. Future studies should use the gold standard for MDD diagnosis to obtain more reliable outcomes to construct predictive models. Fourthly, this study needs more data for external validation to verify the reliability of the model, which needs to be further complemented by future studies. Finally, since the ANSWERS sample collection was conducted for undergraduate students at US universities, this inevitably affects the extrapolation of the model. Our study provides a way for researchers to develop algorithms for special situations, but further research with more extensive data is needed for a more widespread group of adolescents or for adolescents from different national regions and cultures.

Conclusion

In this study, the XGBoost algorithm was considered to have optimal performance in the model using sleep-wake disorder symptoms data to predict the onset of MDD in US adolescents, with an AUC value of 0.804. All sleep-wake disorder symptoms were significantly positively correlated with the onset of adolescent MDD, with insomnia severity being the most significant predictor compared with the other predictors in the present study. The US adolescents with MDD are often associated with severe insomnia symptoms, night terrors, and lower sleep quality. This machine learning predictive model based on sleep-wake disorders symptoms can help to raise the awareness of risk of symptoms between sleep-wake disorders and MDD in adolescents and improve primary care and prevention.