How Sleep-Deprived People See and Evaluate Others’ Faces: An Experimental Study

Introduction

Sleep loss can alter humans’ response to affective stimuli. For example, acute total sleep loss and shorter sleep duration increase the amygdala activation to images with negative valence.^1,2 This brain region plays a role in reactivity toward affective stimuli.³ Relative to sleep, acute sleep deprivation is also associated with enhanced reactivity toward positive stimuli.⁴ For example, acute sleep loss elicits a stronger amygdala activation in response to emotional stimuli judged as pleasant.⁵

The face represents an essential source of information for the observer to identify the other person’s emotional state.⁶ In this context, the amygdala is central to emotional face recognition due to its specific role in stimuli saliency detection.⁷ For example, lesions of the amygdala can impair recognition of fear from facial expressions.^8,9 Similarly, amygdala activation correlates with the correct identification of angry and fearful faces.¹⁰ The amygdala is also activated in response to other facial expressions, suggesting that this brain region has a general-purpose function in processing salient information from faces.¹¹

Several studies have been published regarding how sleep-deprived people respond to emotional and neutral faces, including impaired accuracy and slower speed identifying emotional expression in faces.^12,13 In addition, patients suffering from insomnia exhibit a visual attentional bias toward tired faces and misrecognize angry faces as fearful faces.^14–16 Whether faces with negative or positive emotional valence are more visually salient and differently evaluated regarding social appeal when the observer is sleep-deprived is, however, not well-researched.

Thus, in the present within-subjects experiment, 45 young adults spent one night asleep and one night awake in our laboratory. In the morning after each night, we used eye-tracking technology to examine possible differences in dwell time on standardized images of happy, neutral, angry, and fearful faces. Acute sleep loss causes reductions in sustained attention.¹⁷ Thus, we expected that participants would spent less time fixating on the faces after sleep loss than after sleep. Given the bias toward negative and positive affect following sleep loss,^1,5 we nonetheless expected that the time spent fixating on emotional faces would be longer than for neutral faces after sleep loss. Acute sleep loss may reduce the drive to interact with others socially. For example, using a standardized social distance task, a previous study showed that participants enforce greater social separation from others following one night of sleep deprivation.¹⁸ Thus, we also examined whether participants would rate the attractiveness, trustworthiness, and healthiness of faces differently when sleep-deprived.

Materials and Methods

Participants

We recruited study participants via local billboards and an online recruitment website. A total of 508 potential participants filled out an online survey to assess their general health status, including sleep habits. The following exclusion criteria were applied:

• When participants indicated that their overall sleep quality was either bad or very bad
• Presence of extreme chronotype (Morningness-Eveningness Questionnaire score ≤30 or ≥70; ref.¹⁹)
• Self-reported habitual daily sleep duration of fewer than 7 hours
• Habitual bedtime on working days before 22:00 or later than 24:00
• Participants answered with yes to the question “Do you have any physical and/or mental illnesses (including chronic diseases) ?”
• Regular medication use (eg, medicine used to treat allergies)
• Drugs or nicotine usage
• More than five standard units of alcohol or caffeine beverages per day
• Self-reported uncorrected vision problems (eg, myopia)
• Planned travel across time zones within three months of the study period or between sessions.

Of the 508 people who filled in the online survey, 81% were excluded, as they met at least one of the exclusion criteria. The remaining 96 potential participants were invited for an onsite screening session. Subsequently, we excluded 49 additional potential participants due to discrepancies in their online screening and onsite interview responses. Following these exclusions, 47 participants remained for inclusion into the study (mean ± SD, age: 25 ± 3 years; 20 women). All women included in the study were on hormonal contraceptives to minimize potential confound by the menstrual cycle on facial exploration and evaluation.^20,21 Due to unexpected hardware and software issues with the eye tracker, seven eye-tracker recordings were not available for analysis. One participant only slept 02h:45min according to polysomnography (PSG), ie, the participant slept half shorter than the intended 8 hours. Thus, we excluded all data collected in the sleep condition of this participant. The final analytical sample consisted of 45 participants (including 20 women).

The study was conducted following the Declaration of Helsinki, and the regional ethics board in Uppsala approved experimental procedures (DNR2017/560). All subjects gave written informed consent before experiments and were compensated for participation in the study. The experiment described herein is part of a more extensive study investigating the possible health consequences of sleep loss. No other results have been published at the time of this publication. Experimental sessions took place between March 2018 and November 2020.

Study Design and Procedure

A scheme of the study can be found in Figure 1. The present crossover study consisted of two experimental conditions separated by at least one week: one night of total sleep deprivation vs one night with an 8-hour sleep opportunity. The order of experimental conditions was almost perfectly counterbalanced across participants. Study sessions of female participants were scheduled outside of their menstrual phase. Within the week before the first experimental session, participants visited the lab for an adaptation night. The adaptation night served to adjust subjects to the experimental setting and was conducted under the conditions of the experiment, including the attachment of electrodes for sleep recordings.

Figure 1 Experimental scheme. (A) Order of experimental conditions. (B) Overview of the eye-tracking paradigm. Abbreviations: ET, eye-tracking; PSG, polysomnography. Notes: Reprinted from Ma DS, Correll J, Wittenbrink B. The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods. 2015;47(4):1122–1135 with persmission from the Chicago Face Database23 with graphic images created with BioRender.com.

On experimental nights, subjects were served a standardized dinner upon arrival at the laboratory at 19:00. In the sleep condition, sleep was allowed between ~23:00 and 07:00. PSG recordings were performed using SOMNO HD (10–20 system; SOMNOmedics GmbH, Randersacker, Germany). Sleep stage scoring was performed following the American Academy of Sleep Medicine criteria.²² Participants’ slept on average seven hours and twelve minutes in the sleep condition (for more details, see Supplementary Results and Supplementary Table S1). In the sleep loss session, an experimenter continuously monitored participants, who were allowed to spend their time between 23:00 and 07:00 with a selection of movies, games, and books under normal indoor light conditions (~500 lux); however, no food intake, including caffeinated beverages was allowed. On the morning after sleep and sleep loss, participants performed the eye-tracking task at ~08:00.

Eye-Tracking Paradigm

The eye-tracking paradigm included high-resolution and standardized images of adult male and female actors (18–40 years). All pictures were derived from the Chicago Face Database.²³ For each face category (happy, fearful, angry, and neutral), six images (three male, three female) were randomly selected from the Caucasian subgroup of the Chicago Face Database, resulting in 24 pictures of 24 different individuals (can be retrieved upon request from the correspondence author L.v.E.). We used the same 24 face images in both experimental conditions to avoid variability in dwell time on and evaluation of faces due to different sets of photographed actors. For the experiment, we created two versions of the face paradigm using a random number generator to ensure that participants were not presented the faces in the same order in both conditions. The two versions were almost perfectly counterbalanced across participants’ experimental nights.

Participants sat between 60 and 70 cm in front of the eye tracker screen. We used a headrest to minimize head movement. Participants looked at a fixation cross in the middle of the screen for three seconds before they could freely explore each stimulus for 5 seconds. Then participants rated the actor’s healthiness (“How healthy do you deem this person?”), attractiveness (“How attractive do you deem this person?”), and trustworthiness (“How trustworthy do you deem this person?”) on a 100-mm visual analog scale (0= not at all; 100= very much). After a practice run to ensure that the subject understood the instructions, the eye-tracking paradigm started. We did not ask the participants to identify the emotion communicated by each displayed face, as this would have caused unwanted priming effects.

Eye Tracker Hardware

Participants’ visual gaze was assessed using a Tobii Pro Spectrum eye tracker sampling at 600Hz with binocular tracking and Tobii Pro Lab software (Tobii© Technology AB, Stockholm, Sweden). The screen showing the stimuli measured 23.8” and 1920×1080 pixels.

As summarized in Figure 2, the same procedure was manually applied to each face to determine the area of interest (AOI). For the AOI for the upper face region, including the eyes, we first defined the middle point between both pupils of the eyes. We then drew a vertical line from this midpoint to the tip of the nose (eye-nose line). We inserted a horizontal line at 50% of the eye-nose line in the next step (lower bound). To catch the eyebrow, we vertically flipped the eye-nose line (ie, in the direction of the forehead). We then inserted a horizontal line at 75% of the flipped eye-nose line (eyebrow bound). The root of the helix of each ear was used as a vertical boundary. To catch forehead wrinkles that can affect observers’ interpretation of facial expressions,²⁴ the resulting area between the lower and eyebrow bounds was then duplicated and flipped at the horizontal level of the eyebrow bound and was defined as the AOI for the upper face region, including the eyes. The AOI for the lower face region, including the mouth, included the area between the lower bound, the chin-top, and the vertical boundaries.

Figure 2 Facial locations used to define the areas of interest for analysis. Notes: Facial image reproduced from Ma DS, Correll J, Wittenbrink B. The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods. 2015;47(4):1122–1135 with persmission from the Chicago Face Database.23

Eye-Tracking Outcome Variables

Participants were free to explore each face. However, they were instructed to always look at the fixation cross in the center of the screen during the 3-seconds inter-stimulus interval to ensure that they would start exploring each face from the same screen position. Thus, the first 300 ms of each 5-seconds long face stimulus presentation was discarded.

Fixation duration was defined as the time during which participants’ eyes rested on the face using the classifications from the Tobii I–VT Fixation filter, as described elsewhere.²⁵ For the analysis, the total fixation duration (TFD), expressed as a percentage of the maximum of 4.7 seconds, was used as the primary outcome. TFD is a cumulative measure of the time participants’ eyes fixate a stimulus. Additionally, we measured the total number of fixation points (TFP) during the 4.7 seconds. TFP indicates the number of true fixation points participants’ exhibit when visually exploring a stimulus (here: faces).

To explore possible spatial differences in eye-gaze patterns between sleep loss and sleep, we also compared TFD and TFP for the lower and upper parts of the face. Single probes in either condition were excluded if the eye-tracker did not detect eyes during the presentation of the face.

Statistical Analysis

All analyses were performed using IBM SPSS Statistics 26 (SPSS Inc. Chicago, IL, USA). Generalized linear mixed models (GLMM) were used to compare the main effects of the within-subjects factor SLEEP (ie, sleep vs sleep deprivation) and EMOTION (happy vs neutral vs angry vs fearful). We also modeled possible interactions of SLEEP with EMOTION. Post-hoc testing was performed using pairwise t–tests with LSD adjustment for multiple comparisons. We also tested whether the effects of SLEEP on visual exploration and ratings may vary by individual faces within each emotional face category. Unless otherwise specified, data are reported as estimated mean [95%-CI], as derived from the GLMM. P values<0.05 were considered significant.

Results

Eye-Tracking results

Following sleep loss, the TFD decreased by about 8% compared to sleep (sleep loss vs sleep: 72.7% [71.3%, 74.0%] vs 81.0% [80.0%, 81.9%]; p<0.001 for SLEEP). In contrast, no main effect for EMOTION was found (happy vs fearful vs angry vs neutral faces: 76.9% [75.4%, 78.4%] vs 76.2% [74.5%, 77.8%] vs 76.4% [74.9%, 78.0%] vs 77.7% [76.3%, 79.2%], p=0.497). Also, the SLEEP* EMOTION interaction did not reach significance (p=0.235; differences in TFD between the conditions ranged from 6.3% to 10.6%, Table 1).

Table 1 Total Fixation Duration (Expressed as % from the Maximum Dwell Time of 4.7 Seconds) for Faces with Different Emotional Expressions Following Sleep Loss and Sleep

When using the TFD for the upper face, we found a main effect for SLEEP (sleep loss vs sleep: 49.3% [47.8%, 50.8%] vs 57.5% [56.2%, 58.7%]; p<0.001) and EMOTION (happy vs fearful vs angry vs neutral faces: 51.8% [49.8%, 53.8%] vs 51.4% [49.5%, 53.3%] vs 55.3% [53.3%, 57.3%] vs 55.0% [53.1%, 56.9%], p=0.005). As suggested by the GLMM, the SLEEP effect on the TFD for the upper face did not vary by EMOTION (p=0.770 for SLEEP* EMOTION; differences in TFD-upper face between the conditions ranged from −9.9% to −7.2%, Table 1).

When analyzing the TFD for the lower part of the face, we found no main effect for SLEEP (sleep loss vs sleep: 22.5% [21.3%, 23.8%] vs 23.2% [22.3%, 24.2%]; p=0.381) but EMOTION (happy vs fearful vs angry vs neutral faces: 24.2% [22.7%, 25.8%] vs 24.5% [22.9%, 26.0%] vs 20.4% [19.0%, 20.9%] vs 22.4% [20.8%, 23.9%], p<0.001). Furthermore, there was a trend for an interaction of SLEEP with EMOTION (P=0.065). Subsequent post-hoc comparisons showed that participants in the sleep loss condition spent about 3.6% less time fixating on the lower part of the face than after sleep (p=0.015; Table 1). The difference in TFD for the lower part of the face did not reach significance for the other face categories (p≥0.171; Table 1).

The effects of SLEEP on the eye-tracking outcomes did not vary by the individual face when running a GLMM for each face category separately (p≥0.610 for all interactions). Results when using the TFP as a dependent variable are summarized in the Supplementary Results and Supplementary Table S2.

Ratings

As summarized in Figure 3, participants evaluated the faces as less attractive and trustworthy after sleep loss than sleep (the mean difference between sleep loss and sleep for attractiveness: −3.6 mm [−5.6 mm, −1.7 mm]; and trustworthiness: −2.6 mm [−4.4 mm, −0.8 mm]; p≤0.004 for SLEEP). Ratings regarding healthiness did not differ significantly between sleep loss and sleep (the mean difference between sleep loss and sleep: −1.2 mm [−3.0 mm, 0.6 mm]; p=0.180 for SLEEP; Figure 3). We also found a main effect of EMOTION for attractiveness (happy vs fearful vs angry vs neutral faces: 54.7 mm [52.5 mm, 56.8 mm] vs 49.9 mm [39.1 mm, 42.6 mm] vs 45.1 mm [43.0 mm, 47.2 mm] vs 39.0 mm [37.2 mm, 40.8 mm]; p<0.001), trustworthiness (happy vs fearful vs angry vs neutral faces: 63.3 mm [61.4 mm, 65.3 mm] vs 45.3 mm [43.7 mm, 46.9 mm] vs 39.3 mm [37.5 mm, 41.0 mm] vs 45.2 mm [43.3 mm, 47.1 mm]; p<0.001), and healthiness (happy vs fearful vs angry vs neutral faces: 68.2 mm [66.6 mm, 69.8 mm] vs 53.3 mm [51.6 mm, 55.1 mm] vs 57.7 mm [55.9 mm, 59.4 mm] vs 52.6 mm [50.7 mm, 54.5 mm]; p<0.001).

Figure 3 Face ratings following one night of sleep and sleep loss. Ratings collected during one sleep loss condition were not analyzed since the participant misinterpreted the instructions. Values shown as mean [95%-CI]. **p<0.01; *p<0.05. Abbreviations: ns, non-significant; VAS, visual analogue scale.

Post-hoc comparisons revealed that images of angry faces were rated as less trustworthy (−4.9 mm, P=0.005) and healthy (−4.2 mm, P=0.020), whereas fearful and neutral faces were rated as less attractive after sleep loss (−3.9 mm and −5.5 mm; p≤0.030; Table 2). In contrast, ratings for happy faces did not differ between sleep loss and sleep (Table 2). However, the SLEEP* EMOTION interaction was not significant for any of the ratings (p≥0.187). Finally, the effects of SLEEP on ratings did not vary by the individual face when running a GLMM for each face category separately (p≥0.658 for attractiveness; p≥0.594 for trustworthiness; and p≥0.838 for healthiness).

Table 2 Social Appeal Ratings for Faces with Different Emotional Expressions Following Sleep Loss and Sleep

Discussion

Previous human studies showed that images with negative or positive affective valence more significantly activate brain regions involved in processing relevant stimuli when the observer is sleep-deprived.^1,2,5 With these results in mind, we hypothesized that happy, angry, and fearful faces are more visually salient than neutral faces under conditions of acute sleep loss, ie, they are more likely to catch the observer’s gaze.

We found that participants spent less time fixating on the faces after acute total sleep loss than after sleep. Several central nervous mechanisms could explain the decrease in participants’ total fixation duration after sleep loss. For example, sleep loss causes reductions in the speed of processing and attention.^17,26 Furthermore, sleepiness, inevitably caused by overnight wakefulness, may outcompete other goal-directed behaviors.²⁷ Thus, the sleep-deprived brain may be less capable of exerting efforts to explore the face visually. Reduced attention and sleepiness may also explain why participants, contrary to our hypothesis, did not spend more time fixating on emotional faces than they did on neutral faces.

We also examined whether participants’ total fixation duration for the upper and lower parts of the face differed between sleep loss and sleep. Acute sleep loss diminished gaze duration for the upper part of the faces, irrespective of the expressed emotion. The time spent fixating the lower part of the faces did not differ between sleep loss and sleep, except for reduced gaze duration of angry faces. As suggested by previous work, recognizing human emotional facial expressions mainly relies on the eyes and the mouth regions.²⁸ A reduced dwell time on the eyes (and mouth for angry faces) seen herein could explain why others have shown that sleep-deprived healthy adults are significantly less accurate at identifying happy, sad, and angry faces.^12,13 When sleep-deprived, humans are also slower to recognize emotional facial expressions.¹³ A diminished or delayed ability to identify the emotional state of others can have negative consequences for the sleep-deprived, eg, in occupations where workers who are routinely restricted of sleep are also responsible for making judgments about other people’s emotional state (eg, police, security, and teachers). Of note, these recognition deficits are likely to disappear after recovery sleep.¹²

The present study found that angry faces were deemed less trustworthy and healthy after sleep loss. Intriguingly, as suggested by results from a previous meta-analysis, the greater the amygdala activation – a brain activity pattern seen in sleep-deprived humans viewing images with negative valence after acute sleep loss¹ –, the more likely are observers to rate faces as untrustworthy.²⁹ We also found that neutral and fearful faces were rated less attractive after sleep loss. In contrast, the rating of happy faces, the most socially appealing face category in the present study, remained unaffected by sleep loss. Similar findings for the effects of acute sleep loss on social impressions have been found after chronic sleep loss. Specifically, when undergoing a chronic sleep loss intervention, humans deem individuals with less positive and more negative facial features as more dangerous. However, these biases are not present when participants are rested.³⁰ More negative social impressions of others could explain why acute sleep loss may promote social withdrawal and loneliness.¹⁸

Several strengths apply to our study. The experiment was performed under highly controlled in-laboratory conditions. Moreover, women used hormonal contraceptives to reduce possible confounding by the menstrual cycle on facial exploration and evaluation.^20,21 However, several limitations apply to our study. First, we chose only face images of Caucasian actors. A second limitation is that our participants were young adults. Our study focused on the effects of acute sleep loss on happy, neutral, angry, and fearful facial expression. Thus, our data cannot be extrapolated to other facial expressions. Finally, our results are based on non-dynamic photographs. Future studies should therefore examine if dynamic face representations, eg, using real actors or pre-recorded videos reveal similar results.

Conclusion

Among non-verbal signals, facial expressions are crucial for social interactions.^31–33 Thus, spending less time fixating on faces after acute sleep loss may come along with several problems for social interactions, eg, inaccurate and delayed judgment of the emotional state of others. In addition, more negative social impressions of others, as shown herein, may increase the risk of the sleep-deprived becoming socially isolated.