Impact and Feasibility of Information Technology to Support Adolescent Well-Being and Mental Health at School: A Quasi-Experimental Study

Introduction

Health-related behaviors that arise during adolescence can have important, sometimes lifelong, implications on a person’s health.¹ The number of psychiatric and neurodevelopmental diagnoses among minors has increased.^2,3 There are effective interventions in existence that show significant improvements in adolescents’ self-perceptions, values, cognitive skills, quality of life and mental health literacy.⁴ However, adolescents may experience barriers that prevent them from seeking information and supporting their mental health. It has been reported that adolescents lack knowledge about mental disorders.⁵ They often attach a sense of shame to mental health problems,⁵ and they generally lack insight on the subject and belief in their own capabilities.^6,7

There is great potential for information technology (IT) in the area of mental health for adolescents.^8–10 For example, gamification, interactive exercises/modules, videos/animations, social media/messaging, and chatbots have all shown promise in mental health improvement.¹¹ Technology provides opportunities to alleviate problematic situations in ways that traditional face-to-face methods cannot¹² when it comes to improving well-being, increasing equality, and employing trustworthiness and self-monitoring.¹³ Technological interventions can also complement existing mental health care solutions.¹²

Schools are places where mental health can be promoted with the aim of enhancing well-being, and where mental health care can be provided. Mental health interventions have been developed for schools^14–16 since adolescents spend a good bit of their time there,¹⁷ and schools are points of contact with adolescents.

Transforming IT-based treatment programs into universal interventions in school settings could be one solution for tackling latent mental health symptoms among youth and improving their quality of life. Even though systematic knowledge about interventions that have been delivered as universal prevention programs, those held in schools, those consisting of IT, and those focusing on mental health and depression have been gathered and synthetized,^18,19 there is still a lack of knowledge about the combination of these. Many important outcomes related to adolescents’ lives have been previously explored, such as suicidality,²⁰ depression attitudes,^20,21 literacy and self-esteem,²² and coping strategies,²³ but quality of life has not explicitly been measured. This is surprising, as quality of life has been defined to be among the core set of outcomes in clinical trials by various taxonomies and institutions, such as the Cochrane Community.²⁴ Adding quality of life as an outcome in this area of research would provide a much-needed perspective in future comparative effectiveness research, outcomes research, and in clinical care.²⁵

This paper focuses on the IT-based program “DepisNet”, a user-friendly and feasible support system developed for adolescents with depression. It was tailored to improve self-management skills and increase awareness of adolescents’ well-being and mental health.²⁶ It is essential that mental health interventions are evidence-based, developmentally appropriate, and integrated into curricula.²⁷ Therefore, in this study we tested whether this program could be used as a universal school-based program to support adolescents’ well-being and mental health. This was the first time the adolescents in our target schools had courses similar to the program’s topic. We hypothesized that our preventive intervention could decrease depressive symptoms and improve the quality of life of adolescents. Additionally, we evaluated the program’s impact on self-efficacy, self-esteem and feasibility components (acceptance, adherence). The results can provide guidance in addressing mental health in schools and serve as an essential component of adolescent-focused mental health policy.

IT-Based Mental Health Support Programs

We identified four previously reported universal programs (in 9 papers) in which IT was utilized and tested for adolescents around 14–15 years old. Such programs were related to mental health, or specifically depression, and used comparative study designs. The programs lasted 5 or 6 weeks and had their origins in positive psychology, for example, with “Bite Back”,²⁸ or had elements from cognitive-behavioral therapy, for example with “MoodGym”,^21,22,29,30 ‘e-couch Anxiety and Worry program,^31,32 and SPARX-R.^20,23 Results from these comparative studies were, however, mixed. Some found improvements in depression symptoms (for boys,²⁹ for both genders,²⁰ and for girls only)²² in emotional regulation,²³ and anxiety,²⁹ while others reported a lack of improvement.^{21,28,30–32} Feasibility components was reported in all studies, including mostly low completion rates. The programs are elaborated upon in Table 1.

Table 1 IT-Based Mental Health Support Programs in Schools

Materials and Methods

Study Design and Participants

We used a quasi-experimental, pre-post design with two preference arms (intervention and control groups). In this study, quasi-experimental means an experimental design without random assignment to intervention and control groups. In two preference arms, participants (schools) were allowed to choose whether they wanted to participate in the study as part of an intervention group or a control group. Preference arms are, in general, an attempt to add validity and adherence to a study by taking participants’ preferences regarding treatment or other aspects into account.³³ The impact of the intervention is measured in this study with a pre-post design, meaning measurements were taken before the intervention and after the intervention. A quasi-experimental design suited our purposes as we preliminarily aimed to assess the impact and feasibility of the IT intervention in supporting adolescents’ well-being and mental health. A formal power calculation was not conducted as this small-scale study also focused on feasibility.³⁴ According to a previous study carried out in a school setting,³⁵ given the effect size of 0.64, with a statistical power of 0.8 and a confidence level of 0.05, a sample of about 138 subjects was needed. This is similar to other studies that have tested universal programs (n=146 adolescents;²³ n=157 adolescents;²²), although sample sizes have varied in previous studies from 78²¹ to 1767.³¹ The TREND statement (Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions) was followed.³⁶

The setting for the study comprised two lower secondary schools in one city in Finland. In the Finnish education system, compulsory basic education includes elementary schools (grades 1 to 6) and lower secondary schools (grades 7 to 9).

The two schools were allocated either to be the intervention or the control setting, based on the schools’ preferences. The schools were selected to be distant from each other so that contamination and the possibility that adolescents from the control school would benefit from the activities in the intervention school would be minimal.³⁷ Inside the schools, individual randomization for adolescents was not feasible given the possible information flow between the adolescents.

Our study targeted all adolescents in the 8th and 9th grades at our study schools (n=201 in the intervention school; n=157 in the control school). Adolescents were able to join the study if they were able to read, write and understand Finnish and were able to provide written, informed consent before any procedures. Exclusion criteria were inadequate Finnish skills, adolescents without written informed consent and adolescents under 15 without a written consent from their guardian. Moreover, adolescents were excluded if, according to their school physician, nurse or teacher, study participation would likely worsen their existing mental health problems or be otherwise unsuitable for them (eg, due to a developmental disorder).

Participant recruitment was carried out via a pre-existing IT system used for communication between the schools and homes, in a face-to-face information session for adolescents in the schools and through written information letters delivered to the adolescents. Adolescents willing to participate gave written consent before the study began. In addition, a guardian’s consent was sought from all the adolescents under 15 years old.³⁸ Due to the type of the intervention, eg, in one school adolescents used the IT program for 6 weeks and in another they did not use it at all, it was not possible to mask it for the participants.

The flow diagram of the study is depicted in Figure 1.

Figure 1 Flow diagram.

Intervention

At the intervention school, adolescents used the IT program for 6 weeks. The studied intervention is based on the previously developed²⁶ and studied DepisNet program for adolescents.^39–43 It was developed by searching the relevant literature on the topic, identifying the most useful theories guiding intervention development, conducting interviews with professionals to gain a better perspective on the world of adolescents and their needs and solutions to their problems, and cooperating with nursing staff to ensure that the intervention would be adopted as part of the existing treatment environment and procedures.²⁶ It consists of health information, caters flexibly for the problems that adolescents commonly face in their daily lives, and a variety of self-reflective exercises. Even though it was previously used as a targeted intervention in adolescent psychiatry, it was developed to complement the current services in outpatient care. Therefore, the main content of the intervention was not modified in this universal psychoeducation intervention used in school settings, which also aimed to support adolescents’ self-management and increase their awareness of issues associated with well-being and mental health.

The program is based on innate basic needs that motivate human behavior: autonomy, ie, the feeling of having control over one’s own actions; competence, ie, the sense of mastery over one’s environment; and relatedness, ie, having meaningful relationships and interactions with other people.^44,45 By adding these components of the self-determination theory (SDT) to our intervention,³⁸ we promoted adolescents’ autonomy in seeking information from the program that would support their well-being and mental health by encouraging them to find out information relevant to their needs. We motivated adolescents to be competent in managing their well-being and mental health by encouraging them to do weekly exercises in the program, which have been developed to support them. We also supported their relatedness to their own well-being and mental health through the tutor feedback on their exercises. SDT is used extensively in fields of education and health, and focuses on the individuals’ motivation and the interplay between the extrinsic forces affecting a person and the intrinsic motives and needs inherent to human nature.⁴⁶ The exercises in the modules included such tasks as writing a diary, taking a test related to internet use or health evaluation, and reflecting on the received test results in a diary. The intervention was elaborated by following TIDieR (template for intervention description and replication) checklist,⁴⁷ presented in Table 2. At the control school, the adolescents did not have the possibility to use the DepisNet program; instead, they followed their regular curriculum planned according to local needs and interests.

Table 2 Universal School-Based IT-Program Described According to TIDieR Checklist47

Procedure

Adolescents had individual user accounts and passwords to log in to the program, which were delivered to them by email. The program was available 24/7 during the study period. The adolescents were supposed to use the program during their school day once a week during a designated lesson, but they were also able to use it at home. According to previous literature on web-based interventions supporting adolescents and young people with depressive symptoms,⁴⁸ completing one module per week is typical. The program was used independently on the adolescents’ individual iPads, but during the specified weekly lesson, tutors were present to provide support if needed. The adolescents did the exercises on their own and returned them via an online learning management system (Moodle). A detailed description of the program and exercises are presented in Supplementary Table 1. Table 3 shows the various roles of the intervention providers.

Table 3 Intervention Providers and Their Roles

Instruments

Baseline Characteristics

Age, gender (male, female), grade (8, 9), computer and internet skills ranging from very good (1) to poor (5), and computer and internet attitude ranging from very positive (1) to very negative (5) were collected at baseline based on participant’s self-reporting.

Depression

The primary outcome was depression, which was measured with the Raitasalo Depression scale (RBDI⁴⁹). The scale, originally developed by Beck et al (1961),⁵⁰ measured experienced, and recognized and expressed depression, self-esteem and anxiety (13 items). The sum score of the scale ranges from 0 to 39; lower scores represent a better mental state related to depression. Cronbach’s α values in Finnish adolescent populations have been 0.87 (depressive symptoms) and 0.84 (self-esteem), indicating good reliability.⁵¹ In our study, internal consistency for the whole scale was good (α 0.85).

Quality of Life

The quality of life was measured using PQ-LES-Q,⁵² which consists of 15 self-assessing items regarding an adolescent’s own health and well-being (1=very poor–5=very well). Sum scores range from 14 to 70; higher scores represent better quality of life. For the analysis, the first 14 items were summed to form a total score. The internal consistency values have been found to be high (α 0.87–0.90).⁵² The Cronbach’s α for our study was 0.90.

Self-Esteem

Self-esteem was measured using the Rosenberg Self-Esteem Scale (RSES).⁵³ It consists of 10 questions in which sum scores are counted, ranging from 10 to 40; higher scores represent better self-esteem. Internal consistency of the scale in the Finnish adolescent population has been reported to be good (α 0.88).⁵⁴ In our study, however, the Cronbach’s α was below acceptable (0.55).

Self-Efficacy

Self-efficacy was measured using the General Self-Efficacy Scale by Jerusalem and Schwarzer (GSES).⁵⁵ The sum score ranges from 10 to 40; higher scores represent better reliability on oneself and capability to cope. Reliability has been found to be very good in the target population (α 0.89).⁵⁶ The internal consistency was also high in this study (α 0.90).

Feasibility was measured by collecting information about adherence (number of adolescents in total at the study schools, number of refusals, dropouts, lost to follow-up) and acceptance of the program by means of logging activity.⁵⁷ Activity information was collected from the learning management system, Moodle, and includes the total number of adolescents who logged in to the program, how many logged in to the program content, and the number of exercises done.

Data Collection

Baseline data was collected in November 2014 and the follow-up after the intervention in December 2014. Measures were delivered to each adolescent via their school emails using the Webropol system. The adolescents filled in the measures during a school lesson at baseline and immediately after the intervention.

Statistical Analysis

All statistical analyses were performed using SPSS (version 26.0).⁵⁸ Descriptive statistics were generated for baseline characteristics and outcome data. Baseline characteristics of the two groups were compared with T-tests and chi-square (X²) tests and outcome data (primary and secondary outcomes) with T-tests and Mann Whitney U-tests. To analyze the impact of the program, the change in the outcome measures were compared between the two groups by again using T-tests and Mann–Whitney U-tests. Changes within the groups were analyzed with T-tests and Wilcoxon tests. P values < 0.05 were considered indicative of statistically significant between-group differences.

Feasibility of the intervention was analyzed with descriptive statistics as follows: Adherence to the study was analyzed by calculating frequencies and percentages of adolescents willing to participate out of the whole study population, and adolescents filling in the baseline and follow-up measurements. This information was compared between the intervention and control schools. The data regarding program use (acceptance) was analyzed by calculating frequencies and percentages of adolescents logging in to the program out of the number of adolescents who provided the baseline measurement, adolescents logging in to the program contents, and the favorability of the modules (frequencies and percentage of adolescents returning to exercises per module in total and per exercise). This information was only collected from the intervention school.

Ethical Issues

The study was reviewed and approved by the Ethics Committee of the University of Turku (41/2014). The Ethics Committee approved the consent process, which allowed participants over 15 years of age to provide informed consent without parental consent.³⁸ Permission to conduct the study (403/2014 – 441/2014) was received from the Head Principal of the city. Ethical guidelines and the World Medical Association Declaration of Helsinki were followed and any regulations addressing the conduct of a vulnerable population such as minors were taken into account.⁵⁹ For example, adolescents decided for themselves whether to participate or not, and they were aware that refusal to participate would not in any way affect their studies. Tutors paid special attention to the concerns in adolescents’ depression scores or diary writings (eg, RBDI score ≥ 5, suicidal thoughts, self-harm), according to the protocol specifying the process of how to proceed in need of adolescent help. Adolescents were informed about this process when the study began. The tutor first contacted the adolescent if a concern arose and advised her/him to contact the nurse working at school. The school nurse registered the concern in the adolescents’ health records and proceeded according to the health care protocol used in Finland.⁶⁰ Adolescents (n=22) were contacted because of RBDI scores at baseline, at follow-up, or because of their diary writings. Anonymity and confidentiality were emphasized in the learning management system, Moodle, which is SSL protected with usernames and passwords. Usernames were used instead of real names. Adolescents used the program individually, and there was always a tutor present in the classroom or on the Moodle platform as a moderator to protect the privacy of the participants. Only the tutors and researchers had access to the adolescents’ names, backgrounds and outcome data. All of the tutors and researchers had a professional background in health care, and were trained to handle ethically sensitive data.

Results

Baseline Characteristics and Outcome Data

In the intervention group, participants were slightly older (p=0.009). There were fewer adolescents from the 8th grade in the intervention group (13% vs 33%, p=0.01). Computer and internet skills (p=0.16), and attitudes towards computers and the internet (p=0.19), indicated positive results in the intervention group, although the difference was not statistically significant. Comparison of the baseline information on the primary (depression, p=0.62) and secondary outcomes (quality of life, p=0.77; self-esteem, p=0.51; self-efficacy, p=0.34) showed no statistical differences between the study groups (Table 4).

Table 4 Baseline Demographic and Clinical Data

Impact of the Intervention on Adolescents’ Well-Being

Our analysis revealed no statistically significant differences between the groups regarding any of the outcomes, although the changes indicated positive results in the intervention group. Quality of life measured with the PQ-LES-Q indicated positive change in the intervention group, compared to the control group (Change mean 1.36 vs −0.49), although statistical significance was not achieved (p=0.10) (Table 5). There were no significant differences in any of the outcomes within the groups between baseline and follow-up (data not shown).

Table 5 Comparison of the Change in Primary and Secondary Outcomes in Intervention and Control Group

Feasibility

Adherence

Out of 358 adolescents assessed for eligibility, 151 were willing to participate in the study (Figure 1). At the intervention school, 39% (78/201) of the 8th and 9th grade adolescents participated in our study, compared to the control school, in which 46% of adolescents participated (73/157). In the intervention group, 78% (56/78) participated in the baseline measurement, while in the control group, the corresponding number was 75% (55/73). In the intervention group, 70% (39/56) of those who participated in the baseline measurement continued the study to follow-up, versus 78% in the control group (43/55). There was no significant difference between the numbers of loss to follow-up between the groups.

Acceptance

At the intervention school, out of the adolescents who filled in the baseline measurement (N=56) and participated the first DepisNet session, 95% (n=53) logged in to the program. Figure 2 shows the frequencies and percentages of the adolescents who logged in to the different modules of the program. Logins decreased during the intervention. Favorability of the modules, measured by how many times adolescents returned to different types of exercises (tests, diaries), varied between 43% (n=23) and 68% (n=36). Based on returned diaries, the adolescents most highly favored the last module concerning support for depression, and they were least interested in the third module concerning their rights. In total, 18% (n=10) of the adolescents completed all the exercises in the program. In each module, approximately 58% (n=31) of the adolescents did at least one exercise, and this proportion ranged from 47% (n=25) up to 68% (n=36).

Figure 2 Realized logging in to modules and completed exercises per adolescent in the intervention group.

Discussion

In the present quasi-experimental study, we assessed the preliminary impact and feasibility of the DepisNet program to support adolescents’ well-being and mental health as a universal school-based program. Based on our results, the program does not have an impact on the primary outcome, depressive symptoms. The change in adolescents’ quality of life indicated positive results in the intervention group, although the difference between groups or within groups was not statistically significant. However, we did achieve encouraging results related to the feasibility components of our study, adherence and acceptance.

Our analysis revealed no statistically significant differences between the groups in any of our outcomes, although the changes indicated positive results in the intervention group. This result is similar to those of some previous universal IT interventions realized at schools. For example, in Australia, Calear et al³² tested an e-couch Anxiety and Worry program in a large adolescent population and did not observe any significant differences between the intervention and control conditions. We found one similar intervention study that has been carried out in Finland since the time of our study (2014). Puolakanaho et al⁶¹ assessed the efficacy of a five-week web- and mobile-delivered intervention dealing with stress and coping skills. The program was called Youth COMPASS, and it was delivered among a general sample of ninth grade adolescents. The intervention group showed a small but significant decrease in overall stress, and the intervention gains were the largest among high-stressed participants.⁶¹ According to our knowledge, our study is still one of the first evaluating the impact of a universal IT intervention realized in schools with a focus on quality of life. Although less studied, this positive indication in adolescent quality of life that our program produced may have a significant role in preventing depressive symptoms.⁶² This finding warrants future studies to focus more on this aspect of well-being and mental health. Indeed, we located one registered trial (ClinicalTrials.gov) that is testing a transdiagnostic preventive program delivered in schools and consisting of IT, in which Păsărelu and Dobrean aimed to measure Health-Related Quality of Life.⁶³

Feasibility evaluation of program adherence revealed that 42% volunteered to participate in our study. This is a piece of information that has not been reported very often in previous studies. In our study, participation in the intervention was voluntary for adolescents. We are quite surprised that in a study by O’Kearney et al²² the whole target population consents to participate. In our study, the adolescents in the intervention school were less interested in participating in the study compared to those in the control school. Numbers did even out at the baseline, but at the follow-up, there were fewer participants in our intervention group compared to in our control group. Importantly, although fewer than half of the adolescents were willing to participate in our study, those who did participate adhered to the study well. There have been previous studies in which entire schools have discontinued because the adolescents found the program “boring”,²⁸ or the adolescents were simply not willing to continue.²³

The findings of the present study revealed that the number of logins to the modules decreased during the course of the program. This result is similar to those of previous universal IT interventions run in school settings.^29,30 Participant non-completion of the modules has been reported previously.^{20–22,28,29,31} It is difficult to establish real impacts of these interventions if adolescents do not complete the modules in them. Adolescents may not get involved in these types of interventions if they do not feel that the subject is intended for them or if the program seems to be designed for younger children, as a systematic review by Garrido et al points out.¹⁹ In our study, the number of adolescents who did at least one exercise during the module increased being the highest for the last module. This result may indicate that some exercises were more appealing for adolescents than others. Based on diaries returned by the adolescents, they were most favorable towards the module concerning support for depression. This is in line with a previous study in which adolescents lacked knowledge about mental disorders.⁶ On the other hand, the present study also revealed that one-fifth of the adolescents completed all the exercises in the program, which is less than in previous studies. For example, 56% of the adolescents returned all completed workbooks in a study entitled “Bite Back”,²⁸ and 30% of the adolescents completed the entire SPARX-R intervention.²³ In Finland, 75% of the adolescents did finish the Youth COMPASS program and were relatively satisfied with the intervention.⁶¹ In general, the rates of non-completion have been moderate to high across a number of online youth mental health promotion and prevention interventions.⁸ This may mean that the development of more focused interventions for specialized groups of users is needed. There is also a need for program developers to ensure that materials are presented in ways that build on the natural interests of adolescents, with a game-like feel and relatable, interactive content.¹⁹

Encouraging results in our study regarding the feasibility components may indicate that there is a need for IT-based interventions in school settings. Indeed, the COVID-19 pandemic has shown the need to strengthen our existing health services with technology-based innovations.⁶⁴ DepisNet aims to support adolescents’ relatedness,^44,45 and one explanation of our promising results may be that adolescents benefit from extra support from non-parental adults.⁶⁵ Participant face-to-face and/or web-based support has been found to be an important feature in online youth mental health promotion and prevention interventions in terms of program outcomes and completion.⁸ On the other hand, self-directed interventions may not be as well accepted as those led by professionals.⁶⁶ In the Finnish Youth COMPASS program, the non-completers were mostly male, and one reason for this could have been that the participants were expected to do the exercises in their own leisure time, as the program was not part of the regular school curriculum.⁶¹ Interventions should not be planned to be completed in participants’ own time, especially if hard work is involved in the learning modules, as adolescents have indicated disliking a strong educational focus in interventions.¹⁹ The amount of individual support provided for each participant is also crucial, as reported in a study by Topooco et al,⁶⁷ in which a therapist spent an average of 47 minutes on each participant every week and sent over 30 messages to each participant. The high number of human resources needed should be taken into account to ensure the optimal support for all adolescents and especially for those who have poor self-reliance.⁶⁸ To summarize, the DepisNet program carried out in a school setting seems to be a feasible universal intervention among adolescents, with promising results in promoting adolescents’ awareness of adolescents’ general well-being and mental health issues.

Limitations

The study also yielded limitations. First, it is difficult to estimate how much adolescents spent time with the program separate from their regular school curriculum, which limit comparability to other similar interventions. Second, we were not able to perform analysis of non-participants which could have provided important information to whom such intervention is feasible. It is also possible that the participants that refused to participate were the adolescents that most needed the intervention. In the intervention group, participants were also slightly older than in the control group. Third, we aimed to increase motivation to complete the study by giving participants in both schools a voucher to the cinema. This was not revealed to the adolescents at the time of informed consent in order to not influence their decision to participate. However, this incentive likely increased the program use and study completion rate and limited the generalizability of the acceptance results. Fifth, the name of the program, DepisNet, might have deterred some adolescents from participating in the study because of stigma. Huang and Bashir found that apps with titles relating to symptoms have significantly lower installs, ratings, and reviews than those without symptom-related titles.⁶⁹ Sixth, the internal consistency of our self-esteem measurement was poor, the Cronbach’s α was below acceptable Seventh, this study was not based on individual randomization, which may have caused biases. The characteristics of the adolescents could have been very different between the two schools. This could have potentially led to difficulties in recruiting participants to the control group.⁷⁰ However, the comparison of the adolescents’ baseline characteristics showed no statistical differences between the study groups, and recruitment did not significantly differ between the groups. Therefore, we may assume that the degree of bias related to randomization was low. Eighth, as the schools were allocated based on their preferences, self-selection bias is possible. We may also question if we can actually talk about impact of our study as the sample size is small, we did not use a randomized control trial design, and only short-term outcomes were evaluated. Lastly, much has changed in the technological landscape as well as in society in general since this study was conducted in 2014. If the study were carried out today, there could be more interactive small groups and workshops, and more gaming and commercial features included in the program to engage a wider variety of users.

Conclusion

Universal interventions and programs that support adolescent well-being and mental health can be integrated into school curricula to promote their awareness of general well-being and mental health. The knowledge gained in this study can also be used when there is a need to include up-to-date content in school curricula to support adolescent well-being and mental health. Such needs occur when distance learning increases or during a sudden crisis, such as the COVID-19 pandemic, when adolescents are forced to cope and study in new ways. Moreover, future work needs to consider the changing face of IT, methods for how to engage a variety of user groups, information on how to support aspects of IT that complement traditional health care methods, and how to promote mental health literacy in school environments.