Back to Journals » Journal of Multidisciplinary Healthcare » Volume 15

Effectiveness and Implementation Outcome Measures of Mental Health Curriculum Intervention Using Social Media to Improve the Mental Health Literacy of Adolescents

Authors Hassen HM , Behera MR, Jena PK, Dewey RS , Disassa GA 

Received 5 February 2022

Accepted for publication 14 April 2022

Published 3 May 2022 Volume 2022:15 Pages 979—997


Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Scott Fraser

Hailemariam Mamo Hassen,1 Manas Ranjan Behera,1 Pratap Kumar Jena,1 Rebecca S Dewey,2 Getachew Abeshu Disassa3

1School of Public Health, KIIT Deemed to Be University, Bhubaneswar, Odisha, India; 2School of Medicine, The University of Nottingham, Nottingham, UK; 3College of Education and Behavioural Sciences, Jimma University, Jimma, Ethiopia

Correspondence: Hailemariam Mamo Hassen, School of Public Health, KIIT Deemed to Be University, Email [email protected]

Purpose: Globally, adolescent mental health problems reportedly manifest more severely in individuals with lower mental health literacy. Mental health curriculum interventions using social media have been associated with positive implementation outcomes. This study aimed to investigate whether such an intervention significantly improves adolescent mental health literacy and is associated with positive implementation outcome measures.
Methods: This study employed a quasi-experimental design, comparing the before-and-after difference between the intervention group (n1=77) and the control group (n1=76). A mental health literacy module was delivered using social media. Data were collected using validated questionnaires. Effect size and difference-in-differences model calculations (and associated 95% confidence intervals, CI) were conducted alongside other descriptive analyses.
Results: The estimated intervention effect size and difference-in-differences estimates were greater than zero. The effect was greater in the intervention than in the control group (p< 0.05) despite sex and age differences. The effect size estimate provided by Cohen’s d and Hedges’ g values was medium-to-large (d/g=0.429– 0.767, p< 0.05). The difference-in-differences reflected a significant effect (DID=0.348, CI: 0.154– 0.542, p< 0.001) in increasing mental health literacy despite differences associated with sex, age, and school grade. The intervention program was acceptable, appropriate, feasible, and satisfactory, and more than 80% of participants said they agree to completely agree with these implementation outcomes. Resources and personal and family-related factors were among the perceived influencing factors determining the effectiveness and implementation outcome measures.
Conclusion: The intervention program effectively improved the mental health literacy of adolescents. It has been highlighted that social media interventions for mental health promotion could be practical and scalable; however, there is a need to take into account socio-demographic differences and barriers to inclusion/compliance.

Keywords: adolescent mental health, mental health promotion, difference-in-differences, effect size


Adolescent mental health problems tend to be more serious in individuals with lower mental health literacy.1,2 Mental health interventions using social media are thought to be effective and have shown positive implementation outcome measures for mental health promotion.3,4

Addressing mental health problems in children and adolescents has emerged as a public health priority. Globally, approximately 10–20% of adolescents live with mental health problems,5,6 and most do not have access to the necessary health services. Approximately 17–23% of Ethiopian adolescents have mental health problems,7,8 which is higher than the global prevalence.3 Implementing mental health services in low-income countries like Ethiopia is highly challenging.9 This has been combined with reportedly low levels of mental health literacy, ie, knowledge and awareness of mental health issues.9

Mental health literacy measures, help-seeking behaviour, and adherence to mental health services have been shown to worsen with complex health disparities resulting in a gap in mental health promotion and mental ill-health prevention.7,8 Limited help-seeking behaviours and delays in receiving help have been reported by adolescents with mental health symptoms.1,2,10,11 These adolescents face difficulties recognizing how and when to seek support and help.12 Several studies have revealed that low help-seeking intention and behaviour in adolescents reflect a lack of awareness about the nature and symptoms of mental health problems and alternative sources of help.11 These studies are sometimes limited in their scope for assessing the impact of a mental health curriculum as more attention has been given to individuals with mental health problems than to individuals who appear to be apparently well.13

Mental health promotion has evolved to take on a new positive perspective, termed good mental health,14,15 comprised of several domains, with one of these domains being mental health literacy.15 Interventions to improve modifiable factors, like mental health literacy, in adolescents have been shown to influence mental health outcomes.16–20 In this regard, mental health literacy has been shown to be improvable through the use of a mental health curriculum. Mental health literacy is an individual’s ability to recognize mental disorders and forms of psychological distress, as well as the extent of their knowledge, beliefs, and attitudes towards mental illnesses, their risk factors and causes, and the sources of help available.16–20 However, effective, acceptable, appropriate, and feasible mental health promotion approaches targeting the mental health literacy of adolescents both in and outside of formal education are very limited, and reports of their efficacy have been inconclusive.21

It is imperative to emphasize adolescent mental health promotion for multiple reasons.3,11,21–30 The nature of adolescence as a developmental stage makes this age group a priority for public health scholars and policymakers. The onset of many mental health problems and disorders often occurs in adolescence,11,22,23 and the prevalence of chronic illnesses and their associated mental health problems increases in adolescent populations.30 Physical, cognitive, emotional, and behavioural development during adolescence is critical to the lifelong wellbeing of the individual.24–26 These developmental processes shape health-related behaviours and skills.27 Consequently, mental health during childhood and adolescence significantly affects the health outcomes of the next generation.3,21 Therefore, early intervention is worthwhile to improve the understanding of health information,28 and ensure good health later in life.29

The use of a mental health curriculum to promote positive mental health aims to achieve this by improving adolescents’ knowledge, beliefs, and attitudes towards recognizing mental disorders and their risk factors and being aware of available sources of help and information.16–18 However, the mode of delivery has become a challenge in the face of financial hardship, inefficiencies, and inequitable resource allocation. Inadequate health systems and structural inequalities in low-income countries like Ethiopia and other African countries demand innovative approaches to improving mental health literacy. Furthermore, unexpected global challenges such as the COVID-19 pandemic have created multiple physical and logistic barriers.1

To study the impact of interventions on populations, it is necessary to study groups of people with common backgrounds and characteristics and using communication modalities and strategies amenable and available to these target populations.14,21 As such, schools are ideal settings for health promotion in children and adolescents.31 However, the effectiveness, acceptability, appropriateness, and feasibility of this approach have been challenged,21 demanding a need for innovative collective approaches using the emerging communication technologies of digital and online platforms. These approaches working through schools as supporting platforms to establish direct channels of communication with adolescents are thought to be effective, sustainable, and scalable.3,4,29 Capitalizing on the internet and mobile technology using communication channels such as text messages,32 mobile phone-based peer support,33 and web-based interventions,32 offers many potential benefits for both the intended service users and the service provider.34

Social media is commonly used for health information exchange,35,36 and has become an integral part of adolescent life.37 Compared to any other age group, a higher proportion of adolescents and young adults are social media users.38 Social media allows individuals to exchange information and ideas, including those related to health. Ethiopia has experienced a rapid growth in internet use, and adolescents use social media increasingly in daily life.39 This offers opportunities for improving social connectedness, resilience, and well-being.36,40 Social media is established as a medium of communication for health promotion and intervention in developed countries.

Interventions to promote mental health using social media are effective, acceptable, feasible, appropriate, and popular.35,36 However, in low-income countries, digital inequality and related factors negatively impact the successes of these interventions. There is little quantitative evidence supporting the effectiveness of social media-assisted mental health promotion in adolescents in low-income countries, particularly in African countries such as Ethiopia.39,41 Little is known about the implementation outcome measures of such efforts, or about the factors influencing them.42

Consequently, the present study was conducted to assess whether intervention using a mental health curriculum through guided social media use effectively improves adolescent mental health literacy. A hypothesis stating that an intervention using social media significantly improves the level of mental health literacy in adolescents compared to that in a control cohort (p<0.05) was tested. Evidence from direct beneficiaries is needed to understand the effectiveness of an intervention for health promotion to strive for equitable and resilient health systems for universal health coverage.43

Understanding implementation outcome measures and the factors influencing them is necessary to help develop comprehensive intervention programs. Notwithstanding the importance of implementation outcome measures, little is known about them in the context of conducting such work in a low-income country. Consequently, this study was based on the theory of change aligning with the integrated theory of health behaviour change. The integrated theory of health behaviour change (ITHBC) states that enhancing knowledge, attitudes, and beliefs around mental health issues (mental health literacy) results in behaviour change towards improved mental health.44,45 The theory of change (ToC) explains situations and phenomena from the standpoint of conceptual and empirical evidence by describing the mechanisms by which the intervention is intended to work.46 These theoretical backgrounds provide opportunities for reflection around intervention design, implementation, and outcome analysis. The theory of change framework (Figure 1) reflects the main target of intervention and the secondary objectives in the form of implementation outcome measures.

Figure 1 The intervention outcome mapping in the lens of the theory of change.

This body of work primarily aimed to investigate the effectiveness of a mental health curriculum intervention delivered over social media to improve adolescent mental health literacy using effect size and difference-in-differences. The secondary analysis focused on the four measures of intervention implementation outcomes: the acceptability of the implementation, its appropriateness, its feasibility, and the satisfaction of the beneficiaries. Further, we assessed the perceived factors affecting the effectiveness of the intervention and these outcome measures.

Materials and Methods

Study Population

This study was conducted in public and private schools in urban areas of Dire Dawa province, Ethiopia, from the last week of June to the first week of August 2020. Dire Dawa city is diverse in the ethnicities, cultures, languages, and socio-demographic factors it hosts. In the pretest period, the research team recruited adolescents aged between 15 and 19 years who had smartphone and social media access. Adolescents below 15 years of age were excluded because they were not expected to have smartphones or internet access, or the familiarity with these that is required to engage fully with the intervention activity.47

Study Design

This study employed a quasi-experimental design, comparing the before-and-after difference between the intervention (treatment) group and the control group (Figure 2). Conducting a randomized control trial was unlikely to be feasible because it was costly and had complex ethical issues. As such, the assignment of participants to treatment and control groups was based on the need to avoiding contamination and bias during the intervention and to ensure recruiting a reasonable number of intervention and control participants. Furthermore, it was difficult to control all known and unknown factors that may influence the outcome of the intervention. Participants were recruited into geographically distant intervention and control groups based on the locations of their school and area of residency to reduce contact between the two groups during the intervention period and as such minimize contamination between groups.

Figure 2 Flow chart of the study design.

Sample Size Determination and Sampling Procedures

The power analysis for this study was based on the primary aim of comparing the change in mental health literacy scores from pretest to post-test and comparing this between intervention and control groups. The minimum sample size was based on a systematic review of interventions for mental health literacy, using on the reported effect size, δ=0.823 with a standard deviation, σ2=1.55.14 The calculation assumed a confidence of level of 95% (Za/2 =1.96), power of 90% (Zβ=1.282), and type 1 error α=5%. Using a validated sample size calculation for a quasi-experimental study,48 a minimum of 146 participants (73 for each arm) were needed. However, the aim was to exceed that sample size. Of the 731 potential participants who met the initial inclusion criteria, 156 were recruited and allocated to the intervention and (n1 = 77) and control (n2 = 76) groups and completed the study (Figure 2). This sample size represents approximately 105% of the calculated minimum sample size.

Intervention Materials

The intervention took the form of a mental health curriculum aimed at adolescents to improve their mental health literacy. It was adapted from the school mental health curriculum guides already prepared for African, Australian, and Canadian adolescents that aimed to improve mental health literacy and mental health first aid skills.40,49–54 The intervention material was prepared and presented for the present study by summarizing the manual’s texts, figures, and case vignettes in plain English. The contents of the intervention focused on how to seek information, awareness of common mental disorders and mental health problems, awareness of risk and protective factors, causes of mental illness, awareness of methods for self-help or for helping others, and awareness of the professional or informal help available, attitudes related to promoting recognition of mental health problems and/or appropriate help-seeking behaviour, and recognizing mental illnesses and sources of help.

English is the medium of instruction used in Ethiopian junior schools and teaching above grade levels and is taken as a school subject starting from kindergarten. Furthermore, science subjects are given in English regardless of the area’s native language. Thus, English was used for the intervention.

Intervention Procedures

A pilot intervention was conducted prior to the main intervention study.47 Facebook and Telegram were reportedly the most commonly used social media platforms among Ethiopian adolescents, and similarly, in Dire Dawa. Hence, participants were registered to Facebook and Telegram groups.47 A 45-megabyte internet package was provided to each participant for completing each intervention session as an incentive and to improve recruitment, compliance, and retention of study participants.

For the intervention group, the summarized texts, figures, and case vignettes were posted on both social media platforms every 72 hours (three days) on working days (Monday to Friday) in eight sessions across approximately six consecutive weeks. Discussion between participants was allowed on the platform and a summary was provided at the end of each week (every two sessions). The control group received information with a similar frequency and timescale, but instead, the content was that of “secrets of successful students”. Post-test data collection was conducted from all study participants (ie, both intervention and control groups) at the end of week seven, one week after the conclusion of the social media information delivery.

All reasonable efforts were made to minimize the impact of errors and bias. Participants engaged in orientation and closing discussions throughout the study period. All data entry was double-checked by a data clerk. A thorough descriptive and inferential analysis was conducted objectively using standardized models. The study design controlled for the anticipated effect modifiers and confounding factors through the implementation of strict eligibility criteria.

Variables and Measurement Tools

The study variables included sociodemographic characteristics (sex, school grade, and age), whether they were allocated to the intervention group (Gi=1) or control group (Gi=0), time point as pretest (t=0) or post-test (t=1), mental health literacy, and intervention implementation outcome measures and influencing factors. Mental health literacy was measured using the mental health literacy questionnaire, MHLq,55 a freely available and validated tool used in adolescent populations.17,55–63 We further tested its reliability in the present study setting (Cronbach’s alpha=0.834). The version of the MHLq used in this study was comprised of 33 items, with each item eliciting a response using a five-point Likert scale (1=strongly disagree, 2=slightly disagree, 3=neither agree nor disagree, 4=slightly agree, 5=strongly agree). Items comprised statements covering recognition, awareness, attitudes, and beliefs towards mental health issues.16–20

The intervention implementation outcome measures were evaluated using a validated, questionnaire,64 with items eliciting a response using a five-point Likert scale representing the level of agreement (1=completely disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=completely agree). Outcome measures were used to assess the extent of acceptability, feasibility, appropriateness, and satisfaction of the intervention.

Acceptability was measured using 4 items encompassing whether the intervention met the participant’s acceptance, the extent to which they liked it, whether they found it appealing, and whether they found it welcoming. Feasibility was measured using 4 items eliciting to what extent the intervention approach was implementable, possible, doable, and easy to use. Appropriateness was measured using 4 items covering whether the intervention seemed to fit its purpose, was suitable, applicable, and a good match. Satisfaction was measured using 7 items consisting of the quality of the intervention, whether it was wanted, whether it met the participant’s needs, whether they would recommend it to others, the extent it helped them to deal with their problems, and the extent to which they would like to repeat the experience. Perceived factors influencing the effectiveness of the intervention and implementation outcome measures were measured in a similar manner using a 5-point Likert scale where participants were asked to indicate their level of access to resources, and their experience of personal and family-related challenges, so that these factors could be controlled for in the analyses.

Statistical Analyses

The baseline analysis was performed using descriptive statistics of the sociodemographic characteristics. Subsequently, we analyzed the effectiveness of the intervention in terms of the effect size and the difference-in-differences estimate. Finally, we analyzed the implementation outcome measures and related influencing factors. All statistical analyses were conducted using SPSS version 25.65

Effectiveness of the Intervention

Mental health literacy scores were treated as intervals following the measurement framework used by Stevens.66 Negatively keyed items of the mental health literacy questionnaire (Q7, Q12, Q15, Q17, Q24, and Q26) were reverse scored and individuals’ scores were computed. The analysis was restricted to subsets of groups and characteristics to address confounding factors and to determine which group of adolescent participants most benefited from the intervention, based on their background characteristics. We computed the mean differences associated with the intervention (posttest-pretest score) within each group using a paired t-test and then estimated the effect size. The effectiveness of the intervention program was expressed as effect size and difference-in-differences estimate. Significance was considered at a threshold of p≤0.05.

Effect Size Analysis

Effect size is typically assumed to be the most important outcome measure in practical intervention and impact evaluation studies as recommended by Jacob Cohen.67,68 Effect size is a preferred way of communicating the practical significance of a result in evidence-based and field-based studies of interventions.69,70 Effect size was estimated using a 95% confidence interval and expressed using Cohen’s d and Hedges’s g.

These expressions use a pooled standard deviation in the denominator; however, the latter uses n, and the former uses n-1. These two different methods of calculating the standard deviation reflect the mean score difference between the intervention and control groups. Hence, the absolute effect size is the difference between the mental health literacy scores in these two groups. The meaning of this intervention effect was interpreted from both the conventional approach and the comparison approach based on a broad conventional point of reference and comparison with previous studies, respectively.68,71

Difference-in-Differences Analysis

The effectiveness of the intervention at improving mental health literacy was also estimated using a difference-in-differences analysis (Figure 3). The difference-in-differences (DID) analysis is a regression analysis exploring the time dimension of the data to define the actual and counterfactual effects of the intervention. It was estimated by comparing the difference in the outcomes between the intervention and control groups over the six weeks of the intervention period.

Figure 3 Conceptual framework, regression model, and coefficients for the estimation of the difference-in-differences.

The difference that existed before the intervention program was addressed by assessing the presence of unobserved heterogeneity. Such an analysis approach has the advantage of controlling for differences arising from unobservable characteristics. Hence, the intervention effect was obtained by assessing for a difference between the groups, as shown in Figure 3, which displays the steps taken in the analysis. After computing the before-to-after difference for each group [E(YIai-YIbi | Di=1) and E(YCai -YCbi|Di=0)], we computed the average intervention effect [E(YIai-YIbi | Di=1)-E(YCai -YCbi|Di=0)] where the subscript a denotes “after” and b denotes “before” the intervention, and the superscripts I and C indicate the intervention and control groups, respectively. Dgt =1 if unit g is exposed to the intervention in period t, and Dgt =0 if unit g is exposed to the control condition in period t. Y(1)gt represents the outcome of interest for unit g during period t under the hypothetical scenario g received the treatment during t. Y(0)gt is the outcome of the same unit and time under the alternative scenario in which g received the control condition.

The estimation of the difference-in-differences regression relies on certain assumptions. The unobserved heterogeneity is assumed to be time-invariant, addressed by comparing the before and after states. The common trend assumption states that in the absence of the intervention or treatment, both the control and intervention groups experience the same measurement trend of outcome variables over time. Therefore, any difference from the trend observed in the intervention group is the effect of the intervention. Other assumptions considered for this analysis were the linearity of all relationships, the independence of observations, the absence of perfect collinearity, and the non-zero variances of independent variables; error terms were assumed to be zero and normally distributed.

It is assumed that, in the absence of any intervention, the difference between the “intervention” and the “control” groups is constant over time (the two groups would have evolved in the same way in the absence of any intervention). Therefore, the average outcome in the control group is β0 during the pre-treatment period and β0+β2 in the post-treatment period. Likewise, the average untreated outcome in the intervention group is β0+β1 during the pre-treatment period and β0 + β1 + β2 in the post-treatment period, where β1 is different irrespective of the intervention effect. Thus, the difference in pre-treatment is normal while the difference in post-treatment can be assumed to be a causal effect (difference-in-differences, β3).

In the difference-in-differences analysis, these assumptions restrict the scope of the possible confounders that are not entirely dependent on random assignment. It assumes that confounders varying across the groups are time-invariant and that time-varying confounders are constant. Hence, the difference-in-difference is estimated using a model that accounts for target variable measures, group membership, and time periods. The coefficient of the interaction term (β3) is an estimate of the treatment effect. Importantly, the timing effect in the intervention and control groups is not equal. The two groups are different at the start, and the observational characteristics change over time.

Implementation Outcome Measures and the Influencing Factors

Descriptive analyses were used for each outcome measure, and response. The mean score and differences across sexes, age groups, and school grades were analyzed using independent t-tests. Similarly, descriptive analyses were used for each factor that was being assessed for its effect on the implementation outcome measures and the potential effectiveness of the intervention program.


Sociodemographic Characteristics

Participant socio-demographic characteristics are presented in Table 1. The mean ± standard deviation age of participants was 17.34 ± 1.31 years for the intervention group participants and 17.21 ± 1.36 years for the control group, and there was no statistically significant difference in any demographic characteristic between groups (ie, all p>0.05).

Table 1 Summary of Demographic Characteristics for Intervention and Control Groups

Mental Health Literacy Pre-and Post-Intervention

Pretest and post-test mental health literacy scores of participants in the intervention and control groups are presented by age group and sex (Figure 4). The mean pretest and post-test mental health literacy scores of girls were slightly higher than those of the boys. The mean pretest and post-test mental health literacy scores of the control group were lower than those of the intervention group. The paired-samples t-test showed a significant change between mean pretest and post-test scores.

Figure 4 The pre-test and post-test scores of mental health literacy for both intervention and control groups.

Effect of the Intervention on the Mental Health Literacy of Adolescents

Effect Size Estimate of the Intervention

The paired samples t-test showed a significant change in the mean mental health literacy score between the pretest and post-test time points (p<0.01) for both the intervention and control groups (Table 2). However, effect size estimates in the intervention group were greater than in the control group. The effect size for mental health literacy was greatest among participants aged 15–17 years, and for girls participants. The independent samples t-test on effect size showed a significantly greater effect in the intervention group compared to the control group (p=0.028). This effect size was higher in girls aged 15–17 years (Cohen’s d=0.2.042, Hedges’ g=2.045, p<0.001) than in boys of the same age (Cohen’s d=0.141, Hedges’ g=0.141, p>0.05). The effect size was lower, and non-significant in participants aged 18–19 years (p>0.05) (Table 2).

Table 2 Effect Size for Chang of Total Mental Health Literacy Score in Intervention and Control Groups by Age and Sex

Difference-in-Difference Estimates of the Intervention

The regression model for the difference-in-differences analysis was stratified by sex, age group, and education level (Figure 5 and Table 3). These models were used to estimate results for the intervention and it was estimated that the intervention had a highly significant impact (DID: 0.348, CI: 0.154–0.542, p<0.001), in support of the proposed hypothesis. The increase in mental health literacy scores differed by sex and age.

Table 3 Difference-in-Differences (DID) Estimating the Effect of the Intervention on the Mean Score of Mental Health Literacy

Figure 5 Difference-in-differences showing the effect of the intervention on the mental health literacy score. (A) All participants; (B) Boys (15–17 years); (C) Boys (18–19 years); (D) Girls (15–17 years) and (E) Girls (18–19 years).

Table 3 showed that the mental health literacy score significantly increased despite differences associated with sex (boys DID: 0.308, CI: 0.042–0.574, p<0.05 vs girls DID: 0.398, CI: 0.120–0.677, p <0.01), age group (15–17 years DID: 0.295, CI: 0.020–0.570, p<0.05 vs 18–19 DID: 0.392, CI: 0.117–0.667, p<0.001), and education level (grades 9–10 DID: 0.371, CI: 0.096–0.646, p <0.01 vs grades 11–12 DID: 0.324, CI: 0.050–0.597, p <0.05).

Implementation Outcome Measures and Influencing Factors

The analysis aimed to assess the self-rated implementation outcome measures and potential factors that may influence these implementation outcome measures.

Implementation Outcome Measures

The majority of respondents either agreed or completely agreed about the acceptability, feasibility, appropriateness, and satisfaction of the implementation of the intervention (Figure 6). However, ratings differed significantly across age groups, with a higher proportion of older adolescents reporting complete agreement for acceptability, appropriateness, feasibility, and satisfaction. Ratings did not differ significantly with sex (Table 4). More than 60% of participants responded that they completely agreed with the acceptability of the implementation (range 55–72%), the extent to which they liked it, found it appealing, and found its features welcoming. Ratings of feasibility, consisting of to what extent it is implementable, possible, doable, and easy to use, varied between 35 and 50%. Appropriateness, namely the east of fit, suitability, applicability, and the degree to which the implementation was a good match varied between 57 and 63%. Finally, satisfaction, the perceived quality of the implementation, the degree to which it is wanted, the degree to which it met the participant’s needs, how likely they would be to recommend it to others, the extent to which it helped deal with their problems, and whether they would like to repeat it, also varied between 45 and 71%.

Table 4 Differences in Perceived Intervention Implementation Outcome Measures by Gender, Age Group, and School Grade

Figure 6 Percentage of intervention group participants’ rating on the intervention implementation outcome measures.

With a minimum score of 5 and a maximum of 20, the mean and standard deviation ratings given by participants are separated and analysed by sex, age group, and school grade, along with testing the significance of any differences (Table 4). There was no significant sex difference in the scores given for the implementation outcome measures (p≥0.05). There were significant age differences in the ratings of implementation acceptability (p=0.003), appropriateness (p=0.027), feasibility (p=0.012), and satisfaction (p=0.013), where participants aged 18–19 years rated the implementation more highly. Ratings of satisfaction differed significantly (p=0.016) by school grade, where participants from grades 11–12 gave higher satisfaction scores.

Influencing Factors

The factors likely to influence implementation outcome measures were a lack of resources and personal and family-related challenges (Figure 7). Analysis focused on the proportion of participants who responded either in agreement or complete agreement with each of the items. Lack of access to the technological resources such as smart phone or computer (reported by 71.5% of participants), a lack of internet access (77.9%), or poor internet coverage (72.7%) were commonly reported. Personal influencing factors including a lack of digital skills and difficulty using technology (66.2%), lack of time to engage with the intervention (58.5%), unfamiliarity with social media (62.3%), and lack of motivation (57.2%) were less common but still reported by over half the respondents. Using social media incorrectly or spending time on unnecessary websites (57.2%), lacking confidence and trust in the privacy or confidentiality of the intervention (44.2%), and carelessness, laziness, or having little interest in social media (55.9%) were other factors reported by respondents in the sample. The family restriction of not being allowed to use social media or the internet (53.2%) was also reported as an influencing factor.

Figure 7 Self-ratings of agreement with factors supposedly influencing the intervention implementation outcomes.


The present study primarily investigated the effectiveness of a mental health curriculum intervention delivered over social media to improve adolescent mental health literacy using effect size and difference-in-differences. The secondary analysis evaluated the four measures of intervention implementation outcomes viz; the acceptability of the implementation, its appropriateness, its feasibility, and the satisfaction of the beneficiaries. Further, the perceived factors affecting the effectiveness of the intervention and these outcome measures were assessed. For the notion that mental health literacy is modifiable and is related to individuals’ knowledge, beliefs, and awareness of mental health issues;16–20 these findings are vital for enhancing outcomes such as self-efficacy, help-seeking intention, and health behaviour as well as fostering positive mental health and wellbeing.3,21 These changes are variously immediate and intermediate, with outcomes expressed as either proximal or distal outcomes.45

The overall findings from this study are that the intervention effect was greater than zero, and that the magnitude of the effect can be expressed using the standardized metrics of effect size and difference-in-differences estimate. Both the effect size and the difference-in-differences analysis showed a significant effect of the intervention (p<0.05) compared to a control condition. Nevertheless, the magnitudes of these metrics varied with sex and age. The implementation of the intervention was found to be acceptable, appropriate, feasible, and satisfactory despite several influencing factors that varied with the socio-demographic characteristics of the participant.

Effect sizes for quantitative intervention analyses of outcomes can be interpreted using a variety of approaches, including convention and comparison,68,69,71,72 with much evidence available in the published literature advocating strongly in favor of using a convention approach for studies such as this one. According to the conventional approach, effect sizes expressed primarily using Cohen’s d or Hedges’ g values can be categorized into small (below 0.2), medium (0.2 to 0.5), or large (0.5 or higher).67 The estimated intervention effect size estimate of this study was greater than zero; and the intervention program has potentially improved the mental health literacy of adolescents. The effect size observed in the present study would be conventionally classified as medium-to-high, varying by both sex and age. Despite there being little evidence advocating the comparison approach, the effect size seen in the present study (d=0.429–0.767, p<0.05) was consistent with previously published meta-analysis findings on universal and selective intervention studies to improve mental health literacy using a mental health curriculum (d=0.541–0.774, p<0.05).14 It was also consistent with a result of another metal analysis of a digital interventions, social media included to improve positive mental health outcomes among young people involving supervision (Cohen’s d=0.52) and without involving supervision (Cohen’s d = 0.33).73

Results from regression models that stratified data by sex and age group have reported the coefficient of the estimated difference-in-differences as being reflective of an intervention program having a higher and more significant impact (p<0.001). The level of mental health literacy dramatically increased with the intervention despite sex and age differences. Hence, the intervention implemented using social media was shown to improve adolescents’ mental health literacy significantly, in support of the original hypothesis. Previous qualitative evidence has shown that health interventions using online technology have been able to overcome multiple physical and logistical barriers and challenges.1 Findings from the present study were consistent with those of previous studies despite little quantitative evidence being available.1,74

Socio-demographic and socio-economic factors have been shown to influence the effectiveness and outcomes of health interventions.75 A growing number of studies have revealed that gender dynamics affect outcomes aiming to introduce child and adolescent health-related-behavioural change.76 As such, the regression models were stratified after multiple observable confounders and had been controlled for by assessing the significance of the variability in mental health literacy across these groups. For example, an earlier study conducted in the Ethiopian population showed that girls had slightly higher mental health literacy scores than did boys42 consistent with other studies conducted in Portugal, Australia, and the USA.55,59,77 Further, the effectiveness of the mental health literacy intervention conducted in this study varied across sex and age groups, independent of the methods used to measure it. The difference-in-difference estimate was higher among girls than boys, and the effect size was higher for participants aged 15–17 years (p<0.05).

Age is known to be a factor associated with health intervention outcomes.78 From the present study, effect of the intervention on the mean score of mental health literacy estimated with difference-in-differences (DID) was significantly higher for older adolescents than for younger adolescents. The probable explanation might be that older adolescents spend more time and participate with more attention than the younger adolescents. The younger adolescents may have shorter attention spans and lower focusing. The differences in the cognitive and behavioural skills of adolescents have also been shown to influence such intervention outcomes.27

The findings of this study contribute greatly to the existing literature describing interventions to improve the mental health literacy of adolescents, as well as to the literature assessing implementation outcome measures. First, such quantitative evidence in the form of standardized effect sizes and difference-in-difference estimates provides an opportunity to demonstrate the practical significance and impact of digital and web-based interventions promoting adolescent mental health literacy instead of only reporting the statistical significance. Second, effect sizes and difference-in-difference analyses allow researchers to draw meta-analytic conclusions across studies. Third, these effect size and difference-in-difference findings contribute to the design and planning of new studies, including the estimation of required sample sizes with the desired likelihood.70

Participant ratings of the perceived outcome measures of the intervention implementation process were significantly different across age groups (p<0.05), with older adolescents reporting closer agreement with the implementation outcome measures. This effect of age may explain the difference between technology and internet access and challenges related to digital literacy. Older adolescents may find it easier to access the internet and may have better digital literacy. These findings were consistent with the findings of other similar studies.37,79

Significant variation in observable measures was seen with the demographic characteristics of the participants. Participant sex has been shown to be one of the main factors predicting internet use behaviours,79 with girls spending more time on social media, and being more receptive to digital interventions to behaviour change.79 However, in the present study there was no significant sex difference in implementation outcome measure ratings, which is inconsistent with previous reports.37,79,80 Detailed qualitative explanations are needed to interpret such quantitative findings.

This study assessed several potential factors that were suggested to influence the effectiveness of the intervention and the intervention implementation outcome measures. Unavailability or lack of access to resources (technology or internet access), personal factors (confidence, concerns about privacy, digital skills, frequency of interaction, motivation, using other unnecessary websites, and other personality traits) and family-related factors (being prohibited from accessing social media or the internet) were perceived factors affecting effectiveness and implementation outcome measures. This finding was consistent with the outcomes of related studies of digital or web-based interventions,81–85 whereby behavioural factors,81 digital or internet literacy,82 lack of money or internet access,83 adherence,84 and participant engagement,85 were reported to be associated with the effectiveness of an intervention. Participant interest and motivation, as well as having an understanding of their concerns, emerge as key factors for effective digital or web-based intervention programs.86

The present study has both strengths and limitations. The study used a before-and-after design with the control mechanism utilizing both a control group of participants and a pretest/post-test comparison to assess the effect of the intervention. This study was conducted in a real-life setting and with a low implementation cost, avoiding many confounds to internal validity.87 However, like any quasi-experimental study, these findings may be subject to biases. The conventional interpretation of the effect size has limitations due to the absence of reliable empirical evidence of the effects of such an intervention at the population level. Further, due to the absence of much existing published evidence, subtle differences in study designs, scales, and measurement instruments may limit the validity of a comparison approach.68,69,71,72 Reliability and scalability might be confounded by deviation from pragmatic conditions, offering incentives, increased handholding, selection bias, confined study environment, and lack of follow-up assessments. The present study shared factors related to the disparity of internet access, difference in interest and internet literacy, negative feeling and doubts about online information among anticipated limitations of studies for online web-based interventions.88,89 The study population represents few subsets of adolescent population who reported they had internet access limiting generalizability of findings to other populations. Finally, the study was conducted during the unprecedented COVID-19 pandemic. As such, participants were at home during the social media intervention, likely having an impact on the intervention outcomes measured, either positively or negatively.


These findings contribute novel and innovative approaches to assessing the effectiveness of the implementation of an intervention, as well as directly assessing the effect of the intervention itself and factors associated with it. The intervention used social media as a channel of communication for mental health promotion to improve adolescent mental health literacy. The findings of the study support the assertion that an intervention using an evidence-based mental health curriculum delivered over social media can significantly improve the mental health literacy of adolescents. We found that the implementation of the intervention was considered acceptable, appropriate, feasible, and satisfactory. The programme itself has practical scalability and sustainability despite some factors limiting its effectiveness in some individuals. The effectiveness of the intervention as reflected in the effect size and in the difference-in-differences estimate differed with both sex and age. The factors found to limit the success of the intervention were the availability of resources, and personal and family-related behaviours indicating the need to consider these challenges in the future when scaling up such an intervention to utilize emerging digital and online platforms for the promotion of mental health literacy. Although these findings are preliminary, they highlight potential areas for improvement in the application of adolescent mental health promotion using digital and online technologies including, most importantly, social media. Future interventions should consider other factors contributing towards good mental health, as well as other socio-demographic and contextual factors.


DID, difference-in-differences; ITHBC, the integrated theory of health behaviour change; ToC, theory of change.

Ethics Approval and Informed Consent

The study complied with the International Ethical Guidelines for Health-Related Research Involving Humans, prepared by the Council for International Organizations of Medical Sciences in collaboration with the World Health Organization.90 Ethical approval was obtained from KIIT University Health research institutional ethics review committee (Reference:KIIT/KIMS/IEC/63/2019) and Haramaya University institutional health research review committee (Reference:00.H.M.S./10.0/3763/2020. The Haramaya University health research review committee was the only recognized health research review committee mandated to approve health research in Dire Dawa province and Eastern Ethiopia during the study period, and as such reviewed the study independently from the authors’ affiliation to Haramaya University. All participants and their parents had the study design and objectives explained to them online and gave written informed consent. The principals of the participating schools also had the study explained to them in person and gave written informed consent.


We would like to acknowledge Dire Dawa Education bureau, school administrators, principals, and teachers in Dire Dawa city in Ethiopia for their assistance during data collection.


The authors report no conflicts of interest for this work and declare that they do not have any financial interest or benefit that has resulted from the direct applications of this research study.


1. Divin N, Harper P, Curran E, Corry D, Leavey G. Help-seeking measures and their use in adolescents: a systematic review help-seeking measures and their use in adolescents: a systematic review. Adolesc Res Rev. 2018;3(1):113–122. doi:10.1007/s40894-017-0078-8

2. Tully LA, Hawes DJ, Doyle FL, Sawyer MG, Dadds MR. A national child mental health literacy initiative is needed to reduce childhood mental health disorders. Aust New Zeal J Psychiatry. 2019;000486741882144. doi:10.1177/0004867418821440

3. Freeman JL, Caldwell PHY, Med B, Bennett PA, Sc BAL, Scott KM. How adolescents search for and appraise online health information: a systematic review. J Pediatr. 2018;195:244–255. doi:10.1016/j.jpeds.2017.11.031

4. Korda H, Itani Z. Harnessing social media for health promotion and behavior change. Health Promot Pract. 2013;14(1):15–23. doi:10.1177/1524839911405850

5. Ling J, Tay YF, Klainin-Yobas P. Mental health literacy levels. Arch Psychiatr Nurs. 2018. doi:10.1016/j.apnu.2018.04.007

6. WHO. Adolescents: health risks and solutions; 2018. Available from: Accessed April 15, 2022.

7. Zeleke WA, Nichols LM, Wondie Y. Mental health in Ethiopia: an exploratory study of counseling alignment with culture. Int J Adv Couns. 2019;41(2):214–229. doi:10.1007/s10447-018-9368-5

8. CSA and UNICEF Ethiopia. Multidimensional child deprivation in Ethiopia. Addis Ababa; 2018. Available from: Accessed Jan 10, 2022

9. Ganasen KA, Parker S, Hugo CJ, Stein DJ, Emsley RA, Seedat S. Mental health literacy: focus on developing countries. Afr J Psychiatry. 2008;11(1):23–28. doi:10.4314/ajpsy.v11i1.30251

10. Cornally N, Mccarthy G. Help-seeking behaviour: a concept analysis. Int J Nurs Pract. 2011;17(3):280–288. doi:10.1111/j.1440-172X.2011.01936.x

11. Eschenbeck H, Lehner L, Hofmann H, Bauer S, Becker K, Diestelkamp S, Kaess M, Moessner M, Rummel-Kluge C, Salize HJ. School Based Mental Health Promotion in and adolescents with StresSOS using online or face-to-face interventions: study protocol for a randomized controlled trial within the ProHEAD Consortium. Trials. 2019;20(1):94. doi:10.1186/s13063-018-3159-5

12. Nagata JM, Ferguson BJ, Ross DA. Research priorities for eight areas of adolescent health in low- and middle-income countries. J Adolesc Health. 2016;59(1):50–60. doi:10.1016/j.jadohealth.2016.03.016

13. WHO. Promoting mental health. Concepts, emerging evidence, practice. A summary report 2004. Available from: Accessed Jan 12, 2022.

14. Salazar de Pablo G, De Micheli A, Nieman DH, et al. Universal and selective interventions to promote good mental health in young people: systematic review and meta-analysis. Eur Neuropsychopharmacol. 2020;41:28–39. doi:10.1016/j.euroneuro.2020.10.007

15. Fusar-Poli P, Salazar de Pablo G, De Micheli A, et al. What is good mental health? A scoping review. Eur Neuropsychopharmacol. 2020;31:33–46. doi:10.1016/j.euroneuro.2019.12.105

16. Kelly CM, Mithen JM, Fischer JA, et al. Youth mental health first aid : a description of the program and an initial evaluation. Int J Ment Health Syst. 2011;5(1):4. doi:10.1186/1752-4458-5-4

17. Kelly CM, Jorm AF, Wright A. Improving mental health literacy as a strategy to facilitate early intervention for mental disorders. Med J Aust. 2007;187(7):1–5. doi:10.5694/j.1326-5377.2007.tb01332.x

18. Jorm AF. Why we need the concept of “Mental health literacy.”. Health Commun. 2015;30(12):1166–1168. doi:10.1080/10410236.2015.1037423

19. Jorm AF, Korten AE, Jacomb PA, Christensen H, Rodgers B, Pollitt P. “Mental health literacy”: a survey of the public’s ability to recognise mental disorders and their beliefs about the effectiveness of treatment. Med J Aust. 1997;166(4):182–186. doi:10.5694/j.1326-5377.1997.tb140071.x

20. Jorm AF, Barney LJ, Christensen H, et al. Research on mental health literacy: what we know and what we still need to know. Aust N Z J Psychiatry. 2006;40(1):3–5. doi:10.1111/j.1440-1614.2006.01734.x

21. Salam RA, Sc M, Das JK, et al. Adolescent health interventions: conclusions, evidence gaps, and research priorities. J Adolesc Health. 2016;59:88–92. doi:10.1016/j.jadohealth.2016.05.006

22. David K, Park MJ, Mulye TP The mental health of adolescents: a national profile, 2008. San Francisco: National Adolescent Health Information Center (NAHIC), Univ. Calif; 2008. Available from: Accessed Jan 12, 2022

23. Tay JL, Tay YF, Klainin-Yobas P. Effectiveness of information and communication technologies interventions to increase mental health literacy: a systematic review. Early Interv Psychiatry. 2018;12(6):1024–1037. doi:10.1111/eip.12695

24. Patton GC, Olsson CA, Skirbekk V, et al. Adolescence and the next generation. Nature. 2018;554(7693):458–466. doi:10.1038/nature25759

25. La Maison C, Munhoz TN, Santos IS, Anselmi L, Barros FC, Matijasevich A. Prevalence and risk factors of psychiatric disorders in early adolescence: 2004 Pelotas (Brazil) birth cohort. Soc Psychiatry Psychiatr Epidemiol. 2018;53(7):685–697. doi:10.1007/s00127-018-1516-z

26. Mills KL. Social development in adolescence: brain and behavioural changes. PQDT - UK Irel; 2015. Available from: Accessed December 12, 2021

27. Bröder J, Okan O, Bauer U, et al. Health literacy in childhood and youth: a systematic review of definitions and models. BMC Public Health. 2017;17(1):1–25. doi:10.1186/s12889-017-4267-y

28. Johnson RL. Pathways to adolescent health: early intervention. J Adolesc Health. 2002;31(6SUPPL.):240–250. doi:10.1016/S1054-139X(02)00497-4

29. Manganello JA. Health literacy and adolescents: a framework and agenda for future research. Health Educ Res. 2008;23(5):840–847. doi:10.1093/her/cym069

30. Fok MSMM, Wong TKSS. What does health literacy mean to children? Contemp Nurs. 2002;13(2–3):249–258. doi:10.5172/conu.13.2-3.249

31. Mcluckie A, Kutcher S, Wei Y, Weaver C. Sustained improvements in students’ mental health literacy with use of a mental health curriculum in Canadian schools. BMC Psychiatry. 2014;14(1):1–6. doi:10.1186/s12888-014-0379-4

32. Diestelkamp S, Wartberg L, Kaess M, et al. Effectiveness of a web-based screening and brief intervention with weekly text-message-initiated individualised prompts for reducing risky alcohol use among teenagers: study protocol of a randomised controlled trial within the ProHEAD consortium. Trials. 2019;20(1):73. doi:10.1186/s13063-018-3160-z

33. Chyzzy B, Dennis C-L. Mobile phone-based peer support in the prevention of postpartum depression among adolescent mothers: a pilot randomized controlled trial. J Adolesc Health. 2019;64(2):S8–S9. doi:10.1016/j.jadohealth.2018.10.030

34. Baldofski S, Kohls E, Bauer S, et al. Efficacy and cost-effectiveness of two online interventions for children and adolescents at risk for depression (E.motion trial): study protocol for a randomized controlled trial within the ProHEAD consortium. Trials. 2019;20(1):53. doi:10.1186/s13063-018-3156-8

35. Lin HC, Chang CM. What motivates health information exchange in social media? The roles of the social cognitive theory and perceived interactivity. Inf Manag. 2018;55(6):771–780. doi:10.1016/

36. Szeftel R, Schlozman S, Bostic JQ, et al. Functional assessment of social media in child and adolescent psychiatry. In: Child and Adolescent Psychiatry and the Media; 2018. doi:10.1016/b978-0-323-54854-0.00008-4

37. Martínez V, Espinosa-Duque D, Jiménez-Molina Á, et al. Feasibility and acceptability of “cuida tu Ánimo” (Take care of your mood): an internet-based program for prevention and early intervention of adolescent depression in Chile and Colombia. Int J Environ Res Public Health. 2021;18(18):1–13. doi:10.3390/ijerph18189628

38. Hausmann JS, Touloumtzis C, White MT, Colbert JA, Gooding HC. Adolescent and young adult use of social media for health and its implications. J Adolesc Health. 2017;60(6):714–719. doi:10.1016/j.jadohealth.2016.12.025

39. Mulisa F, Getahun DA. Perceived benefits and risks of social media: Ethiopian secondary school students’ perspectives. J Technol Behav Sci. 2018;3(4):294–300. doi:10.1007/s41347-018-0062-6

40. Kitchener BA, Jorm AF. Mental health first aid: an international programme for early intervention. Early Interv Psychiatry. 2008;2(1):55–61. doi:10.1111/j.1751-7893.2007.00056.x

41. Kim S-Y, Ball K, Chow CK, et al. The role of social media in preventing and managing non-communicable diseases in low-and-middle income countries: hope or Hype? Health Policy Technol. 2019. doi:10.1016/j.hlpt.2019.01.001

42. Hassen HM, Behera MR, Jena PK, Dewey RS. Mental health literacy of adolescents and the effect of socio- demographic characteristics: a cross-sectional study. Online J Health Allied Sci. 2021;20(4):2–8. Available from: Accessed December 12, 2021.

43. O’Connell T, Rasanathan K, Chopra M. What does universal health coverage mean? Lancet. 2014;383(9913):277–279. doi:10.1016/S0140-6736(13)60955-1

44. Ryan P, Weiss M, Traxel N. Testing the integrated theory of health behaviour change for postpartum weight management. J Adv Nurs. 2011;67(9):2047–2059. doi:10.1111/j.1365-2648.2011.05648.x

45. Ryan P. Integrated theory of health behavior change: background and intervention development. Clin Nurse Spec. 2009;23(3):161. doi:10.1097/NUR.0b013e3181a42373

46. Mayne J. Theory of change analysis: building robust theories of change. Can J Progr Eval. 2017;32(2):155–173. doi:10.3138/cjpe.31122

47. Hassen HM, Behera MR, Jena PK, Satpathy SK. A quasi-experimental and guided social media intervention to improve mental health literacy level of urban school adolescents in Ethiopia: a detailed study protocol. Prepr Res Sq. 2020;1–15. doi:10.21203/

48. Bhalerao S, Kadam P. Sample size calculation. Int J Ayurveda Res. 2010;1(1):55. doi:10.4103/0974-7788.59946

49. Kutcher S, Wei Y. Mental health & high school curriculum guide (Guide v.3): understanding mental health and mental illness. Teen Mental Health; 2017. Available from: Accessed Jan 12, 2022.

50. Kutcher S, Bagnell A, Wei Y. Mental health literacy in secondary schools. A Canadian approach. Child Adolesc Psychiatr Clin N Am. 2015;24(2):233–244. doi:10.1016/j.chc.2014.11.007

51. Kutcher S, Wei Y, Morgan C. Successful application of a Canadian mental health curriculum resource by usual classroom teachers in significantly and sustainably improving student mental health literacy. Can J Psychiatry. 2015;60(12):580–586. doi:10.1177/070674371506001209

52. Mental health & high school curriculum guide; March, 2017:1–165. Available from: Accessed Jan 12, 2022.

53. GCYDCA. African school mental health curriculum: understanding mental health and mental illness. Guid Couns Youth Dev Cent Africa; 2012. Available from: Accessed Jan 12, 2022.

54. Caulfield A, Vatansever D, Lambert G, Van Bortel T. WHO guidance on mental health training: a systematic review of the progress for non-specialist health workers. BMJ Open. 2019;9(1):bmjopen–024059. doi:10.1136/bmjopen-2018-024059

55. Campos L, Dias P, Palha F, Duarte A, Veiga E. Development and psychometric properties of a new questionnaire for assessing mental health literacy in young people. Univ Psychol. 2016;15(2):61–72. doi:10.11144/Javeriana.upsy15-2.dppq

56. O’Connor M, Casey L, Clough B. Measuring mental health literacy-a review of scale-based measures. J Ment Health. 2014;23(4):197–204. doi:10.3109/09638237.2014.910646

57. Bale J, Grové C, Costello S. A narrative literature review of child-focused mental health literacy attributes and scales. Ment Health Prev. 2018;12:26–35. doi:10.1016/j.mhp.2018.09.003

58. Lam LT. Mental health literacy and mental health status in adolescents: a population-based survey. Child Adolesc Psychiatry Ment Health. 2014;8:1–8. doi:10.1186/1753-2000-8-26

59. Dias P, Campos L, Almeida H, Palha F. Mental health literacy in young adults: adaptation and psychometric properties of the mental health literacy questionnaire. Int J Environ Res Public Health. 2018;15:7. doi:10.3390/ijerph15071318

60. Wei Y, McGrath PJ, Hayden J, Kutcher S. Mental health literacy measures evaluating knowledge, attitudes and help-seeking: a scoping review. BMC Psychiatry. 2015;15:1. doi:10.1186/s12888-015-0681-9

61. Tomé G, Matos MG, Camacho I, Gomes P, Reis M, Branquinho CS. Mental health promotion in school context – validation of the ES ’ COOL scale for teachers. J Psychiatry Behav Sci. 2018;2:1009.

62. Bjørnsen HN, Eilertsen MEB, Ringdal R, Espnes GA, Moksnes UK. Positive mental health literacy: development and validation of a measure among Norwegian adolescents. BMC Public Health. 2017;17(1):1–10. doi:10.1186/s12889-017-4733-6

63. Reilly MO, Svirydzenka N, Adams S, Dogra N. Review of mental health promotion interventions in schools. Soc Psychiatry Psychiatr Epidemiol. 2018;53(7):647–662. doi:10.1007/s00127-018-1530-1

64. Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):1–12. doi:10.1186/s13012-017-0635-3

65. IBM Corp. IBM SPSS Statistics for Windows, Version 25.0; 2017. Available from: Accessed April 15, 2022.

66. Harpe SE. How to analyze Likert and other rating scale data. Curr Pharm Teach Learn. 2015;7(6):836–850. doi:10.1016/j.cptl.2015.08.001

67. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Vol. 53. 92nd ed. New York, NY: Academic Press.; 1988. Available from: Accessed Jun 19, 2021.

68. Schäfer T, Schwarz MA. The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases. Front Psychol. 2019;10:(APR):1–13. doi:10.3389/fpsyg.2019.00813

69. Kraft MA. Interpreting effect sizes of education interventions. Educ Res. 2020;49(4):241–253. doi:10.3102/0013189X20912798

70. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:(NOV):1–12. doi:10.3389/fpsyg.2013.00863

71. Baguley T. Standardized or simple effect size: what should be reported? Br J Psychol. 2009;100(3):603–617. doi:10.1348/000712608X377117

72. Bakker A, Cai J, English L, Kaiser G, Mesa V. Beyond small, medium, or large: points of consideration when interpreting effect sizes. Educ Stud Math. 2019;102:1–8. doi:10.1016/j.pscychresns.2019.02.003

73. Garrido S, Millington C, Cheers D, et al. What works and what doesn’t work? A systematic review of digital mental health interventions for depression and anxiety in young people. Front Psychiatry. 2019;10:1–19. doi:10.3389/fpsyt.2019.00759

74. Kaess M, Ritter S, Lustig S, et al. Promoting Help-seeking using E-technology for ADolescents with mental health problems: study protocol for a randomized controlled trial within the ProHEAD Consortium. Trials. 2019;20(1):94. doi:10.1186/s13063-018-3157-7

75. Fuller D, Neudorf J, Bermedo-Carrasco S, Neudorf C. Classifying the population by socioeconomic factors associated with support for policies to reduce social inequalities in health. J Public Health. 2016;38(4):635–643. doi:10.1093/pubmed/fdv201

76. Kraft JM, Wilkins KG, Morales GJ, Widyono M, Middlestadt SE. An evidence review of gender-integrated interventions in reproductive and maternal-child health. J Health Commun. 2014;19:122–141. doi:10.1080/10810730.2014.918216

77. Coles ME, Ravid A, Gibb B, George-Denn D, Bronstein LR, McLeod S. Adolescent mental health literacy: young people’s knowledge of depression and social anxiety disorder. J Adolesc Health. 2016;58(1):57–62. doi:10.1016/j.jadohealth.2015.09.017

78. Bouchard DR, Baillargeon JP, Langlois MF. The independent effect of age groups on the effectiveness of lifestyle intervention. Curr Obes Rep. 2013;2(2):107–119. doi:10.1007/s13679-012-0044-5

79. Bradford S, Rickwood D. Adolescent’s preferred modes of delivery for mental health services. Child Adolesc Ment Health. 2014;19(1):39–45. doi:10.1111/camh.12002

80. Lyon AR, Bruns EJ, Ludwig K, et al. The Brief Intervention for School Clinicians (BRISC): a mixed-methods evaluation of feasibility, acceptability, and contextual appropriateness. School Ment Health. 2015;7(4):273–286. doi:10.1007/s12310-015-9153-0

81. Connor PJO, Martin B, Weeks CS, Ong L. Factors that influence young people’ s mental health help-seeking behaviour: a study based on the Health Belief Model. J Adv Nurs. 2014;70(11):2577–2587. doi:10.1111/jan.12423

82. Arnold C, Farhall J, Villagonzalo KA, Sharma K, Thomas N. Engagement with online psychosocial interventions for psychosis: a review and synthesis of relevant factors. Internet Interv. 2021;25:100411. doi:10.1016/j.invent.2021.100411

83. Robotham D, Satkunanathan S, Doughty L, Wykes T. Do we still have a digital divide in mental health? A five-year survey follow-up. J Med Internet Res. 2016;18:11. doi:10.2196/jmir.6511

84. Beatty L, Binnion C, Systematic A. Review of predictors of, and reasons for, adherence to online psychological interventions. Int J Behav Med. 2016;23(6):776–794. doi:10.1007/s12529-016-9556-9

85. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. 2017;7(2):254–267. doi:10.1007/s13142-016-0453-1

86. Espinosa HD, Á C, Moessner M, et al. Acceptability study of “ascenso”: an online program for monitoring and supporting patients with depression in chile. Telemed e-Health. 2016;22(7):577–583. doi:10.1089/tmj.2015.0124

87. Bärnighausen T, Tugwell P, Røttingen JA, et al. Quasi-experimental study designs series—paper 4: uses and value. J Clin Epidemiol. 2017;89:21–29. doi:10.1016/j.jclinepi.2017.03.012

88. Chou WYS, Prestin A, Lyons C, Wen KY. Web 2.0 for health promotion: reviewing the current evidence. Am J Public Health. 2013;103(1):9–18. doi:10.2105/AJPH.2012.301071

89. Benetoli A, Chen TF, Aslani P. Consumer perceptions of using social media for health purposes: benefits and drawbacks. Health Informatics J. 2019;25(4):1661–1674. doi:10.1177/1460458218796664

90. CIOMS. International Ethical Guidelines for Health-Related Research Involving Humans: prepared by the Council for International Organizations of Medical Sciences (CIOMS) in Collaboration with the World Health Organization (WHO); 2016. Available from: Accessed February 8, 2022.

Creative Commons License © 2022 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.