Effectiveness and Implementation Outcome Measures of Mental Health Curriculum Intervention Using Social Media to Improve the Mental Health Literacy of Adolescents

Introduction

Adolescent mental health problems tend to be more serious in individuals with lower mental health literacy.^1,2 Mental health interventions using social media are thought to be effective and have shown positive implementation outcome measures for mental health promotion.^3,4

Addressing mental health problems in children and adolescents has emerged as a public health priority. Globally, approximately 10–20% of adolescents live with mental health problems,^5,6 and most do not have access to the necessary health services. Approximately 17–23% of Ethiopian adolescents have mental health problems,^7,8 which is higher than the global prevalence.³ Implementing mental health services in low-income countries like Ethiopia is highly challenging.⁹ This has been combined with reportedly low levels of mental health literacy, ie, knowledge and awareness of mental health issues.⁹

Mental health literacy measures, help-seeking behaviour, and adherence to mental health services have been shown to worsen with complex health disparities resulting in a gap in mental health promotion and mental ill-health prevention.^7,8 Limited help-seeking behaviours and delays in receiving help have been reported by adolescents with mental health symptoms.^1,2,10,11 These adolescents face difficulties recognizing how and when to seek support and help.¹² Several studies have revealed that low help-seeking intention and behaviour in adolescents reflect a lack of awareness about the nature and symptoms of mental health problems and alternative sources of help.¹¹ These studies are sometimes limited in their scope for assessing the impact of a mental health curriculum as more attention has been given to individuals with mental health problems than to individuals who appear to be apparently well.¹³

Mental health promotion has evolved to take on a new positive perspective, termed good mental health,^14,15 comprised of several domains, with one of these domains being mental health literacy.¹⁵ Interventions to improve modifiable factors, like mental health literacy, in adolescents have been shown to influence mental health outcomes.^16–20 In this regard, mental health literacy has been shown to be improvable through the use of a mental health curriculum. Mental health literacy is an individual’s ability to recognize mental disorders and forms of psychological distress, as well as the extent of their knowledge, beliefs, and attitudes towards mental illnesses, their risk factors and causes, and the sources of help available.^16–20 However, effective, acceptable, appropriate, and feasible mental health promotion approaches targeting the mental health literacy of adolescents both in and outside of formal education are very limited, and reports of their efficacy have been inconclusive.²¹

It is imperative to emphasize adolescent mental health promotion for multiple reasons.^3,11,21–30 The nature of adolescence as a developmental stage makes this age group a priority for public health scholars and policymakers. The onset of many mental health problems and disorders often occurs in adolescence,^11,22,23 and the prevalence of chronic illnesses and their associated mental health problems increases in adolescent populations.³⁰ Physical, cognitive, emotional, and behavioural development during adolescence is critical to the lifelong wellbeing of the individual.^24–26 These developmental processes shape health-related behaviours and skills.²⁷ Consequently, mental health during childhood and adolescence significantly affects the health outcomes of the next generation.^3,21 Therefore, early intervention is worthwhile to improve the understanding of health information,²⁸ and ensure good health later in life.²⁹

The use of a mental health curriculum to promote positive mental health aims to achieve this by improving adolescents’ knowledge, beliefs, and attitudes towards recognizing mental disorders and their risk factors and being aware of available sources of help and information.^16–18 However, the mode of delivery has become a challenge in the face of financial hardship, inefficiencies, and inequitable resource allocation. Inadequate health systems and structural inequalities in low-income countries like Ethiopia and other African countries demand innovative approaches to improving mental health literacy. Furthermore, unexpected global challenges such as the COVID-19 pandemic have created multiple physical and logistic barriers.¹

To study the impact of interventions on populations, it is necessary to study groups of people with common backgrounds and characteristics and using communication modalities and strategies amenable and available to these target populations.^14,21 As such, schools are ideal settings for health promotion in children and adolescents.³¹ However, the effectiveness, acceptability, appropriateness, and feasibility of this approach have been challenged,²¹ demanding a need for innovative collective approaches using the emerging communication technologies of digital and online platforms. These approaches working through schools as supporting platforms to establish direct channels of communication with adolescents are thought to be effective, sustainable, and scalable.^3,4,29 Capitalizing on the internet and mobile technology using communication channels such as text messages,³² mobile phone-based peer support,³³ and web-based interventions,³² offers many potential benefits for both the intended service users and the service provider.³⁴

Social media is commonly used for health information exchange,^35,36 and has become an integral part of adolescent life.³⁷ Compared to any other age group, a higher proportion of adolescents and young adults are social media users.³⁸ Social media allows individuals to exchange information and ideas, including those related to health. Ethiopia has experienced a rapid growth in internet use, and adolescents use social media increasingly in daily life.³⁹ This offers opportunities for improving social connectedness, resilience, and well-being.^36,40 Social media is established as a medium of communication for health promotion and intervention in developed countries.

Interventions to promote mental health using social media are effective, acceptable, feasible, appropriate, and popular.^35,36 However, in low-income countries, digital inequality and related factors negatively impact the successes of these interventions. There is little quantitative evidence supporting the effectiveness of social media-assisted mental health promotion in adolescents in low-income countries, particularly in African countries such as Ethiopia.^39,41 Little is known about the implementation outcome measures of such efforts, or about the factors influencing them.⁴²

Consequently, the present study was conducted to assess whether intervention using a mental health curriculum through guided social media use effectively improves adolescent mental health literacy. A hypothesis stating that an intervention using social media significantly improves the level of mental health literacy in adolescents compared to that in a control cohort (p<0.05) was tested. Evidence from direct beneficiaries is needed to understand the effectiveness of an intervention for health promotion to strive for equitable and resilient health systems for universal health coverage.⁴³

Understanding implementation outcome measures and the factors influencing them is necessary to help develop comprehensive intervention programs. Notwithstanding the importance of implementation outcome measures, little is known about them in the context of conducting such work in a low-income country. Consequently, this study was based on the theory of change aligning with the integrated theory of health behaviour change. The integrated theory of health behaviour change (ITHBC) states that enhancing knowledge, attitudes, and beliefs around mental health issues (mental health literacy) results in behaviour change towards improved mental health.^44,45 The theory of change (ToC) explains situations and phenomena from the standpoint of conceptual and empirical evidence by describing the mechanisms by which the intervention is intended to work.⁴⁶ These theoretical backgrounds provide opportunities for reflection around intervention design, implementation, and outcome analysis. The theory of change framework (Figure 1) reflects the main target of intervention and the secondary objectives in the form of implementation outcome measures.

Figure 1 The intervention outcome mapping in the lens of the theory of change.

This body of work primarily aimed to investigate the effectiveness of a mental health curriculum intervention delivered over social media to improve adolescent mental health literacy using effect size and difference-in-differences. The secondary analysis focused on the four measures of intervention implementation outcomes: the acceptability of the implementation, its appropriateness, its feasibility, and the satisfaction of the beneficiaries. Further, we assessed the perceived factors affecting the effectiveness of the intervention and these outcome measures.

Materials and Methods

Study Population

This study was conducted in public and private schools in urban areas of Dire Dawa province, Ethiopia, from the last week of June to the first week of August 2020. Dire Dawa city is diverse in the ethnicities, cultures, languages, and socio-demographic factors it hosts. In the pretest period, the research team recruited adolescents aged between 15 and 19 years who had smartphone and social media access. Adolescents below 15 years of age were excluded because they were not expected to have smartphones or internet access, or the familiarity with these that is required to engage fully with the intervention activity.⁴⁷

Study Design

This study employed a quasi-experimental design, comparing the before-and-after difference between the intervention (treatment) group and the control group (Figure 2). Conducting a randomized control trial was unlikely to be feasible because it was costly and had complex ethical issues. As such, the assignment of participants to treatment and control groups was based on the need to avoiding contamination and bias during the intervention and to ensure recruiting a reasonable number of intervention and control participants. Furthermore, it was difficult to control all known and unknown factors that may influence the outcome of the intervention. Participants were recruited into geographically distant intervention and control groups based on the locations of their school and area of residency to reduce contact between the two groups during the intervention period and as such minimize contamination between groups.

Figure 2 Flow chart of the study design.

Sample Size Determination and Sampling Procedures

The power analysis for this study was based on the primary aim of comparing the change in mental health literacy scores from pretest to post-test and comparing this between intervention and control groups. The minimum sample size was based on a systematic review of interventions for mental health literacy, using on the reported effect size, δ=0.823 with a standard deviation, σ₂=1.55.¹⁴ The calculation assumed a confidence of level of 95% (Za/2 =1.96), power of 90% (Z_β=1.282), and type 1 error α=5%. Using a validated sample size calculation for a quasi-experimental study,⁴⁸ a minimum of 146 participants (73 for each arm) were needed. However, the aim was to exceed that sample size. Of the 731 potential participants who met the initial inclusion criteria, 156 were recruited and allocated to the intervention and (n₁ = 77) and control (n₂ = 76) groups and completed the study (Figure 2). This sample size represents approximately 105% of the calculated minimum sample size.

Intervention Materials

The intervention took the form of a mental health curriculum aimed at adolescents to improve their mental health literacy. It was adapted from the school mental health curriculum guides already prepared for African, Australian, and Canadian adolescents that aimed to improve mental health literacy and mental health first aid skills.^40,49–54 The intervention material was prepared and presented for the present study by summarizing the manual’s texts, figures, and case vignettes in plain English. The contents of the intervention focused on how to seek information, awareness of common mental disorders and mental health problems, awareness of risk and protective factors, causes of mental illness, awareness of methods for self-help or for helping others, and awareness of the professional or informal help available, attitudes related to promoting recognition of mental health problems and/or appropriate help-seeking behaviour, and recognizing mental illnesses and sources of help.

English is the medium of instruction used in Ethiopian junior schools and teaching above grade levels and is taken as a school subject starting from kindergarten. Furthermore, science subjects are given in English regardless of the area’s native language. Thus, English was used for the intervention.

Intervention Procedures

A pilot intervention was conducted prior to the main intervention study.⁴⁷ Facebook and Telegram were reportedly the most commonly used social media platforms among Ethiopian adolescents, and similarly, in Dire Dawa. Hence, participants were registered to Facebook and Telegram groups.⁴⁷ A 45-megabyte internet package was provided to each participant for completing each intervention session as an incentive and to improve recruitment, compliance, and retention of study participants.

For the intervention group, the summarized texts, figures, and case vignettes were posted on both social media platforms every 72 hours (three days) on working days (Monday to Friday) in eight sessions across approximately six consecutive weeks. Discussion between participants was allowed on the platform and a summary was provided at the end of each week (every two sessions). The control group received information with a similar frequency and timescale, but instead, the content was that of “secrets of successful students”. Post-test data collection was conducted from all study participants (ie, both intervention and control groups) at the end of week seven, one week after the conclusion of the social media information delivery.

All reasonable efforts were made to minimize the impact of errors and bias. Participants engaged in orientation and closing discussions throughout the study period. All data entry was double-checked by a data clerk. A thorough descriptive and inferential analysis was conducted objectively using standardized models. The study design controlled for the anticipated effect modifiers and confounding factors through the implementation of strict eligibility criteria.

Variables and Measurement Tools

The study variables included sociodemographic characteristics (sex, school grade, and age), whether they were allocated to the intervention group (G_i=1) or control group (G_i=0), time point as pretest (t=0) or post-test (t=1), mental health literacy, and intervention implementation outcome measures and influencing factors. Mental health literacy was measured using the mental health literacy questionnaire, MHLq,⁵⁵ a freely available and validated tool used in adolescent populations.^17,55–63 We further tested its reliability in the present study setting (Cronbach’s alpha=0.834). The version of the MHLq used in this study was comprised of 33 items, with each item eliciting a response using a five-point Likert scale (1=strongly disagree, 2=slightly disagree, 3=neither agree nor disagree, 4=slightly agree, 5=strongly agree). Items comprised statements covering recognition, awareness, attitudes, and beliefs towards mental health issues.^16–20

The intervention implementation outcome measures were evaluated using a validated, questionnaire,⁶⁴ with items eliciting a response using a five-point Likert scale representing the level of agreement (1=completely disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=completely agree). Outcome measures were used to assess the extent of acceptability, feasibility, appropriateness, and satisfaction of the intervention.

Acceptability was measured using 4 items encompassing whether the intervention met the participant’s acceptance, the extent to which they liked it, whether they found it appealing, and whether they found it welcoming. Feasibility was measured using 4 items eliciting to what extent the intervention approach was implementable, possible, doable, and easy to use. Appropriateness was measured using 4 items covering whether the intervention seemed to fit its purpose, was suitable, applicable, and a good match. Satisfaction was measured using 7 items consisting of the quality of the intervention, whether it was wanted, whether it met the participant’s needs, whether they would recommend it to others, the extent it helped them to deal with their problems, and the extent to which they would like to repeat the experience. Perceived factors influencing the effectiveness of the intervention and implementation outcome measures were measured in a similar manner using a 5-point Likert scale where participants were asked to indicate their level of access to resources, and their experience of personal and family-related challenges, so that these factors could be controlled for in the analyses.

Statistical Analyses

The baseline analysis was performed using descriptive statistics of the sociodemographic characteristics. Subsequently, we analyzed the effectiveness of the intervention in terms of the effect size and the difference-in-differences estimate. Finally, we analyzed the implementation outcome measures and related influencing factors. All statistical analyses were conducted using SPSS version 25.⁶⁵

Effectiveness of the Intervention

Mental health literacy scores were treated as intervals following the measurement framework used by Stevens.⁶⁶ Negatively keyed items of the mental health literacy questionnaire (Q7, Q12, Q15, Q17, Q24, and Q26) were reverse scored and individuals’ scores were computed. The analysis was restricted to subsets of groups and characteristics to address confounding factors and to determine which group of adolescent participants most benefited from the intervention, based on their background characteristics. We computed the mean differences associated with the intervention (posttest-pretest score) within each group using a paired t-test and then estimated the effect size. The effectiveness of the intervention program was expressed as effect size and difference-in-differences estimate. Significance was considered at a threshold of p≤0.05.

Effect Size Analysis

Effect size is typically assumed to be the most important outcome measure in practical intervention and impact evaluation studies as recommended by Jacob Cohen.^67,68 Effect size is a preferred way of communicating the practical significance of a result in evidence-based and field-based studies of interventions.^69,70 Effect size was estimated using a 95% confidence interval and expressed using Cohen’s d and Hedges’s g.

These expressions use a pooled standard deviation in the denominator; however, the latter uses n, and the former uses n-1. These two different methods of calculating the standard deviation reflect the mean score difference between the intervention and control groups. Hence, the absolute effect size is the difference between the mental health literacy scores in these two groups. The meaning of this intervention effect was interpreted from both the conventional approach and the comparison approach based on a broad conventional point of reference and comparison with previous studies, respectively.^68,71

Difference-in-Differences Analysis

The effectiveness of the intervention at improving mental health literacy was also estimated using a difference-in-differences analysis (Figure 3). The difference-in-differences (DID) analysis is a regression analysis exploring the time dimension of the data to define the actual and counterfactual effects of the intervention. It was estimated by comparing the difference in the outcomes between the intervention and control groups over the six weeks of the intervention period.

Figure 3 Conceptual framework, regression model, and coefficients for the estimation of the difference-in-differences.

The difference that existed before the intervention program was addressed by assessing the presence of unobserved heterogeneity. Such an analysis approach has the advantage of controlling for differences arising from unobservable characteristics. Hence, the intervention effect was obtained by assessing for a difference between the groups, as shown in Figure 3, which displays the steps taken in the analysis. After computing the before-to-after difference for each group [E(YIai-YIbi | Di=1) and E(YCai -YCbi|Di=0)], we computed the average intervention effect [E(YIai-YIbi | Di=1)-E(YCai -YCbi|Di=0)] where the subscript a denotes “after” and b denotes “before” the intervention, and the superscripts I and C indicate the intervention and control groups, respectively. Dgt =1 if unit g is exposed to the intervention in period t, and Dgt =0 if unit g is exposed to the control condition in period t. Y(1)gt represents the outcome of interest for unit g during period t under the hypothetical scenario g received the treatment during t. Y(0)gt is the outcome of the same unit and time under the alternative scenario in which g received the control condition.

The estimation of the difference-in-differences regression relies on certain assumptions. The unobserved heterogeneity is assumed to be time-invariant, addressed by comparing the before and after states. The common trend assumption states that in the absence of the intervention or treatment, both the control and intervention groups experience the same measurement trend of outcome variables over time. Therefore, any difference from the trend observed in the intervention group is the effect of the intervention. Other assumptions considered for this analysis were the linearity of all relationships, the independence of observations, the absence of perfect collinearity, and the non-zero variances of independent variables; error terms were assumed to be zero and normally distributed.

It is assumed that, in the absence of any intervention, the difference between the “intervention” and the “control” groups is constant over time (the two groups would have evolved in the same way in the absence of any intervention). Therefore, the average outcome in the control group is β0 during the pre-treatment period and β0+β2 in the post-treatment period. Likewise, the average untreated outcome in the intervention group is β0+β1 during the pre-treatment period and β0 + β1 + β2 in the post-treatment period, where β1 is different irrespective of the intervention effect. Thus, the difference in pre-treatment is normal while the difference in post-treatment can be assumed to be a causal effect (difference-in-differences, β3).

In the difference-in-differences analysis, these assumptions restrict the scope of the possible confounders that are not entirely dependent on random assignment. It assumes that confounders varying across the groups are time-invariant and that time-varying confounders are constant. Hence, the difference-in-difference is estimated using a model that accounts for target variable measures, group membership, and time periods. The coefficient of the interaction term (β3) is an estimate of the treatment effect. Importantly, the timing effect in the intervention and control groups is not equal. The two groups are different at the start, and the observational characteristics change over time.

Implementation Outcome Measures and the Influencing Factors

Descriptive analyses were used for each outcome measure, and response. The mean score and differences across sexes, age groups, and school grades were analyzed using independent t-tests. Similarly, descriptive analyses were used for each factor that was being assessed for its effect on the implementation outcome measures and the potential effectiveness of the intervention program.

Results

Sociodemographic Characteristics

Participant socio-demographic characteristics are presented in Table 1. The mean ± standard deviation age of participants was 17.34 ± 1.31 years for the intervention group participants and 17.21 ± 1.36 years for the control group, and there was no statistically significant difference in any demographic characteristic between groups (ie, all p>0.05).

Table 1 Summary of Demographic Characteristics for Intervention and Control Groups

Mental Health Literacy Pre-and Post-Intervention

Pretest and post-test mental health literacy scores of participants in the intervention and control groups are presented by age group and sex (Figure 4). The mean pretest and post-test mental health literacy scores of girls were slightly higher than those of the boys. The mean pretest and post-test mental health literacy scores of the control group were lower than those of the intervention group. The paired-samples t-test showed a significant change between mean pretest and post-test scores.

Figure 4 The pre-test and post-test scores of mental health literacy for both intervention and control groups.

Effect of the Intervention on the Mental Health Literacy of Adolescents

Effect Size Estimate of the Intervention

The paired samples t-test showed a significant change in the mean mental health literacy score between the pretest and post-test time points (p<0.01) for both the intervention and control groups (Table 2). However, effect size estimates in the intervention group were greater than in the control group. The effect size for mental health literacy was greatest among participants aged 15–17 years, and for girls participants. The independent samples t-test on effect size showed a significantly greater effect in the intervention group compared to the control group (p=0.028). This effect size was higher in girls aged 15–17 years (Cohen’s d=0.2.042, Hedges’ g=2.045, p<0.001) than in boys of the same age (Cohen’s d=0.141, Hedges’ g=0.141, p>0.05). The effect size was lower, and non-significant in participants aged 18–19 years (p>0.05) (Table 2).

Table 2 Effect Size for Chang of Total Mental Health Literacy Score in Intervention and Control Groups by Age and Sex

Difference-in-Difference Estimates of the Intervention

The regression model for the difference-in-differences analysis was stratified by sex, age group, and education level (Figure 5 and Table 3). These models were used to estimate results for the intervention and it was estimated that the intervention had a highly significant impact (DID: 0.348, CI: 0.154–0.542, p<0.001), in support of the proposed hypothesis. The increase in mental health literacy scores differed by sex and age.

Table 3 Difference-in-Differences (DID) Estimating the Effect of the Intervention on the Mean Score of Mental Health Literacy

Figure 5 Difference-in-differences showing the effect of the intervention on the mental health literacy score. (A) All participants; (B) Boys (15–17 years); (C) Boys (18–19 years); (D) Girls (15–17 years) and (E) Girls (18–19 years).

Table 3 showed that the mental health literacy score significantly increased despite differences associated with sex (boys DID: 0.308, CI: 0.042–0.574, p<0.05 vs girls DID: 0.398, CI: 0.120–0.677, p <0.01), age group (15–17 years DID: 0.295, CI: 0.020–0.570, p<0.05 vs 18–19 DID: 0.392, CI: 0.117–0.667, p<0.001), and education level (grades 9–10 DID: 0.371, CI: 0.096–0.646, p <0.01 vs grades 11–12 DID: 0.324, CI: 0.050–0.597, p <0.05).

Implementation Outcome Measures and Influencing Factors

The analysis aimed to assess the self-rated implementation outcome measures and potential factors that may influence these implementation outcome measures.

Implementation Outcome Measures

The majority of respondents either agreed or completely agreed about the acceptability, feasibility, appropriateness, and satisfaction of the implementation of the intervention (Figure 6). However, ratings differed significantly across age groups, with a higher proportion of older adolescents reporting complete agreement for acceptability, appropriateness, feasibility, and satisfaction. Ratings did not differ significantly with sex (Table 4). More than 60% of participants responded that they completely agreed with the acceptability of the implementation (range 55–72%), the extent to which they liked it, found it appealing, and found its features welcoming. Ratings of feasibility, consisting of to what extent it is implementable, possible, doable, and easy to use, varied between 35 and 50%. Appropriateness, namely the east of fit, suitability, applicability, and the degree to which the implementation was a good match varied between 57 and 63%. Finally, satisfaction, the perceived quality of the implementation, the degree to which it is wanted, the degree to which it met the participant’s needs, how likely they would be to recommend it to others, the extent to which it helped deal with their problems, and whether they would like to repeat it, also varied between 45 and 71%.

Table 4 Differences in Perceived Intervention Implementation Outcome Measures by Gender, Age Group, and School Grade

Figure 6 Percentage of intervention group participants’ rating on the intervention implementation outcome measures.

With a minimum score of 5 and a maximum of 20, the mean and standard deviation ratings given by participants are separated and analysed by sex, age group, and school grade, along with testing the significance of any differences (Table 4). There was no significant sex difference in the scores given for the implementation outcome measures (p≥0.05). There were significant age differences in the ratings of implementation acceptability (p=0.003), appropriateness (p=0.027), feasibility (p=0.012), and satisfaction (p=0.013), where participants aged 18–19 years rated the implementation more highly. Ratings of satisfaction differed significantly (p=0.016) by school grade, where participants from grades 11–12 gave higher satisfaction scores.

Influencing Factors

The factors likely to influence implementation outcome measures were a lack of resources and personal and family-related challenges (Figure 7). Analysis focused on the proportion of participants who responded either in agreement or complete agreement with each of the items. Lack of access to the technological resources such as smart phone or computer (reported by 71.5% of participants), a lack of internet access (77.9%), or poor internet coverage (72.7%) were commonly reported. Personal influencing factors including a lack of digital skills and difficulty using technology (66.2%), lack of time to engage with the intervention (58.5%), unfamiliarity with social media (62.3%), and lack of motivation (57.2%) were less common but still reported by over half the respondents. Using social media incorrectly or spending time on unnecessary websites (57.2%), lacking confidence and trust in the privacy or confidentiality of the intervention (44.2%), and carelessness, laziness, or having little interest in social media (55.9%) were other factors reported by respondents in the sample. The family restriction of not being allowed to use social media or the internet (53.2%) was also reported as an influencing factor.

Figure 7 Self-ratings of agreement with factors supposedly influencing the intervention implementation outcomes.

Discussion

The present study primarily investigated the effectiveness of a mental health curriculum intervention delivered over social media to improve adolescent mental health literacy using effect size and difference-in-differences. The secondary analysis evaluated the four measures of intervention implementation outcomes viz; the acceptability of the implementation, its appropriateness, its feasibility, and the satisfaction of the beneficiaries. Further, the perceived factors affecting the effectiveness of the intervention and these outcome measures were assessed. For the notion that mental health literacy is modifiable and is related to individuals’ knowledge, beliefs, and awareness of mental health issues;^16–20 these findings are vital for enhancing outcomes such as self-efficacy, help-seeking intention, and health behaviour as well as fostering positive mental health and wellbeing.^3,21 These changes are variously immediate and intermediate, with outcomes expressed as either proximal or distal outcomes.⁴⁵

The overall findings from this study are that the intervention effect was greater than zero, and that the magnitude of the effect can be expressed using the standardized metrics of effect size and difference-in-differences estimate. Both the effect size and the difference-in-differences analysis showed a significant effect of the intervention (p<0.05) compared to a control condition. Nevertheless, the magnitudes of these metrics varied with sex and age. The implementation of the intervention was found to be acceptable, appropriate, feasible, and satisfactory despite several influencing factors that varied with the socio-demographic characteristics of the participant.

Effect sizes for quantitative intervention analyses of outcomes can be interpreted using a variety of approaches, including convention and comparison,^68,69,71,72 with much evidence available in the published literature advocating strongly in favor of using a convention approach for studies such as this one. According to the conventional approach, effect sizes expressed primarily using Cohen’s d or Hedges’ g values can be categorized into small (below 0.2), medium (0.2 to 0.5), or large (0.5 or higher).⁶⁷ The estimated intervention effect size estimate of this study was greater than zero; and the intervention program has potentially improved the mental health literacy of adolescents. The effect size observed in the present study would be conventionally classified as medium-to-high, varying by both sex and age. Despite there being little evidence advocating the comparison approach, the effect size seen in the present study (d=0.429–0.767, p<0.05) was consistent with previously published meta-analysis findings on universal and selective intervention studies to improve mental health literacy using a mental health curriculum (d=0.541–0.774, p<0.05).¹⁴ It was also consistent with a result of another metal analysis of a digital interventions, social media included to improve positive mental health outcomes among young people involving supervision (Cohen’s d=0.52) and without involving supervision (Cohen’s d = 0.33).⁷³

Results from regression models that stratified data by sex and age group have reported the coefficient of the estimated difference-in-differences as being reflective of an intervention program having a higher and more significant impact (p<0.001). The level of mental health literacy dramatically increased with the intervention despite sex and age differences. Hence, the intervention implemented using social media was shown to improve adolescents’ mental health literacy significantly, in support of the original hypothesis. Previous qualitative evidence has shown that health interventions using online technology have been able to overcome multiple physical and logistical barriers and challenges.¹ Findings from the present study were consistent with those of previous studies despite little quantitative evidence being available.^1,74

Socio-demographic and socio-economic factors have been shown to influence the effectiveness and outcomes of health interventions.⁷⁵ A growing number of studies have revealed that gender dynamics affect outcomes aiming to introduce child and adolescent health-related-behavioural change.⁷⁶ As such, the regression models were stratified after multiple observable confounders and had been controlled for by assessing the significance of the variability in mental health literacy across these groups. For example, an earlier study conducted in the Ethiopian population showed that girls had slightly higher mental health literacy scores than did boys⁴² consistent with other studies conducted in Portugal, Australia, and the USA.^55,59,77 Further, the effectiveness of the mental health literacy intervention conducted in this study varied across sex and age groups, independent of the methods used to measure it. The difference-in-difference estimate was higher among girls than boys, and the effect size was higher for participants aged 15–17 years (p<0.05).

Age is known to be a factor associated with health intervention outcomes.⁷⁸ From the present study, effect of the intervention on the mean score of mental health literacy estimated with difference-in-differences (DID) was significantly higher for older adolescents than for younger adolescents. The probable explanation might be that older adolescents spend more time and participate with more attention than the younger adolescents. The younger adolescents may have shorter attention spans and lower focusing. The differences in the cognitive and behavioural skills of adolescents have also been shown to influence such intervention outcomes.²⁷

The findings of this study contribute greatly to the existing literature describing interventions to improve the mental health literacy of adolescents, as well as to the literature assessing implementation outcome measures. First, such quantitative evidence in the form of standardized effect sizes and difference-in-difference estimates provides an opportunity to demonstrate the practical significance and impact of digital and web-based interventions promoting adolescent mental health literacy instead of only reporting the statistical significance. Second, effect sizes and difference-in-difference analyses allow researchers to draw meta-analytic conclusions across studies. Third, these effect size and difference-in-difference findings contribute to the design and planning of new studies, including the estimation of required sample sizes with the desired likelihood.⁷⁰

Participant ratings of the perceived outcome measures of the intervention implementation process were significantly different across age groups (p<0.05), with older adolescents reporting closer agreement with the implementation outcome measures. This effect of age may explain the difference between technology and internet access and challenges related to digital literacy. Older adolescents may find it easier to access the internet and may have better digital literacy. These findings were consistent with the findings of other similar studies.^37,79

Significant variation in observable measures was seen with the demographic characteristics of the participants. Participant sex has been shown to be one of the main factors predicting internet use behaviours,⁷⁹ with girls spending more time on social media, and being more receptive to digital interventions to behaviour change.⁷⁹ However, in the present study there was no significant sex difference in implementation outcome measure ratings, which is inconsistent with previous reports.^37,79,80 Detailed qualitative explanations are needed to interpret such quantitative findings.

This study assessed several potential factors that were suggested to influence the effectiveness of the intervention and the intervention implementation outcome measures. Unavailability or lack of access to resources (technology or internet access), personal factors (confidence, concerns about privacy, digital skills, frequency of interaction, motivation, using other unnecessary websites, and other personality traits) and family-related factors (being prohibited from accessing social media or the internet) were perceived factors affecting effectiveness and implementation outcome measures. This finding was consistent with the outcomes of related studies of digital or web-based interventions,^81–85 whereby behavioural factors,⁸¹ digital or internet literacy,⁸² lack of money or internet access,⁸³ adherence,⁸⁴ and participant engagement,⁸⁵ were reported to be associated with the effectiveness of an intervention. Participant interest and motivation, as well as having an understanding of their concerns, emerge as key factors for effective digital or web-based intervention programs.⁸⁶

The present study has both strengths and limitations. The study used a before-and-after design with the control mechanism utilizing both a control group of participants and a pretest/post-test comparison to assess the effect of the intervention. This study was conducted in a real-life setting and with a low implementation cost, avoiding many confounds to internal validity.⁸⁷ However, like any quasi-experimental study, these findings may be subject to biases. The conventional interpretation of the effect size has limitations due to the absence of reliable empirical evidence of the effects of such an intervention at the population level. Further, due to the absence of much existing published evidence, subtle differences in study designs, scales, and measurement instruments may limit the validity of a comparison approach.^68,69,71,72 Reliability and scalability might be confounded by deviation from pragmatic conditions, offering incentives, increased handholding, selection bias, confined study environment, and lack of follow-up assessments. The present study shared factors related to the disparity of internet access, difference in interest and internet literacy, negative feeling and doubts about online information among anticipated limitations of studies for online web-based interventions.^88,89 The study population represents few subsets of adolescent population who reported they had internet access limiting generalizability of findings to other populations. Finally, the study was conducted during the unprecedented COVID-19 pandemic. As such, participants were at home during the social media intervention, likely having an impact on the intervention outcomes measured, either positively or negatively.

Conclusions

These findings contribute novel and innovative approaches to assessing the effectiveness of the implementation of an intervention, as well as directly assessing the effect of the intervention itself and factors associated with it. The intervention used social media as a channel of communication for mental health promotion to improve adolescent mental health literacy. The findings of the study support the assertion that an intervention using an evidence-based mental health curriculum delivered over social media can significantly improve the mental health literacy of adolescents. We found that the implementation of the intervention was considered acceptable, appropriate, feasible, and satisfactory. The programme itself has practical scalability and sustainability despite some factors limiting its effectiveness in some individuals. The effectiveness of the intervention as reflected in the effect size and in the difference-in-differences estimate differed with both sex and age. The factors found to limit the success of the intervention were the availability of resources, and personal and family-related behaviours indicating the need to consider these challenges in the future when scaling up such an intervention to utilize emerging digital and online platforms for the promotion of mental health literacy. Although these findings are preliminary, they highlight potential areas for improvement in the application of adolescent mental health promotion using digital and online technologies including, most importantly, social media. Future interventions should consider other factors contributing towards good mental health, as well as other socio-demographic and contextual factors.