Development and Validation of a Routine Session-by-Session Experience Measure for Youth Mental Health Services: My Youth Mental Health Session Experience (MySE)

Introduction

World-wide, countries, such as Australia, Ireland, Canada, Israel, Norway, New Zealand, and a growing number of others, have invested significantly in reorienting their mental health care services to focus specifically on the needs of adolescents and young adults; aiming to reduce the high burden of disease from mental health problems for these age groups.¹ These youth mental health services are designed to redress the many barriers to access and engagement and the reluctance of young people to seek help from traditional mental health care systems.² Headspace is the largest national youth mental health care network worldwide. Founded in 2006 by the Australian Government,³ the headspace national network of centres and programs has supported over 867,000 young people through almost 6.8 million services.⁴ The foundation of the headspace network is the headspace centre, and there are now over 150 headspace centre-based services across Australia, supported by a suite of other programs, including digital services.⁵ The headspace centre is a community-based, integrated service hub providing highly accessible, youth-friendly services that deliver evidence-based interventions to young people for their mental health, health and wellbeing needs.⁶

The youth mental health service approach has revealed the need for youth-specific measurement tools suitable for the 12–25-year age range.^7,8 Specifically tailored measures have not been available, with few assessment and outcome measures applicable across the entire age range, due to the typical child and adolescent versus adult service focus.⁹ This has led headspace to develop purpose-built measures in conjunction with and appropriate for our client group.

To measure outcomes that determine the impact of the services young people receive, headspace centre services use three main measures: two existing clinical measures and a purpose-built quality of life measure. These include:

• the Kessler Psychological Distress Scale (K10),¹⁰ a measure of self-reported psychological distress, which was developed for use with adults, but is widely used across a broad age range, and is mandated for collection by the Australian Government as the headspace service funders;
• the Social and Occupational Functional Assessment Scale (SOFAS),¹¹ a brief clinician rating of current functioning that also was not developed specifically for young people; and
• MyLifeTracker (MLT),¹² a brief self-reported quality of life measure, which was developed specifically for use in youth mental health services, being co-designed with young people to ensure it measured outcomes that were meaningful across the 12–25-year age range.

The need for youth-specific and youth-designed measures is critical for youth mental health services, as these service approaches strongly prioritise young people’s participation, at all levels, and particularly in their own health care.^7,8 Further, the utility of ultra brief measures in clinical practice has long been recognised, especially to support measurement-based treatment.¹³ Routine monitoring is critical for measurement-informed care,¹⁴ which has been shown to improve retention and engagement, client experience, and clinical outcomes.¹⁵ Along with youth-specific outcome measures, such as MLT, which are beginning to be developed, youth-specific session experience measures are also needed as there are very few tailored measures currently available.

One of the few available and commonly used measures is the Session Rating Scale (SRS), which is a simple four-item visual analogue scale that addresses key dimensions of the therapeutic relationship.¹⁶ The items relate to the client’s perceptions of respect and understanding, relevance of goals and topics, client-practitioner fit, and overall alliance. The measure is recommended to be used for those aged 13 years and up, and the SRS has been translated into several languages and shown to be reliable in varied international contexts.¹⁷

Nevertheless, reviews of the psychometric properties of the SRS have questioned its construct validity as well as its practicality as a measure of therapeutic alliance given its relative stability.^18,19 We found no studies of the psychometric properties specifically for use with youth. Most importantly for youth mental health services, however, is that it was developed based on conceptual dimensions of the therapeutic alliance; it was not co-designed with young people to ensure that it focused on what was meaningful for them in their therapeutic sessions. Youth involvement in measure development is critical for the integrity of youth mental health services, which are based on the fundamental principle of valuing young people’s experiential knowledge.^7,8

Development and Implementation of My youth mental health Session Experience (MySE)

The ‘My youth mental health Session Experience’ (MySE) measure was developed by headspace for use as a routine session experience measure. Real-time feedback from young people about their experience of their youth mental healthcare session is needed for measurement-informed care. Such feedback enables clinicians to adapt and better target the services they are providing to support the young person’s engagement in treatment and the outcomes they achieve. It supports collaboration, shared decision-making, and is empowering for clients, as well as providing valuable information for service evaluation.²⁰

MySE was designed in collaboration with young people and service providers across the headspace centre network. We first surveyed the network and were advised that a session rating scale was needed. We developed a potential list of scale items by reviewing the literature and examining existing items we were already collecting related to service satisfaction. We presented the pool of proposed items to a working group of 20 service providers from across the headspace centre network and workshopped selection and wording of the items. We subsequently held two workshops with young people who were members of the headspace Youth National Reference Group (hYNRG), which comprises young people from diverse backgrounds from each state and territory across Australia, who work with headspace to ensure young people’s voices and opinions remain at the forefront. These workshops finalised the selection and wording of the items. The final set of items was then re-reviewed. The subsequent items were preliminarily implemented in the headspace centre data collection system, and the initial results were shared with the network; feedback confirmed that the items were useful and aligned with the practice experience.

The resulting measure is very brief, with just five items to use every session to determine whether the young person felt: willing to be there, listened to and understood, involved in guiding their own treatment, hopeful, and that things are improving in their lives. These were the aspects of each service session that were identified by young people and headspace clinicians as being meaningful and important for young people’s engagement, satisfaction, and the achievement of outcomes.

MySE was implemented in all headspace centres in July 2019. It is routinely measured at the beginning of each service session, asking young people to reflect on their last session. The five items are:

1. How willing are you to be at [youth mental health service]? (Willingness)
2. Do you feel listened to and understood at [youth mental health service]? (Understood)
3. Do you feel that [youth mental health service] is helping you to work on the issues that are important to you? (Importance)
4. Do you feel things are improving for you since attending [youth mental health service]? (Improving)
5. Do you feel more hopeful for the future since attending [youth mental health service]? (Hopefulness)

Young people are asked to rate each item on a 1–10 scale using a slider anchored by a sad emoji to a happy emoji. Item 1 is asked at each visit, and the other four items are asked from the second visit onwards. The average of the five items is calculated as a summary measure of service session experience. The summary and individual item scores are graphically displayed and made available electronically to clinicians at commencement of the session, so they can be used to inform, adapt, and improve the service that the young person receives to ensure it meets their needs.

The Current Study

This study investigates the psychometric properties of MySE to establish whether it is a reliable and valid measure of youth mental health service session experience. To do this, we examine the descriptives, test–retest reliability, and analyse its concurrent validity against a client satisfaction measure. We determine the factor structure of the five items, expecting that they will show good internal consistency and represent a single latent factor of session experience.

Materials and Methods

Participants

Participants were 37,201 young people aged 12 to 25 years who commenced an episode of care at one of the 150 headspace centre services between 1 July 2021 and 30 June 2022. They were 28% aged 12–14 years, 31% aged 15–17 years; 26% aged 18–21 years, and 15% who were young adults aged 22–25 years. Females were represented (64%) more than males (31%); 5% of the participants identified as gender diverse. Participants were predominantly from metropolitan areas (54%), followed by inner regional (30%), outer regional (13%), and remote areas (3%) of Australia. A total of 9% were First Nations young people; 10% had a culturally and linguistically diverse background; and 28% identified as LGBTIQA+.

Young people can present to headspace with any issue, but anxiety and depressive symptoms are the most common primary presenting issues (36% and 23%, respectively). Other primary presenting issues included stress-related issues (5%), anger issues (4.8%), difficulty with personal relationships (3.4%), conflict in the home environment (3.0%), suicidal thoughts or behaviour (2.7%), trauma (2.3%), and other situational stressors or mental health symptoms (<2% each).

In total, 37,201 young people had received 38,935 episodes of care during this period. For most participants, this was either their first (66%) or second (22%) episode of care with headspace. Each episode of care consists of individual occasions of service (visits). Most episodes are short, consisting of either one single visit (27%), two visits (22%), or three visits (13%). Only 5% of episodes consist of more than 10 visits. The average visit frequency is 3.8. Since MySE is focused on evaluating the young person’s session experience over time, most of our analyses focus on the subsample of participants who had two or more visits at headspace and have a valid MySE measurement at visit 2 (16,484 young people in 16,747 episodes of care).

Procedure

When young people present at a headspace centre, they are provided with an electronic device where they answer a series of questions. They give consent to the use of their data for the primary purpose of service provision and outcome monitoring, but also for research purposes.

Different question sets are presented depending on the number of visits. The first visit in each episode of care has the longest questionnaire, including demographic and other background information. It also asks participants about their willingness to be at headspace, the first item of MySE, as well as gathering baseline information for the outcome measures of psychological distress and quality of life. At each subsequent visit, young people answer all five items of the MySE measure. They are provided with an optional client satisfaction survey at visits 2, 5, and 9. Service providers also input information at every visit, including a measure of social and occupational functioning.

Ethics approval ensured that the study complied with the Declaration of Helsinki, comprising approval through the headspace Board Research subcommittee and the consent processes being reviewed and endorsed by an independent body, the Australasian Human Research Ethics Consultancy Services. Note that the data are collected primarily for service provision and quality improvement purposes, and parental consent is not routinely sought for those under 18 years, who are assessed as mature minors unless otherwise indicated. Young people were able to opt out of data collection if they chose, or at the discretion of their parents.

Measures

Psychological distress was measured using the Kessler Psychological Distress Scale 10 (K10).¹⁰ This self-report scale has 10 questions about the level of anxiety and depressive symptoms experienced in the last four weeks answered on a rating scale from 1 (none of the time) to 5 (all of the time). The total score is the sum of all questions, ranging from 10 to 50, with higher scores indicating greater psychological distress. The K10 is widely used in mental health research and has been validated in the general Australian population.²¹

Quality of life was measured using MyLifeTracker (MLT), a five-item self-report measure developed and validated specifically for youth mental health services.¹² The MLT items focus on areas of importance to young people, such as their general well-being, day-to-day activities, relationships, and coping with life over the last week responded to on a 0 to 100 slider scale. The total score is the average across the five items, ranging from 0 to 100, with higher scores indicating a higher quality of life.

Functioning was assessed at each visit by clinicians via the Social and Occupational Functional Assessment Scale (SOFAS).¹¹ This is a single-item clinician assessment of a client’s social functioning independent of their psychological assessment, with scores ranging from 1 to 100. Higher scores represent increased functioning across a variety of domains, including personal hygiene, school or work functioning, and social activities.

Client satisfaction was determined using the headspace Youth Service Satisfaction Scale (YSSS).²² This comprises 14 items that measure satisfaction with the service centre (three items), staff (four items), outcomes (five items), and general satisfaction (two items) with responses on a 5-point Likert-type scale from “Strongly Disagree” to “Strongly Agree”. Total scores are averaged over all items and can range from 1 to 5, with higher scores indicating greater satisfaction.

Demographic characteristics were gender (female, male, gender diverse), age-group (early-, mid-, late-adolescence, early adulthood), First Nations background, cultural and linguistic background, LGBTIQA+ identity, and geographic location.

Data Analysis

Descriptive statistics for MySE across time (visits) and for different demographic groups are computed, and correlation matrices between measures and inter-items are evaluated. Next, we conduct factor analysis of MySE items using maximum likelihood factoring. Because we expect session experience to change over time and with clinical outcomes, we examine factor structure and other psychometric properties at visit 2 (the first instance where all five items are elicited), visit 3 (the median number of visits), visit 5 (29% of episodes have 5 or more visits) and visit 9 (9% have 9 or more visits). We randomly split the sample into two and use the first half for exploratory factor analysis (EFA) and the second half to perform confirmatory factor analysis (CFA). We assess model fit using root mean squared error of approximation (RMSEA), the comparative fit index (CFI), the Tucker-Lewis Index (TLI), and the standardized root mean squared residual (SRMR).²³

Cronbach’s α coefficient is used to assess internal consistency. For this analysis, we pool all data available from visits 2 to 10. We examine test–retest reliability using Pearson’s correlation coefficient r for visits that fall within a two-week period. To assess concurrent validity, we perform Pearson’s correlation analysis at visits 2, 5, and 9 between MySE and YSSS.

Analyses were conducted using SPSS Version 28 and Stata Version 15. For in-text reporting, we apply a threshold of statistical significance that requires p < 0.001 and d > 0.1 and β > 0.2 to eliminate trivial differences due to our high statistical power.

Results

Descriptives

Descriptive statistics for the MySE summary measure and the outcome measures at visit 2 by age group and gender are shown in Table 1. All subgroups report an overall positive session experience (between 7 and 8 on the 10-point scale), but older age groups report a more positive experience than younger age groups (p < 0.001, η² = 0.033). There was no significant difference between gender groups.

Table 1 Summary Statistics of Measures at Visit 2

Figure 1 plots average MySE scores by visit. To rule out a possible selection effect from young people with on average worse experience dropping out of care, we display averages for the stable subsample of those with 10 or more visits (dashed lines) alongside the full sample (solid lines). In both samples, all dimensions of session experience show an upward trend, and this trend is strongest for feeling that things are improving. Willingness to be at headspace and feeling listened to and understood show increases over the first few visits and remain very high and stable thereafter. While session experience is already quite high at the first measurement (cf. Table 1), all individual measures range between 7.8 and 8.8 by the tenth visit.

Figure 1 Average MySE scores by visit (solid lines = full sample; dashed lines = subsample of young people with 10 or more visits).

The correlation matrices show pairwise correlations between outcome measures and the total MySE score (Figure 2a) and between the individual MySE items (Figure 2b) for the full sample. All correlation coefficients are statistically significant (p < 0.001). The MySE summary measure is most strongly correlated with client satisfaction (r = 0.68), moderately correlated with MLT (r = 0.43), and only weakly correlated with the clinical outcome measures K10 and SOFAS (r = −0.28 and r = 0.24, respectively; Figure 2a).

Figure 2 (a) Correlation matrices: Pairwise correlations between outcome measures. (b) Correlation matrices: Pairwise correlations between MySE items.

Figure 2b shows that inter-item correlations range from 0.52 (between Willingness and Improving) to 0.79 (between Improving and Hopefulness). Factor analysis was deemed appropriate with all correlations >0.3, the determinant of the correlation matrix positive, Bartlett’s test of sphericity significant, and a Kaiser–Meyer–Olkin Measure of Sampling Adequacy of 0.85.²⁴

Factor Structure

Exploratory factor analysis of the five MySE items, conducted on a randomly selected half of the sample, produced single-factor solutions at visits 2, 3, 5, and 9 (Table 2). Even though factor loadings for the five items are uniformly strong and positive, the chi-squared test statistic for the single-factor model indicated an unsatisfactory model fit across all visits. This is expressed in relatively high uniqueness values for the Willingness item, indicating that this variable is not well represented in the single-factor model. Over time (visits), the single-factor solution becomes stronger, with lower unique variances and higher factor loadings for all items. However, the single-factor model does not reach satisfactory levels of goodness-of-fit. A two-factor solution is not indicated, however, as the estimation produces a Heywood case (boundary case with 0 or below 0 estimates for uniqueness).

Table 2 Exploratory Factor Analysis at Different Visits

The CFA confirmed that the model fit is not satisfactory with an RMSEA and its 90% confidence interval well above the acceptable value of 0.06 (see Table 3).²³ Other goodness-of-fit indicators (with the exception of SRMR) also confirm that the fit is below the desired standard (CFI > 0.95; TLI > 0.95; SRMR < 0.08). The addition of a covariance path between the residual variances of Improvement and Hopefulness improves the fit (RMSEA becomes 0.055 and TFI and CFI >0.99, at visit 2).

Table 3 Confirmatory Factor Analysis at Different Visits

Internal Consistency

Across all visits, MySE shows high internal consistency, with Cronbach’s α = 0.89. This holds true for all age and gender groups and at different points in time.

Test–Retest Reliability

Table 4 shows Pearson’s correlation coefficients between consecutive visits, provided these visits occur within two weeks from each other. The number of observations given is the number of paired visits based on which the correlation between the consecutive MySE measures is computed. All correlations were strong and became stronger over subsequent visits.

Table 4 Test–Retest Reliability at Different Visits

Concurrent Validity

Concurrent validity was established using data from the sub-sample of young people who opted to complete the voluntary client satisfaction survey (YSSS). Figure 3a shows concurrent overall satisfaction and session experience (MySE) measures pooled across visits for which both measures are available (n = 3,981), revealing a strong correlation (r = 0.68). The blue line represents a fitted linear regression line.

Figure 3 (a) Concurrent validity of MySE and YSSS: Overall satisfaction and session experience (scatterplot of YSSS and MySE data with fitted regression line). (b) Concurrent validity of MySE and YSSS: Specific satisfaction ratings by MySE ratings.

Figure 3b shows that the strong association between MySE and overall satisfaction holds for individual MySE dimensions and their most closely aligned counterparts in the client satisfaction scale. The MySE dimension Understood (“Feeling listened to and understood”) aligns with the satisfaction item “I felt that headspace staff listened to me”. The MySE dimension Importance (“Feeling that headspace is helping [me] to work on the issues that are important to [me]”) aligns with the satisfaction item “I got help for the things I wanted to get help with”. The MySE dimension Improving (“Feeling that things are improving for [me]”) aligns with the satisfaction item “I feel that my mental health improved because of my contact with headspace”.

Discussion

This study investigated the psychometric properties of MySE, a brief five-item measure to routinely assess young people’s experience of their youth mental health service sessions. The measure was developed to fill the gap in availability of a measure specifically tailored for adolescents and young adults for session-by-session experience monitoring in youth mental health care.

Our results show that the total MySE score has high internal consistency that is invariant over age and gender groups. There is one latent factor of session experience that all MySE items relate to, and all items demonstrate strong factor loadings. Nevertheless, one factor does not adequately capture all the information present in the individual items, although a two-factor model also does not fit well. Adding covariance between the residual variances of Improvement and Hopefulness improves the fit to be adequate, but this does not affect the construction nor interpretation of the scale. It is also evident that the willingness item shows high uniqueness despite also loading strongly on the one factor. This supports the usefulness of considering the individual dimensions of MySE separately, together with the summary measure, which does provide a reliable summary score.

The only demographic effect evident was for age, whereby the young adults reported more positive session experiences than the adolescents. This trend is consistent with earlier studies of overall service (rather than per session) experience for headspace centre services,²² and has also been demonstrated for headspace online services.²⁵ There is surprisingly little published literature on age differences in youth mental health services,²⁶ despite the wide age range and major developmental changes that occur during this period. For example, the youth-specific outcome measure, MLT, shows age and gender group differences that are developmentally congruent and important to consider for clinical interpretation.²⁷ Notably, the willingness of young people to attend mental health services increases with age, demonstrated through increasing self-referral and less influence of parents on help-seeking.²⁵ Growing maturity and subsequent greater autonomy and self-direction are reflected in greater willingness and overall session experience, important issues related to maturity that service providers need to consider regarding service engagement and retention.

MySE is intended to be a dynamic measure for service providers to respond to and use to improve their service delivery at an individual level. The test–retest reliability over a two-week period is high, demonstrating reliability over time. However, session experience improves over time with further service engagement (Figure 1), and our analyses show that this is not due to a retention effect. Session experience should change over time, with the expectation that it would generally improve with ongoing treatment. Our results show that MySE demonstrates this attribute.

Willingness to be at headspace and feeling listened to and understood show increases over the first few visits and then remain very high and stable thereafter. Improvement over time is strongest for feeling that things are improving, which is an important validation of continued service engagement.

The measure shows concurrent validity, being strongly associated with general service satisfaction, and moderately associated with youth-specific outcomes as measured by MLT. It is only weakly associated with the outcomes of self-reported psychological distress and clinician-reported social and occupational functioning, however. These results suggest support for the principle that outcomes and session experience both need to be youth-specific and youth-designed to be congruent. Both MLT and MySE were developed through youth co-design and specifically target the youth mental health service context. The individual items of MySE also show strong concurrent validity with similar items from the overall service satisfaction measure that was developed specifically for the headspace youth mental health centre services, but which is less frequently collected.

Our results need to be interpreted in light of the limitations of our study. These mostly relate to generalisation; it is unclear whether results from the Australian headspace services will apply to other youth mental health services elsewhere internationally. Importantly, however, the measure did not reveal differences for young people from First Nations, culturally and linguistically diverse backgrounds, or who were LGBTIQA+, demonstrating important cultural relativity. This may reflect the prioritisation headspace places on including young people from diverse backgrounds, including young people from each of these demographic groups being members of hYNRG.

Conclusion

Our findings show that MySE has appropriate psychometric properties to measure session-by-session service experience for young people in youth mental health services. The reason for implementing such routine measures is to support measurement-informed care, which should improve young people’s experience and outcomes. Future research is needed to show whether and how MySE improves clinical practice and whether implementing the measure is associated with improved engagement and outcomes for young people in the innovative and rapidly growing sector of youth mental health care.