Testing the Factorial Validity of the Classroom Engagement Inventory with Spanish Students

Introduction

Currently, the need to motivate students to participate in their teaching process is becoming present. To achieve this objective, various investigations conclude that dynamic learning favors the participation and commitment of students compared to students trained in traditional teaching, where students’ main resource is their instructor who only teaches them face-to-face.^1,2

Lipowsky et al³ summarize that cognitive activation needs to occur. Three aspects are the key: The first is to try to connect the new knowledge with the previous knowledge of the students. The second refers to the cognitive demand of students. Make it clear at this point that very high or very low demands do not motivate students.⁴ And finally it has to do with the oratory of the teacher, the speech that is given and its quality. That is why it is necessary to choose well the tasks and activities carried out in class. Although cognitive activation has been found to be positively associated with performance,^5,6 so are positive feelings towards context.⁷ This shift towards a more active education represents an opportunity to improve academic achievement and student engagement.⁸ That is why it is necessary to create instruments that allow evaluating the commitment of the students, to know the satisfaction with the teaching practices of the students.

Educational engagement refers to the degree to which students are immersed in activities within the classroom.⁹ The more engagement the students have, the more focused and participatory they will be in the task.¹⁰ On the other hand, Kahu and Nelson¹¹ affirm that this engagement does not depend solely on the teaching style, but also that a sociocultural context will be necessary that encourages the students’ commitment to the task, this behavior can be inferred by a compendium of dynamic, interconnected, multidimensional behaviors.

Understanding what happens in a classroom can be challenging. Within the school, a multitude of determinants converge that converge in the students and that will result in different levels of motivation and engagement on the part of the students. Motivation is understood as a continuum in which the motivated student is willing to mobilize an amount of energy to obtain a goal. Ryan and Deci¹² from the theory of self-determination, state that when motivation is related or guided by external factors or the achievement of specific goals, we speak of extrinsic motivation, while when it starts from a feeling of accomplishment, improvement or self-efficacy, speaks of intrinsic motivation. In the classroom this translates into being motivated towards the pair in order to achieve a specific goal, such as passing a task for extrinsic motivation or learning about what is proposed because it causes interest for intrinsic motivation.¹³

On the other hand, engagement is another step in the commitment acquired by students in their learning. Not only is the student involved in this process, so is the community, the values and the relationships that are forged in this environment.¹⁴ In this sense, Reeve and Tseng¹⁵ show that engagement is projected into the future and involves changes in behavior since there are feelings, emotions and thoughts involved in the process that make this state last over time. Engagement culminates in a proactive attitude of students towards their learning. This change in the students will be perceived by them with feelings of self-efficacy and autonomy, in which they will mark the way to learn about what they feel motivated by.

Engagement has traditionally been defined by three interrelated dimensions: emotion or affection, which refers to the positive emotions experienced and the absence of negative emotions in the environment, which would encourage the student to continue in that situation.¹⁶ The behavioral aspect that refers to all the energy mobilized to meet expectations related or not to learning but which are in line with the environment.¹⁶ And finally, the cognitive aspect refers to the cognitive strategies that the student performs in order to achieve the goals set.¹⁷

Engagement cannot be understood without motivation and that is why it is a prerequisite for engagement to occur. Engagement and intrinsic motivation are two very close constructs and in fact in the model proposed by Saeed and Zyngier,¹⁸ although they understand that intrinsic motivation and real engagement are not the same, they found that they correlate positively and that both influenced the result of their learning in the same way.

Regarding gender differences in engagement, it has been shown that women may have a higher level of dedication than men.¹⁸ In line with these results, girls may also have higher mean scores on behavioral and emotional engagement than boys.¹⁹ However, in other studies no differences have been found between levels of engagement by gender.^20,21

There are different instruments to evaluate engagement. Assunção et al²² presented the University Student Engagement Inventory (USEI), which is a validated instrument to evaluate the engagement of university students. It was validated with a sample of 3992 university students. The instrument has 3 subdimensions: behavioral engagement, cognitive engagement and emotional engagement. Alpha and omega values ≥ 0.7 were satisfactory indicators of internal consistency. Also, the Schoolwork Engagement Inventory,²³ validated with 679 students, stands out. The instrument has three dimensions: Energy, Dedication and Absorption.

However, it has been decided to adapt and validate the Classroom Engagement Inventory²⁴ for Spanish students because it comprehensively evaluates the engagement dimensions (Affective engagement, Behavioral engagement, Behavioral engagement, Cognitive Engagement and Disengagement) and we consider that it can be a sensitive instrument to assess the impact on engagement and disengagement of educational programs. This instrument has been used in previous research, such as Ayçiçek and Yanpar²⁵ to determine the effect of the flipped classroom model on student participation in teaching English in the classroom. Since the “Classroom Engagement Inventory” was developed for higher students, they again performed the confirmatory factor analysis for high school students. The value of reliability coefficient in their study is 0.93, suggesting that the research instrument is reliable. After the application of the flipped classroom, the experimental group had significant differences in the five subdimensions (p>.05) with respect to the control group. Furthermore, this instrument has been validated in Turkish.²⁶ It was administered to 300 high school students in Ankara and the 5-factor structure of the original form was tested, having a Cronbach-Alpha internal consistency coefficient of 0.930. After the exploratory factor analysis, the structure appeared was tested through confirmatory factor analysis applied to 201 high school students and the structure was confirmed by the analysis.

To validate the Classroom Engagement Inventory with Spanish high school students, once the scale was drawn up, a Confirmatory Factor Analysis (CFA) was done to ensure content validity, internal consistency, and factorial structure.

Method

Participants

The study participants were 546 young people (316 men and 230 women) between 12 and 16 years old. These young people came from various compulsory secondary schools. A non-probabilistic incidental selection procedure was used for their selection.

Measurements

Commitment

An attempt was made to validate and adapt the Classroom Engagement Inventory by Wang, Bergin and Bergin²⁴ into Spanish. The scale is made up of 24 items divided into five factors: Affective engagement (eg, I feel interested), Behavioral Compliance (eg, I listen very carefully), Behavioral Effortful (eg, I get really involved in class activities), Cognitive Engagement (eg, I think deeply when I take quizzes in this class) and Disengagement (eg, I just pretend like I am working). Each of the items begins in the same way “In this class …”. The responses to each of the items are made through a Likert-type scale that ranges from 1 (totally disagree) to 5 (totally agree) except the items belonging to Cognitive Engagement that range between 1 and 7.

Procedure

The direct and reverse translation strategy was used in order to adapt the English version to the Spanish version.²⁷ The translation according to the degree of goodness of fit between the translated version and the original version. Once the questionnaire was obtained, two experts in educational psychology judged whether the items obtained were in line with the object of the scale and had not lost their original meaning.²⁸

With the questionnaire in place, approval was requested from the Bioethics Committee of the University of Almeria in order to start the study. Furthermore, this study respected the tenets of the Declaration of Helsinki. Once approval was obtained, we contacted several compulsory secondary schools to ask them to help us with the study. The schools that gave us their support to access their students were explained the objective of the study. In the same way, the students and their parents were informed and asked to sign an informed consent form.

The questionnaires were filled in at the beginning of the lessons, on paper and individually, and took about 13 minutes. A researcher from the group was present during the completion of the questionnaire to answer any questions.

Analysis of Data

As a necessary process to provide the questionnaire with validity and reliability, its psychometric properties were analysed. For this purpose, a confirmatory factor analysis was carried out to analyse the structure of 24-items of the questionnaire.²⁹ In addition, a gender invariance factorial analysis was carried out to determine whether the structure of the questionnaire is understood in a similar way by both boys and girls, and a temporal stability analysis (test-retest reliability). Finally, a reliability analysis was carried out using Cronbach’s alpha,³⁰ descriptive statistics and bivariate correlations (Pearson). The programmes used in these analyses were the statistical packages SPSS v25 and AMOS v21.

The estimators were not affected despite non-normality (Mardia coefficient = 215.12). Therefore, the maximum likelihood method together with a bootstrapping of 6000 interactions was used for the CFA. In this sense, in order to accept or reject the factor structure of the questionnaire, the fit indices established by Byrne²⁹ and Strijbos, Pat-El and Narciss³¹ will be taken into account. The indices are those set out in Table 1 below.

Table 1 CFA Adjustment Indices

Results

Confirmatory Factor Analysis

In order to provide evidence of an adequate factor structure, two factor analyses were carried out, as detailed in Table 2:

Table 2 Factorial Analysis

As shown in Table 2, the fit indices of the four-factor model (Figure 1) revealed good fit indices for the 24-item model: χ2 (242. N = 546) = 664.14, p <0.001; χ2/df = 2.74; CFI = 0.95; IFI = 0.95; RMSEA = 0.055 (90% CI =.051 - 0.061); SRMR = 0.039. The standardized regression weights ranged between 0.68 and 0.87, being statistically significant (p <0.001). Regarding the correlation between the factors, it ranged between −0.31 and 0.66, being statistically significant (p <0.001).

Figure 1 Results of the structural equation modelling. Note: ***p< 0.001.

In addition, a higher order factor analysis was performed in order to group the five factors into a single factor, revealing good fit indices: χ2 (247. N = 546) = 737.68, p <0.001; χ2/df = 2.99; CFI = 0.95; IFI = 0.95; RMSEA = 0.057 (90% CI = 0.052 - 0.060); SRMR = 0.048.

Analysis of Invariance by Sex

The fit indices obtained (Table 3) allow us to accept the equivalence of the basic measurement models between the two samples. Although the value of χ2/df exceeds that required to accept the hypothesis of invariance, the rest of the indices contradict this conclusion (GFI 0.966; CFI 0.948; RMSEA 0.057) which allows us to accept the base model of invariance (model 1 = unrestricted model).

Table 3 Analysis of Invariance by Sex

Adding restrictions on factor loadings to the base model characterises the metric invariance (model 2). The values in Table 1 allow us to accept this level of invariance. The general fit index (GFI= 0.962) and the root mean square error of approximation (RMSEA= 0.057) continue to provide converging information in this direction. Furthermore, Bentler’s comparative index (CFI= 0.947) does not change much with respect to the previous model.

Taking into account the criterion of nested models proposed by Cheung and Rensvold,³² who suggest that if the calculation of the difference of the CFIs of both nested models decreases by 0.01 or less, the restricted model is considered good and therefore the fulfilment of the factorial invariance; the difference between CFIs obtained (0.001) allows us to accept the metric invariance model. We can conclude so far that the factor loadings are equivalent in the two samples.

Having demonstrated the metric invariance between the samples, we move on to assess the equivalence between intercepts (model 3 = strong factorial invariance). The indices (Table 2) show a good fit of this model, both assessed independently and analysed with respect to its nesting with the metric invariance model. The difference between the comparative Bentler³³ indices is 0.004; the overall fit index is 0.944 and the mean squared error of approximation is 0.056. Accepting strong invariance, the two models evaluated are equivalent with respect to the factor coefficients and the intercepts.

Descriptive Statistics, Correlation and Reliability Analysis

Table 4 shows that the correlation between the five factors is positive among those closest to the factors and negative with the most distant, being significant. Likewise, in order to obtain evidence of the reliability of the scale, an analysis of internal consistency was carried out where the scores were satisfactory, above 0.80.

Table 4 Descriptive Statistics, Cronbach’s Alpha and Bivariate Correlations

Discussion

This study analyzed the validity of the Spanish version of Classroom Engagement Inventory. The educational engagement of the students is important to understand the motivation towards the task and the participation of the students in high schools, which is directly related to their academic success. Although the concept of engagement is not new, the development of measures that allow its evaluation in the classroom to promote motivational teaching practices is something valuable for teachers and families as well as for researchers.

To validate the questionnaire in Spanish, it was necessary to examine its psychometric properties, including a confirmatory factor analysis. The goal was to create a practical and consistent questionnaire that represented the different factors of engagement and disengagement in education, and which was invariant to the gender of the students.

The confirmatory factor analysis provided support for a 24-item four factor model for the Classroom Engagement Inventory. The subscales attained adequate internal consistency, with Cronbach alpha values exceeding 0.70, which is commonly accepted as reflective of internal consistency.³⁰ The confirmatory factor analysis revealed that the set of engagement factors were correlated positively amongst each other, and engagement factors and disengagement were inversely correlated with each other, as anticipated.

Despite the results achieved, some limitations are found in this research. In the first place, the sample was non-probabilistic, dependent on access to the sample in their educational centers. In turn, within the selected sample, it depended on the class not being confined due to COVID-19 and the researchers being able to apply the questionnaire in person, which consequently led to an experimental drop-out. Second, the results show that the questionnaire can be used by both men and women, however, future research should determine if other variables could alter the results, such as age or socioeconomic status.

Previous research suggests that emotional engagement and cognitive engagement are highly necessary to facilitate student participation.³⁴ Future research should examine the relationships between engagement in the classroom and other variables such as academic performance, academic anxiety, self-esteem, educational flow, among others, as well as conduct experimental studies to investigate whether student engagement varies according to the methodology applied by the teacher.

Positive academic engagement is characterized by the vigor, dedication, and immersion of the students.³⁵ Knowing the level of student engagement can be a useful tool for secondary education teachers, since it allows us to analyze the methodological decisions applied and check if they meet the expectations of the students. In the present study, we validated the Spanish version of the Classroom Engagement Inventory, offering quality instruments validated to the Spanish context.

In addition to the formal educational context, this questionnaire can be applied in non-formal contexts such as socio-educational programs or educational reinforcement programs. This last context is of special relevance, since many students at risk of social exclusion participate in these programs and it is important that they feel motivated to reinforce their participation. These programs can achieve that students improve their reading skills and significantly reduce school failure.³⁶

Despite the results achieved in this study, there are a number of limitations that future studies should take into account, such as the lack of construct validity. Therefore, future studies should relate Engagement to classroom climate or satisfaction with classes.

Conclusion

The results of this study support the Spanish version of the Classroom Engagement Inventory (Appendix A) as a valid and reliable instrument to measure academic engagement in the Spanish context.