The Use of Multidimensional Nomial Logistic Model and Structural Equation Model in the Validation of the 14-Item Health-Literacy Scale in Chinese Patients Living with Type 2 Diabetes

Background

Type 2 diabetes mellitus (T2DM) is a challenging public health problem that requires extensive disease management skills. According to the tenth edition Atlas of International Diabetes Federation, approximately 537 million adults (20–79 years) are living with T2DM in 2021 and over a quarter of these population comes from China.¹ Controlling blood glucose requires the patients to have adequate self-care abilities, including reasonable daily healthy diet arrangement, blood glucose self-monitoring, regular exercise and timely medication adjustment according to medical advice.^2,3 Because of this reason, patients’ ability to effectively obtain and use healthcare information and services is of great importance.

Health literacy (HL), defined as the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions, has been discovered to be an important construct that may affect the outcomes in diabetes control.⁴ HL promotes people’s ability to manage their health optimally, and it is shown that good HL can predict better diabetes-related knowledge, self-care behavior, glycemic control and fewer odds of micro- and macrovascular complications.^5–7 A meta-analysis indicated that HL was positively associated with medication adherence and adherence outcomes, and the impact of HL was even greater in samples of lower income and of racial ethnic minority patients.⁸ Moreover, HL may also pose a crucial effect on patients’ self-efficacy for medication use in the context of self-management behaviors.^9–11 However, T2DM patients with poor level of HL were prevalent and associated with increased health care costs.^12–15 This may be because individuals with inadequate health literacy were significantly associated with the increased prevalence of chronic conditions (cerebral, vascular, cancer, diabetes and arrhythmias), higher hospitalization and the poor access to medical care.^16,17

Several instruments were developed to assess the HL in the clinical practice. The Rapid Estimate of Adult Literacy in Medicine (REALM) and the Test of Functional Health Literacy (TOFHLA) are the most commonly used measurements in the clinical study.^18,19 In the former measurement, HL is evaluated with word recognition tests, the latter combined HL and numeracy tests. Although there are Chinese edition of these HL tests, such as the simplified Chinese version of the S-TOFHLA,²⁰ these measurements are not commonly used in the mainland China, this is probably due to different morphological typology between English and Chinese language. Moreover, although multidimensions are indicated in the model of health literacy, most of the measurements such as TOFHLA and REALM pay close attention to the functional HL.²¹ Functional HL, defined as the basic skills in reading and writing to be able to function effectively in daily situations, is only the narrow definition of “health literacy”; together with the more advanced cognitive and literacy skills including communicative literacy (the skills can be used to actively participate in daily activities, extract and derive information from different forms of communication, and to apply new information to changing circumstances) and critical literacy (the skills can be applied to critically analyze information so as to exert greater control over life events and situations), Nutbeam indicated that the different levels of HL progressively allow for greater autonomy and personal empowerment.²² Hence, measurements for more dimensional concepts in health literacy are warranted.

The 14-item health literacy scale (HL-14) is a brief measurement utilized for measuring HL in the clinical and public health contexts.²³ The biggest advantage of the HL-14 is that it evaluates three dimensions of HL including patients’ functional, communicational and critical ability when they are facing healthcare information. The measurement is a Likert scale with good reliability and validity and has been translated into Brazilian Portuguese version which also showed good internal consistency and three-dimensional model fit.²⁴ Using the HL-14, Shirooka reported that non-frailty community-dwelling older adult represented higher HL ability, which indicated the importance of HL in maintaining good status in these people.²⁵

So far, the measurements such as the HL-14 suitable for rapid clinical screening are still unavailable in the clinical practice in China. Therefore, the purpose of the study is to translate the HL-14 into Chinese and validate the measurement in a sample of patients living with T2DM. To avoid the limitations of the Classical Test Theory (CTT), the study adopted the Item Response Theory (IRT) to investigate the psychometric properties of the HL-14 scale, meanwhile the appropriateness of the item difficult settings and differential item functioning could also be checked during the study. Moreover, for further interrogation of the three-dimensional configuration of the HL-14, structural equation modeling (SEM) was used.

Methods

Study Population

Participants were eligible to the study criteria if they were with T2DM and could communicate with the investigators fluently without any others’ help. All participants were approached during their hospitalization in the endocrinology department of Huadong Hospital from August to December 2021. Exclusion criteria included the following: type 1 diabetes and other specific types of diabetes; diabetic ketoacidosis; diabetic coma and those who cannot communicate by themselves or refuse to accept the assessment.

The sample size was computed based on the number of free parameters of the measurement as suggested by the experts. As the original Japanese edition of HL-14 contains fourteen free parameters, the ideally sample size would be 280 patents according to the 1:20 ratio.²⁶ Hence, at least 280 patients were recruited for this study.

Instruments

The Chinese edition of HL-14 is a five-point Likert Scale set that indicates how much the participants agrees or disagrees with the item. The scores on the items were summed up to give the total HL scores, as well as the HL scores of each dimension. Higher scores indicated greater HL possessed by participants.²³ The other two measurements used in the study were the Adherence to Refills and Medication Scale (ARMS) and the Charlson comorbidity index (CCI), which was used to evaluate the medication adherence and the comorbidity state of the study group.^27,28

The ARMS is a four-point Likert scale with 12-item set which is a valid and reliable measurement that has been widely used among patients with chronic disease and low literacy.²⁷ The total score of ARMS ranges from 12 (best) to 48 (worst). Lower scores of ARMS indicate better adherence to the medical regimen. The CCI is the most commonly used comorbidity index which was developed to predict death from comorbid disease.²⁸ It contains 19 issues such as diabetes with diabetic complications, congestive heart failure, peripheral vascular disease etcetera, each of which is weighted according to their potential influence on mortality. The instrument has been widely used in patients living with diabetes in clinical studies.^29–32

Translation and Adaptation of the HL-14 Scale

The translation and adaptation process of the HL-14 scale was conducted according to the Brislin’s model for cross-cultural research as follows:³³ (1) Forward translation: two bilingual clinical pharmacists translated the HL-14 from English to Chinese independently to form the first two Chinese versions of the HL-14; (2) Back translation: another two bilingual clinical pharmacists who were blind to the original English version scale back translated the two translated version from Chinese to English; (3) First group discussion: the four translators discussed the differences among the above mentioned translations and adapted the Chinese version of the HL-14 to achieve the most accurate meaning and the suitable expression for patients living with T2DM; (4) Second group discussion: two new bilingual clinical pharmacists back translated the adapted Chinese version of the HL-14 to English, and then a second discussion of these six translators was held to review the two new back-translation. Any discrepancies were discussed until consensus was achieved. The professor of clinical pharmacy, Chen Z.L., a doctoral supervisor of pharmacy, confirmed the accuracy between the original and back-translated version of the HL-14, and the Chinese version of the HL-14 was considered as the final version for psychometric testing, Table 1.

Table 1 The 14-Item Health Literacy Scale

Data Collection

This is a cross-sectional study. All participants were approached during their hospitalization in the endocrinology department. The HL-14 scale was given to the participants by the clinical pharmacist and finished by the participants themselves. All questionnaires were collected on site and checked for the completeness. Participants’ information, including age, sex, duration of diabetes, education level and their laboratory test results, was recorded according to the Huadong Hospital Information System.

Statistical methods

The IRT and SEM were used to explore the reliability and validity of the HL-14 scale. The strength of IRT in analyzing the Likert Scale has been reported by Bond and Fox.³⁴ The principal advantage of the IRT over the CTT lies in its invariance of the parameters and estimating each item difficulty/person ability on a common logit scale which can be displayed vividly by item-person map (Wright Map). SEM is a statistical method for analyzing the relationship between variables based on the covariance matrix, which enabling the researchers to test a set of regression equations simultaneously.

This study adopted the multidimensional random coefficients multinomial logistic model (MRCMLM) to examine the psychometric properties of the HL-14 scale. MRCMLM is a development of Rasch models of IRT (eg, Dichotomous Model, Rating Scale Model, Partial Credit Model) that incorporates multiple dimensions of the measurement procedure. The mathematical expression of MRCMLM model is presented below:

Where P_nij and P_ni(j-1) refer to the probability of a person of ability θ_n being observed as responding to category j or lower category j-1 respectively of a rating scale on a particular item i of difficulty ξ with d_ij and d_i(j-1) which is held as design vector to express the linear combinations within the ξ. ACER ConQuest 5.12.3: Generalised Item Response Modeling Software was used to perform the MRCMLM analysis in this study. Wright Map would be checked for the items and participants in terms of their difficulty or location in the same logit scale. Moreover, the existence of differential item functioning (DIF) with respect to sex in the HL test was also explored using ACER ConQuest 5.12.3.

In the SEM, fit indices including χ²/df, Comparative Fit Index (CFI), Normed Fit Index (NFI), Root Mean Square Error of Approximation (RMSEA), Tucker-Lewis Index (TLI) and standard residual mean root (SRMR) were computed in the process of confirmatory factor analysis (CFA), and all these model fit indices were set based on Hu and Bentler.³⁵ The factor model is acceptable and appropriate if the results of CFA demonstrate that TLI ≥ 0.95, CFI ≥ 0.95, NFI ≥ 0.95, SRMR < 0.06 and RMSEA < 0.08. Moreover, modification indices (MIs) would be checked to interrogate the independent of residuals. Once the residuals were identified, all authors reviewed and discussed the relative observed variables that may share the conceptually overlapped meanings. AMOS 21.0 (SPSS, Chicago, IL, USA) was used for the CFA in SEM.

The other analyses including demographic information, Pearson correlation (numerical variables) and Spearman correlation (categorical variables) were also performed in this study using SPSS Statistics 23.0.

Results

Demographic Characteristics

Two hundred and ninety-seven patients living with T2DM (161 men [54.2%]) were recruited in this study. All the patients are Chinese citizens from Shanghai, China. The patients had a mean (SD) age of 66.74 (12.16) years (range, 23–90 years), were predominantly with severe comorbidity (124/297, 41.8%), long duration of T2DM (146/296, 49.2% were diagnosed more than 10 years), and relatively low income of 2000–4999 RMB yuan per month (191/297, 64.3%). Of these 297 participants, 84 (28.3%) were well educated (with the level of university), 92 (30.9%) had educational level of senior high school (Table 2).

Table 2 Demographic Characteristics (n = 297)

MRCMLM Summary Statistics

The MRCMLM indicated that the Coefficient Alpha is 0.94 which showed high internal consistency of the scale items. The average dimension estimates of functional HL, communicational HL and critical HL were 0.694, 0.552 and 0.600, respectively. The correlation was moderate between functional HL and communicational HL (0.684) or critical HL (0.672), and high between communicational HL and critical HL (0.949). Table 3 indicates the internal consistency and floor and ceiling effects of the scale.

Table 3 Internal Consistency and Floor and Ceiling Effects of the 14-HL Scale

Item Analysis

The results of response model parameter estimates and item difficulties calibrated by the MRCMLM analysis are presented in Table 4. Item estimates range from −0.597 to 0.635 with the item 1 is the easiest and the item 2 is the hardest. In the present MRCMLM analyses, the infit and outfit range of 0.5 to 1.7 was adopted to investigate the reliability and validity of the HL-14 scale according to Yu Minning.³⁶ The outcome of the response model showed that all items fit in this range except item 9, suggesting that the items in the HL-14 scale fit the MRCMLM sufficiently well to define the three HL construct, ie, patient’s functional HL, communicational HL and critical HL. An examination of the T values (ideally range within −2 to +2) showed that five items (item 5, 8, 9, 11, 14) may produce less or more variation than modeled. Of these five items, item 5, 9, 14 indicated underfit to the model, which represents the guessing risk or unpredictable responses produced by these items, whereas item 8 and 11 are overfit which is closer to the Guttman-style response string.

Table 4 Response Model Item Estimates

Wright Map

The distinctive advantage of IRT analysis is that it can graphically illustrate the location of persons and items on the same interval-level measurement scale using Wright map. The map is presented in Figure 1. The histogram on the left side of the map was showing the distribution of patients in the order of their HL ability which seems distributed normally along with the logit scale. Those located at the upper end were patients with higher ability, whereas those located at the lower end were patients with lower ability. Similarly, the right side of the Figure depicted items plotted to indicate their difficulty level. Given that each person ability and item difficulty were estimated on a common logit scale, therefore, the difficulty level of the HL-14 items seems to be appropriate to most of the study patients as they cluster around the ability between −1 and 1 logit level compared with the ability of the study patients distributed from approximately-5 to 7 logit level.

Differential Item Functioning in Sex

Table 5 shows the sex differences in ability estimates. The parameter estimates indicated that the male participants have performed more poorly than the female counterparts. But the estimated difference of 0.04 is small at just 5.1% of a participant standard deviation (0.785), and the chi-square value of 0.26 on one degree of freedom does not indicate DIF in terms of sex.

Figure 1 Wright Map: Thurstonian Thresholds & Case Latent Distribution.

Table 5 Table of Response Model Sex Estimates

Assessment of CFA Model

All observed variables showed adequate factor loadings from 0.51 (item 09) to 0.95 (item 11) in the configuration of three-dimensional HL model, Figure 2. However, the result of CFA revealed that the model fit did not achieve a good level (x²/df=5.370, CFI = 0.909, NFI = 0.891, TLI = 0.888, RMSEA = 0.122, SRMR = 0.054). Therefore, we checked the MIs and found that the measurement error 10, 12, 13, 14 contributed most to the MIs (Table 6), which present the independent of residuals of the corresponding items. After referring the results of MRCMLM model and the discussion within the study group, all the authors believe that there is conceptually overlapping effect between the item 10 (Apply the obtained information to my daily life) and 14 (Collect information to make my healthcare decisions), or the item 12 (Consider whether the information is credible) and 13 (Check whether the information is valid and reliable). Hence, the covariances between these two groups of measurement errors were drawn and model fit was tested again. The reconstructed CFA model indicated a good fit (x²/df=2.698, CFI = 0.965, GFI = 0.917, NFI = 0.947, TLI = 0.956, RMSEA = 0.076, SRMR = 0.042) (Figure 3), and all observed variables also showed adequate factor loadings in this three-construct model, Figure 3.

Figure 2 The CFA model of HL-14.

Figure 3 The modified CFA model of HL-14.

Table 6 Covariance of the CFA Model

The Relationship Between Patients’ HL Ability and the Study Variables

In this study, the mean score of total HL, functional HL, communication HL and critical HL ability is 47.59 ± 14.68, 16.85 ± 5.96, 17.35±5.84, and 13.39 ± 5.06 respectively. Table 7 shows the correlation between HL ability and study variables. Participants’ total HL score correlated with the number of antidiabetic medicines, monthly income and education level significantly at the 0.01 level (Spearman’s rho=−0.168, 0.272 and 0.328 respectively), and with their comorbidity state at the 0.05 level (Spearman’s rho=−0.148). Their comorbidity state correlated significantly with the communicative and critical HL (Spearman’s rho=−0.195 and −0.158, P < 0.01), but not with their functional HL (Spearman’s rho=−0.075, P = 0.195). On the other hand, patients’ medication adherence correlated significantly with their total HL ability (r=−0.115, P = 0.047) and functional HL ability (r=−0.147, P = 0.011), but not with their communicative (r=−0.071, P = 0.22) and critical HL (r=−0.078, P = 0.178). As higher ARMS scores indicate poorer medication adherence while higher HL scores indicated higher HL ability, the correlation coefficient of these two variables was negative.

Table 7 The Correlation Between HL Ability and Study Variables

Discussion

The present cross-sectional study interrogates the reliability and validity of the HL-14 scale in patients living with T2DM in Shanghai, China. The results showed that the HL-14 scale is a psychometric sound measurement with adequate reliability and validity. To our best knowledge, this is the first study to validate the HL scale based on both MRCMLM and SEM in clinical research, which provides a robust theoretical foundation to the use of the HL-14 scale in clinical practice. Moreover, the test of DIF indicates that there is no statistically significant sex effect among the HL-14 items.

The results of the MRCMLM show that the difficulty settings of the HL-14 scale are appropriate for most of participants in this study as these scale items clustered around the middle part of the logit scale where participants’ latent estimates distributed as shown by Wright map. The average dimension estimates showed that the functional HL items are more difficult than the communicative HL and critical HL items. The correlation between communicative HL and critical HL is very high (r = 0.949) which is even higher than the original Japanese scale (r = 0.66),²³ and this finding can also be identified in the CFA model. We can speculate that there are significant differences in the perception of the communicative and critical HL between Chinese and Japanese, and these two HL abilities can also be regarded as unidimensional construct. As in recent years, there are experts who suggest to create a new term called health literacy fluency, defined as the effective use of health information by those who need it, to shift the focus of the current research.³⁷ Shirooka et al also proposed a concept of comprehensive health literacy which incorporates the concept of communicative and critical HL in addition to the functional HL.²⁵ Therefore, more research on interrogating the dimensionality of this kind of information-handling health literacy is warranted in the future.

In the IRT model, some items (item 5, 8, 9, 11, 14) show less or more variation than modeled. Among these items, only item 9 failed to meet both the infit/outfit and T value criteria. We found that many participants usually had trouble in giving the answer when they were confronting with the item 9 in the investigation. Some participants insisted that they would only talk to their attending physician, while others insisted that they would talk to their family members instead of healthcare professionals. Moreover, many participants wonder that whether they should talk about their illness or not. Hence, we believe that highly guessing risk could be the main reason causing the underfit of item 9 in the MRCMLM. Therefore, Wu emailed the author of the HL-14, Machi Suka, about this result and inquired about her suggestion and permission about rephrasing the item 9 of the HL-14 into the following sentence in Chinese edition: “I tell my opinion about my illness to my doctor or other healthcare professionals”. As we believe it is a crucial trait of health literacy for patients to communicate and acquire healthcare-related information from healthcare professionals, in that a patient may not always get decent information from nonprofessional healthcare workers, and Machi Suka agreed with the modification of item 9 expression in the HL-14 Chinese edition. As for the other items showing the misfit, we think logical correlations exist among the HL-14 items when participants’ functional, communicational and critical HL are evaluated (conceptually overlapping items), which may cause the Guttman-style response or independent measurement errors in the model test. Thus, covariances were drawn between the items (item 10↔14; item 12 ↔13) according to the modification indices and item content. The modified model showed a good fit to the study population (x²/df=2.698, CFI = 0.965, GFI = 0.917, TLI = 0.956, RMSEA = 0.076, SRMR = 0.042), which is even better than the original model reported by Machi Suka (CFI = 0.912, NFI = 0.905, RMSEA = 0.082).²³

Although China had made great progress in improving health literacy in the last decades, geographic disparities were still evident, with the East outperformed the Central and the West, and cities better than rural areas.³⁸ The Chinese Health Literacy Scale is the official instrument to measure Health Literacy in China, but the use of this 80-item scale is time consuming and not suitable for rapid screening especially in clinical practice.³⁹ The HL-14 validated in this study is a unique tool that aims to evaluate functional, communicative and critical HL which provide the clinical a new option appropriate for rapid assessment with adequate reliability and validity. Moreover, together with the medication management tools like ARMS and the Self-Efficacy for Appropriate Medication Use scale (SEAMS),^27,40 the use of the HL-14 can provide the clinical research more evidence in analyzing the sociodemographic determinants in patients’ self-management abilities.⁴¹ The correlation analyses of this study indicated that patients with high health literacy seem to be associated with milder comorbidity status and less medications, meanwhile they possessed better educational level, monthly income and medication adherence. Although a recent meta-analysis indicated the heterogeneous role of health literacy in self-care and glycemic control,⁶ the paucity evidence on multi-dimensional HL in the study give rise to the need of multi-dimensional measurement like the HL-14 in clinical research.

However, limitations should also be noted in this study. Due to the worldwide pandemic of Covid-19, this cross-sectional study was conducted in the endocrinology department of a single tertiary hospital in Shanghai, China. The study findings could be more generalizable if we adopt varied population of patients in more hospitals and districts of China in the near future. Besides the methodology used in this study, more IRT models in validating the scale could be explored.

In conclusion, the HL-14 Chinese edition validated in this study showed adequate reliability and validity. The difficulty setting of this measurement is appropriate for most of patients living with diabetes. Unlike most measurements that focus on functional HL, the HL-14 can be utilized for evaluating more information related HL dimensions (ie, communicational and critical HL) in healthcare practices. The use of the Chinese HL-14 may provide an entirely new view and understanding of the health literacy research in clinical practices in China.