Refinement and Evaluation of a Chinese and Western Medication Adherence Scale for Patients with Chronic Kidney Disease: Item Response Theory Analyses

Introduction

Chronic kidney disease (CKD) is estimated to affect around 8–16% of adults, worldwide¹ and 10.8% of Chinese adults.² As it chronically develops, most CKD sufferers need to maintain long-term, regular medication routines. Medication adherence, therefore importantly, refers to the extent to which a patient’s medication behavior conforms to the agreed recommendations from a healthcare provider.³ Evidence to date has documented that estimates of non-adherence to medication among CKD patients varies from 17–80%,^4,5 and poor medication adherence is associated with the reduced expected medication effects or even increased incident adverse events.⁶

Herbal medication of traditional Chinese medicine (TCM) has been increasingly added into regular CKD regimen as accumulating evidence these days has supported its potential benefits on the reduced ESRD and death risk of CKD patients.⁷ Currently, the generally-accepted scale to evaluate medication adherence is the Morisky, Green and Levine scale (MGL scale);⁸ its Chinese version is also widely considered as a preliminary tool to assess the adherence.^9,10 However, MGL scale may not be optimal to evaluate TCM medication adherence, as TCM administration involves aspects including varied preparation forms (eg, prepared oral solution, herbal decoction) and complicated preparation techniques (eg, to steep, decoct first, decoct later).¹¹ Additionally, the cultivation of medication adherence involves a forwarding process of acquiring knowledge, generating beliefs, and shaping behaviors.¹² Nevertheless, few reliable methods have been particularly developed to assess the TCM medication adherence of CKD patients. Therefore, a CKD medication adherence scale designed for both Chinese and Western medication with a conceptual framework of knowledge, beliefs, and behavior domains is clinically needed. To address this gap, we have previously used classical test theory (CTT) to develop a Chinese and Western medication adherence scale for CKD (version 1) patients, which was tested to have desirable reliability and validity.¹³

Item response theory (IRT) is a robust approach to refine and assess the rating instruments. Different from CTT, IRT models the relationship between a person’s response to an item and his/her level on the measured latent construct, and provides detailed information of each item and the scale as a whole.¹⁴ While CTT assesses the scale using validity and reliability from a more integrative perspective, IRT focuses on the discrimination and difficulty of each individual item. To shorten the scale while preserving as much information as possible, this study therefore aimed to use IRT analyses to further refine the aforementioned CTT-based version-1 Chinese and Western medication adherence scale into a version-2 scale, and also applied CTT as well as IRT analyses to evaluate the performance of the version-2 scale.

Materials and Methods

Part 1. Scale Refinement

Full details describing the development and assessment of the version-1 scale were provided elsewhere.¹³ In brief, a literature review and Delphi study were performed to develop this scale, and it was based upon the knowledge-attitude-belief practice (KABP) theory, it was designed with construct components of medication knowledge, belief and behavior, after the requisite. The 26-item version-1 scale administered in Chinese language was distributed to 222 CKD patients during early 2018 and detected with desirable reliability and validity using CTT analysis. This Likert-5 scale contained both positively worded (PW) and reverse worded (RW) items (items 17–19), with ordinal score of 5–1 or 1–5 points as sequential options of A-E.

In the current study, we re-examined the data of these 222 CKD patients using Samejima graded response model (GRM) of IRT to perform scale refinement from the original version-1 to a simplified version-2. Parameters of 26 items in the version-1 scale, including discrimination parameter (a), difficulty parameter (b), maximum information function peak (Imax), item information curve (IIC) and item characteristic curve (ICC), were estimated using marginal maximum likelihood (MML) analyses of GRM conducted by Multilog 7.03. Discrimination parameter (a) represents the ability of the item to discriminate patients with different levels of latent trait and therefore indicates the extent to which the item is related to the underlying construct. Items with a value of ≤0.3 or ≥3 were considered to be deleted in our study.¹⁵ Difficulty parameter (b) is the value at which the probability of a positive response for a dichotomous item is 50% and reflects the measured construct a respondent must have to endorse a specific item.¹⁶ Items with a b value of ≤-3 or ≥3 were considered to be deleted in our study.¹⁵ ICC is a trace function that models the relationship among the measured construct, item characteristics, and recorded response. IIC is a trace line that reflects the association between information provided by each item and the measured construct of different levels, the peak of which is Imax. Items with flat IIC were considered to be deleted in our study.¹⁵

The scale was refined from the version-1 scale to the new version-2 scale based upon both the statistical results of IRT analyses and the clinical considerations of clinical researchers (QH, LL, BQX, DJZ, CDD, JWT, LZF, FT, XLZ, BNL, YMX, HFC, XSL, and YFW).

Part 2. Scale Evaluation

Another clinical investigation of CKD patients regarding the version-2 scale took place during late 2018. Inpatients or outpatients with regular visits to Guangdong Provincial Hospital of Chinese Medicine were considered eligible if they were 1) diagnosed non-dialysis stage 1-5 CKD patients according to 2012 KDIGO guideline;¹⁷ 2) physically and mentally able to be surveyed independently and signed informed consent; 3) 18–80 years old. Patients were excluded if they were 1) suffering with severe cardiovascular, cerebrovascular, liver, gall bladder, spleen, pancreatic or hematologic comorbidities; 2) suffering with serious mental disorders; 3) having acute complications; 4) a pregnant or lactating female. A sample size requirement of 200 was estimated using methods described by King et al for approximately 5:1 patient-item ratio.¹⁸

Individual members of the research team received planned training on questionnaire survey. Eligible patients were not enrolled to participate unless they were clear about the study content and objectives clarified by the trained research staff as well as they had signed the informed consent. This self-administered questionnaire was only assigned to patients able to complete the questionnaire independently on-site, and checked for the possible omitted or incorrect fill-in by research staff at the survey completion to guarantee survey quality. A test-retest for external reliability was undertaken among at least 50 randomly selected CKD patients within 7–14 days post to the first survey.¹⁹ Two independent research staff entered the data into Excel independently to ensure double-checked data entry.

Items of the version-2 scale were evaluated by both CTT and IRT analyses, and compared with those of the version-1 scale. Firstly, CTT analyses of reliability and validity were performed using SPSS version 20.0. Reliability was examined using internal consistency reliability and test-retest reliability. Validity including content validity, construct validity were tested with exploratory factor analysis (EFA), and criterion-related validity were analyzed. Secondly, Samejima GRM of IRT analyses regarding the parameter (a), parameter (b), Imax, IIC and ICC were performed by Multilog version 7.03.

The present study was approved by the Ethics Committee of Guangdong Provincial Hospital of Chinese Medicine (approval notice ref. B2018-112-01).

Results

Part 1. Scale Refinement

We analyzed the 26-item version-1 scale with the data of 222 CKD patients enrolled in early 2018 using Samejima GRM of IRT analyses (Figure 1, Table 1). Items with a detected a value of ≤0.3 or ≥3 and b value of ≤-3 or ≥3 were deleted (items 22 and 23). Items with detection of either a value of ≤0.3 or ≥3 or b value of ≤-3 or ≥3, were reserved or deleted depending on further clinical considerations (items 2, 3, 5, 7, 9, 10, 11, 17, 18, 20, 21, 24, 25, 26). Of these, items 7, 9, 10, and 11 with a ≥3 were deleted as redundant items pertaining to medication knowledge of herbal decoction. The other 10 items including items 2, 3, 5, 17, 18, 20, 21, 24, 25, and 26 were examined with b ≤-3 or ≥3. Among these 10 items, items 2 and 3 were deleted given that these items were excessively difficult or easy to complete respectively. Items 5, 17, and 21 were deleted due to their obscurity. Item 18, about patient acceptance of Chinese herbal medicine, item 20, pertaining to medication belief, and the cluster of items 24, 25, and 26, about medication administration, were retained given that these items were crucial to the content and construct of the scale as a whole. In summary, 11 items including items 2, 3, 5, 7, 9, 10, 11, 17, 21, 22, and 23 were deleted and the remaining 15 items composed the simplified version-2 scale (Supplementary 1 and 2). The IIC and ICC of the version-1 scale are presented in Supplementary 3.

Table 1 Discrimination Parameter (a), Difficulty Parameter (b) and Maximum Information Function Peak (Imax) of 26 Items in Version-1 Questionnaire

Figure 1 Scree plot of the version-2 scale.

Part 2. Scale Evaluation

Of 280 assigned, 267 copies of (response rate 95.4%) the version-2 scale were returned during late 2018 in our study. Of these, 63 respondents were selected to take test-retest within 7–14 days after the first study visit. The 25^th, 50^th, and 75^th percentiles of the total score of version 2 scale were 48, 53, and 60 points respectively, correspondingly indicating levels of poor, moderate and high medication adherence.

CTT Analyses of Version-2 Scale

1. Internal consistency reliability

Cronbach α of the total 15-item version-2 scale was 0.842, and that of 3 individual components varied from 0.639–0.877, indicating desirable internal consistency overall given that most Cronbach α was beyond 0.7 (Table 2).

2. Test-retest reliability

Table 2 Internal Consistency Reliability of Version-2 Scale

Pearson coefficient r of total score was 0.909, and that of 3 individual components ranged from 0.796–0.905, indicating desirable external reliability overall given that most coefficient was beyond 0.7 (Table 3).

Table 3 Test-Retest Reliability of Version-2 Scale

1. Content validity

The results of the previous 3-round Delphi study on the version-1 scale by 23 experts showed that what the included items conveyed was consistent with our conceived construct of knowledge, belief and behavior. When refining the version-1 scale to the version-2 scale, we still retained specific items key to the aforementioned framework as a whole. Therefore, we may safely conclude that the content validity of the version-2 scale should be similarly meritorious.

2. Construct validity

The detected X² of 1471.567 with P<0.0001 in Bartlett’s test and KMO measure of 0.841 allowed EFA to be performed. After that both varimax rotation (VR) and direct oblimin rotation were explored and we prioritized the use of VR as the number and content of factors produced by these two rotation approaches were very similar (Supplementary 5).^20,21 The principal component analysis (PCA) and VR extracted 4 factors with eigenvalue > 1 and 63.560% total variance explained (Table 4). Similar results were shown in the Scree plot as its turning point revealed in the fourth factor (Figure 1).

Table 4 Total Variance Explained of Version-2 Scale

The component matrix after VR of 15 items in the version-2 scale displayed factor 1 of items 1–5, and 8, factor 2 of items 6, 7, and 9, factor 3 of items 13, 14, and 15, as well as factor 4 of items 10, 11, and 12 (Table 5). Based upon the conceived scale framework, we interpreted factor 1 as basic medication knowledge, factor 2 as TCM medication knowledge, factor 3 as medication behavior and factor 4 as medication belief. Integrating clinical considerations, we combined factor 1 and factor 2 into 1 component as medication knowledge. A scale construct with 3 total components was therefore obtained, including component 1 called medication knowledge (items 1–9), component 2 called medication belief (items 10–12), and component 3 called medication behavior (items 13–15). Given that factors analyzed in EFA were consistent with our previously conceived scale framework of knowledge, belief and behavior, we may conclude that the construct validity of version-2 scale should be desirable.

3. Criterion-related validity

Table 5 Rotated Component Matrix of Version-2 Scale

MGL scale was also assigned to these 267 respondents. The Pearson correlation r for total score of the version-2 scale and MGL scale was calculated as 0.349 (P<0.01), indicating their weak correlation.

IRT Analyses of Version-2 Scale

We also conducted IRT analyses on the retained 15 items using data of the included 267 patients in Part 2. The results of IRT analyses on 15 items in the version-2 scale were compared with those of the version-1 scale. The findings showed, within a and b reference range, slightly decreased difficulty of items 1, 3, and 13 (b<-3), evidently improved discrimination and reduced difficulty of items 10,12,14, and 15, as well as moderately enhanced discrimination and decreased difficulty of items 2,4,5,6,7,9, and 11 in the version-2 scale than in the version-1 scale. Nevertheless, discrimination and difficulty of item 8 was significantly elevated beyond the reference range. Decreased Imax of items 1,2,3,4,6,11, and 14 while evidently increased Imax of items 5, 7, 8, 9, 10, 12, 13, 14, and 15 was detected in the version-2 scale compared to the version-1 scale (Table 6). Details of IIC and ICC are revealed in Supplementary 4.

Table 6 Comparison of Discrimination Parameter (a), Difficulty Parameter (b) and Maximum Information Function Peak (Imax) Between Version-1 and Version-2 Scale

Discussion

KABP is a mode to change health-related behaviors of humans, and involves a progressive process of obtaining knowledge, generating beliefs, as well as shaping behaviors.^12,22 KABP is widely used in the healthcare field. In KABP context, healthcare providers or patients are gradually educated with health knowledge, are helped to build health beliefs and finally cultivate health behaviors. Jose et al²³ conducted a survey regarding antibiotic use in the Omani population using a self-administered questionnaire based upon the knowledge-belief-behavior framework. This indicates the practicability to use KABP as a conceptual framework to develop a scale to evaluate relevant healthcare procedures. Based upon the KABP framework, we therefore developed this scale comprising medication knowledge, medication belief, and medication behavior. To our knowledge, our study may fill the gap that few scales have been developed to assess the medication adherence of CKD patients based upon the KABP framework.

This study refined the 26-item version-1 scale to the 15-item version-2 scale using IRT analyses, and evaluated 15 items of the version-2 scale using both CTT and IRT analyses. When compared with the version-1 scale, results of CTT analyses on the version-2 scale demonstrated its relatively good internal reliability, external reliability, content validity, and construct validity. Findings of the criterion-related validity showed weak correlation between the total score of MGL scale and our scale. This can be attributed to the different content of these 2 scales. While MGL scale focuses on occasions of forgetting, careless at, or stop taking medicine, the content regarding TCM medication preparation and administration are not mentioned in it. The version-2 scale contains items about the preparation and administration of TCM medication, medication beliefs, and medication behaviors. Therefore, the differences in the number and content of the included items, as well as the conceptual framework of these 2 scales may lead to the aforementioned weak correlation indicated by the criterion-related validity.

When compared with the version-1 scale, results of IRT analyses on the version-2 scale documented its enhanced discrimination and reduced difficulty overall. Findings of IRT analyses also showed similar maximum information provided by the version-2 scale when compared with that of the original version-1 scale. Furthermore, the number of items (15 items) and the length of time to complete the version-2 scale (4.6±2.1 mins) were both reduced compared to the version-1 scale (26 items, 7.3±1.2 mins). This may indicate the feasibility to use the version-2 scale in clinical practice to assess Chinese and Western medication adherence of CKD patients.

Methods to assess the performance of scaling instruments generally contain CTT, generalizability theory (GT), and IRT.²⁴ CTT is built on a relatively simple statistical model, and widely used in many sorts of scaling development. However, the results of CTT largely depend on the latent ability of the included respondents, and different levels of the tested ability may have evident impacts on the calculated psychometric property of the scale overall.²⁵ GT is suitable to develop a scale for subjective assessment, such as an interview or examination, due to its strength to control measurement error. Nevertheless, GT is based upon a random sampling model and may introduce bias due to the selected heterogeneous samples.²⁶ IRT assumes that the latent trait presumably causes the item response variance, and it is mostly used for quality control in scale development.^27,28 In this study, making the best of both CTT and IRT, we used IRT analyses to further refine and evaluate the performance of a previously CTT-based CKD Chinese and Western medication adherence scale. The refined version-2 scale was detected with proper reliability, validity, discrimination, difficulty and information provided.

This study had several limitations. Firstly, though, it is relatively good when compared with that of the version-1 scale, the internal consistency of belief and behavior factors of the version-2 scale is middling at a level around 0.7, indicating that the scale needs to be substantially improved in the future. Secondly, all participants enrolled in our study were CKD inpatients or outpatients of Guangdong Provincial Hospital of Chinese Medicine, a public tertiary metropolitan hospital. Most participants were predominantly urban dwellers. Future studies should be, therefore, required to apply this scale to different patients in different hospitals of broader levels for generalizability. In addition, at this initial phase, we only recruited non-dialysis CKD patients as well as not evaluating the correlation between levels of medication adherence and clinical outcomes in this study. Therefore, future studies are still required for further relevant exploration.

Conclusion

We refined the 26-item Chinese and Western medication adherence scale (version-1 scale) to a 15-item version-2 scale using IRT analyses. The following scale evaluation using CTT and IRT analyses reported that the version-2 scale had the desirable reliability, validity, discrimination, difficulty and information provided, indicating that it should be a practicable tool to assess CKD medication adherence. Nevertheless, future studies are required to further improve the internal consistency of belief and behavior factors in the version-2 scale, and it needs to be conducted on different patients in hospitals of broader levels for generalizability. Additionally, further explorations regarding the association between levels of medication adherence and clinical outcomes are needed.