A Longitudinal Study on Psychometric Properties of the Simplified Chinese Version of FACT-Hep for Patients with Hepatocellular Carcinoma by Classical Test Theory and Generalizability Theory

Introduction

Hepatocellular carcinoma (HCC)/liver cancer is a severe disease with a marked heterogeneous geographical distribution in the world. As the fourth most common malignancy in the world, it causes 782,000 deaths annually and accounts for 72% of the total HCC incidence in East, Southeast and North Africa.¹ According to the International Agency for Research on Cancer (IARC), an estimated 905,700 people will be diagnosed with liver cancer and 830,200 people will die from liver cancer worldwide in 2020. More than half of the world’s estimated liver cancer cases and deaths occur in East Asia (54.3% and 54.1%, respectively), with China accounting for 45.3% of the world’s liver cancer cases and 47.1% of liver cancer deaths.² China has a very high prevalence of HCC, causing a severe disease burden, the mortality rate of primary liver cancer in China in 2015 was 325,000 cases, with a mortality rate of 23.72/100,000, accounting for 13.95% of all malignant tumors.^3,4 Also HCC incidence in different countries are increasing sharply, for example Age-adjusted HCC incidence rate in Australia increased from 1.38 per 100000 (95% CI: 1.34–1.43) in 1982 to 4.96 per 100000 (95% CI: 4.89–5.03, P < 0.001) in 2014 with an average annual percentage increase of 4.46% (95% CI: 4.24%–4.69%).⁵

Besides, the survival rate remains poor over the past decades, with overall one year survival being less than 20%.^6,7 Therefore, researchers and clinicians tend to pay more attention to health-related quality of life (HRQOL) of patients with HCC, given the time course of the disease, the difficulty in curing and the burden of treatments. According to WHO, HRQOL is defined as individual’s perceptions of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards and concerns. In the recent 10 years, the assessments of HRQOL have been applied as significant outcome indicators for patients with HCC.^8–10 Consequently, several specific instruments for patients with HCC have been developed and are used in cancer clinical researches, including European Organization for Research and Treatment of Cancer (EORTC) Quality of Life Group questionnaire, the QLQ-HCC18,^11,12 the Functional Assessment of Cancer Therapy (FACT) Hepatobiliary (FACT-Hep) questionnaire,^13,14 the National Comprehensive Cancer Network Functional Assessment of Cancer Therapy (NCCN-FACT) Hepatobiliary-Pancreatic Symptom Index (NFHSI),¹⁵ the Nine-Item Chinese Patient Satisfaction Questionnaire (ChPSQ-9),¹⁶ and the Quality of Life Instrument for Patients with Liver Cancer QOL-LC.^17,18 Of them, the QLQ-HCC18, ChPSQ-9 and QOL-LC were HCC specific instruments, while the FACT-Hep and NFHSI were specific with hepatobiliary cancer. The FACT-Hep is a 45-item self-report instrument developed specifically to measure HRQOL in patients with hepatobiliary cancer, not only HCC but also pancreatic, biliary and metastatic liver cancer.^13,14

The Chinese is the largest population in the world and the HRQOL instruments for the Chinese HCC is seldom, only QOL-LC^17,18 and QLICP-LI (Quality of Life Instruments for Cancer Patients -Liver Cancer).¹⁹ Due to their late development and the fact that QOL-LC was not developed using a modular approach, their applications are not very widespread. Considering that the original English FACT-Hep (V4.0) has clear structure, good psychometric properties, and wide application,^13,14 developing a Chinese version suitable for use in China is both a shortcut and highly needed. Therefore, the Simplified Chinese version of FACT-Hep (V4.0) for HCC was developed by the Center on Outcomes, Research and Education (CORE) to meet this need. However, no formal validation of this scale has been performed for use in China. The purpose of this paper is to evaluate the psychometric properties of the Chinese FACT-Hep used in mainland China using the QLICP-LI¹⁹ as criterion-related validity tool for both of them has developed by modular approach and has similar construct.

Materials and Methods

Instruments and Scorings

Like the original one and similar to the simplified Chinese version of FACT-B for breast cancer,²⁰ the simplified Chinese version of FACT-Hep (V4.0) consists of the general module (FACT-G) for all cancers and the specific module (additional concerns, HCS) for HCC. The FACT-G includes four domains of physical well-being (7 items), social/family well-being (7 items), emotional well-being (6 items) and functional well-being (7 items). While the HCS, which is an 18-item disease specific hepatobiliary cancer subscale, assesses back and stomach pain, gastrointestinal symptoms, anorexia, weight loss and jaundice in patients with hepatobiliary cancers.

Each item in FACT-Hep is rated in a five-point Likert-type scale, with the positively stated items directly obtaining scores from 0~4 and the negatively stated items being reversed scores. The domain/overall score is obtained by adding its own item-score together.^13,14,20 Thus, from the FACT-Hep scale, one can derive five subscales and an overall score (See Table S1 in detail), with higher scores reflecting better QOL.

The QLICP-LI (V2.0), which was one of the Chinese HRQOL system QLICPs (Quality of Life Instruments for Cancer patients) for liver cancer, was used as criterion-related validity tool. The QLICPs was developed by module approach with a general module (QLICP-GM) being used with all types of cancer and some specific modules for different cancers.^21–24 For example, the QLICP-CR is formed by combining the QLICP-GM and the specific module of the colorectal cancer,²² also the QLICP-NA and QLICP-BR were formed by QLICP-GM and the specific modules of the nasopharyngeal cancer and breast cancer respectively.^23,24

The structure and scoring method of QLICP-LI (V2.0) is very similar to FACT-Hep, which consists of a general module (QLICP-GM) and a 12 items specific domain of liver cancer (SLI1-SLI12).¹⁹ The QLICP-GM includes four domains of physical (8 items GPH1-GPH8), psychological (9 items GPS1-GPS9), social (8 items GSO1-GSO8) and common symptoms and side-effects of cancer patients (7 items GSS1-GSS7). The five domains scores and the overall score can be computed by adding all relevant item scores (see Table S2 in detail). All domains/overall scores were linearly converted to a 0–100 scale using the formula: SS=(RS-Min) ×100/R, with higher scores reflecting better HRQOL. The SS, RS, Min and R represent the standardized score, raw score, minimum score, and range of scores, respectively.

Data Collection

This is a longitudinal study on psychometric Properties of the Simplified Chinese Version of FACT-Hep with inpatients investigated three times before and after treatments. The study recruited inpatients diagnosed with HCC at the Yunnan Tumor Hospital (the third affiliated hospital of Kunming Medical University) and the first affiliated hospital of Guangdong Medical University during 2012 to 2013, with the study protocol and the informed consent form being approved by the IRB (institutional review board) of the first affiliated hospital of Guangdong Medical University (Approval No. PJ2012052). All methods were performed in accordance with the relevant guidelines and regulations. A complete assurance was given that all information would be kept confidential. The right was given to the patients not to participate and to discontinue participation in the study with consideration/without penalty.

The inclusion criterion is that the patients should be able to read and understand the questionnaires considering the scales were about patient’s subjective evaluation (HRQOL). Thus, the study population was limited to HCC inpatients at any stages and treatments who can read and understand the questionnaires.

The exclusion criteria are (1) patients with critical illness, combination of other serious diseases, serious mental diseases, etc.; (2) patients with cognitive dysfunction; (3) illiterate patients; (4) patients who refused to participate in the study or were less cooperative.

First, the investigators (Doctor Wang and medical students for Master degree) explained the study and scales to the patients, and then obtained written informed consent from those patients who agreed to participate in the study and met the inclusion criteria. And then, participants were asked to fill out questionnaires at the time of admission to the hospital. In order to evaluate test-retest reliability of the scale, the patients with stable disease (those with no change in their condition) were asked to complete the scales for a second time 1–2 days after hospitalization for treatments effects have not been happened. In order to evaluate responsiveness of the scale, all patients available at the third scheduled assessment time-point were asked to complete the measures at discharge after approximately 4 weeks of treatments. The investigators checked for absent responses each time after receiving the questionnaire and wherever possible asked the patients to respond to the missing items.

Meanwhile, the demographic data (such as gender, age, education level and marital status) and medical data were collected by the face page of our questionnaire and hospital medical records.

Statistical Methods

The psychometric properties of validity, reliability and responsiveness of the simplified Chinese version of the FACT-Hep were analyzed. The multi-trait scaling analyses²⁵ was used to evaluate the construct validity of FACT-Hep by test for item convergent and discriminant validity based on the examination of item-domain correlations. The Pearson correlations of an item with its own domain and other domains were calculated with the two criteria: (1) convergent validity is supported when an item-domain correlation is 0.40 or greater; (2) disciminant validity is revealed when item-domain correlation is higher than that with other domains.

The criterion-related validity was evaluated by correlations between domains of the FACT-Hep and QLICP-LI because of lack of gold standard. Relatively high correlations among conceptually related domains and relatively low correlation among conceptually distinct domains would suggest high criterion-related validity. Also these correlation coefficients with QLICP-LI can reveal convergent and discriminant validity to some extent.

The internal consistency of domains and overalls was estimated by Cronbach’s alpha coefficient calculated using data from the first measurements because of larger sample size, with an alpha coefficient greater than 0.7 being considered acceptable reliability.²⁶ The test-retest reliability of the domains was assessed by the Pearson’s correlation coefficient and intra-class correlation (ICC) with definition of absolute for single measure under two-way mixed model²⁷ between the first and second assessments.

In addition, considering that CTT cannot effectively decompose the factors influencing reliability, nor can it perform expandable decision analysis, the Generalizability Theory (G theory) was also applied to investigate the score dependability of the FACT-Hep, which allows for the simultaneous estimation of multiple sources of variance, including interactions.^28–30 Given the HRQOL of the patients as the target of measurement and the items as one facet of measurement error, the generalizability (G) and decision (D) studies in one-facet crossed design [person-by-item (p × i) design] in measurement mode were conducted in order to estimate the variance components and dependability coefficients. All different variance components were calculated in G study, while in D study, a universe of admissible generalizations was defined to represent the measurement conditions based on the object of measurement and on the measurement facets a researcher is willing to generalize. Simultaneously, the variance components of generalizability coefficients and dependability indexes in each facet, as well as their interactions were calculated. Combining Classical Test Theory (CTT) with Generalizability Theory (GT) to evaluate reliability not only provides more evidence, such as test-retest reliability and internal consistency reliability in CTT, as well as generalizability coefficients and dependability indexes in GT; but also enables expanded decision-making, such as examining the impact of varying numbers of items on reliability.

The responsiveness was evaluated through comparing the mean difference between the pre-treatment and post-treatment by paired t-tests with standardized response mean (SRM) and effect size (ES) being calculated.^31,32 The SRM is the difference of the score between pre-treatment and post-treatment divided by its standard deviation and the ES is the difference divided by baseline standard deviation.

Results

Socio-Demographic and Clinical Characteristics of Participants

The total sample included 114 patients with HCC varied in age from 31 to 73 with a median age of 50.0 and mean age 51.2 ±10.0 (See Table S3 in detail). Most patients were male, married, Han ethnics, and had hepatitis history. Specifically, 92 cases (80.7%) were male while 22 cases (19.3%) were female; the married were 110 cases (96.5%), 84 cases (73.7%) had hepatitis history, and ethnics of Han was 91 cases (79.8%) while others was 23 (20.2%). Thirty-seven (32.5%) patients finished primary school, while 64 (56.1%) completed high school of professional secondary school, and 13 (11.4%) had a college degree. The distributions of occupations were worker 14 cases (12.3), farmer 41 (36.0%) and others 59 (51.7%). Regard to treatments, surgery was 16 cases (14.0%), minimally invasive treatments were 69 cases (60.5%) and others 29 cases (25.5%). Minimally invasive treatments include radiofrequency ablation (RFA), microwave ablation, transarterial chemoembolization (TACE), transarterial radioembolization (TARE), and high-intensity focused ultrasound (HIFU). Other treatments include targeted drug therapy, supportive therapy (nutritional support, symptomatic treatment including antipyretics, analgesics, and regulation of water and electrolyte balance, etc)., and adjuvant therapy after liver resection (chemotherapy and radiotherapy).

Construct Validity

The correlation analysis among items and domains within the FACT-Hep were shown in Table 1. As can be seen, strong correlations between items and their respective domains with all correlation coefficients r were higher than 0.40 and most of them higher than 0.60 (exception of a few items of HCS such as Hep1, Hep2, Hep4 with HCS domain), suggesting item convergent validity. On the other hand there were weak correlations between items and non-relevant domains, implying discriminant validity. For example, the coefficients between domain of FWB and items within this domain (GF1-GF7) were higher than 0.70, higher than the correlation between the domain and any other items in other domains.

Table 1 Correlation Coefficients Among Items and Domains of FACT-Hep (n=114)

Criteria-Related Validity

The Pearson correlation coefficients of scores among domains of FACT-Hep and QLICP-LI were presented in Table 2. It can be seen that the between-instrument correlations were higher for similar domains than across different and non-similar domains. For example, the correlation between EWB of FACT-Hep and PSD of QLICP-LI was 0.53, higher than other any correlations in this row. The correlations of HCS of FACT-Hep with SSD and SPD of QLICP-LI were 0.74 and 0.75 respectively for these domains are associated with symptom/side-effects. The correlation between total scores of FACT-Hep and QLICP-LI was 0.86.

Table 2 Correlation Coefficients Among Domains of FACT-Hep and QLICP-LI (n=114)

Reliability from Classical Test Theory

During the second assessment (two-day follow-up), 63 inpatients completed the questionnaires that would be used for test-retest reliability analysis for the paired t-tests indicated no statistically significant change of domain scores between the first and the second assessments (p>0.05). The Cronbach’s α and test-retest reliability coefficients (correlation coefficients r and ICC) of all domains are presented in Table 3. According to Hays,³³ a test-retest reliability index of 0.7 or higher reliability is better. It can be seen that Cronbach’s α coefficients of all domains were greater than 0.70. The test-retest reliability coefficients both correlation coefficients r and ICC for all domains were larger than 0.80 exception of EWB (0.74).

Table 3 Reliability of the Chinese Version of FACT-Hep (n=114 for α, n=63 for r and ICC)

Reliability from Generalizability Theory

Table 4 showed the estimated G study results based on the p × i design, in which 114 patients filled out the 45 items of FACT-Hep. For most domains, the largest source of variance was due to item, such as 90.25% in PWB, 91.11% in SWB, 97.66% in EWB, 85.78% in HEPCS. D studies were performed to estimate the G coefficients and Φ coefficients for the p × i design, as well as the alternative design with different numbers of items in the five domains (see Table 5 in detail). D study is equivalent to the evaluation of measurement reliability and validity functions in traditional measurement theory, except that GT uses generalization coefficients as validity indicators and reliability indices as reliability indicators. Generalization theory ultimately evaluates the reliability of measurement by two coefficients G (generalization coefficient) and Ф (reliability index), and the larger the two coefficients (usually at least 0.7), the better the results. Generally, the G coefficients and Φ coefficients increased as the number of items in each domain increased. Under the current designs, the G and Φ coefficients were higher or close to 0.70 in four out of five domains, except for EWB. In addition, Table 5 showed the effects of the various levels of items (from 6 to 22) on reliability with G ranging from 0.517 to 0.888, and Ф ranging from 0.335 to 0.883.

Table 4 The Estimated Variance Components and Percentage of Variance Accounted for by Effects (Percent) for P×i Design in G-Study for Four Domains of Quality of Life Instrument FACT-Hep (=113)

Table 5 G-Coefficients and Ф-Coefficients for Different Numbers of Items for p ×I Design in D-Study for Four Domains of Quality of Life Instrument FACT-Hep (V4.0) =113

Responsiveness

At time 3 assessment, 68 patients completed the questionnaires and the data were used for evaluating the responsiveness (see Table 6 in detail). As can be seen, the domain scores of PWB, FWB had statistically significant changes after treatments with the SRM being 0.69 and 0.40, and ES being 0.81 and 0.35. Also the score changes of the general module, overall scale and Trial Outcome Index were of statistically significance with SRM being 0.56, 0.46 and 0.40, and ES being 0.50, 0.47 and 0.30, respectively.

Table 6 Responsiveness of FACT-Hep for Patients with Cancer (n=68)

Discussions

In liver cancer assessments, there are many measures of HRQOL. Jayabalan et al³⁴ reported twenty-nine studies with 10,472 patients using eight HRQOL instruments including QLQ-HCC18, FACT-Hep, the Functional Assessment of Cancer Therapy–Hepatobiliary Subscale (FACT-HS), and the Functional Assessment of Cancer Therapy Hepatobiliary Cancer Symptom Index (FHSI-8) etc. Zhang et al³⁵ systematically reviewed nine studies with patients of hepatocellular carcinoma using FACT‑G, SF‑12, WHOQOL‑BREF, QoL‑LC V2.0, FACIT‑Sp12 etc. Obviously, two very well-known scales are the EORTC’s QLQ-HCC18 (which needs to be used together with QLQ-C30) and the FACT-Hep from the United States. Both of them already have Chinese versions. Our team has provided a detailed introduction to the development and evaluation of the Chinese version of QLQ-HCC18, using CTT methods, which include translation, back-translation, and psychometric property evaluation.³⁶ In contrast, this article mainly focuses on the psychometric properties of the American FACT-Hep scale, using both CTT and Generalizability Theory (GT) methods.

The Functional Assessment of Cancer Therapy (FACT) Quality of Life Measurement System was developed specifically for cancer patients in the English language using a standardized approach to item derivation, reduction and testing. It is a collection of HRQOL questionnaires used in cancer with currently containing 12 subscales specific to different types of cancer (breast, bladder, brain, central nervous system, cervical, colon, esophageal, head and neck, hepatobiliary, lung, ovarian, prostate).³⁷ The FACTs is widely used in international trials with a large database being available to facilitate comparisons across studies, and with selected scales available in over 40 languages.^37–39 The advantage of translating a validated instrument into a local language, instead of developing a new one, is primarily the relatively short amount of time and convenient for cross-cultural HRQOL comparisons. Therefore, in this study, we tested the reliability, validity and responsiveness of the simplified Chinese version of FACT-Hep in order to getting psychometrics used in Mainland China.

On Validity of the FACT-Hep

Indicating how an instrument can capture what it intends to measure, the validity is the most important property. The results presented here showed that the Chinese version of the FACT-Hep had good convergent and divergent validity was as good as the original scale in a group of patients with hepatobiliary cancers.¹³ Our findings confirmed the construct validity and criterion-related validity of the FACT-Hep. Almost all item-domain correlation coefficients met the standards of item convergent validity and discriminant validity. Overall the correlations between similar domains of FACT-Hep and QLICP-LI were higher than those between different and non-similar domains. These correlations supported the criterion-related validity and also demonstrated the domains’ convergent validity and divergent validity to some extent.

On Reliability of the FACT-Hep

The FACT-Hep has been used as a reliable assessment measure for patients with HCC, and has demonstrated very high internal consistency (Cronbach’s α 0.94), excellent test–retest reliability (Spearman correlation 0.91) for the overall scale.¹³ All subscales and aggregated scores showed high internal consistency at initial assessment (Cronbach’s alpha range, 0.72 to 0.94). Measurement stability was also high for all aggregated scales (test-retest correlation range, 0.84 to 0.91; intraclass correlation coefficient range, 0.82 to 0.90).¹³

The results presented here showed that the Chinese version of the FACT-Hep had good reliability as good as the original scale.¹³ The Cronbach’s α coefficient of the Chinese version of the FACT-Hep ranged from 0.72 to 0.90, which is similar to the range of the result of original scale. The test-retest reliability (correlation range, 0.74 to 0.96; intra-class correlation coefficient range, 0.73 to 0.96) of the Chinese version of the FACT-Hep is also similar to that of the original one. However, our findings showed that test-retest reliability for the EWB was lower than that of the original one both in Pearson r (0.74) and ICC (0.73). It is needed further study to find the reasons. One possible reason is they validated original scale in only 51 patients and thus the results on test-retest reliability may not be reliable. Also the other previous studies showed that it was generally less than or close to 0.7.^40,41

On Responsiveness of the FACT-Hep

In terms of responsiveness, the classical paired t-test was employed in this study to test the score changes between the pre-treatment and post-treatment assessments, accompanying with important responsiveness indicators, SRM and ES, with values of 0.20, 0.50 and 0.80 having been proposed to represent small, moderate and large responsiveness, respectively.^31,32 Table 6 showed HRQOL scores changes after treatments were of statistical significance on 2 out of 5 domains of FACT-Hep (PWB, FWB) and the general module, overall scale and Trial Outcome Index, with moderate and large responsiveness SRM. Considering that there were some possible reasons^21,24 for domains which were not statistically significant, it can be inferred that this instrument has good responsiveness: (1) the observation period (about four weeks) might have been too short to observe significant changes; (2) the sample size may be not big enough (n=68); A more heterogeneous study sample from a broader range of patients would leads to a certain degree of mutual cancellation and actually underestimates the SRM; and (3) the score in these domains are of no change in nature. For HCC domain, another possible reason is that some patients would become better after treatments and some patients become worse and thus no change can be found when put all patients scores together. At the same time, it was found that due to the specific Chinese culture, patients with more severe symptoms under the influence of Chinese culture would receive more love and care from those around them compared to those with less severe symptoms.⁴¹

On Results from Generalizability Theory

Traditionally, the scale is assessed by classical test theory analysis, in this research Generalizability Theory was also applied both in G-study and D-study, with both G-coefficients and Ф-coefficients being presented, and also their changes when items assumed to be changed. It can be seen from Table 5 that G-coefficients and index of dependability were all greater than or close to 0.70 for the current design, and changed a little as items changing for four out of five domains (exception of EWB). It can be considered that current items are reasonable and acceptable for these domains. For EWB domain, we estimated a G-coefficient of 0.517 and an index of dependability of 0.335 for the current design, which was below the acceptable level of 0.70. Hence, the domain’s items need improvement. For an alternative design with 13 items, the G-coefficient estimated to be 0.699. Therefore, it will be better to increase the numbers of items of EWB in order to reach an acceptable dependability if possible in future. Another approach would be to reconsider whether the composition of the items in the EWB is inappropriate in the context of Chinese culture. As a matter of fact, it can be noted that the low correlations of EWB with GE2, GE3 and the fact that the GE3-PWB correlation is close to the GE3-EWB correlation, which may exist as a result of a cross-cultural study of the FACT-G factor structure in Asian populations.⁴² As the Cronbach’s alpha of a scale is calculated based on the interrelationships between the individual items of the scale, the very low item-scale correlation (r = 0.44) of the EWB subscale with its item GE2 could explain the suboptimal reliability of the EWB subscale.⁴³

To sum up, the analysis from Generalizability Theory confirmed the reliability of the scale further. However, the numbers of items for EWB domain should be increased in order to obtain better reliability if possible. Randomization should also be followed more closely for patient sampling and quiz item selection, and items should be refined accordingly.

Study Limitations

While the results showed the Chinese version of FACT-Hep is a reliable and valid instrument, some caution is necessary. First, nearly half of the participants did not complete the time 3 assessment because they were not in the wards when the investigators attempted to meet with them at the appointed times, for a variety of reasons (eg, going to other departments for treatments, being discharged early for financial reasons). This decreased sample size may have weakened the responsiveness estimation, even it may be reasonable to infer that these events happened by chance. Second, the participants were selected only from inpatients in hospital. Third, baseline data such as disease type and severity were not collected in this survey. The instrument needs to be further tested in other settings such as outpatients at clinics with more baseline characteristics such as disease severity.

Conclusions

In conclusion, the results of CTT and GT showed that the Chinese version of FACT-Hep has good reliability, validity and responsiveness, it can be used to measure HRQOL for patients with hepatocellular carcinoma in China. The instrument needs to be further tested in other settings such as outpatients at clinics with more baseline characteristics such as disease severity.