Development and Validation of the “Keratoconus End-Points Assessment Questionnaire” (KEPAQ), a Disease-Specific Instrument for Evaluating Subjective Emotional Distress and Visual Function Through Rasch Analysis

Introduction

Keratoconus is the most common primary corneal ectasia worldwide.¹ It is characterized by a progressive distortion of the corneal anatomy, associated with a significant decrease in visual quality. Although so far, there are a considerable number of surgeries and optical aids aimed at improving the visual quality of patients with keratoconus, patients with the disease tend to show significant alterations in their ability to perform their daily work normally.²

At present, the Patient-Reported Outcome Measurements (PROMs) have gained great importance as an effective and straightforward mechanism to collect information on the burden of the disease from the patient’s point of view.³ That means these types of instruments allow to reliably determine how much subjective alteration the patient feels about their quality of life (QoL), according to the disease they suffer. This approach is especially important if considering that the visual alteration is a highly subjective experience and that the visual disturbance referred by the patient is not necessarily associated with the anatomical alteration or with other elements directly measurable by the staff physician. Although general PROMs may be useful in some specific situations,

if the data will be used to provide in-depth information to clinicians and patients on the impact of disease, then a disease-specific measure may be most appropriate.³

Until now, the vast majority of studies on QoL in patients with keratoconus have used questionnaires designed for other types of situations,^4,5 which represents a less than optimal condition.

Nowadays, Keratoconus Outcomes Research Questionnaire (KORQ) is the only questionnaire designed and explicitly validated in patients with keratoconus, developed by Khadka et al⁶ and recently evaluated by Kandel et al.⁷ However, one of the main limitations of this instrument is that it ignores one of the most important elements in the compromise of QoL in patients with keratoconus and the profound emotional effect that the disease has on its daily functioning. Moshos et al⁸ have shown that the presence of keratoconus is associated not only with a higher prevalence of clinical depression but also with a greater intensity of it. It has also been linked to particular different personality coping mechanisms.⁹ Therefore, as has been highlighted before by Kandel et al,⁴ it is of paramount importance that an instrument for the adequate measurement of the QoL in patients with keratoconus must necessarily include a component of the emotional sphere of the patients.

Given the above reasons, the authors of this manuscript have developed and validated an instrument called “Questionnaire for the evaluation of keratoconus endpoints” (KEPAQ), which aims to include not only the visual functioning of the patient but also the compromise the disease generates on its emotional scale. It is expected to provide a more comprehensive view of the impact that ectasia causes in the patient. To date, there is no keratoconus-specific scale to evaluate the emotional aspects of the patient due to the disease.

Methods

This is a prospective and analytical study, using the Rasch methodology, which sought to develop and validate a specific QoL questionnaire for patients with keratoconus, taking into account two constructs: emotional compromise and functional compromise.

The development of the instrument has been careful, taking into account the recommendations of Trakman et al¹⁰ for this type of process. As will be discussed later, a series of successive steps have been taken to obtain a valid and applicable instrument for the population with the disease.

This research adhered to the tenets of the Helsinki’s declaration, and proper approval was obtained at the Comité Ético de Investigación Clínica del Área de Salud de Valladolid-Este. All patients signed written informed consent. When the patient was underage, a parent or tutor signed along with the patient.

Definition of the Construct and Development of a Test Plan

As with any PROM, the development of KEPAQ began by defining from a theoretical point the latent traits (or constructs) that were to be evaluated on it.¹⁰ It was defined that there were two latent traits that should be evaluated to provide an adequate picture of the QoL in patients with keratoconus: the effect of the disease on their emotional aspect (named by the authors as Emotional Compromise, KEPAQ-E) and the effect on its visual functioning (Functional Compromise, KEPAQ-F). From this step, it was defined that both elements were significantly different, so the unidimensionality necessary for a Rasch analysis could not be preserved if they were joined. For this reason, although the concept of unidimensionality was evaluated by joining both segments of the questionnaire (as will be seen below), from the beginning, it was defined that it would be better to take them and analyze them as separate constructions.

Generation of the Item Pool

Initially, from a theoretical point of view, the construction of the item pool began as follows: the lead author (KB) met with a total of five patients with a proven diagnosis of keratoconus, who were currently under his medical care, and He asked them to write all the visual and emotional situations in which they felt the disease affected their QoL. They were also given a copy translated into Spanish from KORQ⁶ so that they could take ideas, but they were allowed to write down any type of element in which they considered the disease to affect them.

Subsequently, the author made a homogenization of all the ideas of the patients and turned them into questions that evaluated the degree of compromise of each one. For example, if a patient had noticed that the illness affected his ability to drive during the day, this point became a question. How much do you feel the disease affects your ability to drive during the day? An Effort was made, so all questions were essentially the same in language and method of questioning, being different only in the specific aspect that is evaluated by every item.

According to the recommendations of Trakman et al,¹⁰ the language of the questions was “kept as simple and concise as possible”, while “double negatives and two-edged questions” were avoided to improve the understanding of the instrument.

In this step, a total of 86 questions potentially included in the KEPAQ were generated, 38 corresponding to the Emotional Compromise area and the remaining 48 to the Functional compromise area.

Choice of the Response Format and Scoring System

Regarding the response format, it was defined to leave the KEPAQ as a completely closed system, that is, with a series of defined responses to be chosen by the patients. Besides, the use of a Likert-Like response system was defined that would allow the patient to express how much or how little they support every given question. In order to keep a standardized language, the authors decided to use the same order and distribution of answers as had been previously used by the Khadka et al before: “Not at all”, “A little”, “Quite a Bit”, and “A Lot”. Another option was given for the patients: “Not Applicable”, for them to use in cases where they felt the question did not directly compel to their regular life.

Regarding scoring, the authors followed Boone et al recommendation^11,12 and scored it with increasing value as the answer represented less compromise (meaning, a “higher value” represents a “better answer”). Therefore, for all questions, answers were scored as follows: “Not at all” = 3; “A little” = 2; “Quite a Bit” = 1; “A Lot” = 0.

Assessment of Content Validity

Once the potential items were developed, these were reviewed by a panel of experts consisting of three Ophthalmologists with experience in managing keratoconus patients. First, all questions that were deemed to be too specific to a certain, reduced group of patients were excluded (for example, the question “how much does your vision interfere with your ability to do cross-stitch” was excluded for this reason). After this step, a total of 72 items remained, 30 for Emotional Compromise and 42 for Functional Compromise.

Then, a Content Validity Index (CVI) was determined as a quantitative way of assessing content validity.¹⁰ Every one of the three Ophthalmology experts was asked to rate every individual item for relevance according to the following 4-point Likert scale criteria: 1 = “Very Irrelevant”; 2 = “Irrelevant”; 3 = “Relevant”; 4 = “Very Relevant”. As has been suggested,

the CVI for each question is calculated by dividing the number of raters who scored the item as 3 or 4 divided by the total number of raters; a score above 0.8 is considered adequate.¹⁰

Taking into account the number of expert raters in our group (three), the only way of getting a CVI above 0.8 was having all raters scoring the item as either 3 or 4. All items in which the CVI did not reach this value were discarded as non-relevant.

After accounting for CVI, a total of 20 potential items were included in the preliminary version of the KEPAQ, eight belonging to the Emotional Compromise sub-scale (KEPAQ-E), and 12 belonging to the Functional Compromise sub-scale (KEPAQ-F).

Pre-Testing and Assessment of Face Validity

Before performing a full-scale evaluation, a small sample of the target audience was selected to perform a pre-testing and face validity evaluation of the KEPAQ. A total of 15 subjects with a confirmed diagnosis of keratoconus were included and underwent the 20-items version of the KEPAQ. After undergoing testing, they were questioned regarding how easy it was to answer the questions, and if they thought the items were clear enough. Feedback on topics such as clarity and understanding of items was obtained and taken into account.

Rasch Analysis to Evaluate Items

To soundly evaluate the psychometric properties of the KEPAQ, a Rasch analysis was performed in a sample of patients with keratoconus. Rasch analysis is currently regarded as “the standard of modern psychometric evaluations of outcome scales”,¹³ significantly surpassing prior methods, now known as Classical Test Theory (CTT). Although an in-depth discussion on the characteristics and advantages of Rasch analysis is well outside the scope of the present paper, a small briefing is offered. The interested reader is forwarded to the excellent textbooks on this specific topic, especially the one by Boone et al¹¹ and the one by Andrich and Marais.¹⁴

Rasch analysis is credited to the Danish mathematician Georg Rasch, and it intends to optimize the way measurement of latent traits is performed. It was originally developed for the Educational area, and its mathematical approach relies upon the assumption that

test takers are more likely to correctly answer easy items than difficult items; furthermore, all items are more likely to be correctly answered by people with a high ability on the construct being assessed than by those with a low ability.¹²

When translating the model into a PROM, the word “ability” could be interpreted as “ableness” or “less disability”. Therefore, as the “ableness” of the patient increases (meaning less disability due to the disease), they will be able to “endorse better answers”, therefore increasing their overall score in a well-constructed test. Rasch analysis also generates a “difficulty” value for every item that comprises the test. Both Person Measures and Item Measures are expressed in an arbitrary unit called logit. A higher logit value in a Pearson Measure represents a better “ableness”, while a higher Item Measure represents a more difficult-to-endorse item.

Rasch analysis was performed in JMetrik version 4.1.1 (Psychomeasurement Systems LLC; Charlottesville, VA, United States) in a MacBook Air computer running macOS Mojave version 10.14.3 (Apple, Inc; Cupertino, CA, United States). Andrich’s polytomous Rating Scale Model was used. Both sub-scales (KEPAQ-E and KEPAQ-F) were analyzed separately. Person Measures and Item Measures were extracted for both sub-scales. For every sub-scale, Unidimensionality and Local Independence Hypothesis were evaluated through a Principal Components Analysis (PCA) of standardized residuals as has been previously suggested by Linacre.¹⁵

Item Fit was evaluated to determine which items fit correctly into the Analysis’ standards. Both the Unweighted Mean Square (UMS) and the Weighted Mean Square (WMS) were obtained as measurements of the Outfit and Infit aspects, respectively. As per Linacre recommendations,¹⁶ a UMS and WMS value between 0.5 and 1.5 was considered adequate. And when one of the measurements was between 1.51 and 2.0, the cause of misfit was carefully evaluated and a decision on whether to remove the misfitting item or not was reached. Items with a fit value over 2.0 were immediately removed due to their potential for score degradation.

Adequate ordering of the thresholds for all items was determined as per Andrich’s Thresholds. In case of disordering of the thresholds, every item was evaluated looking for potential improvement through category collapsing and a decision was made regarding whether to collapse or to remove such item.

As for scale quality statistics, both the Item Separation Index and the Persons Separation Index were obtained, along with the Persons Number of Strata and the Persons Reliability. For both Separation Index, a value over 2.0 was considered as adequate. Cronbach’s Alpha was not measured as the Persons Reliability has been demonstrated to be a somewhat better statistic, and they both tend to behave very similarly. A value of over 0.80 was considered reliable enough for clinical measurement.

Finally, a table translating raw scores to linearly transformed Person Measure values was developed, so that clinicians could hand-score their individual patients according to their raw score, so that it can be transformed to a proper interval value.

Sample Size

Currently, there is no clear consensus regarding the best sample size for the validation of new instruments. Some authors have suggested that the ideal sample size should be somewhere between five and ten subjects for every item to be studied. As the original KEPAQ consisted of 20 items, an arbitrary value of 7.5 subjects for every item was selected; therefore, a sample size of 150 subjects was studied as this is considered enough for proper validation and study through Rasch analysis.

Results

Study Population

A total of 150 patients with a confirmed diagnosis of keratoconus were included in the study. The mean age was 29.84 ± 9.96 years (Minimum 14 years – Maximum 70 years). The mean age at the diagnosis of ectasia was 22.33 ± 9.18 years (Minimum 5 years – Maximum 53 years). 92 (61.33%) patients of the cohort were male.

Upon questioning about their refractive situation, 92 (61.33%) patients referred using glasses as their only refractive aid, while 34 (22.66%) patients referred they were not currently using any kind of refractive aid. The rest of the patients used only contact lenses or a combination of contact lenses and glasses.

Regarding prior keratoconus surgery, 19 (12.66%) patients had had a prior keratoplasty (either penetrating or deep anterior lamellar keratoplasty) while 69 (46.00%) had a corneal ring implanted in at least one of their eyes. A total of 47 (31.33%) patients had a history of corneal crosslinking in at least one of their eyes, while 4 (2.66%) patient had had a phakic toric intraocular lens implantation.

Emotional Compromise (KEPAQ-E)

First analysis of the KEPAQ-E demonstrated an overall good result, but Q8 “How much do you feel the use of glasses or contact lenses alter your facial aesthetics” demonstrated to be highly misfitting, with a UMS of 3.42 and a WMS of 2.75. As per Linacre’s recommendation,¹⁶ the item was considered to deter general status of the scale and was removed.

Rasch analysis was re-run after deleting Q8, and all parameters demonstrated and excellent behavior. Both UMS and WMS for all questions were between accepted values (0.50 to 1.50). The easiest-to-endorse question was Q4 “How much do you feel your ocular disease has affected your confidence to carry on your daily tasks” with an Item Measure of –0.86 Logit. The hardest-to-endorse question was Q7 “How much do you feel your ocular disease has caused on you fear about the future” with an Item Measure of 1.29 Logit. All items were correctly scaled according to Andrich’s thresholds.

Variance explained by the model was 62.48%, and the eigenvalue of the first PCA contrast was 1.43, suggesting an unidimensional scale.

Item Separation Index was 4.58 while Person Separation Index and Person Number of Strata were 2.43 and 3.57, respectively. Person Reliability was 0.85.

DIF analysis was carried out for all items, looking for differences according to sex (male vs female), history of keratoplasty (present vs absent), history of corneal rings (present vs absent) and history of Corneal Crosslinking (present vs absent). All DIF analysis gave out a p value over 0.05 (non-significant) demonstrating a stable scale for all potential groups.

For the total Person Measures of the KEPAQ-E sub-scale in the studied sample, first quartile, median and third quartile were 0.21, 2.22 and 3.72 Logit, respectively.

Score table for transforming KEPAQ-E raw scores to Person Measures can be found in Table 1.

Table 1 Table for Transforming KEPAQ-E Raw Score to Person Measure, Which Is the Value That Should Be Used for Epidemiological and Clinical Applications, According to Rasch Analysis Theory

Functional Compromise (KEPAQ-F)

First Rasch analysis of the KEPAQ-F sub-scale demonstrated Q5 “Does your vision interfere with your ability to avoid objects in your way” to be significantly misfitting, with a UMS of 2.75 and a WMS of 1.94. Upon careful evaluation, the authors determined that the way the question was written may be unclear as it may not be easy for the patient to understand what kind of “object” they should be avoiding and how far away it would be. Therefore, the question was eliminated.

A new run of the Rasch analysis demonstrated better values. However, two questions, Q3 “Does your vision interfere with your ability to drive during the day” and Q4 “Does your vision interfere with your ability to drive during the night” were found to be significantly disordered according to Andrich’s thresholds. Removing them from the test significantly improved all aspects of the scale’s parameters, therefore they were removed.

A final run of the Rasch analysis after having removed Q3, Q4 and Q5 demonstrated excellent results. Both UMS and WMS for all questions were between accepted values (0.50 to 1.50). The easiest-to-endorse question was Q8 “How much does your vision interfere with your ability to perform your daily tasks?” with an Item Measure of –0.38 Logit. The hardest-to-endorse question was, by far, Q10 “How much does your vision interfere with your ability to see objects far away?” with an Item Measure of 1.67 Logit. All items were correctly scaled according to Andrich’s thresholds.

Variance explained by the model was 61.30%, and the eigenvalue of the first PCA contrast was 1.98, suggesting an unidimensional scale.

Item Separation Index was 4.43 while Person Separation Index and Person Number of Strata were 3.19 and 4.59, respectively. Person Reliability was 0.91.

DIF analysis was carried out for all items, looking for differences according to sex (male vs female), history of keratoplasty (present vs absent), history of corneal rings (present vs absent) and history of Corneal Crosslinking (present vs absent). All DIF analysis gave out a p value over 0.05 (non-significant) demonstrating a stable scale for all potential groups.

For the total Person Measures of the KEPAQ-F sub-scale in the studied sample, first quartile, median and third quartile were –0.51, 1.38 and 5.54 Logit, respectively.

Score table for transforming KEPAQ-F raw scores to Person Measures can be found in Table 2.

Table 2 Table for Transforming KEPAQ-F Raw Score to Person Measure, Which Is the Value That Should Be Used for Epidemiological and Clinical Applications, According to Rasch Analysis Theory

Final Version of the Scale

After stringent Rasch analysis, a final version of the KEPAQ was achieved and validated. KEPAQ-E ended up consisting of a total of 7 questions, while the KEPAQ-F ended up consisting of a total of 9 questions. Both sub-scales demonstrate a good behavior, complying with the Rasch model expectations as specified by Khadka et al⁶ (Table 3). For the final version of the KEPAQ, questions in both sub-scales were rearranged from the easiest-to-endorse to the hardest-to-endorse items. Final version of the KEPAQ-E and KEPAQ-F scales can be seen in Figures 1 and 2, respectively.

Table 3 Psychometric Properties of the Final Versions of Both Sub-Scales for the KEPAQ

Figure 1 Emotional Compromise sub-scale of the Keratoconus End-Points Assessment Questionnaire (KEPAQ-E).

Figure 2 Functional Compromise sub-scale of the Keratoconus End-Points Assessment Questionnaire (KEPAQ-F).

Discussion

Currently, there is no global consensus on what is defined under the concept of quality of life. Many authors have focused on purely subjective connotations, defining it, for example, as “a conscious cognitive judgment about life satisfaction itself.” On the other hand, some others have considered that objective elements must be included in the definition of what quality of life represents. Moreover, it is necessary to bear in mind that what is considered quality of life should not represent an arbitrary and cold measure, but it requires contextualization within the specific elements of each patient individually. In this regard, it is worth highlighting the definition of the World Health Organization, who has suggested that the quality of life includes “the perception of an individual about their position in life in the context of culture and value systems in which they live; and in relation to his objectives, expectations, standards and concerns”.

Given these elements, it is not surprising that the quality of life is something intrinsically difficult to measure in medical sciences,¹⁷ and ophthalmology is no exception.

However, until a few years ago there was no questionnaire focused specifically on quality of life on patients with keratoconus, so researchers were forced to use instruments not designed for subjects with this condition.^18,19 For example, Tatematsu-Ogawa et al¹⁸ studied a sample of Japanese patients with keratoconus, comparing their quality of life to a sub-set of age-matched controls, as measured with the National Eye Institute Visual Function Questionnaire-25 (NEI-VFQ-25). The same questionnaire was used by Kymes et al²⁰ in their study including 1166 patients with keratoconus from the CLEK study.

However, as expected, the behavior of standardized questionnaires is not reliable when applied in populations other than those for who they were designed and tested. The above is evidenced in the study by McAlinden et al²¹ who evaluated the performance of the NEI-RQL-42 questionnaire in 44 patients with keratoconus before and after corneal rings implantation. The authors found that only one of the 13 categories of the questionnaire showed adequate behavior in the study population, so they concluded that it was not an applicable instrument in patients with corneal ectasia.

This kind of situations highlight the need for specific measuring instruments for patients with keratoconus. Until now, the only questionnaire designed and evaluated specifically in patients with corneal ectasia is the Keratoconus Outcomes Research Questionnaire (KORQ), initially developed in the Australian population, and published in 2017 by Khadka et al.⁶ This is a questionnaire developed through Rasch analysis, consisting of a total of 29 questions divided into two independent sections: “Activity Limitation” (with 18 questions) and “Symptoms” (with 11 questions). Its psychometric properties have been recently evaluated by an independent group, how found it to be a good-performing scale.⁷

Despite this, an important limitation of KORG is the fact that the mental sphere of patients is not taken into account for the measurement of their QoL.⁴ The importance of the emotional component of patients with keratoconus was revealed by recent studies that have shown that patients with ectasia have different personality coping mechanisms,⁹ and have a higher rate of depression. Given this aspect, it could be suggested that leaving aside the emotional sphere of patients could deprive the researcher and clinician of extremely important information to measure the true QoL of patients with keratoconus.⁴

To solve this problem, the authors of this manuscript have developed and validated the Keratoconus End-Points Assessment Questionnaire (KEPAQ), which consists of two sections, one focused on the Functional Compromise (KEPAQ-F), which has a very similar approach to the KORQ. The second section of the KEPAQ focuses on the Emotional Compromise (KEPAQ-E), which represents the newest aspect of the scale. With this in mind, KEPAQ is the only instrument worldwide that is able to measure QoL both visual functioning and emotional compromise in patients with keratoconus.⁴

The Rasch analysis of the scale demonstrated the need for some small adjustments, especially in the elimination of some of the initially postulated items. After the adjustments, the KEPAQ scale in its final version is a tool that demonstrates very robust, reliable psychometric properties, and that allow a reliable measurement of QoL in patients with keratoconus. One of the great advantages of KEPAQ is the robustness in its conception and in the epidemiological and mathematical process of creation and validation. The process carried out for its development meets the most advanced and demanding guidelines according to global evidence to date. In addition, the KEPAQ is shorter than the KORQ,⁶ so its application in the daily practice can be simpler and faster, without causing damage to the psychometric characteristics of the test.

Although patients with a number of different situations were included (history of keratoplasty, contact lens wear, etc.), Rasch analysis demonstrated that different groups of patients behaved non-differently. Therefore, results can be expected to be applicable to these different groups of subjects.

To facilitate both the investigative and clinical use of the scale, a table is provided by which the clinician can quickly transfer the raw score obtained by adding the values of the scale and then converting them to the Person Measures provided by the Rasch scale. For all statistical and follow-up aspects, the value of the Person Measures meets all the characteristics of a reliable final score, and with interval characteristics, so they can be subjected to parametric statistical analysis, which would be impossible if the scale had been designed using less advanced mathematical processes.

The authors are well aware of the limitations of current research work. First, it has been carried out in the Colombian population, so its applicability in other populations is not yet adequately defined. Currently, the authors are making agreements with Argentine and Indian collaborators to be able to carry out validation studies of the questionnaire in these populations. Definitively, more publications in different languages and cultural settings are necessary to adequately determine the applicability of the KEPAQ in other geographic parts. Second, it is not known to date if the KEPAQ score correlates adequately with clinical or anatomical parameters. Currently, the authors are carrying out a study in which it is intended to asses whether the Belin ABCD classification (in both the best and the worst eye) correlates with the results of the KEPAQ scale. A third limitation is the fact that only one person developed the “item pool”, where a plural number of individuals may have been a better option as it may have opened the door for a more variable pool of potential items to include.

Conclusions

Both the visual function and the emotional distress are important to adequately measure QoL in keratoconus patients. So far, no disease-specific PROM scale had ever been designed to include the emotional aspect of ectasia patients. The KEPAQ is a scientifically robust scale, with an excellent psychometric profile, which can be confidently used to measure QoL in keratoconus patients, both for clinical use and for research.