The PCQ-Infertility Revised: A New Digital Instrument to Measure Treatment Satisfaction of Fertility Patients

Plain Language Summary

It is of great importance to keep improving the quality of healthcare. One important way to do this, is by understanding the needs and wishes of patients. For patients undergoing fertility treatment, a questionnaire was developed that helps clinics to understand which aspects of the fertility care should be improved. This questionnaire was, however, rather long and the questions were difficult to understand. In this study, we show how the questionnaire was improved by removing 24% of questions and rephrasing difficult to understand questions in a way they could be interpreted more easily. We also show that the new questionnaire can capture the patients’ needs and wishes in fertility care more accurately than the original questionnaire. Therefore, this new questionnaire can be used in many fertility clinics and will help improve the quality of fertility care.

Background

Patient-centeredness, which represents how well healthcare is geared towards patients’ needs and wishes, is acknowledged as one of the six key aspects of healthcare.^1,2 Within the fertility care, most healthcare professionals are aware of the meaning and importance of patient-centeredness, but they do not, however, always consistently undertake the necessary steps to improve their patient-centered care.^3,4 Important obstacles are the lack of detailed, concrete feedback from patients to understand their specific needs and the healthcare providers lack of motivation to change practice.⁴ As a result, healthcare professionals may not be aware of which limits their ability to improve the care they provide. From a patient’s perspective, not only medical skills are considered important, but also respect, coordination, accessibility, information, comfort, support, partner involvement, having a lead physician, and a good attitude of and a good relationship with fertility clinic staff.^5–7 Several of these aspects can and should be improved.⁸ Personalized treatment benefits the patients and reduces the substantial physical and emotional burden of fertility treatment.⁹ This may reduce the high dropout rate in fertility care.^8,10

Monitoring patient-centeredness by measuring patients’ experiences via surveys, is a valuable tool to improve fertility care.^5,11 Questionnaires that measure the outcome of the given care (eg the amount of physical pain and quality of life patients experience) are Patient Reported Outcome Measures (PROMs). Patient Reported Experience Measures (PREMs) measure patient satisfaction with a health service and capture overall patient experience including the process of care (eg communication and information provision).¹² In 2010, a PREM was developed based on focus group interviews and a literature review in the Netherlands to measure patient-centeredness within the infertility field, the so called Patient-Centeredness Questionnaire-Infertility (PCQ-Infertility).¹³ This original PCQ-Infertility (PCQ-original, consisting of 51 items) measures patient-centered care by assessing seven dimensions: accessibility, information, communication, patient involvement, respect for patient’s values, continuity and transition, and competence.¹³ The PCQ-original showed a high internal consistency and good construct validity.^13,14 In addition, it has been successfully validated in six countries (the Netherlands, Slovakia, Slovenia, Portugal, Iran, and New-Zealand)^13–19 and it is currently used in clinical practice in countries such as Germany and the Netherlands. However, important limitations of the PCQ-original lies in its format. As a paper-based questionnaire it is less likely to be filled in and data processing is labor-intensive. Furthermore, it’s lengthy and probably due to the diversity in answering categories and the presence of dichotomous items there is a relatively low reliability of dimensions such as “continuity and transition”.^13,19 The response options in the PCQ-original are diverse and include open-ended, multiple-choice, dichotomous (yes/no), rating scales (0–10), and 3-,4-, and 5-point-Likert response scales. Data analysis is therefore unnecessary complicated and could be improved by consistently applying the same response scales to acquire more reliable and easier to interpret results.

Despite its limitations, the PCQ-original discriminates well on the quality of patient-centered healthcare between fertility clinics and hospitals, making it suitable for benchmarking.¹³ It contains many attributes, such as the “Hospitals Experience Questionnaire (American Hospital CAHPS) and the Hospital Comparison System questionnaires (developed by the Consumers” Association and NIVEL measuring hospital service from a patient’s perspective) that are required in Consumer Quality Indexes (CQI), a type of patient satisfaction instrument widely used in the Netherlands.^20,21 However, despite their popularity, CQIs have been criticized for having too many items, being costly to administer, and for the results being difficult to interpret.²² These problems also apply to the PCQ-original. The complicated and long questionnaires could demotivate the patients resulting in nonresponse or careless responding.^23–26

We sought to overcome the shortcomings of the PCQ-original regarding length, interpretability and reliability, and describe the development and validation based upon the essential standards for patient-outcome measures of a revised version of the PCQ-Infertility (PCQ-revised),²⁷ which is now available for large-scale clinical implementation.

Materials and Methods

Data Collection

Sample Population

Women (aged 18–45, single, or in a same sex or heterosexual partnerships) receiving fertility treatment in the Netherlands were invited to fill in the PCQ-original or the PCQ-revised (together with a partner if applicable) by either invitation cards available at the reception or via digital invitation (eg e-mail) which is sent out by the clinic. It was estimated that the response rate was ~50%, but no exact numbers could be collected due to the nature of the invitation procedure. Patients who completed the PCQ-Infertility (either original or revised) consent that data is anonymously collected and used for scientific research. This study was conducted according to the guidelines laid down in the Declaration of Helsinki.

Collection of Feedback on the Original PCQ-Infertility by Fertility Experts and Patients

A medical advisor from Ferring B.V. consulted three fertility healthcare experts from Dutch clinics (one fertility doctor and two nurses specialized in fertility), and the Dutch patient association for people with fertility problems, Freya, to get insights into commonly experienced issues with the use of the PCQ-original by performing semi-structured interviews. Conversations were held 1:1 on location, and a second Ferring B.V. representative was involved to note the mentioned issues and shortcomings of the PCQ-original, and the possible solutions on how to improve the PCQ-original. Questions such as “Which feedback do patients give after filling in the PCQ-Infertility?”, “Which problems do you encounter with the PCQ-Infertility?”, and “How do you think the PCQ-Infertility could be improved”? were asked. Based on these conversations, commonly mentioned issues were noted and concrete shortcomings and solutions to improve the PCQ-original were formulated. In addition, all three fertility healthcare experts and a representative from Freya were asked to critically evaluate the PCQ-original. They were specifically asked to pinpoint essential items, and to give suggestions for improvement regarding the phrasing and wording of items and answers. The acquired insights and the written feedback served as a base to reduce and rephrase items in the PCQ-revised. The development and validation of the PCQ-revised was based upon previously published guidelines for patient-outcome measures.²⁷

Collection of Original PCQ-Infertility Output

LiveResearch, a Dutch research institute that specializes in measuring Customer (Patient) experience, and fertility departments of four fertility care clinics in the Netherlands supplied us with an anonymized dataset of questionnaires completed voluntarily by 844 patients receiving fertility treatment (Materials and methods, Sample Population). Approval by an institutional ethics committee to use this dataset was not required according to the Dutch law, since no patient-sensitive data was collected, data was untraceable, and patients participated voluntarily. Data were collected between 2017 and 2019. The dataset consisted of responses to the questions in the PCQ-original, collected in an excel database that was cleaned from duplicates.¹³

Collection of Revised PCQ-Infertility Output

The same four hospitals that provided the anonymized dataset of the PCQ-original were contacted to participate in this study, in which the PCQ-revised was distributed among new patients. Two hospitals, Nij Geertgen and Hospital Gelderse Vallei, agreed upon participation. The PCQ-revised was completed by 260 new patients who had not previously completed the PCQ-original, of whom 230 came from Nij Geertgen and 30 came from Hospital Gelderse Vallei. Data collection took place between August 2019 and October 2019 in collaboration with LiveResearch.

Data Analyses

Data was exported to SPSS for statistical analysis (Statistical Package for Social Sciences: version 27 for Mac, SPSS Inc., Chicago, IL, USA). For all analyses, normality of data and the proper assumptions such as frequency, kurtosis, skewness, and missing data were checked and statistical significance was set at a p-value of < 0.05. SPSS Amos was used to perform confirmatory analyses.

Psychometric Analyses

First, reliability of the PCQ-original was tested using Cronbach’s alpha for each dimension. Values >0.70 were considered acceptable. Based on the SPSS output of the analysis, an item was flagged if the Cronbach’s alpha increased after deletion of the item. Next, the flagged items for deletion were revised with the requirements of the Patient-Centeredness Quality Index (PCQ index), an index widely used in the Netherlands for measurement of patient satisfaction, and items which were considered essential were not deleted.²⁰ Flagged items for deletion were also cross-referred with the requirements from Freya and deletion only took place if the question was not considered essential. Freya requested items Q4, Q24 and Q26 to remain in the questionnaire, although deletion would have improved the Cronbach’s alpha of the corresponding dimensions. In addition, face validity was applied by fertility healthcare experts to determine which items could be deleted and to decide which items should be reformulated. Reliability analysis was then performed on the PCQ-revised and values >0.70 were considered acceptable.

We performed a confirmatory factor analysis to determine convergent and discriminant validity of the PCQ-revised.¹³ We assessed the model fit by the Comparative Fit Index (CFI) and the Root Mean Square Error of Approximation (RMSEA). The CFI needs to be > 0.9 to indicate a good model fit and the RMSEA needs to be < 0.08 for a good model fit. We calculated standardized factor loadings for each item in relation to the corresponding dimension and considered loadings of > 0.40 to be sufficient.²⁸ Convergent validity was determined by calculation of Average Variance Extracted (AVE) scores, with a score > 0.5 indicating the presence of convergent validity. Discriminant validity was tested with the Fornell and Larcker Criterion, with a criterion of > 0.5.²⁹

Results

We divided our study in three parts. First, we interviewed fertility health care experts and the Dutch patient association Freya to identify shortcomings of the PCQ-original. Second, based on the received feedback, and reliability and validity analyses of the PCQ-original, the scale was revised to make the PCQ-revised parsimonious. Third, we validated the PCQ-revised with the PCQ-original by performing reliability and confirmatory factor analyses.

Identifying Shortcomings of the Original PCQ-Infertility by Expert Opinion

The most commonly mentioned shortcomings in the PCQ-original as indicated by the consulted fertility health care experts and the Dutch patient association Freya, are the high number of items, the difficulty in interpreting the patient’s responses due to differences in answering scale, and inconsistent wording in the items (Table 1). A summary of the shortcomings mentioned for the PCQ-original and their possible solutions are listed in Table 1.

Table 1 Summary of Shortcomings of the PCQ-Original and Proposed Solutions as Indicated by Fertility Health Care Experts and the Dutch Patient Association Freya

Revision of the Original PCQ-Infertility

To improve the response rate, interpretability and the length of the PCQ-original, first the number of items were revised based upon reliability analysis, face validity, and predefined criteria (Materials and Methods, Data Analyses).

The PCQ-original comprises of 51 items in total: four background items (Q1-Q4) in the section “Background Information”, 46 patient’s experience items (Q5-Q50) divided into seven dimensions and one item in the section “To conclude” (Q51, Table 2).¹³ In total, 13 items were removed, as can be seen in Table 2. Most of the items were removed from the section patient’s experience (12/46; 26%).

Table 2 PCQ-Original and PCQ-Revised Items and Answer Types

In the last section of the PCQ-original “To conclude”, the item “What is the probability of recommending this hospital / clinic?” was added (Table 2, “To conclude”). The final PCQ-revised comprises of 39 items: three items in the “Background information” section, 34 items in the main section, and two items “To conclude”. The PCQ-revised is 24% shorter than the PCQ-original (from 51 to 39 items).

After reducing the number of items all items were rephrased. Negatively phrased items were rephrased into positively phrased items, as positively phrased items were expected to be less confusing and more consistent, which would increase the reliability of the results.³⁰ Items that could be considered as vague or ambiguous were also rephrased. The rephrased items are shown in Table 2.

The biggest change that was implemented in the PCQ-revised was the conversion of the dichotomous yes/no into a 5-point Likert scale. For Q35 the dichotomous (yes/no) response scales remained in place and the response scales for items with multiple choice or interval (0–10) scales (Q2, Q3, Q16, Q49, Q50, and Q51) were also not changed.

Testing and Validation of the Revised PCQ-Infertility

Reliability of the Original and the Revised Version of the PCQ-Infertility

Reliability analysis was performed on the PCQ-original, which was completed by 844 patients, and on the PCQ-revised, which was completed by 260 new patients who did not previously complete the PCQ-original (Materials and Methods, Psychometric Analyses). All items from the “Background Information” section were excluded from the reliability analysis, since they either have multiple-choice or dichotomous (yes/no) response scales, which are not eligible for calculation of reliability statistics. The same accounts for Q7, Q12, Q16, Q24, and Q35 of the main section of the PCQ-original. No reliability analysis was performed on the section “Remaining Items”, as these items do not belong to a dimension.

In contrast to the PCQ-original, no Cronbach alpha value was < 0.5 in the PCQ-revised, which indicates that all dimensions are at least reliable to an acceptable standard (Table 3). Importantly, the reduction in number of items did not harm the internal consistency and reliability. For the dimension “Respect for patient”s values’, a higher Cronbach’s alpha was reached when Q24 was deleted (data not shown). However, Freya indicated that this item is essential and should therefore not be deleted from the questionnaire.

Table 3 Reliability Analyses of the PCQ-Original and the PCQ-Revised

Confirmatory Factor Analysis

As the reliability was high in the PCQ-revised, a confirmatory factor analysis was performed to confirm content and discriminant validity of the dimensions, and if the items fit in the previously defined dimensions of the PCQ-original¹³ (Table 4). The confirmatory factor analysis showed good model fit according to the root mean square error of approximation (RMSEA) of 0.072, and the comparative fit index (CFI) was 0.881, which was only slightly below 0.90.^31,32 In five out of seven dimensions, convergent validity was present as shown with an average variance extracted (AVE) scores > 0.5 Table 4. Only the dimensions “Accessibility” (AVE = 0.437) and “Information” (AVE = 0.434) had an AVE < 0.5. Lower factor loadings of Q6 (0.305) and Q7 (0.413) could explain the slight lack of convergent validity in the dimension “Information”. This suggest that these items relate better to a separate latent construct. The dimension “Accessibility” might benefit from the inclusion of extra items for more variance explained, as factor loadings showed that the included items do load well onto the dimension. Discriminant validity for all dimensions was confirmed by the Fornell and Larcker criterion.²⁹

Table 4 Confirmatory Factor Analysis of the PCQ-Revised

Discussion

Our study showed that the PCQ-original was improved by thorough evaluation of the number of items, the phrasing and framing of both items and answers, and by implementation of consistent 5-point-Likert answering scales, while keeping the original seven dimensions. We provide a fully revised PCQ-Infertility that is shorter, more comprehensible, and more reliable than the original PCQ-Infertility (Supplementary Data for the final PCQ-revised). This should improve response rates and offer easier data analysis and data interpretation for healthcare professionals.

The reduction in number of items and the changes we made in wording increased internal consistency on five dimensions while keeping it stable on two dimensions. Two dimensions had a Cronbach’s alpha lower than 0.7, indicating that for these two dimensions, it is also important to evaluate the individual item scores. Importantly, the overall reliability of the revised questionnaire was improved. We expect that the response rate of the PCQ-revised will be higher than the PCQ-original because it will take much less time to complete. In addition, the rephrased items and answers are easier to comprehend, which helps patients to communicate their experiences better. It is likely that this will lead to a reduction in careless responses and improve the measurement of patients’ experience of fertility health-care quality.

Another important improvement we made to the PCQ-original was the implementation of 5-point Likert scales for all items that originally had 3-, 4-point Likert scales or dichotomous (yes/no) answering scales. This increases ease of use of the scores and reduces data analysis time. Raw scores of the PCQ-revised can be used to calculate Quality Improvement (QI) scores to measure patient-centeredness and using 5-point Likert scales increases reliability of such scores.¹³

Our study is not without limitations. This study included only patients from Dutch fertility clinics and the PCQ-revised was validated in a smaller patient population than the PCQ-original. Patients who filled in the PCQ-revised came from two fertility clinics, while the PCQ- original was completed by patients coming from four fertility clinics, causing a possible bias. It is therefore important to validate the PCQ- revised in a larger, more diverse patient population coming from different fertility clinics. It is also important to note that we decided to keep items in the questionnaire upon request of Freya, although deletion of these items would have increased reliability of the dimensions. The reliability of the dimension “Continuity and transition” was previously reported to be less reliable in the PCQ-original,^13,19 and deleting items in this dimension did not improve the reliability in the PCQ-revised. However, the Cronbach’s α was just below 0.7, indicating that minor adaptations could improve the reliability. The reliability of the dimension “Accessibility” was significantly improved in the PCQ-revised compared to the PCQ-original, but it was still below 0.7. Future research should focus on improving the reliability of these two dimensions.

The PCQ-revised is digitalized and can be completed within ten minutes, which fits better in the current day-to-day practice than the paper version of the PCQ-original. This is an important step towards the implementation of a standardized evaluation system that can help fertility health care experts improving their health care. The PCQ-revised is now implemented in more than thirty Dutch fertility clinics. They can access the PCQ-revised outcome data from their clinic via a dashboard, perform data extraction and evaluate the results directly. The dashboard shows the quality improvement (QI) scores of each domain, which gives them the opportunity to immediately act upon the findings, ultimately leading to an improvement of the quality of fertility care. The next steps are to validate the PCQ-revised in other countries and to evaluate patient satisfaction in fertility care with this new digital measurement tool on an even large scale.

Conclusion

The PCQ-revised, measuring patient satisfaction in fertility care, is more valid, reliable, and shorter than the PCQ-original. The questions are easier to comprehend, leading to a higher quality of answers. In addition, improved answering scales can make data analysis easier and faster. The PCQ-revised is digitalized and currently implemented in more than thirty Dutch clinics, indicating that it fits the current day-to-day practice and that it helps fertility health care experts to keep improving the quality of their patient-centered care.