Towards Standardization of CGM Performance Studies: The QUSS-CGM Questionnaire for Assessing User Satisfaction

Introduction

Continuous glucose monitoring (CGM) systems are part of everyday life for many people with diabetes and their increased accuracy in recent years allows them to be used for therapy decision-making.¹ Therefore, adequate device performance is indispensable to ensure reliable diabetes management outcomes. However, studies to determine clinical performance of a CGM system vary tremendously in terms of study design, selection of comparator method, experimental procedures, or statistical analysis, making it difficult both to interpret the data (eg, accuracy) and to compare devices among different studies.² Efforts are being made under the direction of the International Federation of Clinical Chemistry (IFCC) Working Group on CGM to standardize the reporting and procedures for assessing the performance of CGM systems.^2–4

Apart from accuracy and reliability, user satisfaction remains an integral component of CGM performance assessment and constitutes a critical determinant in patient adherence to long-term device utilization. High levels of patient satisfaction have been linked to more frequent CGM use and are key to patient adherence.⁵ A widely used tool to assess user satisfaction are questionnaires,⁶ as shown by the literature review of CGM performance studies from Freckmann et al.² Several validated questionnaires for glucose monitoring user satisfaction already exist, eg, the Glucose Monitoring Experiences Questionnaire (GME-Q),⁷ the Glucose Monitoring System Satisfaction Survey (GMSS),⁸ the CGM Satisfaction Scale (CGM-SAT) and the Glucose Monitoring Survey (GMS).⁹ These questionnaires cover general aspects of usability and user satisfaction with a major focus on quality of life and patient-reported outcomes during long-term usage of a CGM system, as well as its therapeutic impact.^8,9 However, in CGM performance evaluation studies, the wearing time of these minimally invasive systems is comparatively short (generally no longer than two weeks) and factors such as sensor handling, pain or general comfort are primarily addressed rather than long-term experience. In addition, the therapeutic impact cannot be judged by the participant of CGM performance studies since they are usually only allowed to use their own system for therapy decision-making. When reviewing other questionnaires used in CGM performance studies, the selection of questions and surveyed aspects was found to be unsystematic. To our knowledge, there is no validated method specifically tailored to assess user satisfaction with minimally invasive CGM systems in the context of performance studies.

The development of a standardized user satisfaction questionnaire requires a systematic validation approach to ensure high-quality results. Therefore, the objective of this work was to design, develop, and validate the Questionnaire for User Satisfaction Standardized for CGM performance studies (QUSS-CGM). This involved compiling relevant satisfaction questions referred to as items through a systematic literature review, identifying key topics known as attributes, and conducting a structured validation process, including content validation with experts and response process validation with individuals with diabetes to assess item relevance and comprehensibility. A psychometric evaluation followed, incorporating item selection based on corrected item total correlation, exploratory factor analysis (EFA) and reliability assessment. Finally, the QUSS-CGM was translated to English using a forward-backward translation method.

Materials and Methods

Context

The intended target population of the questionnaire are participants (people with type 1 and 2 diabetes aged ≥ 18 years) of two minimally invasive CGM performance evaluation studies.^10,11 Furthermore, the number of questions should be chosen such that the questionnaire can be completed in five to ten minutes. A schematic depiction of the QUSS-CGM development process is shown in Figure 1.

Figure 1 Main steps followed to design, develop and validate the user satisfaction questionnaire. Questionnaire for user satisfaction standardized for CGM (QUSS-CGM).

Questionnaire Design and Item Development

The defined attributes were based on a sub-analysis of a systematic literature search conducted in the MEDLINE database by Freckmann et al,² which included 129 CGM performance studies published between 2002 and 2022. Key inclusion criteria were that studies evaluated ISF-based CGM systems in adults with type 1 or type 2 diabetes under routine care conditions. To retrieve more recent scientific publications on this topic, the search was extended by the time period from 01-Jan-2023 to 28-Aug-2023. All publications meeting the criteria defined by Freckmann et al² and containing relevant content on usability and user satisfaction were included in the analysis to gather items and define the categorizing attributes.

Identified items were grouped by content and assigned to different attributes. Covering the content of each attribute, concise items were positively and negatively phrased and organized in a two-part draft-questionnaire with items for the beginning and end of the study. A 5-point Likert scale was chosen as the response option for the questionnaire (1- “strongly disagree”; 2- “disagree”; 3- “partly agree”; 4-“agree”; 5- “strongly agree”), as it was often used in the questionnaires identified through the systematic search.

Item Selection and Validation

Content validation was performed to confirm that the items mirrored the intended focus and were relevant to the objective of the questionnaire. A panel of experts (n=9) in the field of CGM use and performance evaluation, selected based on their publication track record and experience as investigators or physicians in CGM studies, were instructed to rate the defined draft-questionnaire items according to their relevance and clarity, separately, in addition to suggesting phrasing changes.¹² The established scoring system used consisted of the item-level content validity index (I-CVI) and the overall, scale-level content validity index based on the average method (S-CVI/Ave).¹² Items with an I-CVI <0.78 were excluded,¹² and the remaining items were rephrased according to experts’ suggestions. An overall S-CVI/Ave score of >0.80 was considered to be acceptable.¹³

Response process validation was conducted by a group of people with diabetes (n=10), reflecting the intended target of the questionnaire, to determine the face validity of the items that passed the content validation, evading potential ambiguities.¹⁴ The comprehensibility and clarity of the items were rated, and suggestions could be made for clearer wording.¹⁴ The established the item-level face validity index (I-FVI) and the overall, scale-level face validity index based on the average method (S-FVI/Ave) scoring system was used.¹⁴ Items with an I-FVI <0.80 were revised and rephrased according to the respondents’ suggestions.¹⁴ An overall S-FVI/Ave score of >0.83 was considered to be acceptable.¹⁵

Pilot Study and Psychometric Evaluation

Data from two CGM performance studies^10,11 performed between June and August of 2024 referred to as “Pilot studies” were pooled to increase the sample size and strengthen the statistical significance. For both studies, ethical approval was obtained before the start and they comprised a total of 54 participants, each of whom wore 2 or 3 distinct CGM systems concurrently. All participants provided written informed consent prior to enrollment. Four different systems were tested, one of them in both studies. Subjects completed one questionnaire per system, yielding 132 questionnaires. Questionnaires not fully answered (n=6) were excluded from the analysis. Floor and ceiling effects were investigated (when >15% of the answers focus on the worst and best ratings available on the scale, respectively) to assess the sensitivity of the questionnaire to detect differences at the lower and upper ends of the scale.

Item selection for the EFA was conducted through an iterative process, based on their corrected item total correlations, those below the cut-off of 0.2 were deleted. Items were selected until there was no substantial increase in Cronbach’s α without loss of information. After assessing the adequacy of the sample with the Kaiser–Meyer–Olkin (KMO, >0.5) measure and the Bartlett sphericity test,¹⁶ EFA was carried out to determine the factorial structure of the questionnaire.¹⁷ The number of factors defining the structure of the questionnaire was established using the Kaiser criterion (eigenvalues >1), in addition to an inspection of the Scree Plot using the elbow criterion.¹⁶ Factors were extracted through principal axis factoring using oblique rotation (Oblimin) to simplify the structure. Items were classified under each factor according to their corresponding factor loading score (>|0.4|), cross-loadings for more than one factor were assessed, as well as items with insufficient factor loading scores. Afterwards, questionnaire’s internal reliability was evaluated with the Cronbach’s α coefficient, with values above 0.7 considered as good.¹⁷ Psychometric evaluation consisting of EFA and reliability assessment was performed using SPSS software version 29.

To interpret the outcomes of the questionnaire, a scoring standardization yielding one score per CGM system was carried out. To compute the total score, the established 5-point Likert scale responses (1–5) were recoded to a 0–4 range. Reverse scoring was applied to negatively worded items and similarly converted to the 0–4 range to maintain scoring consistency. The total score was linearly transformed to a scale from 0–100, with higher scores reflecting greater user satisfaction. A scale based on the percentile calculation was defined.¹⁷

Translation

The development and validation of the QUSS-CGM was done in German since this was the mother tongue of the participants in the CGM performance studies. Hence, a validated English translation was developed using a bidirectional translation protocol, where native German bilingual experts translated the German version to English and native English bilingual experts translated the items back to German. Afterwards, the versions were reconciled to avoid cross-cultural differences and establish the final translation.

Results

Systematic Literature Search

The systematic literature search on CGM performance studies yielded a total of 130 articles (n=129 from 2002 to 2022,² n=1 from extended search in 2023), of which 31 manuscripts^18–48 contained relevant content in the form of user satisfaction questionnaires or individual items. Four of these articles were excluded since one evaluated a combined device for insulin infusion and glucose sensing²⁷ and three focused on an implantable CGM system.^31,38,39 Twenty-seven publications containing a total of 205 items were included (Figure 2a). Items were classified according to their content; thereafter, six attributes were defined: sensor handling, ease of use, digital interface, experience, comfort and reliability. Defined attributes were almost evenly represented in the publications, with “sensor handling” being the most frequent attribute and digital interface the least frequent (Figure 2b). This classification was based on the unmodified items, meaning that duplicates and equivalent items were all included.

Figure 2 (a) Literature selection process, based on the systematic literature search published by Freckmann et al 2023;2 (b) Classification of the 205 raw items to six attributes compiled from the user satisfaction surveys used in the 27 CGM performance studies screened.

Item Development

The 205 items pooled from the systematic literature search were classified into six defined attributes (sensor handling, ease of use, digital interface, experience, comfort, and reliability). After removing duplicates and consolidating similar items, 25 unique items remained, forming a draft questionnaire that exemplified the content collected through the literature research, following the methodology described by Yusoff et al (2021)¹⁷ (Figure 3). This substantial reduction from 205 to 25 items was primarily due to the high number of duplicates and repetitive items across publications. Items 1 to 5 were designed to be asked after sensor insertion and items 6 to 25 at the end after sensor removal.

Figure 3 Items constituting the draft-questionnaire before the validation steps; the 25 items were classified according to the attributes defined from the literature search (represented in different colors). Each item has a number assigned, corresponding to the order in which the items are planned to be answered.

The results of the content validation process of the 25 items by nine experts from the diabetes technology field are shown in Table 1. Twenty-one of the 25 initial items were scored with an I-CVI equal to or higher than 0.78; nonetheless, four items (4, 10, 16 and 17) did not reach the cut-off score and were excluded (Table 1). Ratings on clarity and rephrasing suggestions were taken into consideration, and minor item modifications were applied accordingly. The overall S-CVI/Ave of the remaining 21 items was 0.90, indicating an acceptable content validity.

Table 1 Calculation of Item-Level Content Validity Indexes (I-CVIs) and Scale-Level Content Validity Index Based on the Average Method (S-CVI/Ave) of the 25 Items, Done by 9 Experts, Followed by the Calculation of Item-Level Face Validity Indexes (I-FVs) and Scale-Level Face Validity Index Based on the Average Method (S-FVI/Ave) of the 21 Items Remaining After Content Validity, Done by 10 Subjects; the S-CVI/Ave Is 0.90 and the S-FVI/Ave Is 0.93. *I-CVI Cut-off Value Is 0.78 and Therefore Not Met by Items 4, 10, 16 and 17. ¥items with an I-FVI Score Lower Than 0.80 Were Revised and Rephrased

Table 1 also shows the results of the face validity process of the remaining 21 items performed by ten people with diabetes. Nineteen of the total 21 items were scored over 0.80, whereas three received a score equal to or lower than 0.80, indicating that there could be ambiguities. These items were marginally modified according to the suggestions made by the respondents. The overall S-FVI/Ave was 0.93, indicating an acceptable face validity.

Participant Characteristics of Pilot Studies

Among the 54 participants who completed the questionnaire, 65% were male, with a mean age of 51 years (min: 22, max: 71). Participants mostly had type 1 diabetes (96%) with a mean disease duration of 27 years (min: 1, max: 58). Moreover, the majority of participants (96%) reported prior experience with CGM systems.

Psychometric Evaluation

Pilot studies revealed that items about alerts (items 14 and 15) were not answered reliably by almost 25% of participants, since most alerts were switched off as part of the study protocol. These two items were therefore excluded from the EFA, which was performed with a total of 126 questionnaires that had been completely answered. Moreover, through successive iterations of reliability analysis based on corrected total item correlation preceding the EFA, 5 items (items 1, 2, 9, 11 and 16) were sequentially eliminated from the item pool.

Adequacy for EFA of the remaining 14 items (n=126 questionnaires) was confirmed by a KMO value of 0.807 and a significant Bartlett’s test (X2= 764; df= 91; p< 0.001). While the Kaiser criterion suggested a four-factor solution (eigenvalues >1), scree plot analysis indicated a clear break point after two factors (Figure 4). Sequential EFA iterations, beginning with four factors, revealed insufficient loadings (|<0.4|) for the third and fourth factors. A three-factor solution was similarly unsuitable due to single-item loading on the third factor. Consequently, a two-factor solution was adopted as the most appropriate factorial structure. Further iterative analysis of factor loading scores resulted in the exclusion of three items (items 4, 8 and 13), that failed to meet the threshold of |>0.4|. Thus, a final EFA with 11 items was performed and revealed a two-factor solution, accounting for 50.3% of the total variance. The first factor, comprising 6 items, was designated as “User acceptance” based on the content of the individual items. The second factor, consisting of 5 factors, was termed “Daily life impact” (Table 2). Cronbach’s α of each factor was 0.89 and 0.70, respectively. Subsequently, Cronbach’s α of the total scale was 0.84, indicating high internal consistency and good reliability (Table 2) and justifying calculation of a sum score.

Table 2 Factor Loading Scores from the Two-Factor Solution, Including the Final 11 Items (Designated Accordingly with Keywords) Classified According to Factor 1 “User Acceptance” and Factor 2 “Daily Life Impact”

Figure 4 Scree plot, red line pointing the break (elbow criterion). The scree plot displays the eigenvalues for each extracted factor. A clear inflection point or “elbow” is observed after the second factor, as indicated by the red vertical line. This elbow criterion suggests that a 2-factor solution best represents the underlying dimensional structure of the questionnaire.

Scoring

Calculating the total score of the 11-item QUSS-CGM resulted in scores ranging from 0 to maximum 44. To facilitate interpretation, scores were transformed to a 0–100 scale. A percentile analysis of the pilot study data based on the distribution of the total scores was conducted to define thresholds for score categorization: scores ≤66 (24th percentile) were classified as low satisfaction, scores between 67 and 91 (25th to 75th percentiles) as average user satisfaction, and scores ≥92 (76th to 100th percentiles) as high user satisfaction (Figure 5). Descriptive statistics reflected the presence of a ceiling effect along the questionnaire, whereas there was no floor effect.

Figure 5 Total QUSS-CGM score distribution from the final 11-item questionnaire (n=126) and calculated percentiles. Questionnaire for user satisfaction standardized for CGM (QUSS-CGM).

The resulting validated QUSS-CGM, available both in German and English, was established as a two-part questionnaire where the first item should be asked at the beginning of the CGM performance study and the others at the end (Figure 6).

Figure 6 Resulting validated 11-item QUSS-CGM questionnaire (German and English version) and corresponding assessment of user satisfaction based on scoring. Questionnaire for user satisfaction standardized for CGM (QUSS-CGM).

Discussion

This work aimed at validating the QUSS-CGM, a questionnaire for the assessment of user satisfaction to be used as a standardized tool in CGM performance studies.

Pertinent user satisfaction aspects were identified through an extensive literature search of CGM performance studies from the last 20 years. These represented the most frequently addressed factors in such studies, aligning with the intended objective of the QUSS-CGM, to assess aspects that a user can evaluate after an initial, single use of the device.

An initial collection of 205 items regarding user satisfaction in CGM performance studies was compiled after an initial systematic literature search, which was organized into a 25-item “draft-questionnaire”. The development of the “draft-questionnaire” further continued through content and face validation as described in the material and methods section. Four items addressing pain expectations and ambiguous outcomes (eg, post-sensor removal bleeding) were excluded after experts’ content validation based on their potential to introduce response variability and compromise measurement consistency. Items concerning any therapy decisions of the tested CGM systems were also removed since study devices are generally not used for therapy decisions,³⁶ making it difficult to judge, eg, the accuracy and precision of CGM data. In addition, the experts’ opinions on the items about alerts were controversial indicated by an I-CVI below or near the cut-off value, which resulted in one of the three items on this topic being excluded. Assessment of alert-related items is problematic due to default-disabled alerts in CGM performance studies and potential device attribution errors in multi-sensor protocols. Two items on the alerts were classified as initially relevant and left in the questionnaire. Nevertheless, in the pilot studies most alerts were deactivated, thus study participants were not able to rate these items reliably. To guarantee statistical robustness and complete response of the QUSS-CGM, it was decided to exclude these items from the questionnaire. Furthermore, rephrasing some items due to the response process validation was a crucial step since some item elements were recognized as potentially unclear, and by the end of the validation step, efficacy and efficiency were ensured. Overall, the EFA enabled the reduction of the questionnaire to its core 11 components. This reduction eliminated redundancies and enhanced the instrument’s efficiency without compromising reliability, resulting in a concise instrument that allows participants to provide reliable responses within a relatively brief timeframe.

The observed ceiling effect in the questionnaire responses during the pilot studies may be attributed to the exclusive evaluation of CE-certified CGM systems, which are expected to yield a high user satisfaction due to the mandatory usability validation required for CE certification. In addition, the questionnaire was designed for between-group comparison rather than discriminative sensitivity analyses. Regardless, the questionnaire proved to be a reliable tool for a thorough and direct assessment of user satisfaction and effectively captured its variations across different CGM systems. In addition, the development of a standardized scoring system facilitates comparative analysis of CGM systems evaluated across independent studies. While the questionnaire did not explicitly address aspects, such as calibration of the system or sensor insertion, these technical and operational factors are likely reflected in the general user experience assessments. Incorporating general user experience measures can provide indirect insight into how technical and operational parameters may impact the overall acceptability from the end-user’s perspective.

A strength of this work was the screening based on the systematic literature research on CGM performance evaluations,² which allowed for an extensive compilation of study-specific user satisfaction content. Therefore, it served as a foundation comprising all integral aspects to define user satisfaction in the context of CGM performance studies. Another significant strength was the number of experts that participated, nine, which is nearly the highest recommended number of experts,¹² allowing for a comprehensive quantitative analysis and the number of questionnaires for the EFA (n=126), which exceeds the suggested minimum ratio of respondent to items of 5:1.¹⁷ Moreover, sequential EFA was conducted to optimize item fit and verify sufficient factor loadings, thereby ensuring analytical coherence. A limitation of the study was the number of participants in the response process validation, which was the minimum number recommended,¹⁴ still it was within the acceptable range and the results can be considered adequate.

Conclusions

This work presents a proposal for the systematic and standardized assessment of user satisfaction in CGM performance studies through validation of the 11-item QUSS-CGM questionnaire. The insights gained can benefit various groups. Patients can provide feedback to guide device design and improve integration with their daily lives, increasing acceptance. Manufacturers can leverage user-centered data to refine product functionality and user experience. Regulatory bodies can incorporate standardized user assessments into a holistic device performance review. By prioritizing the user perspective, CGM systems can be optimized to meet better the needs of individuals managing diabetes.

Since the assessment of user satisfaction is crucial for clinical performance evaluations and patients’ acceptance, the QUSS-CGM may serve as a valuable tool as part of a future standardized protocol for CGM performance evaluations.