Measurement Equivalence of “Touch-Screen” versus “Paper-Based” Assessments of OHRQoL: A Randomized Crossover Trial
Authors Anowar M, McGrath C, Saub R
Received 5 February 2020
Accepted for publication 21 April 2020
Published 29 May 2020 Volume 2020:12 Pages 199—204
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 3
Editor who approved publication: Professor Christopher E. Okunseri
Maznurfarhatunnisak Anowar,1 Colman McGrath,2 Roslan Saub1
1Department of Community Oral Health & Clinical Prevention, Faculty of Dentistry, University of Malaya, Kuala Lumpur, Malaysia; 2Dental Public Health, Faculty of Dentistry, University of Hong Kong, Hong Kong, People’s Republic of China
Correspondence: Roslan Saub
Department of Community Oral Health & Clinical Prevention, Faculty of Dentistry, University of Malaya, Kuala Lumpur, Malaysia
Email [email protected]
Purpose: To determine the measurement equivalence of computer touch screen assessment (CTSA) and paper based assessment (PBA) of the oral health impact profile (OHIP-14).
Patients and Methods: A randomized crossover trial was conducted. Sixty participants were randomized to either i) Arm A: completed CTSA then PBA of OHIP-14, or ii) Arm B: PBA and then CTSA of OHIP-14 within the same day. User preference and time taken to complete the assessments were recorded. Agreement between CTSA and PBA was determined using directional difference (DD), absolute difference (AD), and intraclass correlation coefficient (ICC).
Results: There was no significant difference in CTSA and PBA OHIP-14 scores (P> 0.05). The magnitude of the DD in scores between assessment methods was small for overall scores and all domains (< 0.3). The AD in OHIP-14 scores was small (∼ 6% for overall score, between 8– 16% for domains). Agreement between CTSA and PBA was high (ICC=0.9; 95% CI=0.8– 0.9) for overall OHIP-14 scores, but ICC values varied across domains. Most (78%) preferred CTSA. There was no significant difference in time taken to complete assessments (P=0.09). Regression analyses did not identify any significant socio-demographic factor associated with absolute difference between CTSA and PBA scores.
Conclusion: There is equivalence of measurements in OHRQoL assessments from CTSA and PBA, and the time taken to complete assessment by either means is similar. There is a greater preference for CTSA. This has implications to support the use of CTSA in OHRQoL assessments.
Keywords: OHIP-14, computer touch screen, written questionnaires, score agreement, feasibility
There is an acceptance of the importance of patient-reported outcome measures (PROMS) in assessing oral health needs and in determining outcomes from care.1 To this end, assessment of oral health-related quality-of-life (OHRQoL) is increasingly employed in population health surveys, in research and in clinical practice.2,3 Traditionally, OHRQoL assessments have been conducted through “paper-based assessments” (PBA). However, PBA have many disadvantages and limitations, particularly when collecting large amounts of data. In PBA, respondents frequently either miss items or mark an item ambiguously, resulting in frequently “missing data”.4 Furthermore, post-data collection PBA have to be scanned or responses manually inputted into statistical packages for analyses, and this may further increase the likelihood of error in the data.5 Moreover, PBA are labor intensive to produce, distribute, and collect; thus not surprisingly more costly to use in research and clinical practice.6
In the new millennium with advances in computer technologies, it has become feasible to collect data by alternative means; namely computerized touch-screen assessments (CTSA). Arguably, CTSA offers several advantages over PBA to ensure high quality data sets (less errors in data entry), less missing data and greater efficiency.7,8 What remains unclear is whether CTSA and PBA questionnaires provide similar assessment outcomes, especially in terms of OHRQoL assessments.
We aimed to determine the measurement equivalence of CTSA and PBA of OHIP-14 (one of the most commonly used OHRQoL measures) specifically to determine agreement at “the group level” and “individual level”. Furthermore, to determine participants’ preference of the modes of OHRQoL assessment and to determine differences (if any) in time taken to complete OHRQoL assessments. Finally, to identify if any socio-demographic factors were associated with absolute differences in OHRQoL assessments (OHIP-14 scores) from CTSA and PBA.
Patients and Methods
Research Design and Sample
This study was a randomized crossover trial (Trial registration: ClinicalTrials.gov NCT02108470; National Medical Research Register NMRR-12-1289-13605). The study was conducted in accordance with the Declaration of Helsinki. Participants (60) were recruited from residential colleges at a university campus in April 2014. The only inclusion criteria were that participants could read and speak Malay. Participants were randomly assigned through block randomization in groups of four (ABBA) into two arms of the trial: i) Arm A, where participants self-completed the CTSA followed by the PBA; or ii) Arm B, where participants self-completed PBA followed by CTSA. The first assessment was conducted in the morning (AM) session and the second assessment was conducted in the afternoon (PM) session. The period of “washout” was a “lunch period” between morning and afternoon sessions. Assignment to trial arm was concealed in envelopes that were opened when written informed consent was obtained from participants. Participation was entirely voluntary but participants did receive a small token in-lieu of their time. A sample size of 60 participants was determined based on the size necessary to assess agreement using the intraclass correlation coefficient (ICC). The null hypothesis for the ICC was set at 0.2 (poor agreement) and the ICC was set at 0.6 as moderate agreement (indicating a significant level of agreement). Consequently, with α at 0.05 and β at 0.2, the minimum number of participants needed was 30 per group; a total of 60 subjects.9 The study was approved by the local institutional review board (Medical Ethics Committee, University of Malaya DF CO 1205/0074(L).
OHRQoL assessments were conducted employing the Malaysian version of OHIP-14 (S-OHIP [M]) comprising 14 items across seven domains (functional limitation, physical pain, psychological discomfort, physical disability, psychological disability, social disability, and handicap).10,11 The response to each item was recorded on a five-point Likert scale representing an event “not having occurred” to “occurring all of time/most of the time” within the past month.
PBA involved “pen and paper” for participants to self-complete questions. Data from PBA were entered manually into the computer package of SPSS for the analyses. For the CTSA a Samsung tablet with Android operating system was used. The respondents entered their responses by touching the relevant buttons on the screen. The navigation buttons at the bottom of the screen allowed the screen to be moved backwards and forwards through the questions. The responses were compiled automatically into an excel file that could be accessed in SPSS. The formats of the questions were similar in both CTSA and PBA. Burden on participants was assessed in two ways; i) time and ii) preference. Time taken for participants to complete PBA and CTSA was recorded by stopwatch. On completion of the trial participants were asked to rate their preference for PBA or CTSA (touch screen, paper, or no preference). Details of their age, gender, and ethnicity was recorded (Figure 1).
Figure 1 Flow chart of data collection.
Statistical analysis was carried out using statistical packages for social sciences software (IBM SPSS, version 20, USA). Data cleaning was performed prior to data analysis. Imputation was carried out for missing data or “do not know“ responses using the mean score of overall respondents for each of the 14 questions. Descriptive statistics of summary total and domain scores (mean (SD) values) were produced and the profile of the participants determined. Agreements with OHIP-14 between PBA and CTSA were determined in a number of ways i) to identify if there was a significant difference in scores between the assessment methods; ii) to calculate the mean directional difference (MDD) in scores (mean of values obtained from PBA scores minus CTSA scores) and to determine the magnitude of this by calculation effect sizes (ES = MDD/SD of the MDD); iii) by calculating the mean absolute difference (the difference between PBA and CTSA in a positive integer); and iv) by determining the Intraclass Correlation Coefficient (ICC) of PBA and CTSA OHIP-14 scores (mixed models) – this assesses agreement at the individual level. Difference in time taken to complete PBA and CTSA was determined. Preference for mode of assessment was determined through descriptive statistics. Following on, regression analyses were conducted where the absolute difference in OHIP-14 scores was the dependent variable and the independent variables were participants age, gender, ethnicity, and sequence of completion of assessments. The datasets used and/or analyzed during this study are available from the corresponding author on reasonable request from April 2020 until April 2022.
The assessment of one participant was excluded because of failure to complete the PBA (5 of the 14 questions were incomplete). The mean age of the participants was 21.4 years (SD=1.3); half were female (51%, 30) and most were of Malay ethnicity (70%, 41) [20% (12) were Chinese, 3% (2) were Indian, and 7% (4) were of other ethnicities]. The internal consistency of OHIP (Cronbach alpha values) for CTSA and PBA were 0.86 and 0.85, respectively. There was no significant difference in summary OHIP-14 scores (or across any of its seven domains) obtained from PBA and CTSA (P>0.05) (Table 1). The MDD was 0.2 (SD=4.7) for summary OHIP-14 scores, and the magnitude of the difference (effect size) was 0.0. Across the domains the ES of directional differences ranged from 0 (physical pain) to 0.3 (physical disability and psychological disability). The mean absolute difference for summary OHIP-14 scores was 3.6 (SD=2.9), constituting approximately a 6% difference in scores (3.6/56). Across domains, the largest difference was in “psychological discomfort” assessments; more than 16% difference (1.3/8). ICC value obtained of PBA and CTSA was 0.90 (95% CI=0.8–0.9). Across domains for two of the seven domains, ICC values were <0.70 (Functional limitation and Psychological discomfort).
Table 1 Agreement Between CTSA and PBA Score
Multiple linear regression analyses did not identify any significant association between socio-demographics (age, gender, ethnicity) or sequence of completing OHRQoL assessments, and (R2=0.091) absolute difference in Table 2.
Table 2 Multiple Regressions for Directional Difference and Absolute Difference of Total OHIP Scores
The mean (SD) time spent for PBA was 163.5 seconds (SD=60.8) compared to 172.2 seconds (SD=38.9). There was no significant difference in time taken to completed assessment: PBA versus CTSA (P=0.09). Among the respondents, 46 respondents (78%) had a preference for the touch screen version, 10 (17%) respondents preferred the written version, and 3 (5%) respondents had no preference.
The value and use of OHRQoL measures has long been accepted, but the challenge has been how to employ them in research and clinical practice in effective and efficiency means.12,13 There have been reports on the equivalence of assessments made by interview and questionnaires that have informed the debate on how to collect OHRQoL.14,15 Our trial sought to inform the debate further by comparing CTSA and PBA. The study benefits from being a randomized cross-over trial with a priori hypothesis of an appropriate sample size and response rate. Some studies have reported that the lack of randomization reduce the quality of the evidence provided by the study, thus introduces an important bias.16
There was no significant difference between scores obtained from CTSA and PBA in terms of overall scores or any of its seven domain scores. Furthermore, the magnitude of the directional difference could be interpreted as “insignificant or small” ES <0.20.17 In the domain of “psychological disability” there was a greater lack of concordance which may reflect that psychological attributes rather than physical attributes may vary.18 In terms of absolute differences, again the proportion of dissimilarity relative to absolute scores was low, at ~6%, but again larger in the aspects of “psychological disability” (~16%) for potential research as aforementioned. Thus, the findings suggest at “the group” level there was good agreement and in practical terms either method could be used to obtain OHRQoL assessments.19,20 At the individual level, here was also high levels of agreement with ICC valued >0.70. Of note again it was lowest in terms of “psychological discomfort”. Thus, at “individual level”, in a case of patient care, either method is appropriate for OHRQoL assessments.21 When comparing respondents by age or gender, there were no significant differences highlighted, as was also recently stated by Casola et al.22
There was no significant difference in time taken to complete assessments. However, given that CTSA are likely to be more time saving because data is automatically entered and amenable to statistical analyses this would imply that overall CTSA offers greater efficiency and most likely greater cost-efficiency.7
Of note most respondents reported to prefer CTSA method over PBA. Perhaps in the era of widespread (and continued use of) hand help devices CTSA have become more “second nature” than PBA. It is worth considering through mix methods and qualitative research whether this is in fact the case.23,24 There are some potential limitations of this trail. First, the study was conducted among university level students who are likely to have greater familiarity with computer-based technologies than perhaps the general population of similar age or older people and, thus, this may influence their ability to complete and preference for CTSA. Second, the study population reported to experience relatively few oral health impacts (ie, had low OHIP scores) which is to be expected given their age and educational attainment. It is plausible that among study populations with poorer oral health who experience more oral health impacts that differences between the CTSA method over PBA may exist. Lastly, both assessments were conducted on the same day so as to ensure oral health state was stable but, given the limited washout period (“lunch time”), it is feasible that participants could recall previous assessments made.
In conclusion, OHRQoL assessments obtained from computer touch screen assessment (CTSA) or paper-based assessments (PBA) are equivalent. There is concordance of measurement at both “the group”, which is refers to the mean absolute difference and “the individual”, which refers to ICC levels. Furthermore, they take an equivalent amount of time to perform assessments and arguably CTSA are likely to be more efficient and cost-effective. Moreover, CTSA appears to be a more preferred means of OHRQoL assessment that PBA. This has implications to inform the practice of OHRQoL assessments in research, epidemiological surveys, and in clinical practice.
CTSA, computer touch screen assessment; ICC, intraclass correlation coefficient; OHIP-14, oral health impact profile-14; OHRQoL, oral health related quality-of-ife; PBA, paper based assessment; SD, standard deviation; DD, directional difference; AD, absolute difference; CI, confidence interval; SE, standard error; ES, effect size.
The authors would like to thank Dr Mahmoud Danaee from the Academic Development Centre (ADeC) University of Malaya and Mr. Vignes Gopal Krishna for guidance for data analysis. This study was supported by the Postgraduate Research Grant (PPP) – Research, University of Malaya, Malaysia (PG026-2014A) and Exploratory Research Grant Scheme (ERGS), Ministry of Higher Education, Malaysia (ER020-2012A).
The authors declare that they have no competing interests in this work.
1. McGrath C, Lam O, Lang N. An evidence‐based review of patient‐reported outcome measures in dental implant research among dentate subjects. J Clin Periodontol. 2012;39:193–201. doi:10.1111/j.1600-051X.2011.01841.x
2. Bennadi D, Reddy CVK. Oral health related quality of life. J Int Soc Prev Community Dent. 2013;3:1–6. doi:10.4103/2231-0762.115700
3. de la Fuente Hernández J, Díaz FDCA, Vilchis MDCV. Oral health related quality of life. In: Virdi M, editor. Emerging Trends in Oral Health Sciences and Dentistry. UK: IntechOpen; 2015:691–715.
4. Ryan JM, Corry JR, Attewell R, Smithson MJ. A comparison of an electronic version of the SF-36 general health questionnaire to the standard paper version. Qual Life Res. 2002;11:19–26. doi:10.1023/A:1014415709997
5. Velikova G, Wright E, Smith A, et al. Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires. J Clin Oncol. 1999;17:998. doi:10.1200/JCO.19184.108.40.2068
6. Morris J, Perez D, McNoe B. The use of quality of life data in clinical practice. Qual Life Res. 1997;7:85–91. doi:10.1023/A:1008893007068
7. Bennett AV, Jensen RE, Basch E. Electronic patient‐reported outcome systems in oncology clinical practice. CA Cancer J Clin. 2012;62:336–347. doi:10.3322/caac.21150
8. Gwaltney CJ, Shields AL, Shiffman S. Equivalence of electronic and paper‐and‐pencil administration of patient‐reported outcome measures: a meta‐analytic review. Value Health. 2008;11:322–333. doi:10.1111/j.1524-4733.2007.00231.x
9. Zhang M, McGrath C, Hägg U. Who knows more about the impact of malocclusion on children’s quality of life, mothers or fathers? Eur J Orthod. 2007;29:180–185. doi:10.1093/ejo/cjl058
10. Saub R, Locker D, Allison P. Derivation and validation of the short version of the Malaysian Oral Health Impact Profile. Community Dent Oral Epidemiol. 2005;33:378–383. doi:10.1111/j.1600-0528.2005.00242.x
11. Saub R, Locker D, Allison P, Disman M. Cross-cultural adaptation of the Oral Health Impact Profile (OHIP) for the Malaysian adult population. Community Dent Health. 2007;24:166–175.
12. Desai R, Durham J, Wassell RW, Preshaw PM. Does the mode of administration of the Oral Health Impact Profile-49 affect the outcome score? J Dent. 2014;42:84–89. doi:10.1016/j.jdent.2013.10.016
13. Reissmann DR, John MT, Schierz O. Influence of administration method on oral health‐related quality of life assessment using the Oral Health Impact Profile. Eur J Oral Sci. 2011;119:73–78. doi:10.1111/j.1600-0722.2010.00805.x
14. Tsakos G, Bernabé E, O’Brien K, Sheiham A, de Oliveira C. Comparison of the self-administered and interviewer-administered modes of the child-OIDP. Health Qual Life Outcomes. 2008;6:40. doi:10.1186/1477-7525-6-40
15. Watson PD, Denny SJ, Adair V, et al. Adolescents’ perceptions of a health survey using multimedia computer‐assisted self‐administered interview. Aust N Z J Public Health. 2001;25:520–524. doi:10.1111/j.1467-842X.2001.tb00316.x
16. Peñarrocha-Oltra D, Peñarrocha-Diago M, Aloy-Prosper A, Covani U, Peñarrocha M. Immediate versus conventional loading of complete-arch implant-supported prostheses in mandibles with failing dentition: a patient-centered controlled prospective study. Int J Prosthodont. 2015;28(5):499–508. doi:10.11607/ijp.3951
17. Cohen J. Statistical Power Analysis for the Behavioral Science.
18. Frennered K, Hägg O, Wessberg P. Validity of a computer touch-screen questionnaire system in back patients. Spine. 2010;35:697–703. doi:10.1097/BRS.0b013e3181b43a20
19. Lee EH, Lee YW, Lee KW, et al. Measurement equivalence of touch screen computerized and paper based diabetes specific quality of life questionnaires. Int J Nurs Pract. 2014;20:382–389. doi:10.1111/ijn.12184
20. Ribeiro C, Moreira L, Silveira A, et al. Development and use of touch-screen computer-assisted self interviewing in Portuguese patients with chronic immune diseases: evaluation of an electronic version of SF-36v2. Acta Reumatol Port. 2010;35:208–214.
21. Broering JM, Paciorek A, Carroll PR, et al. Measurement equivalence using a mixed-mode approach to administer health-related quality of life instruments. Qual Life Res. 2014;23:495–508. doi:10.1007/s11136-013-0493-7
22. Cosola S, Marconcini S, Giammarinaro E, Poli GL, Covani U, Barone A. Oral health-related quality of life and clinical outcomes of immediately or delayed loaded implants in the rehabilitation of edentulous jaws: a retrospective comparative study. Minerva Stomatol. 2018;67(5):189–195. doi:10.23736/S0026-4970.18.04134-1
23. Palmer C, Farhan B, Nguyen N, et al. Are electronic and paper questionnaires equivalent to assess patients with overactive bladder? J Urol. 2018;200:369–374. doi:10.1016/j.juro.2018.03.117
24. Salaffi F, Gasparini S, Grassi W. The use of computer touch-screen technology for the collection of patient-reported outcome data in rheumatoid arthritis: comparison with standardized paper questionnaires. Clin Exp Rheumatol. 2009;27:459.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]